[Feature] Updated AWS UC storage credential to include permissions for file events #4406
Conversation
borremosch-db
requested review from
hectorcast-db
and removed request for
a team
borremosch-db
force-pushed
the
add-storage-credential-file-events-permissions-aws
branch
from
3ae3e94 to
9085086
Compare
borremosch-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
borremosch-db
changed the title
borremosch-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
borremosch-db
changed the title
aws/data_aws_unity_catalog_policy.go
Outdated
| "arn:aws:sqs:*:*:*", | ||
| "arn:aws:sns:*:*:*", |
There was a problem hiding this comment.
Are customers comfortable with granting us permission on all SQS queues and SNS destinations? It may be sensible as a default but I expect they will want to be more selective. I wonder if e.g. there is a specific prefix that Databricks always uses so we can restrict this somewhat?
There was a problem hiding this comment.
these sns/sqs will be created by Databricks, and will follow the pattern arn:aws:sqs:<region>:<aws_account_id>:csms-*
There was a problem hiding this comment.
I added ARN resource ID prefix. LMK if this works.
| @@ -60,6 +60,59 @@ func generateReadContext(ctx context.Context, d *schema.ResourceData, m *common. | |||
| Resources: []string{kmsArn}, | |||
| }) | |||
| } | |||
| policy.Statements = append(policy.Statements, &awsIamPolicyStatement{ | |||
There was a problem hiding this comment.
agree with @mgyucht - these can be left as opt-out/in, as our official documentation mentioned this is optional but strongly recommended
There was a problem hiding this comment.
We're moving from opt-in/out to mandatory (see PRD: Maximizing coverage of managed file events)
borremosch-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
|
@mgyucht thanks for the review. Ideally we would not make these changes opt-in/out as we're moving towards making file events mandatory (see PRD: Maximizing coverage of managed file events) |
borremosch-db
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
|
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Changes
Databricks documentation for storage credentials contains instructions to add permissions for file events, but as of yet these are missing from the terraform provider. This PR adds them for AWS. PRs for Azure and GCP will follow soon
Tests
Updated test:
aws/data_aws_unity_catalog_policy_test.gomake testrun locallydocs/folderinternal/acceptance