Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(s3): replicating objects (#30966)
### Issue # (if applicable) Closes #1680. ### Reason for this change AWS S3 supports configuring [object replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) , but the `s3.Bucket` construct does not support it. ### Description of changes Added `replicationRules` to `BucketProps`. #### Replication configuration version There are two versions of [replication configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-add-config.html#replication-backward-compat-considerations). This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC). To use V2 replication configuration, this PR explicitly specifies [Filter.Prefix](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrulefilter.html#cfn-s3-bucket-replicationrulefilter-prefix) property. ```ts const prefix = rule.prefixFilter ?? ''; const filter = isAndFilter ? { and: { prefix, tagFilters: rule.tagFilter, }, } : { prefix, }; ``` V2 replication configuration has some restriction: - Must specify [DeleteMarkerReplication](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-deletemarkerreplication) ```sh ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED | AWS::S3::Bucket | SourceBucket (SourceBucketDDD2130A) Resource handler returned message: Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX) ``` - Must specify [Priority](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-priority) ```sh ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED | AWS::S3::Bucket | SourceBucket (SourceBucketDDD2130A) Resource handler returned message: Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX) ``` These restriction is not documented but there are some posts about these points. - https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets To resolve these problems,I made the `priority` required and explicitly set the `deleteMarkerReplication`. ```ts const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration const filter = isAndFilter ? { and: { prefix, tagFilters: rule.tagFilter, }, } : { prefix, }; return { id: rule.id, priority: rule.priority, status: 'Enabled', destination: { bucket: rule.destination.bucket.bucketArn, account: rule.destination.account, storageClass: rule.storageClass?.toString(), accessControlTranslation: rule.destination.accessControlTransition ? { owner: 'Destination', } : undefined, encryptionConfiguration: rule.kmsKey ? { replicaKmsKeyId: rule.kmsKey.keyArn, } : undefined, replicationTime: rule.replicationTimeControl !== undefined ? { status: rule.replicationTimeControl ? 'Enabled' : 'Disabled', time: { minutes: 15, }, } : undefined, metrics: rule.replicationTimeControlMetrics !== undefined ? { status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled', eventThreshold: { minutes: 15, }, } : undefined, }, filter, // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication, // CDK explicitly set the deleteMarkerReplication if it is undefined. deleteMarkerReplication: { status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled', }, sourceSelectionCriteria, }; ``` #### IAM permission There is a [documentation to setup IAM permissions for service role](https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html). ```json { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:GetReplicationConfiguration", "s3:ListBucket" ], "Resource":[ "arn:aws:s3:::SRC-BUCKET" ] }, { "Effect":"Allow", "Action":[ "s3:GetObjectVersionForReplication", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource":[ "arn:aws:s3:::SRC-BUCKET/*" ] }, { "Effect":"Allow", "Action":[ "s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags" ], "Resource":"arn:aws:s3:::DST-BUCKET/*" } ] } ``` However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation. Generated Policy: ```json { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetReplicationConfiguration", "s3:GetObjectVersionForReplication", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging", "s3:GetObjectRetention", "s3:GetObjectLegalHold" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::SRC-BUCKET", "arn:aws:s3:::SRC-BUCKET/*" ] }, { "Action": [ "s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags", "s3:GetObjectVersionTagging", "s3:ObjectOwnerOverrideToBucketOwner" ], "Effect": "Allow", "Condition": { "StringLikeIfExists": { "s3:x-amz-server-side-encryption": [ "aws:kms", "aws:kms:dsse", "AES256" ] } }, "Resource": [ "arn:aws:s3:::DST-BUCKET/*" ] }, { "Action": [ "kms:Decrypt" ], "Effect": "Allow", "Condition": { "StringLike": { "kms:ViaService": "s3.ap-northeast-1.amazonaws.com", "kms:EncryptionContext:aws:s3:arn": [ "arn:aws:s3:::SRC-BUCKET/*" ] } }, "Resource": [ "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga" ] }, { "Action": [ "kms:Encrypt" ], "Effect": "Allow", "Condition": { "StringLike": { "kms:ViaService": [ "s3.ap-northeast-1.amazonaws.com" ], "kms:EncryptionContext:aws:s3:arn": [ "arn:aws:s3:::DST-BUCKET*" ] } }, "Resource": [ "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga" ] } ] } ``` I adopted the policy from the document. I look forward to hearing your thoughts on this matter. ### Description of how you validated changes Added both unit and integ tests. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
- Loading branch information