Skip to content

Commit

Permalink
feat(s3): replicating objects (#30966)
Browse files Browse the repository at this point in the history
### Issue # (if applicable)

Closes #1680.

### Reason for this change

AWS S3 supports configuring [object replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) , but the `s3.Bucket` construct does not support it.

### Description of changes

Added `replicationRules` to `BucketProps`.

#### Replication configuration version

There are two versions of [replication configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-add-config.html#replication-backward-compat-considerations). This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC).

To use V2 replication configuration, this PR explicitly specifies [Filter.Prefix](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrulefilter.html#cfn-s3-bucket-replicationrulefilter-prefix) property.
```ts
        const prefix = rule.prefixFilter ?? '';
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };
```

V2 replication configuration has some restriction:
- Must specify [DeleteMarkerReplication](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-deletemarkerreplication)
```sh
ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)
```
- Must specify [Priority](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-priority)
```sh
ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)
```

These restriction is not documented but there are some posts about these points.
- https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets

To resolve these problems,I made the `priority` required and explicitly set the `deleteMarkerReplication`.

```ts
       const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

        return {
          id: rule.id,
          priority: rule.priority,
          status: 'Enabled',
          destination: {
            bucket: rule.destination.bucket.bucketArn,
            account: rule.destination.account,
            storageClass: rule.storageClass?.toString(),
            accessControlTranslation: rule.destination.accessControlTransition ? {
              owner: 'Destination',
            } : undefined,
            encryptionConfiguration: rule.kmsKey ? {
              replicaKmsKeyId: rule.kmsKey.keyArn,
            } : undefined,
            replicationTime: rule.replicationTimeControl !== undefined ? {
              status: rule.replicationTimeControl ? 'Enabled' : 'Disabled',
              time: {
                minutes: 15,
              },
            } : undefined,
            metrics: rule.replicationTimeControlMetrics !== undefined ? {
              status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled',
              eventThreshold: {
                minutes: 15,
              },
            } : undefined,
          },
          filter,
          // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication,
          // CDK explicitly set the deleteMarkerReplication if it is undefined.
          deleteMarkerReplication: {
            status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled',
          },
          sourceSelectionCriteria,
        };
```

#### IAM permission

There is a [documentation to setup IAM permissions for service role](https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html).

```json
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource":"arn:aws:s3:::DST-BUCKET/*"
      }
   ]
}
```

However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation.

Generated Policy:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SRC-BUCKET",
                "arn:aws:s3:::SRC-BUCKET/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms",
                        "aws:kms:dsse",
                        "AES256"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::DST-BUCKET/*"
            ]
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": "s3.ap-northeast-1.amazonaws.com",
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::SRC-BUCKET/*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga"
            ]
        },
        {
            "Action": [
                "kms:Encrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "s3.ap-northeast-1.amazonaws.com"
                    ],
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::DST-BUCKET*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga"
            ]
        }
    ]
}
```

I adopted the policy from the document. I look forward to hearing your thoughts on this matter.

### Description of how you validated changes

Added both unit and integ tests.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
badmintoncryer authored Jan 16, 2025
1 parent 717d91d commit 9d8a7e2
Show file tree
Hide file tree
Showing 25 changed files with 68,147 additions and 33 deletions.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 9d8a7e2

Please sign in to comment.