Conflicting S3 Lifecycle Rule Filter #367
Assignees
Labels
bug
This issue is a bug.
module/powershell-cmdlets
p2
This is a standard priority issue
response-requested
Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Comments
AaronStarr-pga
added
bug
ashishdhingra
added
module/powershell-cmdlets
p2
|
Analysis Looks like in customer's use case, Lifecycle rule configuration the rule scope is set as ...
Amazon Information: 5 : Request metrics: AsyncCall = True; CanonicalRequest = GET\n/\nlifecycle=\nhost:testbucket-issue367.s3.us-east-2.amazonaws.com\nuser-agent:AWSPowerShell.Common/4.1.711.0 ua/2.0 os/windows#10.0.19045.0 md/ARCH#X64 lang/.NET_Core#8.0.8 md/aws-sdk-dotnet-core#3.7.400.59 api/S3#3.7.410.1 md/PowerShellCore/7.-1 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241204T210217Z\nx-amz-security-token:<<TOKEN-REDACTED>>\n\nhost;user-agent;x-amz-content-sha256;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; StringToSign = AWS4-HMAC-SHA256\n20241204T210217Z\n20241204/us-east-2/s3/aws4_request\n49be41a57d6041713c2e188ad199eeed4daa7c44a7e785a1a50ebf0be03a711a; ServiceName = AmazonS3; ServiceEndpoint = https://testbucket-issue367.s3.us-east-2.amazonaws.com/; MethodName = GetLifecycleConfigurationRequest; AmzId2 = DmWjzQT8WlzmNqDT0ZydJmxhs/GH8Em/uLeJeRj5r9xesf6C3uJEzhEWvPAkM8NKYiIe+RU83hmu8eFzkn8C2S6v5sVp1aG9O+CJ3fHXAEY=; StatusCode = OK; BytesProcessed = 338; AWSRequestID = S0G0WWK71YYY92BM; CredentialsRequestTime = 00:00:01.9339616; RequestSigningTime = 00:00:00.0004549; HttpRequestTime = 00:00:00.2713438; ResponseUnmarshallTime = 00:00:00.0001935; ResponseProcessingTime = 00:00:00.0005850; ClientExecuteTime = 00:00:02.2071163;
Amazon Information: 3 : Resolved DefaultConfigurationMode for RegionEndpoint [us-east-2] to [Legacy].
Amazon Information: 6 : Starting a process with the following ProcessInfo: UseShellExecute - False RedirectStandardError - True, RedirectStandardOutput - True, CreateNoWindow - True
Amazon Information: 7 : Process started
Amazon Information: 3 : Process ends with exitcode - 0
Amazon Verbose: 6 : Received response (truncated to 1024 bytes): [<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>testglacierrule</ID><Filter/><Status>Enabled</Status><Transition><Days>0</Days><StorageClass>GLACIER_IR</StorageClass></Transition></Rule></LifecycleConfiguration>]
...As noted, it returns empty Using the similar AWS CLI command Clearly the S3 GetBucketLifecycleConfiguration service API operation is returning empty filter element, which is processed properly properly by AWS Tools for PowerShell (and AWS CLI). While executing Capturing network traffic shows the below request body being sent: <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<Transition>
<Days>0</Days>
<StorageClass>GLACIER_IR</StorageClass>
</Transition>
<ID>testglacierrule</ID>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>Changing the Need to do more analysis, like comparing with Java SDK and/or AWS CLI. Most likely we need to open issue with S3 service team. |
ashishdhingra
added
investigating
|
Running below Java SDK code: package org.example;
import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import software.amazon.encryption.s3.S3EncryptionClient;
public class Handler {
public void sendRequest() {
String bucketName = "testbucket-issue1880";
S3Client s3Client = S3Client.builder().httpClientBuilder(
ApacheHttpClient.builder().maxConnections(100)
).build();
GetBucketLifecycleConfigurationResponse getBucketLifecycleConfigurationResponse =
s3Client.getBucketLifecycleConfiguration(
GetBucketLifecycleConfigurationRequest.builder().bucket(
bucketName
).build()
);
PutBucketLifecycleConfigurationResponse putBucketLifecycleConfigurationResponse =
s3Client.putBucketLifecycleConfiguration(
PutBucketLifecycleConfigurationRequest.builder()
.bucket(bucketName)
.lifecycleConfiguration(
BucketLifecycleConfiguration.builder()
.rules(getBucketLifecycleConfigurationResponse.rules()
).build()
).build());
}
}logs below in verbose logs: Java SDK is sending below request body while invoking <?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>testglacierrule</ID>
<Filter></Filter>
<Status>Enabled</Status>
<Transition>
<Days>0</Days>
<StorageClass>GLACIER_IR</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>AWS Tools for PowerShell uses AWS SDK for .NET behind the scenes. Using AWS .NET SDK code below: using Amazon.S3.Model;
using System.Net;
using System.Text;
string bucketName = "<<bucket-name>>";
Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
Amazon.AWSConfigs.LoggingConfig.LogMetrics = true;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());
using (AmazonS3Client amazonS3Client = new AmazonS3Client(RegionEndpoint.USEast2))
{
var getbucketLifecycleConfig = await amazonS3Client.GetLifecycleConfigurationAsync(new GetLifecycleConfigurationRequest()
{
BucketName = bucketName,
});
var putbucketLifecycleConfig = await amazonS3Client.PutLifecycleConfigurationAsync(new PutLifecycleConfigurationRequest()
{
BucketName = bucketName,
Configuration = new LifecycleConfiguration() { Rules = getbucketLifecycleConfig.Configuration.Rules }
});
}Send the following request to AWS S3 service (captured using network monitoring tool): It is not sending empty Analysis:
For customer's scenario, the PutBucketLifecycleConfiguration somehow expects empty @AaronStarr-pga As a workaround, you would need to parse response from Just FYI, the handling of processing empty elements by AWS SDK .NET unmarshaller would be fixed in next major version Thanks, |
ashishdhingra
added
needs-review
response-requested
|
Thank you for the investigation Ashish and additional information, I have incorporated the workaround in my code. |
@AaronStarr-pga Thanks. I will close this issue for now since the fix is already in place in AWS .NET SDK v4-development branch. |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When calling Get-S3LifecycleConfiguration on a bucket that has a lifecycle policy + rule on it, where the rule scope is applied to all items in the bucket (So no Filter), you will receive a LifecycleRule as expected, however, the Filter field in the rule will be empty.
If you were to take that rule and then run Write-S3LifecycleConfiguration with it, you'd receive and XML format error. This is because the Filter is missing. To update a policy, you need to pull down the rules and fill in all the empty Filter fields with new Amazon.S3.Model.LifecycleFilter objects.
Expected Behavior
Either the Get-S3LifecycleConfiguration should provide rules with the empty filter object or the Write-S3LifecycleConfiguration should not require a Filter object to have correct XML format
Current Behavior
AbortIncompleteMultipartUpload : Amazon.S3.Model.LifecycleRuleAbortIncompleteMultipartUpload
Expiration :
Id : DeleteIncompleteMultipart
NoncurrentVersionExpiration :
Filter :
Status : Enabled
NoncurrentVersionTransitions : {}
Transitions : {}
Write-S3LifecycleConfiguration : The XML you provided was not well-formed or did not validate against our published schema
Reproduction Steps
Create a bucket in S3, add a Policy/rule to it so the rule's scope is the entire bucket.
Use AWS.Tools.S3.Get-S3LifecycleConfiguration to pull it down and isolate the Amazon.S3.Model.LifecycleRule inside.
Take this rule and use it to run Write-S3LifecycleConfiguration.
XML format error
Possible Solution
As stated above, either have the filter get correctly pulled down as an empty Amazon.S3.Model.LifecycleFilter object ([Amazon.S3.Model.LifecycleFilter]::new()) or don't require Write-S3LifecycleConfiguration's parameter Configuration_Rule to need an empty filter to determine it has a scope that includes the whole bucket.
Additional Information/Context
No response
AWS Tools for PowerShell version used
AWS.Tools.S3 | 4.1.705
AWS.Tools.Common | 4.1.705
PowerShell version used
Major Minor Build Revision
5 1 22621 4391
Operating System and version
Windows 11 Enterprise, 23H2
The text was updated successfully, but these errors were encountered: