You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello,
I have KafkaConnect with 2 workers and FilePulse connector with 4 tasks. During the rolling restart of the workers, connector stops cleaning up files from S3. At the same time once the restart is finished, new files are processed and cleaned up successfully.
All not-cleaned files have the status "COMMITTED" in the status topic.
here is my connector config:
There are no errors in the logs.
Just a thought, is it possible/makes sense to check file status by AmazonS3MoveCleanupPolicy retrospectively and if it has "COMMITTED" status clean it up?
The text was updated successfully, but these errors were encountered:
arnitolog
changed the title
AmazonS3MoveCleanupPolicy doesn't cleanup after restart
AmazonS3MoveCleanupPolicy doesn't cleanup files during KafkaConnect workers restart
Sep 23, 2024
Hello,
I have KafkaConnect with 2 workers and FilePulse connector with 4 tasks. During the rolling restart of the workers, connector stops cleaning up files from S3. At the same time once the restart is finished, new files are processed and cleaned up successfully.
All not-cleaned files have the status "COMMITTED" in the status topic.
here is my connector config:
There are no errors in the logs.
Just a thought, is it possible/makes sense to check file status by AmazonS3MoveCleanupPolicy retrospectively and if it has "COMMITTED" status clean it up?
The text was updated successfully, but these errors were encountered: