Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 data mover to support checksum #3

Open
richard-mansfield opened this issue Feb 5, 2018 · 1 comment
Open

S3 data mover to support checksum #3

richard-mansfield opened this issue Feb 5, 2018 · 1 comment

Comments

@richard-mansfield
Copy link

I understand the S3 data mover does not currently support checksum for put/get operations. Is this on the roadmap?

@rread
Copy link

rread commented Nov 30, 2018

Currently (at least when I originally wrote this a couple years ago), the checksums are calculated while the file is being read or written. This works because the local mover is copying the file sequentially, so the checksum is basically "free." However, the S3 library copies the files in chunks and not necessarily in order, so it would be difficult to calculate a checksum as the file is being copied. At the time I didn't see any way of doing it other than scanning the file again before/after the operation but didn't get a chance to add that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants