Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In 3.4.1 version when uploading data , progress is jumping #198

Open
mvictor27 opened this issue Nov 29, 2024 · 9 comments
Open

In 3.4.1 version when uploading data , progress is jumping #198

mvictor27 opened this issue Nov 29, 2024 · 9 comments

Comments

@mvictor27
Copy link

In the latest version, we’ve noticed that when attempting to upload data, bytesUploaded can occasionally exceed totalBytes. Additionally, it sometimes reports a lower value for bytesUploaded before suddenly jumping to a higher one. I'm seeing this happening on progressFor(id...
@donnywals
Copy link
Collaborator

Hi Victor, some questions that would help me investigate:

Is there any pattern to where this is more likely to happen (large files, multiple files, etc)?

And are you seeing this for a default URLSession configuration or one that's in the background?

Which platform and OS version are you seeing this on?

@mvictor27
Copy link
Author

I tested both single and multiple file uploads, and the issue occurred with smaller video files ranging from 68 MB to 200 MB. However, I haven’t tested with larger files yet. We are using sessionConfiguration: .background(withIdentifier) and have tested this only on iOS. I couldn’t reproduce this issue in version 3.3.0.

@donnywals
Copy link
Collaborator

Hi @mvictor27

Investigating this had to take a backseat for a bit due to some other things popping up on my calendar. I will take a look at this in the New Year. Thanks for providing extra input!

@sylvain-villet
Copy link

Hi @donnywals, I tested v3.4.2 and had the same problem (iPad and iPhone on iOS 18.2).
I've added some debug logs in the progressFor and totalProgress delegate methods.
For a single 35MB file upload, the totalProgress goes as expected from 0 to 100%, but the progressFor goes from 0 to 100% for the first block (which is 15% of the total in this case), and then it goes from 15% to 115% for the second block, then from 30% to 130% for the third block, etc...

totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 330280, totalBytes: 34766353, progress: 0.949998983212303
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 6, totalBytes: 34766353, progress: 1.7258065578520705e-05
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 5676296, totalBytes: 34766353, progress: 16.32698143518246
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 9185412, totalBytes: 34766353, progress: 26.4204071102885
progressFor: bytesUploaded: 12177840, totalBytes: 34766353, progress: 35.02766022078876
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 15377488, totalBytes: 34766353, progress: 44.230949389485865
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 20126288, totalBytes: 34766353, progress: 57.89013302603238
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 23014654, totalBytes: 34766353, progress: 66.1980679998273
progressFor: bytesUploaded: 26628740, totalBytes: 34766353, progress: 76.59342353222956
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 29517105, totalBytes: 34766353, progress: 84.90135562968022
progressFor: bytesUploaded: 33028035, totalBytes: 34766353, progress: 94.9999989932795
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
totalProgress: bytesUploaded: 0, totalBytes: 34766353, progress: 0.0
progressFor: bytesUploaded: 34766353, totalBytes: 34766353, progress: 100.0
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 5242886, totalBytes: 34766353, progress: 15.080345068118016
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 5242886, totalBytes: 34766353, progress: 15.080345068118016
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 16698856, totalBytes: 34766353, progress: 48.03165865571232
progressFor: bytesUploaded: 26194643, totalBytes: 34766353, progress: 75.34481111665639
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 28775128, totalBytes: 34766353, progress: 82.76717434238789
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 33522114, totalBytes: 34766353, progress: 96.42114029044116
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
progressFor: bytesUploaded: 38270915, totalBytes: 34766353, progress: 110.08032680333194
progressFor: bytesUploaded: 40009233, totalBytes: 34766353, progress: 115.08032781005244
totalProgress: bytesUploaded: 5242880, totalBytes: 34766353, progress: 15.080327810052438
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 10485766, totalBytes: 34766353, progress: 30.160672878170455
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 10485766, totalBytes: 34766353, progress: 30.160672878170455
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 15955064, totalBytes: 34766353, progress: 45.89225680358248
progressFor: bytesUploaded: 23696974, totalBytes: 34766353, progress: 68.16065521741668
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 25347695, totalBytes: 34766353, progress: 72.90869709572355
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 34018008, totalBytes: 34766353, progress: 97.84750215244033
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 35255879, totalBytes: 34766353, progress: 101.40804530173182
progressFor: bytesUploaded: 41345706, totalBytes: 34766353, progress: 118.92448425637282
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
progressFor: bytesUploaded: 43513795, totalBytes: 34766353, progress: 125.16065461338438
progressFor: bytesUploaded: 45252113, totalBytes: 34766353, progress: 130.16065562010488
totalProgress: bytesUploaded: 10485760, totalBytes: 34766353, progress: 30.160655620104876
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 15728646, totalBytes: 34766353, progress: 45.24100068822289
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 15728646, totalBytes: 34766353, progress: 45.24100068822289
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 25326902, totalBytes: 34766353, progress: 72.8488892694612
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 30899362, totalBytes: 34766353, progress: 88.87720262174177
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 41633020, totalBytes: 34766353, progress: 119.75089823197733
progressFor: bytesUploaded: 45246879, totalBytes: 34766353, progress: 130.14560083423189
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 45866495, totalBytes: 34766353, progress: 131.92782976114867
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 48756675, totalBytes: 34766353, progress: 140.24098242343683
totalProgress: bytesUploaded: 15728640, totalBytes: 34766353, progress: 45.240983430157314
progressFor: bytesUploaded: 50494993, totalBytes: 34766353, progress: 145.2409834301573
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 20971526, totalBytes: 34766353, progress: 60.32132849827533
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 20971526, totalBytes: 34766353, progress: 60.32132849827533
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 28092906, totalBytes: 34766353, progress: 80.8048690065363
progressFor: bytesUploaded: 32220957, totalBytes: 34766353, progress: 92.67856481811596
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 36865695, totalBytes: 34766353, progress: 106.03843031795714
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 42026212, totalBytes: 34766353, progress: 120.88185378546896
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 46257418, totalBytes: 34766353, progress: 133.05225888950733
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 53999555, totalBytes: 34766353, progress: 155.32131023348927
totalProgress: bytesUploaded: 20971520, totalBytes: 34766353, progress: 60.32131124020975
progressFor: bytesUploaded: 55737873, totalBytes: 34766353, progress: 160.32131124020975
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 26214406, totalBytes: 34766353, progress: 75.40165630832777
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 26214406, totalBytes: 34766353, progress: 75.40165630832777
progressFor: bytesUploaded: 30239294, totalBytes: 34766353, progress: 86.9786198166946
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 33542098, totalBytes: 34766353, progress: 96.47862115419468
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 35296429, totalBytes: 34766353, progress: 101.52468106159999
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 47165936, totalBytes: 34766353, progress: 135.6654694267184
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 48404487, totalBytes: 34766353, progress: 139.22796849010882
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 59242435, totalBytes: 34766353, progress: 170.4016380435417
totalProgress: bytesUploaded: 26214400, totalBytes: 34766353, progress: 75.40163905026219
progressFor: bytesUploaded: 60980753, totalBytes: 34766353, progress: 175.4016390502622
progressFor: bytesUploaded: 31457289, totalBytes: 34766353, progress: 90.48199274741299
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 31457289, totalBytes: 34766353, progress: 90.48199274741299
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 41595575, totalBytes: 34766353, progress: 119.64319352104606
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 46828521, totalBytes: 34766353, progress: 134.69494772718897
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 57785453, totalBytes: 34766353, progress: 166.210856226421
progressFor: bytesUploaded: 64485315, totalBytes: 34766353, progress: 185.48196585359415
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 64485315, totalBytes: 34766353, progress: 185.48196585359415
totalProgress: bytesUploaded: 31457280, totalBytes: 34766353, progress: 90.48196686031463
progressFor: bytesUploaded: 66223633, totalBytes: 34766353, progress: 190.48196686031463

@donnywals
Copy link
Collaborator

Thanks for looking at this @sylvain-villet! Very useful information to have since I haven't been able to sit down and investigate this just yet.

@sylvain-villet
Copy link

@donnywals any news on that? It makes the new versions unusable which means that we still have plenty of crashes in production due to the other issue I fixed.

@donnywals
Copy link
Collaborator

Hi Sylvain, I started investigating yesterday. Not quite sure on why it's happening exactly and I'm reluctant to agressively clamp values which would hide the problem but wouldn't actually fix anything. Hopefully today I'll figure this one out and get a fix in.

Apologies for the delay on the investigation on this one; with the christmas holidays and some other stuff that came up last week I haven't really been able to find time earlier...

@donnywals
Copy link
Collaborator

Hi @sylvain-villet This bug seems to have been around for a long, long time for chunked uploads. I've got a fix up on the main branch at the moment that I will be testing some more. I would love if you could take a look too to ensure that this fix works as expected.

My plan is to tag and release this fix on Monday.

There is a slight change in behavior...

  • progressFor should report progress for the entire upload, so not for chunks like it does now. You'll find that it will go from 0-100 only once in the fix even when you upload a 35MB file in several chunks.
  • totalProgress should report progress for all in flight uploads. So if you upload multiple files, totalProgress will run from 0-100 as all uploads make progress.

As a small sidenote, even though we fully support chunked uploads, our recommendation is to upload files in their entirety since TUS would allow you to resume failed uploads from where they've left off without chunking.

@sylvain-villet
Copy link

Thanks, that's great news. We're using the v3.3.0 currently and I think it behaves as you're describing here.
I'm out of work today but I can have a look on Monday!
Thanks for the additional information!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@donnywals @mvictor27 @sylvain-villet