Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not schedule all bulk import subtasks at once #2042

Open
rkorytkowski opened this issue Jan 24, 2025 · 0 comments
Open

Do not schedule all bulk import subtasks at once #2042

rkorytkowski opened this issue Jan 24, 2025 · 0 comments
Assignees

Comments

@rkorytkowski
Copy link
Contributor

Celery does not handle well a chain of multiple chords with groups that have a lot of tasks. See celery/celery#5286 From my testing with 8 chained chords and groups with ~200 tasks in total it starts to fail for us.

I tried to limit the number of tasks, but the more chords you have in a chain, the lower the limit is. It is undesired for us to schedule less that 200 tasks for an import with e.g. 60 000 resources.

My solution is to schedule just one chord at a time and have a callback task that schedules another chord.

@rkorytkowski rkorytkowski self-assigned this Jan 24, 2025
rkorytkowski added a commit to OpenConceptLab/oclapi2 that referenced this issue Jan 24, 2025
rkorytkowski added a commit to OpenConceptLab/oclapi2 that referenced this issue Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant