Report generation performance improvements #310
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ted into a random structure.
jesseleite
changed the title
This was referenced Dec 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR massively improves report generation performance overall (up to 10x faster compared to current version 5.3.0, and up to 4x faster compared to master branch right now). It also adds the ability to chunk generation into smaller jobs for super large sites with thousands of entries.
For example, report generation on 5.3.0 (current version) for even 1000 pages is 60+ seconds with a request timeout...
But with this PR, 1000 pages takes me ~7 seconds, which is about 10x faster 🏎️...
But where this PR really shines is when generating reports for massive sites with thousands of entries, which you can't even do on 5.3.0 (as shown above in our 1000 page example).
On master (untagged), we can at least generate a report for 5000 pages, but it takes me about ~52 seconds...
With this PR though, the same 5000 pages takes about ~29 seconds, which is nearly 2x faster 🏎️
And if I enable async queue driver w/ 5 workers (chunked in jobs of 1000 pages each), the report generation endpoint takes about ~13 seconds, which is about 4x faster compared to master, and not even possible on 5.3.0 🏎️
Configuration
I have made the chunk size configurable in
config/statamic/seo-pro.phpunderreports.queue_chunk_size, so that users can balance chunk size vs. worker count how they like.Note on CPU
Of course your mileage will vary. It's pretty CPU intensive when generating a report for this many entries. The above numbers are just locally on my M1 Macbook.