Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance test: Without BOR and year #88

Open
timrobertson100 opened this issue Oct 28, 2024 · 0 comments
Open

Performance test: Without BOR and year #88

timrobertson100 opened this issue Oct 28, 2024 · 0 comments

Comments

@timrobertson100
Copy link
Member

timrobertson100 commented Oct 28, 2024

A performance test to evaluate potential gains.

Get a baseline duration of current tiles and points jobs on C5 when full capacity is allocated.
Remove the year and basis of record and capture the timings, with same compute capacity, ensuring it is all allocated.
This can be done by changing the SQL to singular values to e.g. SELECT ... 2024 AS year, 'HUMAN_OBSERVATION' AS basisOfRecord in the relevant queries.

Rationale:
We are aware of growing capacity constraints, and removing these controls on the revised GBIF.org interface can be considered (possibly desirable). Users can still filter by year and basis of record driven from the adhoc mapping searches. It would of course require a deprecation and removal of the v2 map API and a simpler v3 version. Removal of pre-calculated views may also be less wasteful (electricity) for rarely used views. An alternative could be to stop precalculating altogether but make use of a fast aggregation store, such as clickhouse (see demo and blog)

For now though, let's just understand the potential savings in compute with existing code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant