8000 global contributions endpoint quick optimization by lsabor · Pull Request #1513 · Metaculus/metaculus · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

global contributions endpoint quick optimization #1513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 25, 2024

Conversation

lsabor
Copy link
Contributor
@lsabor lsabor commented Nov 25, 2024

This takes a local 12 second operation down to 1.5 seconds.
On production, the process took up to 30 seconds.

More optimization is possible by stored "global_leaderboard_dates" directly on a question object instead, if necessary.

@lsabor lsabor requested review from elisescu and hlbmtc November 25, 2024 18:46
@lsabor lsabor changed the title preload global_leaderboard_dates global contributions endpoint quick optimization Nov 25, 2024
@lsabor
Copy link
Contributor Author
lsabor commented Nov 25, 2024

So the obvious problem was with leaderboard.get_questions() because for global leaderboards, it iterated through each question on the relevant project and tested if it's derived "global_leaderboard_dates" corresponded exactly to the leaderboard's open/close time. In this process, it rederived the global leaderboard windows each time, sometimes calling the relevant query 10,000 times. This loads those windows once and then checks through them.

Still not as elegant as it could be, but likely not worth a further refactor unless this issue arises again.

Optimal situation would be that questions are either linked directly to their leaderboards in a many-to-many relationship, or the global leaderboard dates get stored directly on them.

@hlbmtc hlbmtc merged commit 434ff13 into main Nov 25, 2024
2 checks passed
@hlbmtc hlbmtc deleted the fix/contributions-optimization branch November 25, 2024 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
0