perf: Apply backpressure on link counts #32712
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Context: This is QoL feature to highlight most used items. But in high throughput environments where a lot of new documents are being created this becomes a bottleneck. Fix: Limit the size of counts that can be buffered before they're flushed. Statistically this will still work just as well as it did before.
|
Merging for now, will port later w/ other things |
mergify bot
pushed a commit
that referenced
this pull request
May 29, 2025
Context: This is QoL feature to highlight most used items. But in high throughput environments where a lot of new documents are being created this becomes a bottleneck. Fix: Limit the size of counts that can be buffered before they're flushed. Statistically this will still work just as well as it did before. (cherry picked from commit 4d8ebcc) # Conflicts: # frappe/model/utils/link_count.py
mergify bot
pushed a commit
that referenced
this pull request
May 29, 2025
Context: This is QoL feature to highlight most used items. But in high throughput environments where a lot of new documents are being created this becomes a bottleneck. Fix: Limit the size of counts that can be buffered before they're flushed. Statistically this will still work just as well as it did before. (cherry picked from commit 4d8ebcc)
This was referenced May 29, 2025
ankush
added a commit
that referenced
this pull request
May 29, 2025
Context: This is QoL feature to highlight most used items. But in high throughput environments where a lot of new documents are being created this becomes a bottleneck. Fix: Limit the size of counts that can be buffered before they're flushed. Statistically this will still work just as well as it did before. (cherry picked from commit 4d8ebcc) Co-authored-by: Ankush Menat <ankush@frappe.io>
frappe-pr-bot
pushed a commit
that referenced
this pull request
Jun 3, 2025
## [15.69.3](v15.69.2...v15.69.3) (2025-06-03) ### Bug Fixes * allow `Float` fields to be empty ([d4ebe1d](d4ebe1d)) * default fields filter not working in web form ([6bd1c83](6bd1c83)) * Implement backpressure for background jobs (backport [#31489](#31489)) ([#32745](#32745)) ([4cee4ba](4cee4ba)) * permission error when permission docname is none ([776d810](776d810)) * Sample link counts and flush frequently ([#32713](#32713)) ([#32725](#32725)) ([56b03c8](56b03c8)) ### Performance Improvements * Apply backpressure on link counts ([#32712](#32712)) ([#32722](#32722)) ([d5ab9eb](d5ab9eb))
ankush
added a commit
that referenced
this pull request
Jun 4, 2025
* perf: Apply backpressure on link counts (#32712) Context: This is QoL feature to highlight most used items. But in high throughput environments where a lot of new documents are being created this becomes a bottleneck. Fix: Limit the size of counts that can be buffered before they're flushed. Statistically this will still work just as well as it did before. (cherry picked from commit 4d8ebcc) # Conflicts: # frappe/model/utils/link_count.py * chore: conflicts --------- Co-authored-by: Ankush Menat <ankush@frappe.io>
PratikBGhuge
pushed a commit
to PratikBGhuge/frappeApp
that referenced
this pull request
Jun 10, 2025
## [15.69.3](frappe/frappe@v15.69.2...v15.69.3) (2025-06-03) ### Bug Fixes * allow `Float` fields to be empty ([d4ebe1d](frappe/frappe@d4ebe1d)) * default fields filter not working in web form ([6bd1c83](frappe/frappe@6bd1c83)) * Implement backpressure for background jobs (backport [#31489](frappe/frappe#31489)) ([#32745](frappe/frappe#32745)) ([4cee4ba](frappe/frappe@4cee4ba)) * permission error when permission docname is none ([776d810](frappe/frappe@776d810)) * Sample link counts and flush frequently ([#32713](frappe/frappe#32713)) ([#32725](frappe/frappe#32725)) ([56b03c8](frappe/frappe@56b03c8)) ### Performance Improvements * Apply backpressure on link counts ([#32712](frappe/frappe#32712)) ([#32722](frappe/frappe#32722)) ([d5ab9eb](frappe/frappe@d5ab9eb))
frappe-pr-bot
pushed a commit
that referenced
this pull request
Jun 10, 2025
## [14.96.8](v14.96.7...v14.96.8) (2025-06-10) ### Bug Fixes * conflicts ([296acc0](296acc0)) * ensure proper type conversion for disable_prepared_report update ([#32329](#32329)) ([28c17db](28c17db)) * throw permission error ([#32860](#32860)) ([e848182](e848182)) ### Performance Improvements * Apply backpressure on link counts (backport [#32712](#32712)) ([#32721](#32721)) ([1be11bd](1be11bd))
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Brought to you by https://github.com/frappe/erpc
Context: This is a QoL feature to highlight the most used items. But in high throughput environments where a lot of new documents are being created, this becomes a major bottleneck.
Fix: Limit the size of counts that can be buffered before they're flushed.
Statistically, this will still work just as well as it did before... maybe not that well on large datasets because of the small sample size.
Before: 67K transactions/hour @ ~7s p99 latency
After: 98K transactions/hour @ ~2s p99 latency
Note: these runs have few more optimizations, but this PR is dominant one.