10000 Add support for limiting same table concurrency by hanefi · Pull Request #803 · dimitri/pgcopydb · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add support for limiting same table concurrency #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 6, 2024
Merged

Add support for limiting same table concurrency #803

merged 4 commits into from
Jun 6, 2024

Conversation

hanefi
Copy link
Contributor
@hanefi hanefi commented Jun 4, 2024 •
edited
Loading

This commit adds a new option --split-count-max-limit to limit the number of jobs that can be run concurrently for the same table. This option is useful when the number of tables is large and the system is running out of resources.

TODO:

  • Add a proper test. Possibly a unit test
  • Add a new env var for the new option
  • Update docs

Fixes: #799

dimitri
dimitri reviewed Jun 4, 2024
View reviewed changes
@hanefi hanefi force-pushed the split-count-max-limit branch from e9c48b5 to 1088871 Compare June 5, 2024 21:22
@hanefi hanefi marked this pull request as ready for review June 5, 2024 21:23
@hanefi hanefi force-pushed the split-count-max-limit branch 2 times, most recently from ff8f947 to 2719020 Compare June 5, 2024 21:28
@hanefi
Add support for limiting same table concurrency
2719020 
This commit adds a new option --split-max-parts to limit the number of
jobs that can be run concurrently for the same table. This option is
useful when the number of tables is large and the system is running out
of resources.

Optionally the user can also set the environment variable
COPYDB_SPLIT_MAX_PARTS to the desired value.
dimitri
dimitri requested changes Jun 6, 2024
View reviewed changes
Copy link
Owner
@dimitri dimitri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly nitpicking now.

dimitri
dimitri previously approved these changes Jun 6, 2024
View reviewed changes
Copy link
Owner
@dimitri dimitri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider one last change before merge.

@dimitri dimitri added the enhancement New feature or request label Jun 6, 2024
@dimitri dimitri added this to the v0.17 milestone Jun 6, 2024
@dimitri dimitri merged commit 047c842 into dimitri:main Jun 6, 2024
19 checks passed
@hanefi hanefi deleted the split-count-max-limit branch June 6, 2024 12:58
@hanefi hanefi mentioned this pull request Jun 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dimitri dimitri dimitri approved these changes

Assignees

@hanefi hanefi

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.17
Development

Successfully merging this pull request may close these issues.

Implement an option to limit maximum number of table splits
2 participants
@hanefi @dimitri
0