8000 [Data] Fix parallelism deriving heuristic to ensure parallelism stays w/in min/max bounds by alexeykudinkin · Pull Request #47695 · ray-project/ray · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Data] Fix parallelism deriving heuristic to ensure parallelism stays w/in min/max bounds #47695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[Data] Fix parallelism deriving heuristic to ensure parallelism stays w/in min/max bounds #47695

wants to merge 2 commits into from

Conversation

alexeykudinkin
Copy link
Contributor
8000

Why are these changes needed?

Currently, min/max parallelism isn't actually being enforced correctly -- for large enough clusters we will be scaling out too aggressively purely based on the # of available CPUs disregarding the target block-sizes.

This change

  • Fixes parallelism detection heuristic to appropriately respect min/target block-sizes
  • Makes block-sizes configs' defaults env-var-configurable
  • Adjusts default min-block-size from 1Mb to 16Mb

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@alexeykudinkin alexeykudinkin added the go add ONLY when ready to merge, run all tests label Sep 17, 2024
@alexeykudinkin alexeykudinkin force-pushed the ak/prlsm-drv-fix branch 2 times, most recently from 206c0a6 to b99aea7 Compare October 1, 2024 07:26
raulchen
raulchen approved these changes Oct 18, 2024
View reviewed changes
python/ray/data/_internal/util.py

assert (
min_safe_parallelism <= max_reasonable_parallelism
), f"Parallelism boundaries have to overlap: {estimation_context}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, use an if statement to avoid generating the error message string when not needed.

@alexeykudinkin alexeykudinkin linked an issue Nov 12, 2024 that may be closed by this pull request
[data] ray.data.read_parquet not reading in one file per block #47437
Open
@alexeykudinkin alexeykudinkin mentioned this pull request Nov 12, 2024
Open
@stale Stale
Copy link
stale bot commented Feb 1, 2025

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

  • If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@stale stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Feb 1, 2025
@alexeykudinkin alexeykudinkin removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Feb 7, 2025
@pcmoritz pcmoritz requested a review from a team as a code owner March 26, 2025 22:34
@stale Stale
Copy link
stale bot commented Apr 26, 2025

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

  • If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@stale stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Apr 26, 2025
@alexeykudinkin
Making block-size configs' defaults env-var configurable
2eedffc 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
lint
51ddca6 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin alexeykudinkin removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label May 7, 2025
@github-actions GitHub Actions
Copy link
github-actions bot commented Jun 2, 2025

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jun 2, 2025
@github-actions GitHub Actions
Copy link

This pull request has been automatically closed because there has been no more activity in the 14 days
since being marked stale.

Please feel free to reopen or open a new pull request if you'd still like this to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for your contribution!

@github-actions github-actions bot closed this Jun 17, 2025
@github-actions github-actions bot removed the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jun 20, 2025
@richardliaw richardliaw added the data Ray Data-related issues label Jun 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raulchen raulchen raulchen approved these changes

Assignees

@raulchen raulchen

Labels
data Ray Data-related issues go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[data] ray.data.read_parquet not reading in one file per block
3 participants
@alexeykudinkin @raulchen @richardliaw
0