8000 [Data] Replaced `get_object_locations` with `get_local_object_locations` by alexeykudinkin · Pull Request #53942 · ray-project/ray · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Data] Replaced get_object_locations with get_local_object_locations #53942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open

[Data] Replaced get_object_locations with get_local_object_locations #53942

wants to merge 22 commits into from

Conversation

alexeykudinkin
Copy link
Contributor
@alexeykudinkin alexeykudinkin commented Jun 19, 2025 •
edited
Loading

Why are these changes needed?

This change

  1. Replaces usages of get_object_locations API with get_local_object_locations to minimize overhead of fetching Object's metadata by always interacting exclusively with the local Object Manager.
  2. Abstracts common utils inside RefBundle.
  3. Removes misleading dataset_bytes_spilled stat

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@alexeykudinkin alexeykudinkin requested a review from a team as a code owner June 19, 2025 01:55
@alexeykudinkin alexeykudinkin changed the base branch from master to ak/pq-rd-mtd-fix June 19, 2025 06:10
@alexeykudinkin alexeykudinkin added the go add ONLY when ready to merge, run all tests label Jun 19, 2025
Base automatically changed from ak/pq-rd-mtd-fix to master June 19, 2025 07:35
@alexeykudinkin
Abstracted get_bytes_spilled util;
d4091ef 
Fixed metrics to track amount of *output* bytes spilled instead of input bytes;

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Tidying up
d3a19e9 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Fixing types
98c5e4e 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Tidying up more
191c0c3 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Make sure fetching object locations is gated
b8f50cb 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Relocated get_bytes_spilled into RefBundle to leverage local & ca…
78370c7 
…ched metadata;

Cleaning up

Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Deleting unnecessary configs
5bc76a0 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Deleting dataset_bytes_spilled as it's a pointless metric
2b17cc7 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Tidying up
99e3948 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Removed spilled_bytes metric (it's pointless and misleading)
3afd5f9 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Deleted refs to deprecated config
ed4a9b7 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Make sure cached object metadat is invalidated after 1s
dbc5bd9 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Relocated trace_locality to RefBundle
0b86db7 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Rebased iter_batches onto RefBundle.trace_locality
f06e48b 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Replaced get_object_locations with get_local_object_locations
0d257b4 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Missing clean ups
bcdab89 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Fixing usages
841b53d 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Cleaned up orphaned dataset_bytes_spilled usages
ba21a85 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
lint
ce7b638 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Deleting dead code
06b91c0 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
Fixed test_spilled_stats by breaking it down
e20ebd0 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
@alexeykudinkin
lint
b3804f5 
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
go add ONLY when ready to merge, run all tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@alexeykudinkin
0