Open
Description
What happened + What you expected to happen
I'm running ray serve with k8, and I set up the object spilling config as follows:
containers:
- name: ray-worker
env:
- name: "RAY_object_spilling_config"
value: "{\"type\":\"smart_open\",\"params\":{\"uri\":\"gs://some/bucket/path\"}}"
But this is the error that I got for all of the writes.
Traceback (most recent call last):
File "python/ray/_raylet.pyx", line 1249, in ray._raylet.spill_objects_handler
File "python/ray/_raylet.pyx", line 1252, in ray._raylet.spill_objects_handler
File "//.pyenv/versions/3.10.4/lib/python3.10/site-packages/ray/_private/external_storage.py", line 668, in spill_objects
return _external_storage.spill_objects(object_refs, owner_addresses)
File "//.pyenv/versions/3.10.4/lib/python3.10/site-packages/ray/_private/external_storage.py", line 541, in spill_objects
return self._write_multiple_objects(
File "//.pyenv/versions/3.10.4/lib/python3.10/site-packages/ray/_private/external_storage.py", line 150, in _write_multiple_objects
assert written_bytes == payload_len
Under which circumstances would this be false?
Versions / Dependencies
Docker base image: rayproject/ray:2.2.0-py310-gpu
ray[serve]==2.2.0
Reproduction script
I cannot reproduce it in dev setup unfortunately. : ( This works fine locally.
import ray
import smart_open
from ray._private import external_storage
buf = b'0' * 1024 * 1024
object_ref = ray.put(buf)
f = smart_open.open('gs://some/path/test.pkl'), 'wb')
e = external_storage.ExternalStorageSmartOpenImpl('gs://some/path')
e._write_multiple_objects(f, [object_ref], [b'test_address'], 'gs://some/path')
Issue Severity
Medium: It is a significant difficulty but I can work around it.