Closed
Description
What happened + What you expected to happen
A recent update to PyArrow introduced a backwards incompatibility (apache/arrow#45471 (comment)). This causes unexpected keyword argument 'maps_as_pydicts'.
ds.take_all()
...
File "pyarrow/array.pxi", line 987, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 5174, in pyarrow.lib.Table._to_pandas
File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/pyarrow/pandas_compat.py", line 825, in table_to_dataframe
_reconstruct_block(item, column_names, ext_columns_dtypes)
File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/pyarrow/pandas_compat.py", line 765, in _reconstruct_block
arr = pandas_dtype.__from_arrow__(arr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/ray/air/util/object_extensions/pandas.py", line 146, in __from_arrow__
return PythonObjectArray(array.to_pylist())
^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1380, in pyarrow.lib.ChunkedArray.to_pylist
File "pyarrow/array.pxi", line 1775, in pyarrow.lib.Array.to_pylist
TypeError: ArrowPythonObjectScalar.as_py() got an unexpected keyword argument 'maps_as_pydicts'
Versions / Dependencies
Ray and PyArrow latest.
Reproduction script
I'm in a rush, but this should do it:
ray.data.from_pandas(pd.DataFrame([{'id': i} for i in range(10)])).take_all()Issue Severity
Low: It annoys or frustrates me.