Open
Description
Not quite sure why this is happening yet, but for some reason GraphFrame.filter is hanging here:
Line 351 in f7ea385
I see all processes putting their result in the queue and returning, but things are hanging in the queue -- the problem seems to be a dataframe pickle issue. After adding a sleep(1) at the end of parallel_apply, i can see the error:
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/blaschke/local/virtualenv/py3/lib/python3.8/site-packages/dill/_dill.py", line 941, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 855, in save
6021
_str
self.write(SHORT_BINUNICODE + pack("<B", n) + encoded)
File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/pickle.py", line 243, in write
return self.current_frame.write(data)
RecursionError: maximum recursion depth exceeded while calling a Python object
Any ideas, anyone who knows more about dataframes and pickling? I am testing on a python cProfile