Open
Description
ray/rllib/utils/torch_utils.py
Line 702 in 168ae2e
and
ray/rllib/utils/debug/deterministic.py
Line 38 in 575d6af
The above lines manually set the value of the environment variable CUBLAS_WORKSPACE_CONFIG. There is a comment above the line which points to this pytorch issue. However, I suspect this is an error and the intended default value should be preceded with a colon, ie. :4096:8 rather than the current value of 4096:8.
This intuition is taken from this comment on a pytorch issue.
This minor difference makes my application logs extremely difficult to read, as they become cluttered with:
2024-09-16 11:11:26,220 INFO multi_agent_fire_harness.py:222 -- Agent agent_5 is now moving right.
2024-09-16 11:11:26,220 INFO agent.py:178 -- Agent agent_6 has completed a control line of length 2953 ft.
2024-09-16 11:11:26,221 INFO multi_agent_fire_harness.py:209 -- Agent agent_6 has completed the current mitigation. The current mitigation count for this agent is: 1
2024-09-16 11:11:26,221 INFO multi_agent_fire_harness.py:222 -- Agent agent_6 is now moving down.
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
2024-09-16 11:11:26,316 INFO fire_harness.py:324 -- Agent agent_4 is starting to set a control line on fuel model 'Brush' at position (0, 63).
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
2024-09-16 11:11:32,693 INFO fire_harness.py:324 -- Agent agent_5 is starting to set a control line on fuel model 'Brush' at position (126, 75).
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
/home/afennelly/simharness/.venv/lib/python3.9/site-packages/torch/nn/modules/linear.py:114: UserWarning: Could not parse CUBLAS_WORKSPACE_CONFIG, using default workspace size of 8519680 bytes. (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:60.)
return F.linear(input, self.weight, self.bias)
and so on. Please make this update ASAP !!!