Description
Hi there,
We test the new version verl 0.4.0, using SGLang as the rollout backend. However, we found that the performance of verl 0.4.0 is slower than verl 0.3.0 when running the same script. In verl 0.3.0, the running time of 10 steps in average is as follows:
{
'timing_s/gen': 81.85025,
'timing_s/old_log_prob': 56.87424999999999,
'timing_s/ref': 54.18241666666666,
'timing_s/update_actor': 151.79700000000003,
'timing_s/step': 348.087
}In verl 0.4.0, the running time of 10 steps in average is as follows:
{
'timing_s/gen': 70.84241666666667,
'timing_s/old_log_prob': 74.38166666666667,
'timing_s/ref': 71.17658333333333,
'timing_s/update_actor': 167.71325000000002,
'timing_s/step': 387.78775
}The running time of old_log_prob, ref and update_actor in verl 0.4.0 increases obvisously compared to 0.3.0. And the GPU untilization rate decreased from 95%+ to around 85% when executing old_log_prob, ref and update_actor part.
The key differences of the two running environment is the version of verl and the version of verl dependency. The differences of the verl dependency are listed below:
| verl 0.3.0 dependency | verl 0.4.0 dependency |
|---|---|
| sglang 0.4.5 | sglang 0.4.6.post5 |
| sgl-kernel 0.0.5.post3 | sglkernel 0.1.4 |
| megatron 0.11.0 | megatron 0.12.0 |
| - | transformer-engine 2.2.0 |
| - | cuda-bindings 12.8.0 |
| flashinfer 0.2.4 | flashinfer 0.2.5 |
I think the performance of actor is most related to megatron, which is upgraded from 0.11.0 to 0.12.0, and megatron 0.12.0 depends on transformer-engine package. But I can not understand why the higher version of megatron get worse performance. Is there any suggestions to solve this problem? Thanks in advance.