8000 verl v0.2.1 & v0.3 release checklist · Issue #354 · volcengine/verl · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

verl v0.2.1 & v0.3 release checklist #354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
11 of 17 tasks
eric-haibin-lin opened this issue Feb 23, 2025 · 21 comments
Closed
11 of 17 tasks

verl v0.2.1 & v0.3 release checklist #354

eric-haibin-lin opened this issue Feb 23, 2025 · 21 comments
Labels
roadmap

Comments

@eric-haibin-lin
Copy link
Collaborator
eric-haibin-lin commented Feb 23, 2025 •
edited
Loading

v0.2.1

v0.3

feel free to propose features (contributions are welcome!)
@eric-haibin-lin eric-haibin-lin pinned this issue Feb 23, 2025
@BearBiscuit05
Copy link
Collaborator

What can I help about the 'tool calling examples' part?

@eric-haibin-lin
Copy link
Collaborator Author

What can I help about the 'tool calling examples' part?

related to:
#344
#340

under the hood chat calls generate, so the design is supposed to work. just need to provide a working/stable example

@liyu199809
Copy link

Will megatron context parallelism be supported in the future?

@vermouth1992
Copy link
Collaborator

Will megatron context parallelism be supported in the future?

Yes. We will use mcore that supports cp by default.

@casper-hansen
Copy link

@BearBiscuit05 See #344, I outlined the main challenge. I think it should be relatively straightforward if veRL can start using chat or vLLM directly adds support for tool calling in generate.

I imagine we can have GRPO-trained reasoners in the future that learns when to use tools as part of their <think> tags, e.g. to execute code for a feedback loop or retrieve additional information.

@vermouth1992
Copy link
Collaborator

@BearBiscuit05 See #344, I outlined the main challenge. I think it should be relatively straightforward if veRL can start using chat or vLLM directly adds support for tool calling in generate.

I imagine we can have GRPO-trained reasoners in the future that learns when to use tools as part of their <think> tags, e.g. to execute code for a feedback loop or retrieve additional information.

I talked to vllm maintainer yesterday. It seems that there should be no blocking if we switch from generate to chat. Do you mind give it a try to call chat using SPMD style offline inference?

@BearBiscuit05
Copy link
Collaborator

Not very familiar with inference, but I think I’m starting to get the hang of it. Does this mean I need to build a new chat function and add extra params that include tool calls to invoke generate? Or should I just replace generate directly with the chat function from vllm?

@casper-hansen
Copy link

You should be able to replace generate directly with chat. The only problem is that we currently pass tokenized inputs into generate where as chat expects List[ChatCompletionContentPartTextParam] or List[List[ChatCompletionContentPartTextParam]]. Not sure what the best design would be in this case.

Case 1: Detokenize the tokenized 8000 inputs we use for generate.
Case 2: Change veRL to not tokenize datasets before-hand (relatively big change)

class ChatCompletionContentPartTextParam(TypedDict, total=False):
    text: Required[str]
    """The text content."""

    type: Required[Literal["text"]]
    """The type of the content part."""

@vermouth1992
Copy link
Collaborator
vermouth1992 commented Feb 24, 2025 •
edited
Loading

The second choice would incur significant overhead when tokenizing on-the-fly (typically 2x slowdown in generation, which is basically unacceptable). I guess we will need to seek solution for case 1

@BearBiscuit05
Copy link
Collaborator

Got it. I'll give it a try.

@liyu199809
Copy link

未来会支持 megatron 上下文并行吗？

是的。我们将默认使用支持cp的mcore。

It seems that the context parallelism in the model part has not been implemented yet. Is this function currently available?

@BearBiscuit05
Copy link
Collaborator

未来会支持 megatron 上下文并行吗？

是的。我们将默认使用支持cp的mcore。

It seems that the context parallelism in the model part has not been implemented yet. Is this function currently available?

Not right now, but if you check this roadmap, once verl upgrades MCore, cp will be support.

@AIBionics AIBionics mentioned this issue Feb 25, 2025
Open
@casper-hansen
Copy link

Is it possible to optimize startup time? I noticed when using veRL, it is significantly slower to launch a job than when using Huggingface TRL
#384

@maksimstw
Copy link
Contributor
maksimstw commented Mar 2, 2025 •
edited
Loading

Disabling torch.compile is useful, as it can also hang PPO training when enabling use_remove_padding. #387

@eric-haibin-lin
Copy link
Collaborator Author

Disabling torch.compile is useful, as it can also hang PPO training when enabling use_remove_padding. #387

@maksimstw thanks for the feedback! Would you like to provide a PR with this option?

@Llipengll
Copy link

when will you release the "sglang integration" part?

@casper-hansen casper-hansen mentioned this issue Mar 6, 2025
Closed
@hongpeng-guo hongpeng-guo mentioned this issue Mar 12, 2025
Merged
@wuxibin89 wuxibin89 mentioned this issue Mar 13, 2025
Merged
@JarvisFei
Copy link

v0.2.1

v0.3

feel free to propose features (contributions are welcome!)

how to install v0.3

vermouth1992 pushed a commit that referenced this issue Mar 14, 2025
@wuxibin89
doc: add multinode training and debug tutorial (#585)
9ae01af 
#354
8000 PeterSH6 pushed a commit that referenced this issue Mar 14, 2025
@hongpeng-guo
[Config] Providing an option to turn off torch.compile in actor (#554)
5457469 
## Summary

Providing an option in the config to turn off the `torch.compile` used
in `dp_actor.py`

## Usage

Adding the following line to the driver or cli scripts to turn off
`torch.compile`.
```python
+actor_rollout_ref.actor.use_torch_compile=False
```
Otherwise, `torch.compile` will be used by default

## Related Issue

#354 #245

---------

Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
@hongpeng-guo
Copy link
Collaborator

add an option to remove the call of torch.compile

Item solved in #554

@eric-haibin-lin
Copy link
Collaborator Author

hi @JarvisFei , v0.3 is not fully released but you are welcome to try verl main branch with pip install -e with the source code

@eric-haibin-lin
Copy link
Collaborator Author

As we are already making quite some progress in the main branch, I suggest we freeze code week for v0.3 and push the rest of the features to v0.4

@eric-haibin-lin
Copy link
Collaborator Author

Moving discussions to #710

@vermouth1992 vermouth1992 unpinned this issue Apr 2, 2025
wangyuchen333 pushed a commit to wangyuchen333/verl that referenced this issue Apr 25, 2025
@wuxibin89
doc: add multinode training and debug tutorial (volcengine#585)
32513d2 
volcengine#354
wangyuchen333 pushed a commit to wangyuchen333/verl that referenced this issue Apr 25, 2025
@hongpeng-guo
[Config] Providing an option to turn off torch.compile in actor (vo…
1928606 
…lcengine#554)

## Summary

Providing an option in the config to turn off the `torch.compile` used
in `dp_actor.py`

## Usage

Adding the following line to the driver or cli scripts to turn off
`torch.compile`.
```python
+actor_rollout_ref.actor.use_torch_compile=False
```
Otherwise, `torch.compile` will be used by default

## Related Issue

volcengine#354 volcengine#245

---------

Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
histmeisah pushed a commit to SJTU-IAAR/verl that referenced this issue Apr 27, 2025
@wuxibin89
doc: add multinode training and debug tutorial (volcengine#585)
ef9b885 
volcengine#354
histmeisah pushed a commit to SJTU-IAAR/verl that referenced this issue Apr 27, 2025
@hongpeng-guo
[Config] Providing an option to turn off torch.compile in actor (vo…
d2c8985 
…lcengine#554)

## Summary

Providing an option in the config to turn off the `torch.compile` used
in `dp_actor.py`

## Usage

Adding the following line to the driver or cli scripts to turn off
`torch.compile`.
```python
+actor_rollout_ref.actor.use_torch_compile=False
```
Otherwise, `torch.compile` will be used by default

## Related Issue

volcengine#354 volcengine#245

---------

Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
yumc-afk pushed a commit to yumc-afk/verl that referenced this issue May 18, 2025
@wuxibin89
doc: add multinode training and debug tutorial (volcengine#585)
2d5b2cb 
volcengine#354
yumc-afk pushed a commit to yumc-afk/verl that referenced this issue May 18, 2025
@hongpeng-guo
[Config] Providing an option to turn off torch.compile in actor (vo…
3e94e10 
…lcengine#554)

## Summary

Providing an option in the config to turn off the `torch.compile` used
in `dp_actor.py`

## Usage

Adding the following line to the driver or cli scripts to turn off
`torch.compile`.
```python
+actor_rollout_ref.actor.use_torch_compile=False
```
Otherwise, `torch.compile` will be used by default

## Related Issue

volcengine#354 volcengine#245

---------

Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
roadmap
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@eric-haibin-lin @vermouth1992 @hongpeng-guo @JarvisFei @casper-hansen @maksimstw @Llipengll @BearBiscuit05 @liyu199809
0