8000 feat: improve eval by yuki-666 · Pull Request #325 · NVIDIA-NeMo/RL · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: improve eval #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
May 9, 2025
Merged

feat: improve eval #325

merged 10 commits into from
May 9, 2025

Conversation

yuki-666
Copy link
Contributor
@yuki-666 yuki-666 commented May 7, 2025 •
edited
Loading

What does this PR do ?

  1. update eval doc
  2. support average pass@1 over n samples per prompt
  3. update threshold in eval unit test

Issues

Closes #323

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 7, 2025
@yuki-666 yuki-666 added the CI:L1 Run doctests, unit tests, and functional tests label May 7, 2025
@yuki-666 yuki-666 force-pushed the yukih/improve-eval branch from be04f82 to c2e0544 Compare May 7, 2025 13:26
@yuki-666 yuki-666 requested a review from terrykong May 7, 2025 13:44
@yuki-666 yuki-666 force-pushed the yukih/improve-eval branch from eb20f36 to 41dc02a Compare May 7, 2025 15:00
yuki-666 added 5 commits May 8, 2025 05:31
@yuki-666
suppoort average pass@1 over n samples per prompt
a236b73 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666
add seed in eval
a5c306b 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666
update doc
ee4eae3 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666
rewrite eval.md
d0cdb1e 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666
update threshold in eval unit test
ad2f8c5 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666 yuki-666 force-pushed the yukih/improve-eval branch from 41dc02a to f54544d Compare May 8, 2025 09:26
@yuki-666
add eval to front page and add some notes
e350482 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666 yuki-666 force-pushed the yukih/improve-eval branch from f54544d to e350482 Compare May 8, 2025 10:03
@yuki-666 yuki-666 added CI:L1 Run doctests, unit tests, and functional tests CI:L0 Run doctests and unit tests and removed CI:L1 Run doctests, unit tests, and functional tests CI:L0 Run doctests and unit tests labels May 8, 2025
@yuki-666 yuki-666 marked this pull request as ready for review May 8, 2025 10:44
@terrykong
fix doctest
e433296 
Signed-off-by: Terry Kong <terryk@nvidia.com>
parthchadha
parthchadha previously approved these changes May 8, 2025
View reviewed changes
@parthchadha parthchadha added this pull request to the merge queue May 8, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 8, 2025
8000
jgerh
jgerh reviewed May 8, 2025
View reviewed changes
Copy link
Contributor
@jgerh jgerh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed the tech pubs review of the md files and left a few copyedits and suggested text additions.

@yuki-666
update doc
f8c605d 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666
Copy link
Contributor Author
yuki-666 commented May 9, 2025

Completed the tech pubs review of the md files and left a few copyedits and suggested text additions.

thanks @jgerh , have updated from your suggestions.

terrykong
terrykong previously approved these changes May 9, 2025
View reviewed changes
@terrykong terrykong added this pull request to the merge queue May 9, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch May 9, 2025
@terrykong terrykong added this pull request to the merge queue May 9, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 9, 2025
@yuki-666
fix link
9f99daa 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
@yuki-666 yuki-666 added CI:docs Run doctest and removed CI:L0 Run doctests and unit tests labels May 9, 2025
@parthchadha parthchadha added this pull request to the merge queue May 9, 2025
Merged via the queue into main with commit 790888f May 9, 2025
21 checks passed
@parthchadha parthchadha deleted the yukih/improve-eval branch May 9, 2025 16:28
YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jun 10, 2025
@yuki-666 @terrykong
feat: improve eval (NVIDIA-NeMo#325)
2a40212 
Signed-off-by: Yuki Huang <yukih@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Co-authored-by: Terry Kong <terryk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgerh jgerh jgerh left review comments

@parthchadha parthchadha parthchadha approved these changes

@terrykong terrykong terrykong left review comments

Assignees
No one assigned
Labels
CI:docs Run doctest documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve eval: improve doc and support multiple samples
4 participants
@yuki-666 @terrykong @parthchadha @jgerh
0