8000 re config eval by XuhuiZhou · Pull Request #315 · sotopia-lab/sotopia · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

re config eval #315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 26, 2025
Merged

re config eval #315

merged 2 commits into from
Apr 26, 2025

Conversation

XuhuiZhou
Copy link
Member
@XuhuiZhou XuhuiZhou commented Apr 23, 2025 •
edited
Loading

Closes #

📑 Description

Fix the issue of re-evaluating sotopia episodes

Previously we closed this by removing the rewards prompt being saved, which render the reveal script not really working. This pull request fix that.
#21

✅ Checks

  • My pull request adheres to the code style of this project
  • My code requires changes to the documentation
  • I have updated the documentation as required
  • All the tests have passed
  • Branch name follows type/descript (e.g. feature/add-llm-agents)
  • Ready for code review

ℹ Additional Information

@codecov Codecov
Copy link
codecov bot commented Apr 23, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 72.67%. Comparing base (7a573df) to head (1527e14).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sotopia/server.py 0.00% 1 Missing ⚠️ 
@@           Coverage Diff           @@
##             main     #315   +/-   ##
=======================================
  Coverage   72.67%   72.67%           
=======================================
  Files          67       67           
  Lines        3820     3820           
=======================================
  Hits         2776     2776           
  Misses       1044     1044
Files with missing lines Coverage Δ
sotopia/database/logs.py 94.54% <100.00%> (ø)
sotopia/server.py 42.01% <0.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@XuhuiZhou XuhuiZhou requested a review from ProKil April 24, 2025 00:47
@ProKil
Copy link
Member
ProKil commented Apr 24, 2025

Sorry can you specify what issues with the reevaluation? And how this PR solves them?

@XuhuiZhou
Copy link
Member Author

Sorry can you specify what issues with the reevaluation? And how this PR solves them?

@ProKil Just added!

ProKil
ProKil approved these changes Apr 24, 2025
View reviewed changes
examples/benchmark_evaluator.py
@@ -15,8 +15,8 @@

target_model_patterns: list[list[str]] = [
["gpt-4", "gpt-4", "gpt-3.5-turbo"],
["gpt-4", "gpt-4o-mini", "gpt-4"],
["gpt-4", "gpt-4o-mini", "togethercomputer/llama-2-70b-chat"],
["gpt-4", "gpt-3.5-turbo", "gpt-4"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would 3.5 turbo be deprecated soon?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just a step to fetch the human annotated data, does not matter if it will be dperecated or not

@XuhuiZhou XuhuiZhou merged commit 5a6fa7a into main Apr 26, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ProKil ProKil ProKil approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@XuhuiZhou @ProKil
0