10000 [run] parallelize benchmark runs · Issue #167 · clp-research/clemcore · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[run] parallelize benchmark runs #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
phisad opened this issue Mar 27, 2025 · 0 comments
Open

[run] parallelize benchmark runs #167

phisad opened this issue Mar 27, 2025 · 0 comments
Labels
dev: enhancement A new feature or improvement to existing functionality priority: low

Comments

@phisad
Copy link
Collaborator
phisad commented Mar 27, 2025

The adventuregame takes about 1-3 hours to run, when we run every episode on after the other.

However, the episodes run independently, so they running an episode could be parallelized.

We should add a "-p" option to the "run" command to enable parallel runs of the benchmark.

Possible side effects:

  • for not HF models this might flood the API services. Hence an appropriate delay must be set anyways.
  • for HF models we cannot easily load the model N times. However, the model would require to join all threads to batch the requests. Possibly slowing down again the execution.

A workaround for the HF case could be to start the model separately as a service that allows incoming streams of requests and batches them automatically. Possibly vllm can do this.
@phisad phisad added the dev: enhancement A new feature or improvement to existing functionality label Mar 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dev: enhancement A new feature or improvement to existing functionality priority: low
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@phisad
0