8000 Added Ray-Serve Config For LLMs by Blaze-DSP · Pull Request #3517 · ray-project/kuberay · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Added Ray-Serve Config For LLMs #3517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Added Ray-Serve Config For LLMs #3517

wants to merge 3 commits into from

Conversation

Blaze-DSP
Copy link

Added Example Config For Ray-Serve LLM

kouroshHakha
kouroshHakha reviewed May 1, 2025
View reviewed changes
Copy link
@kouroshHakha kouroshHakha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config looks good to me. (tho I haven't run the config myself)

@kouroshHakha kouroshHakha requested a review from kevin85421 May 1, 2025 05:39
@Blaze-DSP
Copy link
Author

Should I also add config for autoscaling?

@kevin85421
Copy link
Member

Chatted with @Blaze-DSP offline

@kouroshHakha
Copy link
kouroshHakha commented May 6, 2025 •
edited
Loading

What is the plan? @kevin85421

@kevin85421
Copy link
Member

What is the plan? @kevin85421

Add a doc in Ray repo and make this example simpler (e.g. remove LoRA).

@Blaze-DSP
Copy link
Author

I have updated the ray serve llm config and added doc for it in the ray repo.

PR For Doc.: ray-serve llm doc

@pcmoritz pcmoritz mentioned this pull request May 8, 2025
Open
8 tasks
secret added
2c84dc1 
Signed-off-by: DPatel_7 <dpatel@gocommotion.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewsykim andrewsykim andrewsykim requested changes

@kouroshHakha kouroshHakha kouroshHakha left review comments

@kevin85421 kevin85421 Awaiting requested review from kevin85421

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Blaze-DSP @kevin85421 @kouroshHakha @andrewsykim
0