Pass vLLM (VLLMModel) model client params as `client_kwargs` #1137

sergiopaniego · 2025-04-03T09:31:34Z

This support is already present in another model client such as HfApiModel and OpenAIServerModel.

I've tested it with the following code:

from smolagents import CodeAgent, DuckDuckGoSearchTool, VLLMModel

# Import search tool
search_tool = DuckDuckGoSearchTool()

# Load model
model_name = "google/gemma-3-4b-it"
# model = VLLMModel(model_id=model_name)
model = VLLMModel(model_id=model_name, client_kwargs={"max_model_len": 65536}) # new functionality!

# Initialize the agent with search_tool
agent = CodeAgent(tools=[search_tool], model=model)

# Run it!
result = agent.run("How did Real Madrid do in their last La Liga match?")
result

BTW, there are currently no tests for VLLMModel in test_models.py and it's not described in the guided tour docs.

albertvillanova

Thanks for addressing this issue.

While I agree with providing users the ability to customize their model configuration, I have concerns about the proposed naming convention.

The term client_kwargs is typically used in our codebase for API-based models that need a client to request remote inference (like in HfApiModel, OpenAIServerModel, etc.). However, vLLM is a local inference library rather than a remote API client.

A more appropriate approach might be to rename this to model_kwargs(?) or llm_kwargs to better reflect that these parameters configure a local model instance rather than a remote API client. This would maintain consistency with how we distinguish between local models and API-based models elsewhere in the codebase.

Otherwise, I think the functionality itself is valuable and should be implemented.

sergiopaniego · 2025-04-03T13:53:13Z

Thanks for the review!
I've updated it to model_kwargs, as you suggested. It feels more natural now

albertvillanova

Thanks!

Hellisotherpeople · 2025-04-03T18:32:55Z

Big love for this!

sergiopaniego added 3 commits April 3, 2025 11:14

Pass vLLM model params as client_kwargs

2a08f5f

Updated code quality

b5c068a

Updated code quality using ruff

ad89bc0

sergiopaniego requested a review from albertvillanova April 3, 2025 09:41

albertvillanova reviewed Apr 3, 2025

View reviewed changes

client_kwargs to model_kwargs

df2d50d

albertvillanova approved these changes Apr 3, 2025

View reviewed changes

aymeric-roucher merged commit b781114 into huggingface:main Apr 3, 2025
3 checks passed

sergiopaniego deleted the vllm-model-params branch April 4, 2025 07:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pass vLLM (VLLMModel) model client params as `client_kwargs` #1137

Pass vLLM (VLLMModel) model client params as `client_kwargs` #1137

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pass vLLM (VLLMModel) model client params as client_kwargs #1137

Pass vLLM (VLLMModel) model client params as client_kwargs #1137

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pass vLLM (VLLMModel) model client params as `client_kwargs` #1137

Pass vLLM (VLLMModel) model client params as `client_kwargs` #1137