Agent.run() can return RunResult object #1337

aymeric-roucher · 2025-05-16T11:10:33Z

No description provided.

aymeric-roucher · 2025-05-19T10:06:13Z

examples/agent_from_any_llm.py



 # Choose which inference type to use!

-available_inferences = ["hf_api", "hf_api_provider", "transformers", "ollama", "litellm", "openai"]
-chosen_inference = "hf_api_provider"
+available_inferences = ["inference_client", "transformers", "ollama", "litellm", "openai"]


Aligns this with the new name since #1198.

aymeric-roucher · 2025-05-19T10:06:37Z

examples/agent_from_any_llm.py

-elif chosen_inference == "hf_api_provider":
-    model = InferenceClientModel(provider="together")
+if chosen_inference == "inference_client":
+    model = InferenceClientModel(model_id="meta-llama/Llama-3.3-70B-Instruct", provider="nebius")


Specify provider "nebius" since they don't error out when using tool_call="required"

aymeric-roucher · 2025-05-19T10:07:13Z

src/smolagents/agents.py

+    """Holds extended information about an agent run."""
+
+    result: Any
+    token_usage: TokenUsage | None


This can be None in case the token usage cannot be obtained.

aymeric-roucher · 2025-05-19T12:04:57Z

src/smolagents/gradio_ui.py

-                            )
-                    except Exception as e:
-                        raise e
+                    msg = msg.replace("<", r"\<").replace(">", r"\>")  # HTML tags seem to break Gradio Chatbot


@albertvillanova escaping HTML tags fixe a hairy bug where HTML tagged messages wouldn't appear in the Gradio Chatbot. @yvrjsharma do you know why this is?

I agree this should be fixed in Gradio if possible.

Thanks for reporting this, I have tagged you both in a related issue on gradio repo.

aymeric-roucher · 2025-05-19T12:06:00Z

src/smolagents/models.py

@@ -1226,7 +1225,7 @@ class InferenceClientModel(ApiModel):

    def __init__(
        self,
-        model_id: str = "Qwen/Qwen2.5-Coder-32B-Instruct",
+        model_id: str = "Qwen/Qwen3-32B",


Update the default to the current best Qwen model!

This may raise an error for some providers?

Good point, reverting this and leaving it for later.

aymeric-roucher · 2025-05-19T12:07:12Z

@albertvillanova could you take a look, then if the UI appears good to you I'll add tests!

albertvillanova

Thanks. Just some comments/questions below. And as you suggested, more tests to be added.

src/smolagents/gradio_ui.py

albertvillanova · 2025-05-19T13:42:16Z

src/smolagents/gradio_ui.py

-                            )
-                    except Exception as e:
-                        raise e
+                    msg = msg.replace("<", r"\<").replace(">", r"\>")  # HTML tags seem to break Gradio Chatbot


I agree this should be fixed in Gradio if possible.

albertvillanova · 2025-05-19T13:59:29Z

src/smolagents/models.py

-        self.last_input_token_count: int | None = None
-        self.last_output_token_count: int | None = None


I like the new approach, but this is a breaking change: should we deprecate these parameters?

albertvillanova · 2025-05-19T14:02:00Z

src/smolagents/models.py

@@ -1226,7 +1225,7 @@ class InferenceClientModel(ApiModel):

    def __init__(
        self,
-        model_id: str = "Qwen/Qwen2.5-Coder-32B-Instruct",
+        model_id: str = "Qwen/Qwen3-32B",


This may raise an error for some providers?

albertvillanova · 2025-05-19T14:13:38Z

src/smolagents/agents.py

+    token_usage: TokenUsage | None
+    messages: list[dict]
+    timing: Timing
+    state: str


Could you describe the meaining of all attributes in the docstring? For example, we should describe what values can have state...

aymeric-roucher · 2025-05-20T12:07:19Z

src/smolagents/models.py

        self.model_id: str | None = model_id

+    @property
+    def last_input_token_count(self) -> int | None:


@albertvillanova WDYT of this implementation?

But you should emit a warning instead of logging one, no?

albertvillanova

Thanks. Just some comments.

albertvillanova · 2025-05-20T12:13:27Z

src/smolagents/models.py

@@ -1309,17 +1339,24 @@ def generate_stream(
        for event in self.client.chat.completions.create(
            **completion_kwargs, stream=True, stream_options={"include_usage": True}
        ):
+            if getattr(event, "usage", None):
+                print("EV:", event)


I guess you forgot this print

albertvillanova · 2025-05-20T12:14:26Z

src/smolagents/models.py

-            if getattr(event, "usage", None):
-                self.last_input_token_count = event.usage.prompt_tokens
-                self.last_output_token_count = event.usage.completion_tokens


You forgot these?

albertvillanova · 2025-05-20T12:14:37Z

src/smolagents/models.py

-            if getattr(event, "usage", None):
-                self.last_input_token_count = event.usage.prompt_tokens
-                self.last_output_token_count = event.usage.completion_tokens


albertvillanova · 2025-05-20T12:17:30Z

src/smolagents/monitoring.py

-    def get_total_token_counts(self):
-        return {
-            "input": self.total_input_token_count,
-            "output": self.total_output_token_count,
-        }
+    def get_total_token_counts(self) -> TokenUsage:
+        return TokenUsage(
+            input_tokens=self.total_input_token_count,
+            output_tokens=self.total_output_token_count,
+        )


For backward compatibility, maybe better:

keep get_total_token_counts as it was implemented before, but raising a deprecation warning

implement a new get_token_usage method?

I think it'll be more robust to not only keep the method get_total_token_counts, but also the underlying attributes self.total_input_token_count and self.total_output_token_counts, because people might be accessing these directly!

albertvillanova · 2025-05-20T12:19:06Z

src/smolagents/models.py

        self.model_id: str | None = model_id

+    @property
+    def last_input_token_count(self) -> int | None:


But you should emit a warning instead of logging one, no?

src/smolagents/models.py

albertvillanova

I just realized this.

src/smolagents/monitoring.py

Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

aymeric-roucher · 2025-05-20T13:38:11Z

@albertvillanova about your suggestion with defining duration through post_init: I applied it to TokenUsage : for Timing however, since this object will often be defined at step start with an empty end_time and the end_time is only filled later, thus the duration as well, it made more sense to keep the property definition!

…__`. After huggingface/smolagents#1337 we need to instrument all exported subclasses instead of the parent class. - bump "smolagents[mcp]>=1.17.0".

#350) * fix(smolagents): Instrument `Model.generate` instead of `Model.__call__`. After huggingface/smolagents#1337 we need to instrument all exported subclasses instead of the parent class. - bump "smolagents[mcp]>=1.17.0". * Fix uninstrument

Start work on run results

f1c6fa5

aymeric-roucher changed the title ~~Start work on run results~~ Agent.run() returns RunResult object May 16, 2025

aymeric-roucher added 2 commits May 16, 2025 16:31

Create Timing and Usage objects

46eb7c8

Improve usage class, add property caculations

f4c8acb

aymeric-roucher commented May 19, 2025

View reviewed changes

aymeric-roucher added 6 commits May 19, 2025 12:08

Revert deletion of step callbacks after max steps error

bbf194d

Rename attributes to token_usage

962922f

Update gradio UI for token usage

8dd6730

Merge branch 'main' into add-run-results

b076d11

Fix gradio chatbot as much as possible

795f76e

Fix gradio chatbot by escaping HTML tags

c7be43b

aymeric-roucher commented May 19, 2025

View reviewed changes

aymeric-roucher marked this pull request as ready for review May 19, 2025 12:06

aymeric-roucher added 2 commits May 19, 2025 14:28

Pass monitoring tests

3446f7e

Pass more tests

816466a

albertvillanova approved these changes May 19, 2025

View reviewed changes

aymeric-roucher added 8 commits May 19, 2025 16:47

Revert default LLM upgrade

9343611

Remove sleep

7a944b3

Re-add last_input_token_count attribute for Model

4de35e9

Add tests

d476420

Pass tests

f93846c

Pass memory test

eefc8a8

Pass agents test

55bd6c7

Revert model change in GAIA

3b796d9

aymeric-roucher commented May 20, 2025

View reviewed changes

albertvillanova reviewed May 20, 2025

View reviewed changes

src/smolagents/monitoring.py Outdated Show resolved Hide resolved

src/smolagents/monitoring.py Outdated Show resolved Hide resolved

yvrjsharma mentioned this pull request May 20, 2025

Style leakage in gr.Chatbot when rendering HTML code snippet gradio-app/gradio#11168

Open

1 task

aymeric-roucher and others added 6 commits May 20, 2025 15:03

Update src/smolagents/models.py

4e6b593

Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

Update src/smolagents/monitoring.py

4c92beb

Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

Update src/smolagents/monitoring.py

F438

a87ef70

Co-authored-by: Albert Villanova del Moral <8515462+albertvillanova@users.noreply.github.com>

Revert suggestion to avoid None durations

18b1849

Use post-init suggestion for TokenUsage, property for Timing

582a09e

Re-add deprecated token count increment in stream methods

e25715e

aymeric-roucher added 4 commits May 20, 2025 15:40

Fix dict conversion error

3a321e7

Test ActionStep.dict()

6dc4c6d

Fix edge case

56117ad

Fix even more tests

dbdb3fa

aymeric-roucher merged commit 1d90caf into main May 20, 2025
4 checks passed

aymeric-roucher changed the title ~~Agent.run() returns RunResult object~~ Agent.run() can return RunResult object May 20, 2025

aymeric-roucher mentioned this pull request May 23, 2025

Fix smolagents benchmark #1377

Merged

daavoo mentioned this pull request May 27, 2025

fix(smolagents): Instrument Model.generate instead of `Model.__call… mozilla-ai/any-agent#350

Merged

		self.last_input_token_count: int \| None = None
		self.last_output_token_count: int \| None = None

Agent.run() can return RunResult object #1337

Agent.run() can return RunResult object #1337

Uh oh!

Conversation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!