8000 feat: tool calling custom interfaces tasks extension by jmatejcz · Pull Request #636 · RobotecAI/rai · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: tool calling custom interfaces tasks extension #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 18 commits into
base: development
Choose a base branch
from
Open

feat: tool calling custom interfaces tasks extension #636

wants to merge 18 commits into from

Conversation

10000
jmatejcz
Copy link
Contributor
@jmatejcz jmatejcz commented Jun 17, 2025 •
edited
Loading

Purpose

Make new custom interfaces tasks for tool calling benchmark, test and adjust task prompts and system prompt
For now custom interfaces tasks revolve around checking the interface of given topic/service and publish/call it once. Also majority of them aren't predefined and ready to import.

Proposed Changes

  • Changed prompts in existing tasks

  • Added 3 hard tasks that require calling multiple services/topics

  • Total of 12 different tasks

  • Predefined total of 18 tasks

  • Added tests for predefined tasks

  • Refactored timeout to depend on number of required calls

Issues

Testing

  1. Mark only cusotm interfaces tasks, i recommend:
model_names = ["gpt-4o-mini"]
    vendors = ["openai"]

    # Define benchmarks that will be used
    mani_conf = ManipulationO3DEBenchmarkConfig(
        o3de_config_path="src/rai_bench/rai_bench/manipulation_o3de/predefined/configs/o3de_config.yaml",  # path to your o3de config
        levels=[  # define what difficulty of tasks to include in benchmark
            "trivial",
        ],
        repeats=1,  # how many times to repeat
    )
    tool_conf = ToolCallingAgentBenchmarkConfig(
        extra_tool_calls=[5],  # how many extra tool calls allowed to still pass
        task_types=[  # what types of tasks to include
            # "basic",
            # "spatial_reasoning",
            # "navigation",
            "custom_interfaces",
            # "manipulation",
        ],
        complexities=["easy"]
        N_shots=[2],  # examples in system prompt
        prompt_detail=["descriptive"],  # how descriptive should task prompt be
        repeats=1,
    )
  1. then run;
python src/rai_bench/rai_bench/examples/benchmarking_models.py
  1. View results, check if everything works
  2. Run tests
pytest tests/rai_bench/tool_calling_agent
  1. Check the task prompts in rai_bench/tool_calling_agent/tasks/basic.py if they make sense to you
  2. Check the subtasks in rai_bench/tool_calling_agent/predefined/tasks_tasks.py if they make sense to you

@jmatejcz jmatejcz force-pushed the jm/feat/custom-interfaces-tasks branch from 9326585 to a70250c Compare June 17, 2025 14:20
@jmatejcz jmatejcz mentioned this pull request Jul 1, 2025
Merged
@jmatejcz jmatejcz force-pushed the jm/feat/basic-tasks branch from 8fa85fd to f84f5b6 Compare July 1, 2025 10:42
@jmatejcz jmatejcz changed the base branch from jm/feat/basic-tasks to jm/feat/basic-tasks-extension July 1, 2025 13:20
Base automatically changed from jm/feat/basic-tasks-extension to development July 3, 2025 07:17
@jmatejcz jmatejcz force-pushed the jm/feat/custom-interfaces-tasks branch from 89293c0 to d996e02 Compare July 3, 2025 08:23
@jmatejcz
Copy link
Contributor Author
jmatejcz commented Jul 4, 2025 •
edited
Loading

@boczekbartek
I've tries to adjust these changes to things we disscused in basic tasks, so to make default validators, I've reduced the number of parameters and variables required, but these custom interfaces tasks has especially large number of consts, variables and parameters as they require filling Interfaces which can be long.

I didnt change logic in these last commits but moved a lot of code, so please let me know if this is somewhat clear and made with sense now, because i'm a little bit dizzy with all these changes haha

This commits also have some changes to basic tasks, as i just removed or moved some redundant code, which i missing in last PR

@jmatejcz jmatejcz marked this pull request as ready for review July 4, 2025 12:29
@jmatejcz jmatejcz requested a review from boczekbartek July 4, 2025 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boczekbartek boczekbartek Awaiting requested review from boczekbartek

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jmatejcz
0