Custom LLM API timeout instead of default 10 minutes for self-hosted LLMs #4076

Belerafon · 2025-05-28T13:29:37Z

Related GitHub Issue

Closes: #3621

Description

Why: Roo Code often emits API socket-timeout errors when using self-hosted, slow LLMs — either while reading a large file or even during the initial prompt phase. Yet these slow models can be usable in autonomous scenarios for working on some private code source. It would be great to have a user-tunable setting to configure the API timeout for self-hosted providers (OpenAI-compatible, Ollama, LMStudio, etc.). The timeout can be up to 30 minutes to allow big file chunks or prompts to be fully processed. This way, even an older laptop could produce meaningful results overnight with roo code.

What This PR adds a timeout option to the settings for LMStudio, Ollama, and OpenAI-Сompatible providers, and forwards this value into the OpenAI class constructor. Translations for all supported languages have been included.

Test Procedure

Connected an OpenAI-compatible client to a local slow LLM and tested timeout settings of 5, 10, and 20 minutes.

Observed socket-timeout errors and retries occurring according to the configured timeout.

Repeated manual testing for LMStudio provider with the same timeout values.

Type of Change

🐛 Bug Fix: Non-breaking change that fixes an issue.
✨ New Feature: Non-breaking change that adds functionality.
💥 Breaking Change: Fix or feature that would cause existing functionality to not work as expected.
♻️ Refactor: Code change that neither fixes a bug nor adds a feature.
💅 Style: Changes that do not affect the meaning of the code (white-space, formatting, etc.).
📚 Documentation: Updates to documentation files.
⚙️ Build/CI: Changes to the build process or CI configuration.
🧹 Chore: Other changes that don't modify src or test files.

Pre-Submission Checklist

Screenshots / Videos

Before

After

Documentation Updates

The UI guide should include:

Timeout for API requests to provider (min 5 min). If no response is received within this period, the request is retried. Increase this value for slower models.

Additional Notes

This is my first PR ever. So... do what you must.
Also, I am not a frontend developer, these code changes were done by LLM with my revisions.

Get in Touch

Important

This PR adds customizable API timeout settings for self-hosted LLMs, updating backend logic and frontend UI to support user-defined timeouts up to 30 minutes.

Behavior:
- Adds openAiApiTimeout, ollamaApiTimeout, and lmStudioApiTimeout settings in provider-settings.ts for customizable API timeouts.
- Updates LmStudioHandler, OllamaHandler, and OpenAiHandler to use the new timeout settings.
UI:
- Adds timeout input fields to LMStudio.tsx, Ollama.tsx, and OpenAICompatible.tsx for user configuration.
- Updates translations in multiple language files to include new timeout descriptions.
Testing:
- Manual testing with different timeout settings (5, 10, 20 minutes) for OpenAI-compatible and LMStudio providers.

^{This description was created by}^{for 55685aa. You can customize this summary. It will automatically update as commits are pushed.}

ellipsis-dev · 2025-05-28T13:31:52Z

webview-ui/src/components/settings/providers/LMStudio.tsx

+					})(),
+				}}
+				className="w-full mt-4">
+				<label className="block font-medium mb-1">{t("settings:providers.openAiApiTimeout")}</label>


The label translation key here is settings:providers.openAiApiTimeout, but this is in the LMStudio component and the value comes from lmStudioApiTimeout. Consider renaming the translation key to be consistent (e.g., lmStudioApiTimeout).

Suggested change

<label className="block font-medium mb-1">{t("settings:providers.openAiApiTimeout")}</label>

<label className="block font-medium mb-1">{t("settings:providers.lmStudio.apiTimeout")}</label>

Since the LMStudio, Ollama, and OpenAI-Compatible providers all use the OpenAI API library, the timeout name, description, and behavior are identical. Therefore, a common translation key is intentionally used for all three providers.

ellipsis-dev · 2025-05-28T13:31:52Z

webview-ui/src/components/settings/providers/Ollama.tsx

+				})}
+				type="text"
+				inputMode="numeric"
+				placeholder={t("settings:placeholders.numbers.maxTokens")}


The placeholder text is using the translation key settings:placeholders.numbers.maxTokens, which doesn't seem appropriate for a timeout field. Consider using a key that matches the timeout setting for consistency.

Suggested change

placeholder={t("settings:placeholders.numbers.maxTokens")}

placeholder={t("settings:providers.openAiApiTimeout")}

ellipsis-dev · 2025-05-28T13:31:53Z

webview-ui/src/components/settings/providers/Ollama.tsx

+					})(),
+				}}
+				className="w-full mt-4">
+				<label className="block font-medium mb-1">{t("settings:providers.openAiApiTimeout")}</label>


The label is using the translation key settings:providers.openAiApiTimeout, but this file is for Ollama. It looks like a copy-paste error. Consider updating this to a key that reflects Ollama (e.g., settings:providers.ollamaApiTimeout).

Suggested change

<label className="block font-medium mb-1">{t("settings:providers.openAiApiTimeout")}</label>

<label className="block font-medium mb-1">{t("settings:providers.ollamaApiTimeout")}</label>

ellipsis-dev · 2025-05-28T13:31:53Z

webview-ui/src/components/settings/providers/OpenAICompatible.tsx

+				})}
+				type="text"
+				inputMode="numeric"
+				placeholder={t("settings:placeholders.numbers.maxTokens")}


Typo / copy-paste issue: The placeholder translation key in this field is set to settings:placeholders.numbers.maxTokens, which doesn't match the field's purpose (API timeout). Consider using a more appropriate translation key (e.g. one related to timeout) for clarity.

Suggested change

placeholder={t("settings:placeholders.numbers.maxTokens")}

placeholder={t("settings:placeholders.numbers.timeout")}

This seems like a valid feedback, would using openAiApiTimeout here make more sense?

daniel-lxs · 2025-05-28T16:46:17Z

Hey @Belerafon, can you take another look at the translations? It seems that you could use translations that are more relevant.

FrancoFun · 2025-05-28T18:39:43Z

src/api/providers/lmstudio.ts

 		this.client = new OpenAI({
 			baseURL: (this.options.lmStudioBaseUrl || "http://localhost:1234") + "/v1",
 			apiKey: "noop",
+			timeout: timeoutMs,


Isn't the OpenAI class timeout in seconds though? https://github.com/openai/openai-python?tab=readme-ov-file#timeouts

Node.js OpenAI library’s TypeScript definitions (see screenshot), the explicitly states:

“The maximum amount of time (in milliseconds) that the client should wait for a response from the server before timing out a single request.”

Moreover, the built-in default is set here:

timeout: options.timeout ?? 600000 /* 10 minutes */
—so here it’s definitely milliseconds, not seconds. May be a python library in seconds...

Belerafon · 2025-05-28T20:18:22Z

Hey @Belerafon, can you take another look at the translations? It seems that you could use translations that are more relevant.

I’ve fixed the placeholder translation—it was clearly a bug. However, I’ve kept using
t("settings:providers.openAiApiTimeout") and
t("settings:providers.openAiApiTimeoutDescription")
for all three providers, since the texts are identical and they all rely on the same OpenAI library.

By the way, I’ve noticed an issue with the tests for my PR: while npm test passes locally on my Windows 11 host, the platform-unit-test (ubuntu-latest) job on GitHub is failing. I’m not entirely sure what’s going wrong and would appreciate any help.

daniel-lxs · 2025-05-28T21:03:23Z

Hey @Belerafon, can you try rebasing your branch against main?

Belerafon · 2025-05-29T07:17:59Z

Hey @Belerafon, can you try rebasing your branch against main?

Looks like I did it

FrancoFun · 2025-05-30T01:47:21Z

My guess is that you will also have to update the test. It probably fails because it doesn't expect a timeout parameter:

api/providers/__tests__/openai.spec.ts:101:30
it("should set default headers correctly", () => {
// Check that the OpenAI constructor was called with correct parame…

Adding:
timeout: expect.any(Number),
in openai.spec.ts will hopefully do the trick.

I don't see this specific test for Ollama and LMStudio, so they should be fine.

Roo Code often emits API socket-timeout errors when using self-hosted, slow LLMs — either while reading a large file or even during the initial prompt phase. Yet these slow models can be usable in autonomous (e.g. overnight) scenarios for working on some private code source. It would be great to have a user-tunable setting to configure the API timeout for self-hosted providers (OpenAI-compatible, Ollama, LMStudio, etc.). The timeout can be up to 30 minutes to allow big file chunks or prompts to be fully processed. This way, even an older laptop could produce meaningful results overnight with roo code.

Update the test to verify that the OpenAI client is initialized with a timeout parameter, while maintaining the existing test structure and assertions for other configuration options.

vercel · 2025-05-30T12:38:10Z

@Belerafon is attempting to deploy a commit to the Roo Code Team on Vercel.

A member of the Team first needs to authorize it.

Belerafon · 2025-05-30T12:51:37Z

Adding: timeout: expect.any(Number), in openai.spec.ts will hopefully do the trick.

Thanks for your help, now the test passes! except [Vercel] - Requires authorization to deploy. I have no idea why I would need to deploy something somewhere with my small commit, but I guess it's beyond my capabilities.

Belerafon requested review from mrubens and cte as code owners May 28, 2025 13:29

github-project-automation bot added this to Roo Code Roadmap and Roo Code Roadmap May 28, 2025

github-project-automation bot moved this to New in Roo Code Roadmap May 28, 2025

github-project-automation bot moved this to 8000 Triage in Roo Code Roadmap May 28, 2025

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels May 28, 2025

ellipsis-dev bot reviewed May 28, 2025

View reviewed changes

daniel-lxs moved this from Triage to PR [Draft / In Progress] in Roo Code Roadmap May 28, 2025

hannesrudolph added the PR - Draft / In Progress label May 28, 2025

FrancoFun reviewed May 28, 2025

View reviewed changes

daniel-lxs added PR - Changes Requested and removed PR - Changes Requested labels May 28, 2025

Belerafon force-pushed the API_Request_Timeout branch from 46ce635 to 9827fd3 Compare May 29, 2025 07:09

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels May 29, 2025

Belerafon added 3 commits May 30, 2025 15:34

Placeholder text bugfix

fe4c263

test(openai): add timeout check to OpenAI client initialization test

60c6926

Update the test to verify that the OpenAI client is initialized with a timeout parameter, while maintaining the existing test structure and assertions for other configuration options.

Belerafon force-pushed the API_Request_Timeout branch from 9827fd3 to 60c6926 Compare May 30, 2025 12:38

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels May 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Custom LLM API timeout instead of default 10 minutes for self-hosted LLMs #4076

Custom LLM API timeout instead of default 10 minutes for self-hosted LLMs #4076

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	<label className="block font-medium mb-1">{t("settings:providers.openAiApiTimeout")}</label>
	<label className="block font-medium mb-1">{t("settings:providers.lmStudio.apiTimeout")}</label>

	placeholder={t("settings:placeholders.numbers.maxTokens")}
	placeholder={t("settings:providers.openAiApiTimeout")}

Custom LLM API timeout instead of default 10 minutes for self-hosted LLMs #4076

Are you sure you want to change the base?

Custom LLM API timeout instead of default 10 minutes for self-hosted LLMs #4076

Conversation

Uh oh!

Related GitHub Issue

Description

Test Procedure

Type of Change

Pre-Submission Checklist

Screenshots / Videos

Documentation Updates

Additional Notes

Get in Touch

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!