8000 add a default value for endpoint url option by reidliu41 · Pull Request #2839 · instructlab/instructlab · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

add a default value for endpoint url option #2839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+1 −1
Open

add a default value for endpoint url option #2839

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/instructlab/model/chat.py
View file
10000
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ def is_openai_server_and_serving_model(
@click.option(
"--endpoint-url",
type=click.STRING,
help="Custom URL endpoint for OpenAI-compatible API. Defaults to the `ilab model serve` endpoint.",
help="Custom URL endpoint for OpenAI-compatible API. Defaults to the `ilab model serve` endpoint (http://127.0.0.1:8000/v1).",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something that gets read from the config if not passed in as a flag? If so, what happens if it's a different value?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it(endpoint_url) will read config file. It based on host and port. If value like --port 1234, it need to specify --endpoint-url.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. But I check, if the option set it read from config, it should read it from configuration, not config file(please correct me if I am wrong), because change the value from config file, it will not change value from --help.
    e.g.
default:
@click.option(
    "--gpu-layers",
    type=click.INT,
    cls=clickext.ConfigOption,
    config_sections="llama_cpp",
    required=True,  # default from config
)

  --gpu-layers INTEGER        Number of model layers to offload to GPU. -1
                              means all layers. [required; default: -1; <<<<<<---
                              config: 'serve.llama_cpp.gpu_layers']

$ ilab config show -k serve.llama_cpp -wc
gpu_layers: -1     <<<<<<---
llm_family: ''
max_ctx_size: 4096

change:
$ ilab config show -k serve.llama_cpp -wc
gpu_layers: -2  <<<<<<---
llm_family: ''
max_ctx_size: 4096
  --gpu-layers INTEGER        Number of model layers to offload to GPU. -1
                              means all layers. [required; default: -1; <<<<<<---
                              config: 'serve.llama_cpp.gpu_layers']
  1. So based on this default value, the default endpoint_url value in --help should based on host: 127.0.0.1 and port: 8000 in the config field, unless change it. If it need to change, seems other also need to change to read it from config file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @RobotSail any suggestions that I need to modify? Actually, my purpose is simple just share the api example, may try to avoid the situation like #2287

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RobotSail any thoughts?

)
@click.option(
"--api-key",
Expand Down
Loading
0