add a default value for endpoint url option #2839
Conversation
Signed-off-by: reid_liu <guliu@redhat.com>
| @@ -124,7 +124,7 @@ def is_openai_server_and_serving_model( | |||
| @click.option( | |||
| "--endpoint-url", | |||
| type=click.STRING, | |||
| help="Custom URL endpoint for OpenAI-compatible API. Defaults to the `ilab model serve` endpoint.", | |||
| help="Custom URL endpoint for OpenAI-compatible API. Defaults to the `ilab model serve` endpoint (http://127.0.0.1:8000/v1).", | |||
There was a problem hiding this comment.
Is this something that gets read from the config if not passed in as a flag? If so, what happens if it's a different value?
There was a problem hiding this comment.
yes, it(endpoint_url) will read config file. It based on host and port. If value like --port 1234, it need to specify --endpoint-url.
There was a problem hiding this comment.
- But I check, if the option set it read from config, it should read it from
configuration, not config file(please correct me if I am wrong), because change the value fromconfig file, it will not change value from--help.
e.g.
default:
@click.option(
"--gpu-layers",
type=click.INT,
cls=clickext.ConfigOption,
config_sections="llama_cpp",
required=True, # default from config
)
--gpu-layers INTEGER Number of model layers to offload to GPU. -1
means all layers. [required; default: -1; <<<<<<---
config
8000
: 'serve.llama_cpp.gpu_layers']
$ ilab config show -k serve.llama_cpp -wc
gpu_layers: -1 <<<<<<---
llm_family: ''
max_ctx_size: 4096
change:
$ ilab config show -k serve.llama_cpp -wc
gpu_layers: -2 <<<<<<---
llm_family: ''
max_ctx_size: 4096
--gpu-layers INTEGER Number of model layers to offload to GPU. -1
means all layers. [required; default: -1; <<<<<<---
config: 'serve.llama_cpp.gpu_layers']
- So based on this default value, the default endpoint_url value in
--helpshould based onhost: 127.0.0.1andport: 8000in the config field, unless change it. If it need to change, seems other also need to change to read it fromconfig file.
There was a problem hiding this comment.
hi @RobotSail any suggestions that I need to modify? Actually, my purpose is simple just share the api example, may try to avoid the situation like #2287
There was a problem hiding this comment.
@RobotSail any thoughts?
Yeah, sometimes it might be confused for the users, seems only the developer who created this option and he/she should very clear about this option/function. So make some simple tips, user can easy to use it, even a word. |
|
This pull request has merge conflicts that must be resolved before it can be |
|
This pull request has merge conflicts that must be resolved before it can be merged. @reidliu41 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork |
Some users might not know the default endpoint url is
url+v1=http://127.0.0.1:8000/v1, so add it in description.Checklist:
conventional commits.