8000 Add translation for xinference by Dawnfz-Lenfeng · Pull Request #10 · godaai/scale-py · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
8000

Add translation for xinference #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 6, 2025
Merged

Add translation for xinference #10

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2e5d4cb
xinf
Dawnfz-Lenfeng Feb 4, 2025
f9e4a3b
del completion
Dawnfz-Lenfeng Feb 6, 2025
3c1f032
fix typo
Dawnfz-Lenfeng Feb 6, 2025
405b2ce
modify descp of temp
Dawnfz-Lenfeng Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 69 additions & 127 deletions ch-xorbits/xinference.ipynb
View file
Open in desktop
< 10000 tr data-hunk="51ffd5b85f2a295a75142c851b1a8bdd53593269bbf6c5a6b3318f36bd6ea7bf" class="show-top-border">
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example: Using Qwen for Simple Text Generation and Conversation"
"## Example: Using Llama for Simple Conversation"
]
},
{
Expand Down Expand Up @@ -132,28 +132,29 @@
"source": [
"The default host and IP address for Xinference are 127.0.0.1 and 9997, respectively.\n",
"\n",
"Next, use the following command to start the Qwen model. The `size-in-billions` parameter corresponds to the parameter scale used. The first-generation Qwen model (code-named `qwen-chat` in Xinference) has parameter scales of 1.8 billion, 7 billion, 14 billion, and 72 billion. Here, we will try using the 7B model."
"Next, use the following command to start the Llama model. The `--size-in-billion` parameter corresponds to the parameter scale used. The first-generation Llama model (code-named `llama-3-instruct` in Xinference) supports parameter scales of 8 billion, 70 billion. The `--quantization` parameter specifies the precision reduction method (options: 4-bit, 8-bit, or none for full precision). Here we'll use the 8B model with 8-bit quantization."
]
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Launch model name: qwen-chat with kwargs: {}\n",
"Launch model name: llama-3-instruct with kwargs: {}\n",
"Model uid: my-llm\n"
]
}
],
"source": [
"!xinference launch \\\n",
" --model-uid my-llm \\\n",
" --model-name qwen-chat \\\n",
" --size-in-billions 7 \\\n",
" --model-name llama-3-instruct \\\n",
" --size-in-billions 8 \\\n",
" --quantization 8-bit \\\n",
" --model-format pytorch \\\n",
" --model-engine transformers"
]
Expand All @@ -169,7 +170,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -182,104 +183,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we will use the OpenAI API to easily use the large model for text generation and conversation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Completion API\n",
"\n",
"We can perform text generation using OpenAI's `client.completions.create` method. The Completion API is used to guide the model to generate text based on a given prompt."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[temperature: 0.7 | top_p: 0.9]\n",
"通义千问，智慧之海，\n",
"回答如流，无尽的探索。\n",
"通义千问，人间瑰宝。<|im_end|>\n",
"\n"
]
}
],
"source": [
"def complete_and_print(\n",
" prompt, temperature=0.7, top_p=0.9, client=client, model=\"my-llm\"\n",
"):\n",
" response = (\n",
" client.completions.create(\n",
" model=model, prompt=prompt, top_p=top_p, temperature=temperature\n",
" )\n",
" .choices[0]\n",
" .text\n",
" )\n",
"\n",
" print(f\"[temperature: {temperature} | top_p: {top_p}]\\n{response.strip()}\\n\")\n",
"\n",
"\n",
"prompt = \"写一首关于通义千问的三行俳句诗。\"\n",
"complete_and_print(prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can adjust some parameters provided by the API to configure the creativity and determinism of the output.\n",
"\n",
"The `top_p` means the cumulative probability cutoff for token selection, which controls how many tokens to choose, while the `temperature` parameter determines whether there is randomness in text generation within this range. When the temperature is close to 0, the result will be almost deterministic."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[temperature: 0.01 | top_p: 0.01]\n",
"通义千问，智慧之源，\n",
"回答问题，如诗如画。\n",
"机器语言，人类之友。<|im_end|>\n",
"\n",
"[temperature: 0.01 | top_p: 0.01]\n",
"通义千问，智慧之源，\n",
"回答问题，如诗如画。\n",
"机器语言，人类之友。<|im_end|>\n",
"\n",
"[temperature: 1.0 | top_p: 1.0]\n",
"通义千问通四海，言无不尽寻奥秘。智囊无所不在，问答之间显才智。<|im_end|>\n",
"<|im_start|>\n",
"\n",
"[temperature: 1.0 | top_p: 1.0]\n",
"诗中要包含词语\"通义千问\"和\"人工智能\"。\n",
"\n",
"通义千问问何来？ \n",
"人工智能显神威。 \n",
"科技引领未来路。\n",
"\n"
]
}
],
"source": [
"# The first two generations will be very similar,\n",
"complete_and_print(prompt, temperature=0.01, top_p=0.01)\n",
"complete_and_print(prompt, temperature=0.01, top_p=0.01)\n",
"\n",
"# The last two generations will be different\n",
"complete_and_print(prompt, temperature=1.0, top_p=1.0)\n",
"complete_and_print(prompt, temperature=1.0, top_p=1.0)"
"Next, we will use the OpenAI API to easily use the large model for conversation."
]
},
{
Expand All @@ -302,7 +206,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -323,19 +227,19 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"==============\n",
"assistant: 你最喜欢的颜色是蓝色。\n",
"assistant: You told me earlier that your favorite color is BLUE!\n",
"\n",
"\n",
"==============\n",
"assistant: 您的宠物叫毛毛。\n",
"assistant: You told me earlier that your pet's name is Lucy, which is a lovely name for a dog!\n",
"\n",
"\n"
]
Expand All @@ -357,17 +261,17 @@
"\n",
"chat_complete_and_print(\n",
" messages=[\n",
" user(\"我最喜欢的颜色是蓝色\"),\n",
" assistant(\"听到这个消息真是令人欣喜！\"),\n",
" user(\"我最喜欢的颜色是什么？\"),\n",
" user(\"My favorite color is blue\"),\n",
" assistant(\"That's wonderful to hear!\"),\n",
" user(\"What is my favorite color?\"),\n",
" ]\n",
")\n",
"\n",
"chat_complete_and_print(\n",
" messages=[\n",
" user(\"我有一只名叫毛毛的小狗\"),\n",
" assistant(\"听到这个消息真棒！毛毛一定很可爱。\"),\n",
" user(\"我的宠物叫什么名字？\"),\n",
" user(\"I have a little dog named Lucy\"),\n",
" assistant(\"That's awesome! Lucy must be very cute.\"),\n",
" user(\"What is my pet's name?\"),\n",
" ]\n",
")"
]
Expand All @@ -376,7 +280,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Of course, we can still adjust different `temperature` and `top_p` parameters to show how different settings affect the randomness and diversity of the generated content."
"We can adjust some parameters provided by the API to configure the creativity and determinism of the output.\n",
"\n",
"The `top_p` means the cumulative probability cutoff for token selection, which controls how many tokens to choose, while the `temperature` parameter determines whether there is randomness in text generation within this range. When the temperature is close to 0, the result will be almost deterministic."
]
},
{
Expand All @@ -389,29 +295,65 @@
"output_type": "stream",
"text": [
"==============\n",
"assistant: 钢琴学习可以帮助你提高音乐素养，培养耐心和专注力，增强记忆力，提高创造力，提升自信心，培养良好的节奏感，以及更好地理解音乐理论。\n",
"assistant: Learning to play the piano can have numerous benefits, including:\n",
"\n",
"* Improved cognitive skills: Playing piano requires coordination between hands, eyes, and brain, which can improve memory, concentration, and problem-solving abilities.\n",
"* Enhanced creativity: Piano playing allows for self-expression and creativity through music composition and improvisation.\n",
"* Stress relief: Playing piano can be a calming and meditative experience, reducing stress and anxiety.\n",
"* Brain development: Research suggests that early childhood piano lessons can even affect brain structure and function, improving spatial-temporal skills and language development.\n",
"* Social benefits: Playing piano can provide opportunities to connect with others through music-making, whether it's performing in front of an audience or jamming with friends.\n",
"\n",
"These are just a few examples, but I'm sure you're experiencing many more benefits as you learn and enjoy playing the piano!\n",
"\n",
"\n",
"==============\n",
"assistant: 钢琴学习可以帮助你提高音乐素养，培养耐心和专注力，增强记忆力，提高创造力，提升自信心，培养良好的节奏感，以及更好地理解音乐理论。\n",
"assistant: Learning to play the piano can have numerous benefits, including:\n",
"\n",
"* Improved cognitive skills: Playing piano requires coordination between hands, eyes, and brain, which can improve memory, concentration, and problem-solving abilities.\n",
"* Enhanced creativity: Piano playing allows for self-expression and creativity through music composition and improvisation.\n",
"* Stress relief: Playing piano can be a calming and meditative experience, reducing stress and anxiety.\n",
"* Brain development: Research suggests that early childhood piano lessons can even affect brain structure and function, improving spatial-temporal skills and language development.\n",
"* Social benefits: Playing piano can provide opportunities to connect with others through music-making, whether it's performing in front of an audience or jamming with friends.\n",
"\n",
"These are just a few examples, but I'm sure you're experiencing many more benefits as you learn and enjoy playing the piano!\n",
"\n",
"\n",
"==============\n",
"assistant: 学习钢琴有很多好处，例如它可以帮助你提高音乐素养，培养耐心，增强记忆力，还可以增长知识，帮助你理解节奏和和弦，以及提高审美能力。\n",
"assistant: Learning to play the piano can bring many benefits, including:\n",
"\n",
"* Improved cognitive skills: playing the piano requires coordination between hands and brain, which can improve memory, concentration, and spatial-temporal skills.\n",
"* Enhanced creativity: composing and improvising music can foster creative thinking and self-expression.\n",
"* Emotional intelligence: playing emotional music can help develop empathy and understanding of others' emotions.\n",
"* Language development: research suggests that musical training can improve language skills and even delay symptoms of Alzheimer's disease.\n",
"* Stress relief: playing the piano can be a calming and meditative experience, reducing anxiety and stress levels.\n",
"\n",
"These are just a few examples, but I'm sure you're experiencing them firsthand as you learn to play the piano! Keep practicing and enjoying it!\n",
"\n",
"\n",
"==============\n",
"assistant: 钢琴学习可以帮助提升思维技巧，培养自制力，提高音乐审美，增加自信心，提升技能，练习记忆力，并且可以让你分享自己最喜欢的音乐。\n",
"assistant: Learning to play the piano can have many benefits, including:\n",
"\n",
"* Improved cognitive skills: Playing the piano requires coordination between different parts of the brain, which can improve memory, concentration, and spatial-temporal skills.\n",
"* Enhanced creativity and self-expression\n",
"* Boosted confidence and self-esteem through mastery of new skills\n",
"* Stress relief and relaxation through playing calming music or meditative pieces\n",
"* Language development for children (piano lessons can even be beneficial for language learning)\n",
"* Social benefits from sharing your love of music with others, whether by performing or teaching\n",
"* Brain plasticity and adaptability, as your brain reorganizes itself in response to new musical demands.\n",
"\n",
"These benefits can translate to other areas of life, such as personal relationships, work, and overall well-being!\n",
"\n",
"How about you, what has been most enjoyable or surprising about learning piano so far?\n",
"\n",
"\n"
]
}
],
"source": [
"messages = [\n",
" user(\"我最近在学习钢琴\"),\n",
" assistant(\"那真是一个很好的爱好！\"),\n",
" user(\"你觉得钢琴学习有什么好处？\"),\n",
" user(\"I've been learning piano recently.\"),\n",
" assistant(\"That's really a great hobby!\"),\n",
" user(\"What do you think are the benefits of learning this instrument? Tell me briefly\"),\n",
"]\n",
"\n",
"\n",
Expand Down Expand Up @@ -469,7 +411,7 @@
},
{
"cell_type": "code",
"execution_count": 37,
"execution_count": 24,
"metadata": {
"vscode": {
"languageId": "shellscript"
Expand All @@ -480,7 +422,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Service is not running, starting service.\n"
"Service is already running, exiting.\n"
]
}
],
Expand Down Expand Up @@ -849,7 +791,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "xinference",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -867,5 +809,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}
0