First, I would like to thank Voyager's original authors and users who have contributed to the project in discussion and PRs in the original repo. This fork is mainly for personal interest, learning, and having fun with kids.
The original model is bundled with OpenAI API calls, and I am curious to explore the capabilities of other LLMs. Also, it would be good to save money to utilize local LLM applications. The fork made some changes so that users can specify the API URL and embedding model and evaluate various LLMs depending on their interests.
The following python packages are required for utilizing 3rd party embedding model:
pip install langchain-community
pip install sentence-transformers
The embedding model can be specified when initializing the Voyager:
embedding_model = '/path/to/embedding/model'
openai_api_key = 'your openai api key'
openai_api_base = 'URL to OpenAI compatiable API'
mc_port = 55566
voyager = Voyager(
mc_port=mc_port,
openai_api_key=openai_api_key,
openai_api_base=openai_api_base,
embedding_model = embedding_model
)
- Windows user can use node.js v20.9.0 and npm v10.1.0. Reference.
- When install mineflayer-collectblock, you may encounter errors. If so, please try remove the {package_root}/voyager/env/mineflayer/node_modules/mineflayer/node_modules/prismarine-block and run
npx tscagain. Reference.
- Check this PR for running a dedicated minecraft server.
- The User should set the game mode to survival. Otherwise, bricks are not collected into bot's inventory. And the bot should have the op privilege in the game. It needs to execute some op commands during the learning.
- This fork is tested with the Meta Llama 3.1 70B (llama-3.1-70b-versatile provided by Groq). The bot took about ~50 iterations to learn how to craft iron tools from scratch.
- Be aware of API usage if you are using paid LLM API services.
Happy Minecrafting!
We introduce Voyager, the first LLM-powered embodied lifelong learning agent in Minecraft that continuously explores the world, acquires diverse skills, and makes novel discoveries without human intervention. Voyager consists of three key components: 1) an automatic curriculum that maximizes exploration, 2) an ever-growing skill library of executable code for storing and retrieving complex behaviors, and 3) a new iterative prompting mechanism that incorporates environment feedback, execution errors, and self-verification for program improvement. Voyager interacts with GPT-4 via blackbox queries, which bypasses the need for model parameter fine-tuning. The skills developed by Voyager are temporally extended, interpretable, and compositional, which compounds the agent’s abilities rapidly and alleviates catastrophic forgetting. Empirically, Voyager shows strong in-context lifelong learning capability and exhibits exceptional proficiency in playing Minecraft. It obtains 3.3× more unique items, travels 2.3× longer distances, and unlocks key tech tree milestones up to 15.3× faster than prior SOTA. Voyager is able to utilize the learned skill library in a new Minecraft world to solve novel tasks from scratch, while other techniques struggle to generalize.
In this repo, we provide Voyager code. This codebase is under MIT License.
Voyager requires Python ≥ 3.9 and Node.js ≥ 16.13.0. We have tested on Ubuntu 20.04, Windows 11, and macOS. You need to follow the instructions below to install Voyager.
git clone https://github.com/MineDojo/Voyager
cd Voyager
pip install -e .
In addition to the Python dependencies, you need to install the following Node.js packages:
cd voyager/env/mineflayer
npm install -g npx
npm install
cd mineflayer-collectblock
npx tsc
cd ..
npm install
Voyager depends on Minecraft game. You need to install Minecraft game and set up a Minecraft instance.
Follow the instructions in Minecraft Login Tutorial to set up your Minecraft Instance.
You need to install fabric mods to support all the features in Voyager. Remember to use the correct Fabric version of all the mods.
Follow the instructions in Fabric Mods Install to install the mods.
Voyager uses OpenAI's GPT-4 as the language model. You need to have an OpenAI API key to use Voyager. You can get one from here.
After the installation process, you can run Voyager by:
from voyager import Voyager
# You can also use mc_port instead of azure_login, but azure_login is highly recommended
azure_login = {
"client_id": "YOUR_CLIENT_ID",
"redirect_url": "https://127.0.0.1/auth-response",
"secret_value": "[OPTIONAL] YOUR_SECRET_VALUE",
"version": "fabric-loader-0.14.18-1.19", # the version Voyager is tested on
}
openai_api_key = "YOUR_API_KEY"
voyager = Voyager(
azure_login=azure_login,
openai_api_key=openai_api_key,
)
# start lifelong learning
voyager.learn()- If you are running with
Azure Loginfor the first time, it will ask you to follow the command line instruction to generate a config file. - For
Azure Login, you also need to select the world and open the world to LAN by yourself. After you runvoyager.learn()the game will pop up soon, you need to:- Select
Singleplayerand pressCreate New World. - Set Game Mode to
Creativeand Difficulty toPeaceful. - After the world is created, press
Esckey and pressOpen to LAN. - Select
Allow cheats: ONand pressStart LAN World. You will see the bot join the world soon.
- Select
If you stop the learning process and want to resume from a checkpoint later, you can instantiate Voyager by:
from voyager import Voyager
voyager = Voyager(
azure_login=azure_login,
openai_api_key=openai_api_key,
ckpt_dir="YOUR_CKPT_DIR",
resume=True,
)If you want to run Voyager for a specific task with a learned skill library, you should first pass the skill library directory to Voyager:
from voyager import Voyager
# First instantiate Voyager with skill_library_dir.
voyager = Voyager(
azure_login=azure_login,
openai_api_key=openai_api_key,
skill_library_dir="./skill_library/trial1", # Load a learned skill library.
ckpt_dir="YOUR_CKPT_DIR", # Feel free to use a new dir. Do not use the same dir as skill library because new events will still be recorded to ckpt_dir.
resume=False, # Do not resume from a skill library because this is not learning.
)Then, you can run task decomposition. Notice: Occasionally, the task decomposition may not be logical. If you notice the printed sub-goals are flawed, you can rerun the decomposition.
# Run task decomposition
task = "YOUR TASK" # e.g. "Craft a diamond pickaxe"
sub_goals = voyager.decompose_task(task=task)Finally, you can run the sub-goals with the learned skill library:
voyager.inference(sub_goals=sub_goals)For all valid skill libraries, see Learned Skill Libraries.
If you have any questions, please check our FAQ first before opening an issue.
If you find our work useful, please consider citing us!
@article{wang2023voyager,
title = {Voyager: An Open-Ended Embodied Agent with Large Language Models},
author = {Guanzhi Wang and Yuqi Xie and Yunfan Jiang and Ajay Mandlekar and Chaowei Xiao and Yuke Zhu and Linxi Fan and Anima Anandkumar},
year = {2023},
journal = {arXiv preprint arXiv: Arxiv-2305.16291}
}Disclaimer: This project is strictly for research purposes, and not an official product from NVIDIA.