Technical Paper: https://arxiv.org/pdf/2306.08161.pdf
h2oGPT is a large language model (LLM) fine-tuning framework and chatbot UI with document(s) question-answer capabilities. Documents help to ground LLMs against hallucinations by providing them context relevant to the instruction. h2oGPT is fully permissive Apache V2 open-source project for 100% private and secure use of LLMs and document embeddings for document question-answer.
Welcome! Join us and make an issue or a PR, and contribute to making the best fine-tuned LLMs, chatbot UI, and document question-answer framework!
Turn ★ into ⭐ (top-right corner) if you like the project!
- Try h2oGPT now
- Supported OS and Hardware
- Apache V2 ChatBot with LangChain Integration
- Apache V2 Data Preparation code, Training code, and Models
- Roadmap
- Getting Started
- Development
- Help
- Acknow 8000 ledgements
- Why H2O.ai?
- Disclaimer
Live hosted instances:
For questions, discussing, or just hanging out, come and join our Discord!
GPU mode requires CUDA support via torch and transformers. A 6.9B (or 12GB) model in 8-bit uses 7GB (or 13GB) of GPU memory. 8-bit or 4-bit precision can further reduce memory requirements.
CPU mode uses GPT4ALL and LLaMa.cpp, e.g. gpt4all-j, requiring about 14GB of system RAM in typical use.
GPU and CPU mode tested on variety of NVIDIA GPUs in Ubuntu 18-22, but any modern Linux variant should work. MACOS support tested on Macbook Pro running Monterey v12.3.1 using CPU mode.
- LangChain equipped Chatbot integration and streaming responses
- Persistent database using Chroma or in-memory with FAISS
- Original content url links and scores to rank content against query
- Private offline database of any documents (PDFs, Images, and many more)
- Upload documents via chatbot into shared space or only allow scratch space
- Control data sources and the context provided to LLM
- Efficient use of context using instruct-tuned LLMs (no need for many examples)
- API for client-server control
- CPU and GPU support from variety of HF models, and CPU support using GPT4ALL and LLaMa cpp
- Linux, MAC, and Windows support
Light mode with soft colors talking to cat image:
Dark mode with H2O.ai colors:
- Variety of models (h2oGPT, WizardLM, Vicuna, OpenAssistant, etc.) supported
- Fully Commercially Apache V2 code, data and models
- High-Quality data cleaning of large open-source instruction datasets
- LoRA and QLoRA (low-rank approximation) efficient 4-bit, 8-bit and 16-bit fine-tuning and generation
- Large (up to 65B parameters) models built on commodity or enterprise GPUs (single or multi node)
- Evaluate performance using RLHF-based reward models
Screen.Recording.2023-04-18.at.4.10.58.PM.mov
All open-source datasets and models are posted on 🤗 H2O.ai's Hugging Face page.
Also check out H2O LLM Studio for our no-code LLM fine-tuning framework!
- Integration of code and resulting LLMs with downstream applications and low/no-code platforms
- Complement h2oGPT chatbot with search and other APIs
- High-performance distributed training of larger models on trillion tokens
- Enhance the model's code completion, reasoning, and mathematical capabilities, ensure factual correctness, minimize hallucinations, and avoid repetitive output
- Add other tools like search
- Add agents for SQL and CSV question/answer
First one needs a Python 3.10 environment. For help installing a Python 3.10 environment, see Install Python 3.10 Environment. On newer Ubuntu systems and environment may be installed by just doing:
sudo apt-get install -y build-essential gcc python3.10-devCheck your installation by doing:
python --version # should say 3.10.xx
pip --version # should say pip 23.x.y ... (python 3.10)On some systems, pip still refers back to the system one, then one can use python -m pip or pip3 instead of pip or try python3 instead of python.
After Python 3.10 environment installed:
git clone https://github.com/h2oai/h2ogpt.git
cd h2ogpt
# broad support, but no training-time or data creation dependencies
for fil in requirements.txt reqs_optional/requirements_optional_langchain.txt reqs_optional/requirements_optional_gpt4all.txt reqs_optional/requirements_optional_langchain.gpllike.txt ; do pip install -r $fil ; done
# Optional: support docx, pptx, ArXiv, etc.
sudo apt-get install -y libmagic-dev poppler-utils tesseract-ocr libreoffice
# Optional: for supporting unstructured package
python -m nltk.downloader allPlace all documents in user_path or upload in UI.
UI using GPU with at least 24GB with streaming:
python generate.py --base_model=h2oai/h2ogpt-oasst1-512-12b --load_8bit=True --score_model=None --langchain_mode='UserData' --user_path=user_pathUI using CPU (streaming not yet supported in UI):
python generate.py --base_model='llama' --prompt_type=wizard2 --score_model=None --langchain_mode='UserData' --user_path=user_pathFor help installing cuda toolkit, see CUDA Toolkit
git clone https://github.com/h2oai/h2ogpt.git
cd h2ogpt
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu117
python generate.py --base_model=h2oai/h2ogpt-oig-oasst1-512-6_9b --load_8bit=TrueThen point browser at http://0.0.0.0:7860 (linux) or http://localhost:7860 (windows/mac) or the public live URL printed by the server (disable shared link with --share=False). For 4-bit or 8-bit support, older GPUs may require older bitsandbytes installed as pip uninstall bitsandbytes -y ; pip install bitsandbytes==0.38.1. For production uses, we recommend at least the 12B model, ran as:
python generate.py --base_model=h2oai/h2ogpt-oasst1-512-12b --load_8bit=True
and one can use --h2ocolors=False to get soft blue-gray colors instead of H2O.ai colors. Here is a list of environment variables that can control some things in generate.py.
Note if you download the model yourself and point --base_model to that location, you'll need to specify the prompt_type as well by running:
python generate.py --base_model=<user path> --load_8bit=True --prompt_type=human_bot
for some user path <user path>.
For quickly using a private document collection for Q/A, place documents (PDFs, text, etc.) into a folder called user_path and run
pip install -r reqs_optional/requirements_optional_langchain.txt
python -m nltk.downloader all # for supporting unstructured package
python generate.py --base_model=h2oai/h2ogpt-oig-oasst1-512-6_9b --load_8bit=True --langchain_mode=UserData --user_path=user_pathFor more ways to ingest on CLI and control see LangChain Readme. For example, for improved pdf handling via pymupdf (GPL) and support for docx, ppt, OCR, and ArXiV run:
sudo apt-get install -y libmagic-dev poppler-utils tesseract-ocr tesseract-ocr libreoffice
pip install -r reqs_optional/requirements_optional_langchain.gpllike.txtFor 4-bit support, the latest dev versions of transformers, accelerate, and peft are required, which can be installed by running:
pip uninstall peft transformers accelerate -y
pip install -r reqs_optional/requirements_optional_4bit.txtwhere uninstall is required in case, e.g., peft was installed from GitHub previously. Then when running generate pass --load_4bit=True, which is only supported for certain architectures like GPT-NeoX-20B, GPT-J, LLaMa, etc.
Any other instruct-tuned base models can be used, including non-h2oGPT ones. Larger models require more GPU memory.
CPU support is obtained after installing two optional requirements.txt files. This does not preclude GPU support, just adds CPU support:
- Install base, langchain, and GPT4All, and python LLaMa dependencies:
git clone https://github.com/h2oai/h2ogpt.git
cd h2ogpt
pip install -r requirements.txt # only do if didn't already do for GPU support, since windows needs --extra-index-url line
pip install -r reqs_optional/requirements_optional_langchain.txt
python -m nltk.downloader all # for supporting unstructured package
pip install -r reqs_optional/requirements_optional_gpt4all.txtSee GPT4All for details on installation instructions if any issues encountered.
- Change
.env_gpt4allmodel name if desired.
model_path_llama=WizardLM-7B-uncensored.ggmlv3.q8_0.bin
model_path_gptj=ggml-gpt4all-j-v1.3-groovy.bin
model_name_gpt4all_llama=ggml-wizardLM-7B.q4_2.bin
For gptj and gpt4all_llama, you can choose a different model than our default choice by going to GPT4All Model explorer GPT4All-J compatible model. One does not need to download manually, the gp4all package will download at runtime and put it into .cache like Hugging Face would. However, gpjt model often gives no output, even outside h2oGPT.
So, for chatting, a better instruct fine-tuned LLaMa-based model for llama.cpp can be downloaded from TheBloke. For example, 13B WizardLM Quantized or 7B WizardLM Quantized. TheBloke has a variety of model types, quantization bit depths, and memory consumption. Choose what is best for your system's specs. However, be aware that LLaMa-based models are not commercially viable.
For 7B case, download WizardLM-7B-uncensored.ggmlv3.q8_0.bin into local path. Then one sets model_path_llama in .env_gpt4all, which is currently the default.
- Run generate.py
For LangChain support using documents in user_path folder, run h2oGPT like:
python generate.py --base_model='llama' --prompt_type=wizard2 --score_model=None --langchain_mode='UserData' --user_path=user_path