This project builds upon LAVIS library.
cd LAVIS
pip install -e .The main idea focuses on making better use of external knowledge to instruct MLLM to output more accurate generation. The detail can refer to lavis\models\blip2_models\blip2_vicuna_instruct_okvqa.py
bash run_scripts/blip2/train/train_okvqa.shThis publication version was made in a rush due to intensive workload that the author currently have. We will add follow-up patches to make codes more readible and ensure reproducibility. (of course, the speed depends on the number of people who are interested in using this framework.)