├─agentverse
│ ├─...
│ └─tasks
│ └─kgqa
│ ├─freebase
│ │ └─three_role_one_turn_sequential_freebase
│ └─metaqa
│ └─three_role_one_turn_sequential_metaqa
├─eval_helper
│ └─get_evaluation.py
├─KBQA_TASK
│ ├─freebase
│ │ ├─dataset
│ │ │ ├─cwq.json
│ │ │ ├─grailqa.json
│ │ │ ├─WebQSP.json
│ │ │ └─WebQuestions.json
│ │ ├─freebase_func.py
│ │ ├─main_cwq.py
│ │ ├─main_grailqa.py
│ │ ├─main_webqsp.py
│ │ ├─main_webquestions.py
│ │ └─prompt_list.py
│ └─metaqa
│ ├─dataset
│ ├─prompt_list.py
│ ├─metaqa_func.py
│ ├─main_metaqa_1hop.py
│ ├─main_metaqa_2hop.py
│ └─main_metaqa_3hop.py
└─requirements.txtpip install -r requirements.txtIf you want to call OpenAI's API, you also need to export your OpenAI key as follows before running our code.
- Using Environment Variable:
export OPENAI_API_KEY="your_api_key_here"- Or, directly specifying in the Python file:
import os
os.environ["OPENAI_API_KEY"] = "your_api_key_here"What's more, DoG also supports the use of local open-source large language models.
import os
os.environ["OPENAI_API_BASE"] = "your_api_base_here"Additionally, we recommend deploying the Virtuoso server locally by the instruction.
We select five public datasets to evaluate the reasoning ability over knowledge graphs: MetaQA, WebQSP, CWQ,WebQuestions, and GrailQA. You can get them directly from dataset folder.
- MetaQA
python main_metaqa_1hop.py --task "kgqa/metaqa/three_role_one_turn_sequential_metaqa" --output_path "./output/metaqa_1hop_output.txt"
python main_metaqa_2hop.py --task "kgqa/metaqa/three_role_one_turn_sequential_metaqa" --output_path "./output/metaqa_2hop_output.txt"
python main_metaqa_3hop.py --task "kgqa/metaqa/three_role_one_turn_sequential_metaqa" --output_path "./output/metaqa_3hop_output.txt"- CWQ
python main_cwq.py --task "kgqa/freebase/three_role_one_turn_sequential_freebase" --output_path "./output/cwq_output.txt"- GrailQA
python main_grailqa.py --task "kgqa/freebase/three_role_one_turn_sequential_freebase" --output_path "./output/grailqa_output.txt"- WebQSP
python main_webqsp.py --task "kgqa/freebase/three_role_one_turn_sequential_freebase" --output_path "./output/webqsp_output.txt"- WebQuestions
python main_webquestions.py --task "kgqa/freebase/three_role_one_turn_sequential_freebase" --output_path "./output/webquestions_output.txt"Feel free to modify the YAML files under agentverse\tasks\kgqa\freebase to change the strategy for Question Simplifying. We provide basic prompt and encourage exploring more interesting and efficient debate strategies.
This repository builds the multi-agent debate team based on the following paper and code:
- Chan et al., "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate," 2023 [Paper Link] [Repo Link]
Thanks for their nice contribution.
@inproceedings{ma2025debate,
title={Debate on graph: a flexible and reliable reasoning framework for large language models},
author={Ma, Jie and Gao, Zhitao and Chai, Qi and Sun, Wangchun and Wang, Pinghui and Pei, Hongbin and Tao, Jing and Song, Lingyun and Liu, Jun and Zhang, Chen and others},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
pages={24768--24776},
year={2025}
}