A inference benchmark tool, you can base on this tool to extend usage for different benchmark purpose. You can add a dataloader to provide your dataset for benchmarking, and also you can add your backend for benchmarking on different framework or models.
This tool provides several basic benchmarkers, like mlperf, direct, nlp_generative.
pip install -r requirement.txt
python runner.py -m facebook/opt-1.3b -b 1 -s 16
deepspeed --num_gpus 2 runner.py -m facebook/opt-1.3b -b 1 -s 16
- facebook/opt-1.3b
- t5-3b
- EleutherAI/gpt-j-6B
- decapoda-research/llama-7b-hf
- decapoda-research/llama-13b-hf
- decapoda-research/llama-30b-hf
- decapoda-research/llama-65b-hf
- bigscience/bloom-7b1
- bigscience/bloom
- microsoft/bloom-deepspeed-inference-fp16
python runner.py -h