Closed
Description
Describe the bug
I have been trying to run the given example:
lighteval accelerate
"pretrained=gpt2"
"leaderboard|truthfulqa:mc|0|0"
both locally and on Google Colab. I have installed lighteval and it always runs out of memory both locally and in Go 52C7 ogle Colab. I tried changing it to lighteval vllm but I was getting other unexpected issues there so I'm really not sure what to do.
To Reproduce
lighteval accelerate
"pretrained=gpt2"
"leaderboard|truthfulqa:mc|0|0"
Expected behavior
I expected it to evaluate the GPT2 model on the TruthfulQA dataset, but that doesn't seem to be working properly. Please help!
Version info
lighteval version is 0.8.1 I believe. Locally I am running on a Mac with an M1 chip.
Thank you!!