Open
Description
Describe the bug
When using LiteLLMClient we are forced to hit a warning:
[2025-06-06 18:21:13,616] [ WARNING]: You cannot select the number of dataset splits for a generative evaluation at the moment. Automatically inferring. (data.py:237)
To Reproduce
- From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/abstract_model.py#L58,
DATASET_SPLITS = 4. - From https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/models/litellm_model.py#L229, we pass that to
num_dataset_splits. - Then from https://github.com/huggingface/lighteval/blob/v0.10.0/src/lighteval/data.py#L83-L85, we are forced to hit the above warning.
Expected behavior
Not being forced to hit a warning.
Version info
lighteval==0.10.0