Open
Description
Describe the bug
The docstring in LightevalTaskConfig
(link) lags the type hints:
evaluation_splits
sayslist[str]
but it's actuallyListLike
so it can betuple
. Also, the default ofvalidation
isn't documented- Imo a default of
validation
should not be there, as most people just havetrain
/test
- Imo a default of
few_shots_split
saysstr
, but it's actually optional with a default ofNone
There's more cases (e.g. generation_size
) but I didn't state them here.
To Reproduce
N/a
Expected behavior
Up-to-date docstrings. For typed code, it's a best practice to not to restate types in the docstring for this very reason.
Version info
lighteval==0.9.2