Open
Description
Is your feature request related to a problem? Please describe.
Labels: enhancement, parser, good-first-issue
Description:
As an eval author, I want to include structured metadata in my .yml files to better describe and categorize each evaluation. This data will be foundational for future features like advanced filtering and reporting.
Currently, our parser only extracts the core functional keys. We need to extend it to parse additional descriptive keys: id, description, difficulty, and tags.
Describe the solution you'd like
Acceptance Criteria:
- The
Evals.from_yml/1function is updated to parse the following new optional keys from the root of a YAML eval file:id(string)description(string)difficulty(string, e.g., "easy", "medium", "hard")tags(list of strings)
- The parsed keys should be converted to atoms (e.g.,
:id,:difficulty). - The parsed values should be stored in the resulting
evalmap that is passed through the evaluation pipeline. - The parser must remain backward-compatible and not fail if these new keys are absent.
- The
difficultystring should be converted to an atom (e.g.,"easy"->:easy).
Implementation Notes:
- This work will primarily be in
evals.ex, within thefrom_yml/1function. - The existing
remap_key/4helper can be used for most of this. - For
difficulty, a custom mapping function might be needed to convert the string to an atom.
# Example of how from_yml should transform the data
# from_yml(%{"id" => "...", "difficulty" => "easy", ...})
# -> %{id: "...", difficulty: :easy, ...}Describe alternatives you've considered
No response
Additional context
No response