feat(py/dotpromptz): test harness that creates suites and test case methods dynamically #279
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rutujaD1999
approved these changes
May 14, 2025
r8zr
reviewed
May 14, 2025
67fe8cc to
c7fbf8e
Compare
…ethods dynamically
The previous implementation of the test harness simply iterated
over each suite and test case within it to run it. It made it hard
to identify and scope failures.
This technique generates test suites and test case methods dynamically
adding them to the module environment before the `unittest.main`
entry-point takes over.
CHANGELOG:
- [ ] Update `spec_test.py` with the newer implementation.
USAGE:
```
zsh❯ uv run spec_test.py
...2025-05-14 02:29:57 [info ] [TEST] history > basic_history > inserts conversation history with proper metadata and roles
.2025-05-14 02:29:57 [info ] [TEST] history > empty_history > handles empty history by only rendering the template content
.2025-05-14 02:29:57 [info ] [TEST] ifEquals > basic > renders false branch when values are not equal
.2025-05-14 02:29:57 [info ] [TEST] ifEquals > basic > renders true branch when values are equal
.2025-05-14 02:29:57 [info ] [TEST] ifEquals > type_safety > treats different types as not equal
.2025-05-14 02:29:57 [info ] [TEST] json > basic > renders json in place
.2025-05-14 02:29:57 [info ] [TEST] json > indented > renders json in place
.2025-05-14 02:29:57 [info ] [TEST] media > basic > renders media part
.2025-05-14 02:29:57 [info ] [TEST] metadata > ext > extension fields are parsed and added to 'ext'
.2025-05-14 02:29:57 [info ] [TEST] metadata > metadata_state > accesses state object from metadata
.2025-05-14 02:29:58 [info ] [TEST] metadata > metadata_state > handles missing state values
.2025-05-14 02:29:58 [info ] [TEST] metadata > metadata_state > handles nested state objects
.2025-05-14 02:29:58 [info ] [TEST] metadata > raw > raw frontmatter is provided on top of parsed frontmatter
.2025-05-14 02:29:58 [info ] [TEST] partials > basic_partial > renders a basic partial
2025-05-14 02:29:58 [debug ] {'event': 'partial_registered', 'name': 'greeting'}
.2025-05-14 02:29:58 [info ] [TEST] partials > nested_resolved_partial > renders a resolver partial inside a resolver partial.
.2025-05-14 02:29:58 [info ] [TEST] partials > partial_with_context > renders a partial with context
2025-05-14 02:29:58 [debug ] {'event': 'partial_registered', 'name': 'userGreeting'}
.2025-05-14 02:29:58 [info ] [TEST] partials > resolved_partial > renders a partial provided by a resolver.
.2025-05-14 02:29:58 [info ] [TEST] picoschema > any_field > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > array_of_scalars > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > enum_field > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > inferred_json_schema_from_properties > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > input_and_output > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > line_endings_crlf > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > named_schema_override_description > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > nested_named_schema > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > nested_object_in_array_and_out > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > required_field > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_json_schema_type > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_object > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_scalar_description > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_scalar_description_extra_whitespace > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_scalar_description_no_whitespace > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_scalar_description_with_commas > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > simple_scalar_no_description > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > wildcard_fields_with_other_fields > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] picoschema > wildcard_fields_without_other_fields > returns as expected
.2025-05-14 02:29:58 [info ] [TEST] role > all_roles > allows system, user, and model roles
.2025-05-14 02:29:58 [info ] [TEST] role > system_only_prompt > inserts history after system prompt
.2025-05-14 02:29:58 [info ] [TEST] role > system_role > renders variables in system and user role
.2025-05-14 02:29:58 [info ] [TEST] section > basic_section > renders sequential sections with proper metadata and content boundaries
.2025-05-14 02:29:58 [info ] [TEST] section > nested_sections > handles nested and reopened sections with proper metadata boundaries
.2025-05-14 02:29:58 [info ] [TEST] unlessEquals > basic > renders false branch when values are equal
.2025-05-14 02:29:58 [info ] [TEST] unlessEquals > basic > renders true branch when values are different
.2025-05-14 02:29:58 [info ] [TEST] unlessEquals > type_safety > treats different types as not equal
.2025-05-14 02:29:58 [info ] [TEST] variables > basic > does not escape HTML
.2025-05-14 02:29:58 [info ] [TEST] variables > basic > overrides a default variable with a provided variable
.2025-05-14 02:29:58 [info ] [TEST] variables > basic > uses a default variable
.2025-05-14 02:29:58 [info ] [TEST] variables > basic > uses a provided variable
.
----------------------------------------------------------------------
Ran 51 tests in 0.054s
```
c7fbf8e to
0f1fcc3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous implementation of the test harness simply iterated
over each suite and test case within it to run it. It made it hard
to identify and scope failures.
This technique generates test suites and test case methods dynamically
adding them to the module environment before the
unittest.mainentry-point takes over.
CHANGELOG:
spec_test.pywith the newer implementation.USAGE: