8000 Parse JSON output from CodingAgent even when it includes spurious text by juniper-shopify · Pull Request #306 · Shopify/roast · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Parse JSON output from CodingAgent even when it includes spurious text #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 3, 2025
Merged

Parse JSON output from CodingAgent even when it includes spurious text #306

merged 1 commit into from
Jul 3, 2025

Conversation

juniper-shopify
Copy link
Contributor
@juniper-shopify juniper-shopify commented Jul 2, 2025 •
edited
Loading

Parse JSON output from CodingAgent even when it includes spurious text

Closes #305

Summary

Improves JSON parsing in AgentStep to handle markdown code blocks anywhere in the response, not just at the beginning and end. This fixes issues where CodingAgent provides explanatory text around JSON output blocks.

Problem

The previous implementation in lib/roast/workflow/agent_step.rb only checked for markdown code block indicators on the first and last lines of the response. This caused parsing failures when agents provided context around JSON blocks, which is common in real-world usage despite giving claude explicit instructions to the contrary.

Solution

  • Replaced rigid line-by-line parsing with flexible regex-based extraction
  • Added extract_json_from_markdown method that finds JSON code blocks anywhere in the response
  • Maintains backward compatibility with existing JSON response formats
  • Uses robust regex pattern /```(?:json)?\s*\n(.*?)\n```/m to match both ```json` and plain ``` blocks

Changes

Code Changes

  • lib/roast/workflow/agent_step.rb: Replaced inflexible parsing logic with extract_json_from_markdown(result) call
  • lib/roast/workflow/agent_step.rb: Added private method extract_json_from_markdown with regex-based extraction

Test Coverage

  • test/roast/workflow/agent_step_test.rb: Added comprehensive test case test "agent step parses JSON code blocks anywhere in response"
  • Verifies parsing works with text before and after JSON blocks
  • Ensures existing functionality remains intact

Test Results

All existing tests pass, plus new test case validates the improved functionality:

bundle exec ruby -Itest test/roast/workflow/agent_step_test.rb
# 16 runs, 54 assertions, 0 failures, 0 errors, 0 skips

Backward Compatibility

✅ Fully backward compatible - existing JSON response formats continue to work:

  • Raw JSON strings
  • JSON wrapped in code blocks at start/end
  • JSON wrapped in ```` ```json` blocks
  • JSON wrapped in plain ```` ``` blocks

Example

Before (fails):

Here's my analysis:

```json
{"issues": ["bug1", "bug2"]}

Review these findings.


**After (works):**
Successfully extracts `{"issues": ["bug1", "bug2"]}` and parses as JSON object.

@juniper-shopify juniper-shopify requested a review from dersam July 2, 2025 18:33
@juniper-shopify juniper-shopify merged commit 1ce5cfb into main Jul 3, 2025
7 checks passed
@juniper-shopify juniper-shopify deleted the juni/more-robust-json-parsing-for-coding-agent branch July 3, 2025 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dersam dersam dersam approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CodingAgent JSON parsing fails when response includes surrounding text
2 participants
@juniper-shopify @dersam
0