fix: RecursiveSplitter bug in the case when the recursive chunking is triggered
#9316
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RecursiveSplitter bug in the case where a split_text is found to be longer than split_length and recursive chunking is triggered
RecursiveSplitter bug in the case where a split_text is found to be longer than split_length and recursive chunking is triggeredRecursiveSplitter bug in the case when the recursive chunking is triggered
Pull Request Test Coverage Report for Build 14727388496Details
💛 - Coveralls |
julian-risch
approved these changes
Apr 30, 2025
| @@ -426,7 +426,7 @@ def test_run_separator_exists_but_split_length_too_small_fall_back_to_character_ | |||
| result = splitter.run(documents=[doc]) | |||
| assert len(result["documents"]) == 10 | |||
| for doc in result["documents"]: | |||
| if re.escape(doc.content) not in ["\ "]: | |||
| if re.escape(doc.content) not in ["\\ "]: | |||
There was a problem hiding this comment.
I remember seeing a warning about a regex. I assume this helps to get rid of that warning. 👍
There was a problem hiding this comment.
yes, that's it 👍🏽
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes
Removed the line that was making a cumulative count in the case where a
split_textis found to be longer thansplit_lengthand recursive chunking is triggered.A new test case was added to the
RecursiveDocumentSplitterthat verifies its behavior regarding the issue mentioned above.How did you test it?
test_run_complex_text_with_multiple_separators()to the existing test fileChecklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.