8000
Load_dataset text "sample_by=paragraph" not working, only returns 1 item · huggingface datasets · Discussion #7152 · GitHub
More Web Proxy on the site http://driver.im/
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello!
I have a text file on windows. I already ran a script to insert a "\n" at certain points in the file.
It is now in the format:
"
AParagraphHere
Line1OfParagraph
Line2OfParagraph
...
DifferentParagraph now
Line2...
"
When I use "load_dataset("text", data_files={"train": ["my_file.txt"]}, sample_by="paragraph"), only one item is returned, which is the text of the entire file without seperation by paragraphs:
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hello!
I have a text file on windows. I already ran a script to insert a "\n" at certain points in the file.
It is now in the format:
"
AParagraphHere
Line1OfParagraph
Line2OfParagraph
...
DifferentParagraph now
Line2...
"
When I use "load_dataset("text", data_files={"train": ["my_file.txt"]}, sample_by="paragraph"), only one item is returned, which is the text of the entire file without seperation by paragraphs:
DatasetDict({
train: Dataset({
features: ['text'],
num_rows: 1
})
})
When I leave out "sample_by='paragraph'", it returns one item for every line in the file.
How can I make it return the actual paragraphs?
Beta Was this translation helpful? Give feedback.
All reactions