10000 Tags · benbrandt/text-splitter · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Tags: benbrandt/text-splitter

Tags

v0.27.0

Verified

This commit was signed with the committer’s verified signature.
benbrandt Ben Brandt
GPG key ID: D4618C5D3B500571
Verified
Learn about vigilant mode

v0.26.0

Toggle v0.26.0's commit message 
Bump check in ci

v0.25.1

Toggle v0.25.1's commit message 
chore: attempt to lower requirement on memchr

v0.25.0

Toggle v0.25.0's commit message 
ci: remove caches

v0.24.2

Toggle v0.24.2's commit message

Verified

This commit was signed with the committer’s verified signature.
benbrandt Ben Brandt
GPG key ID: 7DDA95F5EAC98E73
Verified
Learn about vigilant mode 
prep v0.24.2

v0.24.1

Toggle v0.24.1's commit message 
chore: update deps

v0.24.0

Toggle v0.24.0's commit message

Verified

This commit was signed with the committer’s verified signature.
benbrandt Ben Brandt
GPG key ID: 7DDA95F5EAC98E73
Verified
Learn about vigilant mode 
deps update

v0.23.0

Toggle v0.23.0's commit message

Verified

This commit was signed with the committer’s verified signature.
benbrandt Ben Brandt
GPG key ID: 7DDA95F5EAC98E73
Verified
Learn about vigilant mode 
chore: prep v0.23

v0.22.0

Verified

This commit was signed with the committer’s verified signature.
benbrandt Ben Brandt
GPG key ID: 7DDA95F5EAC98E73
Verified
Learn about vigilant mode

v0.21.0

Toggle v0.21.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode 
feat!: special tokens encoded by default (#512)

* feat!: special tokens encoded by default

Special tokens are now also encoded by both Huggingface and Tiktoken tokenizers. This is closer to the default behavior on the Python side, and should make sure if a model adds tokens at the beginning or end of a sequence, these are accounted for as well.

* test: fix python tests

* docs: clarify which tokenizers are affected
0