8000 Commits · sugarme/tokenizer · GitHub

More Web Proxy on the site http://driver.im/

Commits

Commits on May 20, 2025

Commits on May 19, 2025

Fix merge field being list not string
RJKeevil
committed

Commits on May 16, 2025

fix(cache): use thread-safe accessor to get cache value.
Zenithar
committed
fix(cache): race condition detected when batch tokenizr executions.
Zenithar
committed

Commits on Dec 13, 2024

Merge pull request #49 from terwey/utf8normalize
sugarme
authored

Commits on Dec 11, 2024

use stdlib utf8 to validate; add tests
terwey
committed

Commits on Jun 14, 2024

Commits on Jun 13, 2024

pretrained/tokenizer: fix FromFile, add FromReader
yarcat
committed

Commits on Jun 12, 2024

Fix TruncateEncodings for No Pair Encoding
cwarden
committed

Commits on Feb 24, 2024

Merge pull request #40 from jackielii/safer-input
sugarme
authored

Commits on Feb 23, 2024

use private() for better type safety
jackielii
committed

Commits on Aug 29, 2023

Commits on Aug 23, 2023

Commits on Jul 5, 2023

Commits on Jul 4, 2023

Commits on Jun 25, 2023

Commits on Jun 24, 2023

Commits on Jun 20, 2023

WIP: pretokenizer
sugarme
committed

Commits on Jun 19, 2023

WIP: pretrained/processor
sugarme
committed

0