8000 Commits · pytorch-labs/tokenizers · GitHub

More Web Proxy on the site http://driver.im/

Commits

Commits on May 29, 2025

Add regex unit tests and enable shared linkage in fbcode
larryliu0820
authored

Commits on May 22, 2025

Commits on May 16, 2025

Add look ahead tiktoken target
larryliu0820
authored

Commits on May 13, 2025

Commits on May 8, 2025

Accept custom pattern string and special tokens
sxu
authored

Commits on May 2, 2025

Fix tokenizer special token handling (#67)
jackzhxng
authored

Commits on May 1, 2025

Handle null bos and eos token (#66)
jackzhxng
authored

Commits on Apr 30, 2025

Forward fix missing vtable for bpe tokenizer
jackzhxng
authored

Commits on Apr 29, 2025

Commits on Apr 25, 2025

Commits on Apr 24, 2025

Commits on Apr 23, 2025

Pcre2 buck target in third-party (#55) (#58)
jackzhxng
authored

Commits on Apr 21, 2025

Commits on Apr 19, 2025

Decouple tokenizers from Re2 and use IRegex interface
jackzhxng
authored

Commits on Apr 18, 2025

Add regex interface with re2 and std::regex implementations
jackzhxng
authored

Commits on Apr 15, 2025

TikToken uses common bpe base functions (#45)
jackzhxng
authored

Commits on Apr 14, 2025

update ignore list (#46)
jathu
authored

Commits on Apr 9, 2025

Update pattern key for split pretokenizer (#38)
jackzhxng
authored

Commits on Apr 5, 2025

Make it public
helunwencser
authored

Commits on Apr 2, 2025

fix qnn export
cccclai
authored

Commits on Mar 30, 2025

Remove old tokenizer/ directory in ExecuTorch
larryliu0820
authored

Commits on Mar 27, 2025

Port string_integer_map and changes to tiktoken to pytorch
larryliu0820
authored

Commits on Mar 25, 2025

Fix build for tokenizer tool script (#36)
jackzhxng
authored

Commits on Mar 18, 2025

Commits on Mar 17, 2025

Rely on runtime_wrapper to provide supported platforms (#33)
larryliu0820
authored

Commits on Mar 14, 2025

0