Releases · simonw/ttok

New --allow-special option for allowing special tokens: ttok '<|endoftext|>' --encode --allow-special #13

--encode now encodes text to integer tokens and displays them. This has been renamed from --tokens.
--decode against a sequence of integer tokens now turns those back into text. #7
--tokens for either of those options outputs the raw tokens in a format that helps show if they have a leading space or similar. #4

Initial release. echo input.txt | ttok to count tokens, use -t 100 to truncate to 100 tokens, or --tokens to see what those integer tokens would be. #1

Provide feedback