Login
From text to token: How tokenization pipelines work
(paradedb.com) by philippemnoel | view | 20 comments
BlackRock's Larry Fink: "Tokenization", Digital IDs, & Social Credit
(thewinepress.substack.com) by sbuttgereit | view | 83 comments
The Broken Token: Tokenization for Malayalam Language Models
(thottingal.in) by sthottingal | view | 0 comments
Building Yomu: furigana, tokenization, and offline dictionaries on iOS
(blog.kulman.sk) by ig0r0 | view | 0 comments
LLMs – Part 1: Tokenization and Embeddings
(vasupasupuleti.substack.com) by vpasupuleti10 | view | 1 comments
Show HN: Beat (SRF) – No Parsing, No Tokenization, Binary-Level Speed
(github.com) by aidgn | view | 0 comments
Tokenization in Transformers v5: Simpler, Clearer, and More Modular
(huggingface.co) by ibobev | view | 0 comments
PatchDNA: Flexible and Biologically-Informed Alternative to Tokenization for DNA
(biorxiv.org) by mapmeld | view | 0 comments