Login
From text to token: How tokenization pipelines work
(paradedb.com) by philippemnoel | view | 20 comments
BlackRock's Larry Fink: "Tokenization", Digital IDs, & Social Credit
(thewinepress.substack.com) by sbuttgereit | view | 83 comments
Bond Tokenization: Reinventing Fixed Income for the Digital Age
(catenaa.com) by malindasp | view | 1 comments
Do All Languages Cost the Same? Tokenization in the Era of Commercial LLMs
(arxiv.org) by Anon84 | view | 0 comments
BlackRock: Tokenization makes mobile investing as easy as sending a payment
(coincu.com) by adrianwaj | view | 0 comments
LiTo: Surface Light Field Tokenization
(machinelearning.apple.com) by aeontech | view | 0 comments
Reducing TTFT by CPUMaxxing Tokenization
(crusoe.ai) by AlonKejzman | view | 0 comments
Tokenization Is Killing Our Multilingual LLM Dream
(huggingface.co) by omneity | view | 0 comments
IDs+ Protocol: Solving the CJK Tokenization 'Byte-Premium' in LLMs
(github.com) by oruc001 | view | 0 comments
The Broken Token: Tokenization for Malayalam Language Models
(thottingal.in) by sthottingal | view | 0 comments
Building Yomu: furigana, tokenization, and offline dictionaries on iOS
(blog.kulman.sk) by ig0r0 | view | 0 comments
LLMs – Part 1: Tokenization and Embeddings
(vasupasupuleti.substack.com) by vpasupuleti10 | view | 1 comments
Show HN: Beat (SRF) – No Parsing, No Tokenization, Binary-Level Speed
(github.com) by aidgn | view | 0 comments
Tokenization in Transformers v5: Simpler, Clearer, and More Modular
(huggingface.co) by ibobev | view | 0 comments
PatchDNA: Flexible and Biologically-Informed Alternative to Tokenization for DNA
(biorxiv.org) by mapmeld | view | 0 comments