Login
From text to token: How tokenization pipelines work
(paradedb.com) by philippemnoel | view | 20 comments
BlackRock's Larry Fink: "Tokenization", Digital IDs, & Social Credit
(thewinepress.substack.com) by sbuttgereit | view | 83 comments
What Makes LLM Tokenization Slow?
(healeycodes.com) by ibobev | view | 0 comments
What Makes LLM Tokenization Slow?
(healeycodes.com) by healeycodes | view | 0 comments
Show HN: Parmar – BPE tokenization as a pre-filter for LZMA (+9.6%, and faster)
(github.com) by ronak_parmar | view | 0 comments
GigaToken: ~1000x faster Language model tokenization
(github.com) by syrusakbary | view | 0 comments
UK Tokenization Roadmap Puts £33B on the Table and a Clock on the Wall
(disruptionbanking.com) by emsidisii | view | 0 comments
Tokenization's Next Big Conundrum: What Do Buyers Get?
(bloomberg.com) by petethomas | view | 0 comments
Tokenization Is the Bottleneck You're Not Measuring
(ranvier.systems) by mindsaspire | view | 0 comments
Compute Optimal Tokenization: Scaling Laws for Data Compression in LLMs
(co-tok.github.io) by matt_d | view | 0 comments
Reducing TTFT by CPUMaxxing Tokenization
(crusoe.ai) by intrepidsoldier | view | 0 comments
The Language Tax in LLM Pricing: How Tokenization Create Price Disparity
(tokenstree.com) by vfalbor | view | 0 comments
Bond Tokenization: Reinventing Fixed Income for the Digital Age
(catenaa.com) by malindasp | view | 1 comments
Do All Languages Cost the Same? Tokenization in the Era of Commercial LLMs
(arxiv.org) by Anon84 | view | 0 comments
BlackRock: Tokenization makes mobile investing as easy as sending a payment
(coincu.com) by adrianwaj | view | 0 comments
LiTo: Surface Light Field Tokenization
(machinelearning.apple.com) by aeontech | view | 0 comments
Reducing TTFT by CPUMaxxing Tokenization
(crusoe.ai) by AlonKejzman | view | 0 comments
Tokenization Is Killing Our Multilingual LLM Dream
(huggingface.co) by omneity | view | 0 comments
IDs+ Protocol: Solving the CJK Tokenization 'Byte-Premium' in LLMs
(github.com) by oruc001 | view | 0 comments
The Broken Token: Tokenization for Malayalam Language Models
(thottingal.in) by sthottingal | view | 0 comments
Building Yomu: furigana, tokenization, and offline dictionaries on iOS
(blog.kulman.sk) by ig0r0 | view | 0 comments
LLMs – Part 1: Tokenization and Embeddings
(vasupasupuleti.substack.com) by vpasupuleti10 | view | 1 comments
Show HN: Beat (SRF) – No Parsing, No Tokenization, Binary-Level Speed
(github.com) by aidgn | view | 0 comments
Tokenization in Transformers v5: Simpler, Clearer, and More Modular
(huggingface.co) by ibobev | view | 0 comments
PatchDNA: Flexible and Biologically-Informed Alternative to Tokenization for DNA
(biorxiv.org) by mapmeld | view | 0 comments