the.com/tokens
the smallest fragments of meaning, each one a coin the machine spends to think.
means Tokens are the small units a system breaks language into — words, parts of words, or characters — so a computer or model can count, process, or generate text from them.
from From the Old English 'tācen,' meaning a sign, mark, or symbol — a cousin of 'teach' and German 'Zeichen' ('sign'), all rooted in the idea of showing or signaling. For centuries a token was a physical sign: a coin-like piece standing in for money, proof, or a promise. That sense of a small stand-in for something larger carried into computing, where a 'token' became a discrete chunk a machine recognizes — and from there into language models, where text is sliced into tokens to be counted and processed.
- not words
- AI reads in chunks, not whole words
- old origin
- meant a physical sign or symbol since the 1500s
- costly
- language models are billed per token, not per thought
- arcade economy
- replaced coins to dodge cash-handling laws
- hidden split
- strawberry can become straw-berry to a model
bitcoin — cryptocurrency with ~21 million tokens as max supply, created 2009
ethereum — blockchain platform that issued ~120 million eth tokens since 2015
uniswap — decentralized exchange that distributed 400 million uni governance tokens in 2020