DictionaryToken Economics
Token
A small chunk of text, often part of a word, that AI models read, count, and charge by.
Definition
The basic unit of text an LLM reads and writes; a token can be a word, part of a word, a character, or a byte. API pricing and context limits are counted in tokens. On earlier Claude models a token is about 3.5 English characters; the tokenizer introduced with Claude Opus 4.7 produces about 30% more tokens for the same text (1M tokens is roughly 555K words, versus about 750K before). Use a token counter for exact figures.
Example
A word like tokenization may be split into pieces such as token and ization. Paste text into the calculator to see its count per model.
Appears in
- The Token Efficiency Index: A Peer-Benchmarked Composite Indicator for AI Token EfficiencyarXiv
- SkillReducer: Optimizing LLM Agent Skills for Token EfficiencyarXiv
- Token-Efficient Change Detection in LLM APIsarXiv
- A*-Decoding: Token-Efficient Inference ScalingarXiv
- Token-Efficient RL for LLM ReasoningarXiv
- Token-Efficient Leverage Learning in Large Language ModelsarXiv