A token is a unit used to process text
A token is a small unit into which a large language model divides text before processing it. A token may be a whole word, part of a word, punctuation, or a character sequence.
The division is performed by a tokenizer and differs between model families. The same sentence may therefore use a different number of tokens in two systems.
Why do tokens matter?
- API pricing: Usage is commonly billed by the number of input and output tokens
- Input limits: A model’s context window is measured in tokens
- Usage limits: Some services use token consumption as one factor in plan limits
Token counts often explain why a long document is rejected or why an API request costs more than expected.
How much text is one token?
There is no exact universal conversion. In English, one token is often shorter than a full word on average. Japanese and other scripts can use a different number of tokens for the same meaning. Use the tokenizer or usage report for the specific model when cost or limits matter.
Modern models may support hundreds of thousands or even around a million tokens of context, enough for long books or large codebases. That capacity does not guarantee the model will use every part of the material equally well.
Managing token use
For metered APIs, remove irrelevant boilerplate and send only the sources needed for the task. Summarizing material before a later step can reduce cost, but do not compress away details required for accuracy.
An answer that stops abruptly may have reached an output-token limit. Asking for the remainder can continue the response, although the next request will also consume tokens.