What is a token in AI? Tokens in LLMs explained

If you've used an AI API, you've seen the word token everywhere: prices per million tokens, context windows measured in tokens, usage reports counting input and output tokens. Yet tokens are rarely explained. They're simple once you see them, and knowing how they work helps you predict what a request will cost and why a model sometimes runs out of room.

What is a token?

A token is the basic unit of text a large language model (LLM) works with. Before a model can read your prompt, the text is broken into tokens; the model then processes those tokens and generates its answer one token at a time. When the answer comes back, the tokens are turned back into ordinary text.

A token is usually a chunk of a word rather than a whole word. Common short words are often a single token. Longer or rarer words are split into several pieces. Spaces, punctuation, numbers and symbols all take tokens too.

Why models use tokens instead of words

Models need a fixed vocabulary of pieces they can recognise. A vocabulary of whole words would be enormous, and it would still fail on names, typos, new slang and code. A vocabulary of single characters would be tiny but make every text extremely long. Tokens sit in between: a vocabulary of common word pieces lets a model handle any text, in any language, while keeping sequences reasonably short.

The process of splitting text into tokens is called tokenization, and the component that does it is the tokenizer. Frequent sequences of characters become single tokens; everything else is built from smaller pieces. That's why a common English word might be one token while an unusual technical term takes three or four.

How many words is a token?

There's no exact conversion, because it depends on the text and the model. For ordinary English prose, a widely used rule of thumb is:

• One token is roughly four characters of English text.
• One token is roughly three quarters of a word, so 100 tokens is about 75 words.
• 1,000 words is therefore roughly 1,300 to 1,400 tokens.

Treat these as estimates, not rules. Code, numbers, tables and structured data such as JSON usually take more tokens per word, because they're full of symbols. Many languages other than English also use more tokens for the same meaning. And each model family has its own tokenizer, so the same text can be a different number of tokens for Claude, GPT or another model.

Input tokens and output tokens

Every request has two kinds of tokens. Input tokens are everything you send: the system instructions, the conversation so far, any documents or tool results, and the user's message. Output tokens are what the model writes back. Some models also produce reasoning or thinking tokens before their final answer; these are usually billed as output.

The distinction matters because the two are priced differently. Output tokens typically cost around five times as much as input tokens, since the model generates them one at a time. A short question that produces a long answer can cost more than a long document that produces a short summary.

Tokens and cost

AI APIs price text models per million tokens, with separate rates for input and output. To estimate a request, multiply each count by its rate. For example, at GreenTokens prices Claude Sonnet 5 costs $1 per million input tokens and $5 per million output tokens. A request with 2,000 input tokens and 500 output tokens costs:

• Input: 2,000 × $1 ÷ 1,000,000 = $0.002
• Output: 500 × $5 ÷ 1,000,000 = $0.0025
• Total: $0.0045 per request, or $450 for 100,000 of them.

Prices for every model are on the models page. For a fuller walk-through, including images and video, see how AI API pricing works.

Tokens and limits

Tokens also set a model's limits. The context window, the most a model can consider at once, is measured in tokens, and it has to hold both your input and the output the model writes. Most APIs also let you cap output with a maximum token setting. If a conversation outgrows the context window, older content has to be removed or summarised. We explain this in detail in what a context window is.

How to count tokens

Two reliable ways: read the usage the API returns with every response, or count before sending. The Anthropic-format token counting endpoint returns the exact input size for a request, and on GreenTokens it's free:

from anthropic import Anthropic

client = Anthropic(base_url="https://api.greentokens.io", api_key="sk-gt-…")

count = client.messages.count_tokens(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": "How many tokens is this sentence?"}],
)
print(count.input_tokens)

Using fewer tokens

Because you pay per token, the cheapest token is the one you don't send. Keep system prompts tight, don't resend documents the model doesn't need, cap output length, and use prompt caching for context that repeats. How AI API pricing works shows where the rest of a bill comes from.

Common questions

Do spaces and punctuation count as tokens?

Yes. Spaces are usually folded into the token that follows them, but punctuation marks, line breaks, symbols and formatting characters all take tokens. That's one reason code, JSON and heavily formatted text use more tokens than plain prose of the same length.

Why did my request use more tokens than I expected?

The usual causes are things you didn't write yourself: a long system prompt, conversation history resent with each turn, tool definitions, or documents added by your application. Some platforms also add hidden default instructions. The usage figures in the response always show the true count, so start there.

In short

A token is a small piece of text, often part of a word, that language models read and write. English averages about three quarters of a word per token, but code and other languages use more. Tokens determine what a request costs, with output priced higher than input, and they define the limits of what a model can handle at once. Once you think in tokens, AI pricing and limits stop being mysterious.