AI Tokens and Context Windows Explained: How They Shape Cost, Limits and Accuracy

Every time you send a prompt to ChatGPT, Gemini or Claude, the model never sees your words as words. It sees tokens, small chunks of text that drive almost everything that matters in practice: what a request costs, how much you can paste in, how long an answer can be and, less obviously, how accurately the model uses what you gave it.

This guide explains tokens and context windows in plain terms, using the figures the major AI providers publish in their own documentation, and shows how to estimate, count and manage them, whether you are a developer paying per token or simply someone who keeps hitting "message too long".

Key facts at a glance

• One token is roughly four characters of English text, according to both OpenAI and Google. OpenAI puts 100 tokens at about 75 words; Google says 60 to 80 words.
• A context window is the model's working memory: everything in one request, including the system prompt, the conversation history, files, tool definitions and the answer itself.
• Hidden "thinking" counts too. OpenAI and Anthropic both say reasoning or thinking tokens are billed as output tokens, even though you never see them as answer text.
• Today's large windows hold about 1 million tokens. Google documents 1,048,576 tokens for Gemini 4, and Anthropic lists a 1M-token window for its current Claude models.
• More context is not automatically better. Research from Stanford and others found that models use information at the start and end of a long prompt far better than information in the middle.

Who this guide is for

It is written for three groups: developers building on AI APIs who need to predict costs and avoid errors; teams deciding whether to paste whole documents into a chat or build a retrieval system; and everyday users who want to understand why a long chat starts forgetting details or why an upload is rejected.

What a token actually is

A tokenizer splits text into pieces from a fixed vocabulary. Google's developer documentation describes tokens as anything from a single character such as "z" to a whole word such as "cat", with long words broken into several tokens. OpenAI's help centre adds that a token can also be punctuation, and that spaces and capitalisation change how text is split: "red", "Red" and " red" with a leading space can all be encoded differently.

Two practical consequences follow.

Word counts and token counts are not the same. OpenAI's rule of thumb for English is that one token is about three-quarters of a word, so a 1,500-word report is roughly 2,000 tokens. Google's equivalent estimate is 100 tokens for every 60 to 80 English words, which would put the same report closer to 2,000 to 2,500 tokens. Both companies stress these are estimates, and OpenAI notes that other languages can have very different ratios.

Each model family counts differently. OpenAI warns that the same text can produce different token counts depending on the model, its encoding and the language. That is why a lower price per million tokens on one model does not guarantee a cheaper bill: the same job may simply use more tokens there.

Tokens beyond text: images, audio and video

Tokens are not just for words. Google's Gemini documentation spells out how other media are converted:

Source: Google AI for Developers token documentation.

Google says its newer "agentic" video mode loads only the transcript, frames or audio needed to answer, and estimates that a one-hour lecture using about 1.08 million tokens in static mode might use about 108,000 instead, depending on the prompt. Anthropic, for its part, limits a single Claude request to 600 images or PDF pages on its 1M-token models, or 100 on models with a 200,000-token window, and warns that big files can hit request size limits before the token limit.

How context windows work

Anthropic's documentation describes the context window as all the text a model can reference when generating a response, including the response itself, and contrasts it with the much larger body of data the model was trained on. Think of it as a desk rather than a library: only what is on the desk right now can be used.

In a chat, the desk fills up turn by turn. Each new message is sent together with the full history, so the tenth reply in a conversation costs far more input tokens than the first. Everything counts: the system prompt, every earlier message, tool results, images, documents and tool definitions.

Current limits vary by provider and model:

• Google says Gemini 4 models share a single 1,048,576-token window across input, thinking and output. Synchronous responses are capped at 65,536 output tokens by default, and if input plus the requested output would exceed the window, the API automatically lowers the output limit. Older Gemini 3 and 2.5 models use separate input (1,048,576) and output (65,536) budgets.
• Anthropic lists a 1M-token window for its current Claude models, with up to 128,000 output tokens in one request, while some older models stay at 200,000 tokens. It says long-context requests on 1M models are billed at standard rates, except Claude Haiku 5.5, where prompts over 100,000 tokens cost more.

What happens when you overflow also differs. Anthropic says that if the input alone exceeds the window, the API returns an error with the message "prompt is too long". OpenAI suggests shortening prompts, removing repeated context, splitting large inputs or summarising text first.

Why hidden tokens change the bill

Reasoning models think before they answer, and that thinking is made of tokens. OpenAI's help centre states that reasoning tokens are not visible as answer text but count toward output usage and are billed as output tokens, so "a short visible answer can therefore use more tokens than its displayed text suggests". Anthropic says the same of Claude's thinking tokens, which are a subset of the max_tokens setting and also count toward rate limits.

Other categories matter too. OpenAI separates input, output, cached input and reasoning tokens, and notes that cached input can be priced differently. Anthropic stresses that cached prompt prefixes still occupy the context window: caching changes what you pay for those tokens, not whether they count. And when an API generates several alternative answers at once, every one of them is billed.

Bigger windows, weaker attention

A 1M-token window sounds like permission to paste in everything. The evidence says be careful.

The 2023 paper "Lost in the Middle", by researchers from Stanford University, the University of California, Berkeley and Samaya AI, tested how models use long inputs in a multi-document question-answering task. Performance followed a U-shaped curve: models did best when the relevant passage was at the very beginning or end of the prompt, and worst when it sat in the middle. For GPT-3.5-Turbo, accuracy fell by more than 20 per cent in the worst positions, and with 20 or 30 documents it dropped below the model's 56.1 per cent score with no documents at all. Adding more retrieved documents beyond 20 improved results by only about 1.5 per cent for GPT-3.5-Turbo and about 1 per cent for Claude 1.3, while raising cost and latency.

Those models are now old, and newer systems handle long inputs better. Still, Anthropic's current documentation makes the same point in its own words: as token count grows, accuracy and recall degrade, a problem it calls "context rot", so curating what goes into the window matters as much as how big it is.

How to estimate and count tokens

1. Use the rules of thumb for quick planning. About 4 characters or three-quarters of a word per English token is enough to size a document or a monthly budget.
2. Count exactly before you send. OpenAI offers a web Tokenizer, the open-source tiktoken library and an input-token counting API for full requests including images, files and tools. Google provides a count_tokens call in its SDKs. Anthropic offers a token counting API.
3. Check actual usage after each call. Every provider returns a usage record. OpenAI reports input, output and total tokens; Google breaks out thinking, cached and tool-use tokens; Anthropic splits cached input into separate fields.
4. Set an output cap. Use the provider's maximum output setting so a runaway answer or long reasoning cannot surprise you.
5. Test with real tasks. OpenAI recommends comparing models on the total tokens and cost needed to finish a representative job, not on price per token alone.

What to do when you hit the limits

• Trim and restructure. Remove repeated instructions and boilerplate; keep only the passages the task needs.
• Put the important material at the start or end, and restate the question after a long document, in line with the "Lost in the Middle" findings.
• Retrieve, do not dump. For large knowledge bases, fetch only the most relevant sections instead of pasting everything.
• Summarise long conversations. Anthropic offers server-side compaction, which summarises earlier turns so a conversation can continue past the window; in a chat app, starting a fresh chat with a short summary does the same job by hand.
• Watch images and video. Lower resolution or shorter clips can cut token use sharply.

Conclusion

Tokens are the unit behind price, limits and quality in every major AI system. A few habits go a long way:

• Estimate with "4 characters per token", then confirm with the provider's counting tool before large jobs.
• Budget for hidden reasoning tokens, which are billed as output even though you never see them.
• Remember the whole conversation is resent each turn, so long chats grow expensive and less precise.
• Treat a huge context window as capacity, not a strategy. Put key facts first or last, and send only what the task needs.
• Compare models on cost per finished task, not price per million tokens.