The Seven Token Categories Enterprises Should Understand

Discussion of LLM pricing often oversimplifies costs to just input and output tokens. While this may suffice for demos, it is inadequate — and potentially risky — for enterprise production systems.

Modern AI architectures generate tokens in at least seven categories, many of which are not visible to those managing costs. As organizations scale, the gap between perceived and actual token usage becomes a significant blind spot in enterprise AI economics. This is not an official taxonomy — it is a practical architecture lens for understanding where token consumption hides.

What a token actually is

A token is the atomic unit an LLM uses to represent information as numbers; it is the translation layer between human-readable input and machine-processable form. For text, a token is typically a short word or common subword fragment: “unbelievable” might become three tokens (“un,” “believ,” “able”), while “cat” is one. As a rough rule of thumb, one token is about four characters of English, so a 1,000-word document is roughly 1,300 tokens. For images the unit is a small patch of pixels; for audio or video, a discrete segment of the stream.

This is where a costly misconception enters: the model has no awareness of where its context came from. A thoughtful user query, a 400-page regulatory document injected via retrieval, a JSON blob from an API, and a system prompt written three months ago are all processed identically — as tokens in a context window, each one metered and billed. There is no discount for irrelevant content, and no distinction between signal and noise.

The same logic applies in reverse for output. Every token the model produces — whether a precise answer or a verbose preamble — is billed at the output rate, typically two to five times higher than input. In agentic systems, where outputs frequently become the inputs to subsequent steps, this compounds quickly: verbose output in one layer becomes bloated input in the next.

The seven token categories

1. Input tokens

Prompts, system instructions, conversation history, RAG-retrieved passages, and any text sent to the model at inference. The most visible and usually the least expensive — which often leads teams to underestimate their accumulation.

2. Output tokens

Generated responses, typically more expensive than input tokens. Yet in agentic systems they often account for only a minor portion of total cost, with most expense arising elsewhere.

3. Thinking / reasoning tokens

Advanced models can perform internal analysis — synthesis, classification, planning, extraction, multi-step reasoning — before producing a response. These intermediate steps are usually invisible to the user but still consume compute. Vendors account for this differently:

• Some bill them directly as a distinct category (a separate line item).
• Some absorb them into premium pricing tiers (you control model choice, but pricing is hidden).
• Some expose a separate metric for reasoning depth, letting teams set limits.
• Some hide them entirely within standard output charges (visible only as output volume).

A request that looks cheap by input/output metrics can become unexpectedly expensive when it triggers extended reasoning. High ambiguity, conflicting context, complex tool planning, and long context windows all increase reasoning-token consumption.

4. Retrieval and embedding tokens (RAG)

When a system retrieves documents to ground a response, those documents are injected into the prompt as additional context — and become input tokens, indistinguishable from anything a human typed. RAG has a second cost layer too: embeddings. Documents must be embedded during ingestion, and queries are often embedded at runtime. Poor retrieval design is one of the most common sources of token waste: a pipeline that embeds too much, retrieves too broadly, or injects ten weakly relevant chunks “just in case” pays multiple times.

5. Tool-call tokens

When an agent invokes an external function, every stage of the cycle consumes tokens:

• planning the tool call
• generating the invocation payload
• receiving the API response
• injecting the response into context
• reasoning through the result

In agentic systems, this loop repeats — often many times per user request.

6. Orchestration tokens

Multi-agent architectures present significant risk. A typical agent swarm includes a planner, multiple executors, a critic, a memory manager, and a safety layer. A single question can trigger numerous prompts, tool calls, and hundreds of thousands of tokens — none visible to the user. Orchestration tokens are the overhead of coordinating work: routing tasks, managing handoffs, running tool loops, and synthesizing results.

7. Memory tokens

Long-running systems require memory. Past decisions, user preferences, prior outcomes, and corrective overrides must be available at inference time — and the context window is the only place to store them. Memory differs from orchestration: orchestration coordinates work, memory supplies history. A personalized chatbot may have high memory overhead with minimal orchestration; an agent swarm may have the reverse. They scale and fail differently, and conflating them leads to ineffective solutions.

Closing thoughts

The transition from AI experimentation to a production platform is not primarily a technical problem — it is an economic one. In traditional software, efficiency means optimizing CPU, memory, bandwidth, and I/O. In enterprise AI, architecture must treat tokens with the same rigor as hardware resources.

The gap between a compelling demo and a profitable production system is almost always found in these seven categories.

The organizations that close this gap first will not necessarily be the ones with the best models. They will be the ones that treat token efficiency as a first-class engineering discipline: instrumented, audited, and governed with the same rigor they bring to any other critical infrastructure cost.