What Are Tokens and How They Are Counted: The Architectural Foundation in the Era of LLMs

The Atomic Anatomy: The Bridge Between Human Language and Vectors

At the center of every Large Language Model (LLM) architecture lies a fundamental and frequently misunderstood concept by software engineers: the token, which is the atomic piece of text that models convert into numbers to perform math. To understand the internal mechanics of neural networks like GPT-4, Llama 3, or Claude 3, we must first accept that models do not read words, characters, or full sentences the same way humans do. They operate purely in the mathematical domain, manipulating high-dimensional matrices, which are giant grids of numbers. The token is the atomic unit of this translation, acting as the essential bridge between human language semantics and numerical embedding vectors, which are lists of numbers representing the meaning of words.

When we submit a text string to an AI API, it undergoes a strict preprocessing pipeline before touching the model's synaptic weights, representing the internal adjustable parameters that store learned patterns. If computers processed text character by character, computational cost and memory consumption would explode due to the quadratic complexity of self-attention mechanisms, a mathematical process where every piece of text compares itself to every other piece. Conversely, if they processed whole words, the required vocabulary would be infinitely large, generating out-of-vocabulary (OOV) problems, which happen when the model encounters a word it has never seen before, for rare words, grammatical flexions, or newly coined technical terms. The token resolves this dilemma by splitting text into subwords, balancing semantic granularity with computational efficiency.

Under the Hood of Tokenization Algorithms: BPE, WordPiece, and SentencePiece

Converting raw text into a sequence of numeric IDs is governed by sophisticated tokenization algorithms. The current state of the art relies primarily on three approaches: Byte Pair Encoding (BPE), WordPiece, and SentencePiece. Each possesses algorithmic peculiarities that directly impact context size and model performance across different languages.

Byte Pair Encoding (BPE), initially a data compression algorithm adapted for natural language processing, starts by treating each character as an individual token. Iteratively, it scans the training corpus to count the frequency of all adjacent token pairs, merging the most frequent pair into a new composite token. This process repeats until the desired vocabulary size is reached. The code below demonstrates a simplified conceptual implementation of the pair-counting concept in BPE:

def get_stats(vocab):
    pairs = {}
    for word, freq in vocab.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pair = (symbols[i], symbols[i+1])
            pairs[pair] = pairs.get(pair, 0) + freq
    return pairs

# Conceptual example of subword frequency mapping
initial_vocab = {"l o w": 5, "l o w e r": 2, "n e w e s t": 6}
print(get_stats(initial_vocab))

On the other hand, WordPiece (used by BERT) operates similarly, but instead of selecting the most frequent pair based purely on raw counts, it uses Maximum Likelihood Estimation (MLE), a statistical method to find the most probable explanation for data. The algorithm evaluates how much merging two tokens improves the overall language model probability. Meanwhile, SentencePiece treats the text stream as a raw byte sequence, bypassing the need for explicit whitespace characters as word delimiters. This makes SentencePiece language-agnostic, meaning it works equally well across any human language, and it is widely adopted in modern multilingual models like T5 and Llama.

The Multilingual Disparity: Why Portuguese and Code Consume More Tokens

One of the greatest architectural and financial bottlenecks for companies operating in global markets is the tokenization inefficiency in languages other than English. The tokenizers of major commercial models were trained predominantly on English-language corpora (usually above 70% or 80% of total data). As a result, the tokenizer's internal vocabulary has far more subwords optimized for English than for Portuguese, Spanish, or structured source code.

In practice, this means common words in Portuguese are fragmented into a significantly higher number of tokens. The same phenomenon occurs when processing code blocks in languages like TypeScript, Python, or Rust, where special characters, indentation, and compound variable names trigger aggressive splits. The table below illustrates this stark disparity in real engineering scenarios:

This disparity does not just affect the developer's wallet via API token billing; it directly impacts the model's usable Context Window, which is the maximum amount of text the model can read and remember at once. If a prompt in Portuguese consumes double the tokens to convey the same semantics as in English, the model will exhaust its context window much faster, limiting the amount of code or documentation that can be injected via RAG (Retrieval-Augmented Generation), a technique that fetches external data to give the model extra facts.

Cost Mathematics and Context Window Limits

In the architecture of LLM-based systems, budget and context window management is a critical software engineering discipline. The context window—whether 8k, 128k, or 1 million tokens—represents the maximum token limit the attention mechanism can process simultaneously in a single inference, meaning one single run of generating a response. However, it is vital to understand that computational cost and latency scale non-linearly, growing much faster than the text size due to QKV attention matrix calculations, which manage how words relate to each other.

Furthermore, commercial API pricing models clearly separate costs into two fronts:

Input Tokens (Prompt): Tokens sent by the client to contextualize the model (instructions, chat history, retrieved documents). They generally carry a lower cost per million tokens.
Output Tokens (Completion): Tokens generated autoregressively by the model token by token, meaning the AI predicts the next piece of text one step at a time. The cost is considerably higher (often 3x to 4x the input price) due to the sequential nature of generation, where parallelization, running multiple tasks at once, is severely limited.

Software architects must design systems that avoid redundant transmission of static context with every request. Sending a 50-page technical documentation manual on every API call is a severe financial and architectural anti-pattern, a practice that seems helpful at first but causes long-term problems.

Engineering Techniques for Optimization, Caching, and Prompt Compression

To mitigate the high costs and latency constraints imposed by token counting, several software engineering and prompt optimization strategies must be applied in production:

Prompt Caching: Modern providers like Anthropic (Claude) and OpenAI offer caching mechanisms for long prompt prefixes. If you systematically send a voluminous system prompt (e.g., 20k tokens of business rules and corporate documentation), the system caches the attention states, reducing input cost by up to 90% and drastically lowering Time-to-First-Token (TTFT), the delay before the AI starts typing its response.
Semantic Compression and Summarization: Before injecting long chat histories into context, use smaller, cheaper models (or heuristic algorithms, which use practical rules of thumb) to summarize older messages, retaining only key entities and intents.
Verbosity Reduction and Prompt Minification: Avoid excessive polite preambles ("Please, could you help me with..."). Direct, imperative instructions and the use of structured delimiters (like XML tags or clean JSON) drastically reduce token waste on filler words.

Observability and Budgetary Control in Enterprise Systems

Finally, putting LLM systems into production requires robust observability, the practice of measuring how a system behaves from the inside. You cannot optimize what you do not measure. Enterprise architectures must implement dedicated middleware, acting as a helper software layer between systems, to intercept, log, and audit every token consumed across microservices. Tools like LangSmith, Langfuse, or custom OpenTelemetry exporters should track token usage partitioned by user, tenant, feature, and endpoint. Establishing hard spending caps, rate limiters per user session, and token-based circuit breakers, which automatically stop requests if traffic gets too high, ensures that unexpected loops in autonomous agent architectures do not drain corporate cloud budgets overnight. Mastering the token is, ultimately, mastering the unit of currency in the modern AI era.