5  Inside the Language Model

“What I cannot create, I do not understand.”

— Richard Feynman, on his blackboard at the time of his death, 1988

After this chapter you will understand what a large language model actually is underneath the chat window: how it turns your words into numbers, how it generates a reply one piece at a time, how it was trained to be helpful, and why each of those facts quietly decides how you must build an agent around it.

5.1 Opening the engine before we drive it

Part I handed us a loop: sense the world, decide what to do, act, and repeat. We kept the middle box, the one labelled decide, deliberately shut. We drew it as a clean rectangle and promised that Part II would open it and look at the engine we put inside it today, the large language model. The next chapter will ask what that engine can and cannot reason about. But you cannot judge an engine you have never seen, so this chapter opens the casing first and looks at the machine itself.

Figure 5.1: The model sits at the center of the agent loop we drew in Part I. A request and the environment’s observations come in; the model reasons and decides in tokens; it acts, often by calling a tool; and the result feeds back to inform its next move. The strip along the bottom previews the machine this chapter opens up: the single generation step in which text becomes tokens, tokens flow through the network, and one next token is chosen and appended.

The order matters. Almost every hard decision in the rest of this book, how much to trust an answer, how to shape a prompt, when to reach for a tool, how to keep costs down, is really a reaction to some concrete fact about how this machine is built. Why does the model miscount the letters in a word, give a different answer to the same question twice, or slow to a crawl on a long conversation, yet still follow instructions no one ever wrote down for it? None of these are mysteries once you have seen the parts. They are direct, almost mechanical consequences of four things: how text becomes numbers, how those numbers flow through the network, how the next word is chosen, and how the whole thing was trained. This chapter walks through those four things in plain language.

Think of it the way you would think about hiring a brilliant new colleague whose work you will have to stake your own reputation on. Before you hand them the account, you want to know how they think: what they are astonishingly good at, where they cut corners, what makes them guess instead of check. You are not trying to become a neuroscientist. You just need a working model of the person, good enough to know which tasks to trust them with and which to double-check. That is exactly the goal here. We are not going to derive the mathematics of deep learning. We are going to build a mental model of the language model that is accurate enough to make good engineering decisions, and honest enough about the machine’s habits that those decisions hold up when real money and real users are on the line. Keep that new colleague in mind. We will check back in with them as each part of the machine comes into view, because every part we open up explains one more of their quirks.

A quick word on what this chapter is not. It is not a machine-learning course, and you will not need one to follow it. Wherever there is a choice between a precise equation and a picture that carries the same intuition, we will draw the picture. The payoff comes at the very end, where a short list of practical consequences, the kind you can tape above your desk, falls straight out of the parts we assemble along the way. By then, “the model is limited,” “the output is probabilistic,” and “long prompts cost money” will not be slogans you have to memorize. They will be things you can see.

The model itself starts with a problem so basic it is easy to miss: before it can do anything with your words, it has to turn them into numbers. How it does that, and what quietly goes missing in the translation, turns out to be the root of half the model’s strangest habits.

5.2 From words to numbers: tokens

A neural network cannot read. It has no notion of letters, words, or spaces. It is, at heart, a machine that multiplies long lists of numbers together. So the very first thing that happens to your prompt, before any “intelligence” enters the picture, is a translation step: the text is chopped into pieces and each piece is swapped for a number. Those pieces are called tokens, and almost everything surprising about how a model handles spelling, arithmetic, and word counting traces back to this one quiet step.

5.2.1 How a tokenizer splits text

You might expect the model to split text into words, or maybe into single letters. It does neither. It uses something in between, a subword scheme, most commonly a method called byte-pair encoding [1]. The idea is delightfully simple. Start with raw characters, then repeatedly glue together the pair of symbols that appears most often in a huge pile of text. Do that tens of thousands of times and you end up with a fixed list of chunks, the model’s vocabulary, where common words like ” the” or ” and” become a single token, while rarer words get built from a few pieces. The word “tokenization” might split into token and ization; a name the tokenizer has never seen might shatter into five or six fragments.

Each token in that vocabulary has a fixed slot number, its token ID. The tokenizer’s whole job is a lookup: it turns your sentence into a list of these integers, and turns the model’s output integers back into text. Nothing more. The model itself only ever sees the numbers.

Figure 5.2: The same short sentence, split four ways. Word-level and character-level are the two extremes; modern models use a subword scheme in between, where common words stay whole and rarer ones are built from pieces. Whatever the scheme, the model ultimately reads a list of token IDs, not letters.

Why chop words into subwords at all, rather than keeping a giant list of whole words? Because language is open-ended. New words, typos, product names, code identifiers, and other languages appear endlessly, and no fixed word list could ever hold them all. Subwords give the model a finite vocabulary, usually somewhere around fifty to a hundred thousand tokens, that can still spell out anything by combining pieces. It is the same trick the alphabet plays: twenty-six letters, infinite words. Subword tokens are a coarser alphabet, tuned so the common stuff is cheap and the rare stuff is still expressible.

Figure 5.3: How byte-pair encoding grows that vocabulary. It begins with raw characters, then repeatedly merges the pair of symbols that occurs most often, one merge at a time, until frequent chunks like “est” become single tokens. Because those chunks are reused across words, a small fixed vocabulary can still spell almost anything.

5.2.2 Why tokens are also the meter

Tokens are not just an internal detail. They are the unit the model is billed and bounded in. When a provider quotes a price “per thousand tokens,” or a model advertises a “128,000-token context window,” that is this exact currency. A rough rule of thumb for English is that a token is about four characters, or roughly three-quarters of a word, so a page of text is somewhere near five hundred tokens. Code, unusual names, and languages that do not use spaces tend to fracture into more tokens, which is why the same idea can cost more to process in one form than another. We will return to the money side of this later in the chapter, but hold on to the core fact: the model reads, writes, remembers, and charges in tokens, not words.

5.2.3 Why this breaks exact reasoning

This is the moment tokenization stops being trivia. A famous party trick is to ask a capable model how many times the letter “r” appears in “strawberry” and watch it stumble. The reason is not stupidity. It is that the model never saw the letters. “strawberry” may have arrived as two or three tokens, and inside the model those tokens are single opaque symbols, not a sequence of characters it can count. Asking it to count letters is like asking someone to count the strokes in a word they only ever heard spoken. The information was thrown away at the door. It is our new colleague again: fluent, widely read, and yet unable to see the very letters you are pointing at.

Figure 5.4: The same blind spot, four ways. Because the model works over tokens rather than letters, anything that depends on exact characters tends to fragment: numbers split unevenly, casing changes the tokens, code and symbols break into pieces, and non-English text splits into more tokens. Work like this belongs to a tool, not the model’s intuition.

The same root cause explains a cluster of related weaknesses. Digits get grouped into tokens in irregular ways, so “1234” and “12345” may be split quite differently, which is part of why models are shakier at long multiplication than their fluency suggests [2]. Reversing a string, counting words, or manipulating text character by character all run against the same grain: the model reasons over tokens, not over the raw symbols you see. None of this makes the model useless at these tasks. It makes them tasks you should hand to a tool, a calculator or a few lines of code, rather than trust to the model’s intuition. That instinct, “let the model decide and a tool compute,” is one we will build on for the rest of the book, and it starts here, with a clear-eyed view of what the model literally cannot see.

With the input now reduced to a list of numbers, we can follow those numbers inward and ask the central question: what actually happens between the tokens going in and a prediction coming out?

5.3 The transformer, without the math

Almost every model you will use is built on an architecture called the transformer, introduced in a 2017 paper with the fitting title “Attention Is All You Need” [3]. You do not need its equations to build agents, but you do need its central intuition, because that intuition explains why models are so good at context and so bad at a few stubborn things. That intuition arrives in three moves, each building on the one before: first tokens become meaning, then words learn to look at each other, then we stack the trick deep.

5.3.1 Meaning as coordinates

The token IDs from the last section are just arbitrary slot numbers. Token 4,912 is no closer in meaning to token 4,913 than to token 50,000. The first thing the model does is replace each ID with a long list of numbers called an embedding, a point in a space with hundreds or thousands of dimensions. Think of it as a very elaborate map. On an ordinary map, two dimensions place every city by latitude and longitude, and cities that are near each other on the ground are near each other on paper. An embedding does the same thing for meaning: it places every token in a space where “king” sits near “queen,” “Paris” sits near “France,” and “delighted” sits near “thrilled.” These coordinates are not hand-assigned. They are learned during training, squeezed out of the model’s one relentless task of predicting the next token, because words that behave alike in text end up needing similar coordinates to be predicted well.

5.3.2 Attention: letting words look at each other

A pile of word-points is not yet understanding, because the meaning of a word depends on its neighbors. In “I sat by the river bank” and “I deposited it at the bank,” the word “bank” is identical on the page but means two different things, and the only way to tell them apart is to look at the surrounding words. This is the job of attention, the mechanism the transformer is named for.

The idea, stripped to its core, is this: at every position, the model looks back over all the earlier words and decides which ones are relevant to the word it is currently processing, then pulls in information from those and largely ignores the rest. When it processes “it” in “the trophy did not fit in the suitcase because it was too big,” a well-trained model learns to attend from “it” back to “trophy” rather than “suitcase,” which is exactly the piece of context needed to resolve the pronoun. Attention is a learned spotlight: for each word, it brightens the handful of earlier words that matter and dims the rest, so every word’s representation gets updated in light of its real context rather than its dictionary definition.

Figure 5.5: Attention for one head, step by step. Each token’s query is compared against every earlier token’s key to score relevance; a causal mask blocks any look at tokens that come later; a softmax turns those scores into weights; and the weights blend the earlier tokens’ values into a new, context-aware representation of the current token. Stacking many such heads and blocks is what turns a pile of word points into understanding.

5.3.3 Depth: stacking the trick

One round of this look-and-update is not enough for anything subtle. So the transformer stacks the operation into many layers, dozens in a small model and far more in a large one. Each layer takes the context-enriched words from the layer below and refines them again, and the intuition is that meaning gets assembled in stages. Early layers tend to settle low-level matters like grammar and which word attaches to which; middle layers build up phrases and relationships; later layers work with more abstract meaning, the drift of an argument, the tone of a request, the intent behind a question. Nobody programmed this division of labor. It emerges because a deep stack, trained to predict the next token across a staggering amount of text, discovers that building understanding in layers is the way to do the job well.

If you want to open the block itself and see that there is no magic inside, only attention and arithmetic repeated at scale, the full picture is worth one careful look.

Figure 5.6: One transformer block, opened up, and the stack it lives in. On the left, a decoder-only model is little more than token and position embeddings, then the same block repeated L times, topped by a head that turns the final vectors into next-token probabilities. On the right, inside one block: masked multi-head self-attention lets each position gather from the tokens before it, never ahead, and a position-wise feed-forward network then transforms each position on its own, with residual connections and normalization keeping a deep stack trainable.

That is the whole engine, in intuition if not in arithmetic. Tokens become points in a meaning-space; attention lets each point absorb its relevant context; depth repeats that until the top of the stack holds a rich, context-aware representation of everything read so far. The output of that top layer is used for one thing above all, the thing the model was built to do: guess what comes next. How that single guess grows into whole paragraphs is where we turn now.

5.4 Generating one token at a time

So far we have described the model as something that predicts “the next word.” Take that literally, because the way a single prediction grows into a whole answer is the source of two facts you will design around constantly: its outputs are probabilistic, and its memory is nothing more than the text in front of it.

5.4.1 A probability for every token

When the model finishes processing your prompt, it does not produce a sentence. It produces a single thing: a probability for every token in its vocabulary, a score for how likely each one is to come next. Given “The capital of France is,” the token ” Paris” might get a very high probability, ” a” a modest one, and ” banana” a vanishingly small one. That entire distribution, tens of thousands of numbers, is the model’s raw output for one step. Everything the model “knows” at that moment is compressed into this one ranked list of guesses.

Figure 5.7: One generation step, end to end. The current tokens are looked up as embeddings, run through the decoder layers, and turned into a probability over the entire vocabulary; a decoding strategy then picks a single token, which is appended so the whole thing can repeat. Every word a model writes is produced one pass of this loop at a time.

5.4.2 Sampling, and why answers vary

To turn that distribution into an actual token, the system has to pick one. The simplest rule is to always take the highest-scoring token, which is called greedy decoding. But most systems instead sample: they roll a weighted die, so a token with probability 0.6 is chosen about sixty percent of the time and lower-ranked tokens occasionally win. This is why the same prompt can give different answers on different runs. Our new colleague, asked the same question on Monday and again on Friday, answers a little differently each time, not from indecision but because a weighted roll of the dice is simply how they think. It is not a bug or a sign of creativity in any mystical sense. It is a deliberate coin-flip baked into how text is produced.

Two dials control that coin-flip. Temperature flattens or sharpens the distribution: low temperature makes the model almost always take its top guess, giving steady, repetitive, predictable text; high temperature spreads the odds, giving surprising and sometimes incoherent text. A related setting, top-p or nucleus sampling, keeps only the smallest set of top tokens whose probabilities add up to some threshold and samples from those, which trims off the long tail of absurd options while still allowing variety [4]. The practical upshot is a knob you will reach for often: turn it down when you need reliable, near-deterministic output, such as producing structured data or following a strict format, and turn it up when you want range and are willing to tolerate the occasional odd turn.

Figure 5.8: The same step, run in a loop, is all it takes to write. Each row feeds the current sequence to the model, which predicts one token; that token is appended and the sequence rolls forward into the next row, adding “Paris,” then “France,” then a period, and on. A next-token predictor becomes a system that writes arbitrary text, which is also why more tokens mean more compute, latency, and cost.

5.4.3 The loop that writes paragraphs

A single token is not much of an answer, so generation is a loop. The model predicts one token, the system appends it to the text, and then the whole, now slightly longer, text is fed back in to predict the next token. Predict, append, feed back, repeat, until a special stop token is produced or a length limit is hit. This is what autoregressive means: each new token is generated in the context of every token before it, including the ones the model itself just wrote. The reply you read was not planned as a whole and then typed out. It was grown one token at a time, each choice committing the model a little further down a path.

This detail has a subtle consequence worth naming now, because the next chapter leans on it. Because each token is conditioned on the words already written, the act of writing intermediate steps genuinely changes what the model predicts next. Getting the model to “think out loud” before answering is not decoration; it literally feeds its own reasoning back in as context for the final answer. That is the mechanical seed of chain-of-thought prompting [5], which the next chapter treats in full.

5.4.4 The context window is all it has

There is a hard limit on how much text the model can be fed at once, its context window, measured, of course, in tokens. Everything the model can use to answer, the system instructions, the conversation history, any documents you pasted, the tool results, and its own reply so far, has to fit inside that window. The surprise is what happens between requests: the model remembers nothing. It does not carry a running memory of your conversation. The illusion of memory is created by resending the whole transcript on every turn, so it fits back inside the window and the model can read what was said before.

Figure 5.9: The context window is the whole of the model’s working memory. The conversation so far is flattened into one token sequence that must fit inside a fixed window; the model attends across everything in the window to generate the next token. Anything that falls outside has to be resent, retrieved, or summarized back in, because nothing persists between calls on its own.

That single fact drives an enormous amount of agent design. It is why long conversations eventually forget their own beginnings, why important instructions can get lost in the middle of a huge prompt [6], and why the entire discipline of memory systems, which gets its own chapter later, exists at all. The model’s working memory is a fixed-size scratchpad that gets wiped and rewritten on every call. Everything durable has to be stored outside it and fed back in. Keep that picture close: it is the reason so much of building agents is really the craft of deciding what to put in the window, and what to leave out.

So the model reads, thinks, and writes. What we have not asked is where any of its competence, its grammar, its facts, its manners, its willingness to follow instructions, actually came from. That is a story about training, and it is the key to predicting how a model will behave.

5.5 How a model is trained

A useful model is not built in one step. It is raised in stages, and each stage installs a different kind of competence. This matters to you as a builder because a model’s habits, what it is good at, what it refuses, how it phrases things, how hard it tries to be correct versus merely agreeable, are not random personality quirks. They are the direct fingerprints of how it was trained. If the earlier sections introduced our new colleague, this one is their upbringing and their work history, the record that explains every habit you will later rely on or work around. Learn these stages and you can usually predict, and sometimes fix, why a model behaves the way it does.

Figure 5.10: How a raw predictor becomes a helpful, careful reasoner, one training stage at a time. Pretraining reads the world to build broad competence; supervised fine-tuning teaches it to answer rather than merely continue; preference and reward modeling capture what people actually like; alignment tunes the model toward those preferences to be helpful, honest, and safe; and reasoning training rewards getting verifiable problems actually right. Each stage installs a capability the one before it lacks.

5.5.1 Pretraining: reading almost everything

The first and by far the largest stage is pretraining. The model is shown an enormous corpus of text, much of the public internet, a large slice of digitized books, code, and more, and set to the single task from the last sections: predict the next token, over and over, trillions of times. Every time it guesses wrong, its internal numbers are nudged a little to make the right token more likely next time. That is the entire learning signal. No labels, no human grading the answers, just an endless stream of “what comes next” drawn from real text [7].

The astonishing thing, the puzzle the next chapter unpacks, is that this humble objective forces the model to absorb an enormous amount about the world, because you cannot predict text well without it. To guess the next word in a physics explanation you must pick up some physics; to finish a line of code you must learn to program; to complete a story you must track characters and motives. Grammar, facts, styles, and a rough working knowledge of how the world tends to go all get squeezed in as a side effect of the prediction game. And it scales in a strikingly regular way: bigger models trained on more data with more computation get predictably better, a relationship smooth enough that researchers charted it as scaling laws [8] and later refined the recipe to balance model size against data so the compute is not wasted [9]. Some abilities seem to switch on only past a certain scale [10], which is part of why the last few years have felt like a series of sudden jumps.

What comes out of pretraining, though, is not an assistant. It is a raw text continuer. Ask a purely pretrained model “What is the capital of France?” and a perfectly valid completion is another question, because that is the kind of text that often follows a question on the internet: “What is the largest city? What language do they speak?” It has knowledge but no notion that it is supposed to answer you. Fixing that is the next stage.

5.5.2 Supervised fine-tuning: learning to answer

To turn a text continuer into something that responds to requests, the model goes through supervised fine-tuning, often called instruction tuning. Here it is trained on a curated set of example conversations: a prompt, and a high-quality response of the kind we want the model to produce, many thousands of them, written or vetted by people. The learning signal is the same next-token prediction as before, but now the text it practices on is example after example of an instruction being followed well [11].

The effect is like the difference between someone who has read the entire library and someone who has also been shown how a good assistant actually behaves. The knowledge was already there from pretraining; supervised fine-tuning teaches the model the format and intent of being helpful: when you ask a question, give an answer; when asked for a list, produce a list; when given an instruction, carry it out rather than continue it. This stage is where the raw model first starts to feel like the chat assistants you know.

5.5.3 Alignment: learning what people prefer

Supervised fine-tuning teaches the model to answer, but “an answer” is a low bar. Of two truthful, on-topic replies, one might be clear and well-organized and the other rambling or subtly rude, and it is hard to capture that difference by writing more example answers. So a third stage tunes the model against human preferences rather than fixed targets.

The classic recipe is reinforcement learning from human feedback, or RLHF. People are shown several of the model’s responses to the same prompt and asked which they prefer. Those comparisons train a second model, a reward model, to predict how much a human would like any given response [12]. The language model is then optimized to produce responses the reward model scores highly, effectively practicing against an automated stand-in for human taste [11]. This alignment stage is where much of a model’s felt personality comes from: its politeness, its habit of adding caveats, its refusals, its house style. A newer family of methods, of which direct preference optimization is the best known, reaches a similar result more simply, by optimizing the model directly on the preference comparisons without training a separate reward model first [13].

It is worth naming the double edge here, because it shapes real behavior. Training a model to produce responses people rate highly is not the same as training it to produce responses that are true. A confident, agreeable, well-formatted answer can win the preference even when it is wrong, which is one of the pressures behind a model’s tendency to sound sure of itself and to tell you what you seem to want to hear. Alignment makes models pleasant and usable; it does not, by itself, make them honest. The next chapter takes that limitation as one of its central themes.

5.5.4 Reasoning training: rewarding a right answer

The most recent stage to become standard targets a specific weakness: on hard problems in math, code, and logic, a model that was only ever rewarded for sounding good has no particular reason to be correct. The fix is to train it on problems where the answer can be checked automatically, and to reward it for actually getting them right. Because the reward comes from a verifier, a test suite that passes or a math answer that matches, rather than from human ratings, this is often called reinforcement learning from verifiable rewards.

The striking recent finding is how far this can go. The DeepSeek team showed that letting a model attempt verifiable problems and rewarding only correct final answers, using an efficient method they call Group Relative Policy Optimization [14], caused sophisticated reasoning behavior to emerge on its own: the model learned to work through long chains of steps, check itself, and change strategy mid-solution, without ever being shown human examples of how to reason [15]. This is the training stage behind the “reasoning models” that pause to think before answering. It also sharpens the distinction the next chapter draws: a model tuned this way is genuinely better at problems with a checkable answer, and no better, sometimes worse, at open-ended judgment where nothing can be verified.

5.5.5 Tool use is a trained behavior too

One last point closes the loop between this chapter and the rest of the book. A model’s ability to call a tool, to emit a neatly structured request for a calculator, a search, or a database, is not a separate faculty bolted on at runtime. It is a behavior installed by training, taught with examples of when and how to call functions and reinforced when the calls are well-formed and useful [16]. This is why “tool calling” is best understood as learned behavior plus orchestration: the model has been trained to recognize when a tool is needed and to produce the request, while your surrounding code does the actual calling and feeds the result back. Chapters ahead build whole systems on this handshake. For now, simply notice that even the model’s agentic instincts trace back to how it was raised.

Training explains what the model can do. The last piece of the machine is more mundane but just as consequential for anyone shipping a real system: what it costs, in time and money, to run.

5.6 Making it fast and affordable

Everything so far has been about what the model computes. This section is about what that computation costs, because in a real agent the bill and the wait are not afterthoughts. They decide which designs are viable. The good news is that the cost model falls straight out of the machine we have already built, so it takes only a little more to see it clearly.

5.6.1 Two phases, and why writing is the slow part

Running the model on a request happens in two phases. First it reads your whole prompt and builds up its internal representation of it. This is fast, because all of your prompt tokens can be processed together in parallel. Then it writes the answer, and writing is where the time goes, because generation is that strict one-token-at-a-time loop from earlier: the model cannot produce the tenth token of its reply until it has produced the ninth. Reading is a group activity; writing is a queue. That asymmetry is why a long answer feels slower to stream out than a long prompt felt to send in, and why cutting the length of what you generate usually saves more time than cutting what you send.

5.6.2 The KV cache: don’t redo the reading

Now recall a detail that should bother you. To produce each new token, the model feeds the entire text so far, prompt plus everything generated, back through all its layers. Taken literally, that means when writing the hundredth token it would reprocess the previous ninety-nine from scratch, and the token after that would reprocess a hundred, and so on. A staggering amount of repeated work.

The fix is a piece of bookkeeping called the KV cache. As the model processes each token, the intermediate results that attention needs from it, its “keys” and “values,” are computed once and then stored. When the next token is generated, the model reuses those saved results for all the earlier tokens and only does fresh work for the new one. The cache turns an impossibly repetitive process into a practical one, and it is the single biggest reason models can generate long outputs at usable speed [17].

Figure 5.11: Why the KV cache makes generation practical. On the first pass the model computes each token’s key and value once and stores them. As it then generates, it reuses those cached keys and values for every earlier token and computes only the new one, so each step does a small amount of work instead of reprocessing the entire history. This is the single biggest reason long outputs generate at usable speed.

5.6.3 Why long prompts quietly cost you

The cache is a huge win, but it also explains why context is not free. That stored cache grows with every token in the window, so a longer prompt takes more memory to hold and makes every subsequent generated token do a little more work, since attention now has more history to look across. This is the mechanical reason behind a fact you will feel in the bill: providers charge for the input tokens on every single turn, not just the output. In a long agent conversation, where the entire growing transcript is resent each turn, those input tokens are often the dominant cost, quietly accumulating turn after turn even when each individual reply is short. An agent that stuffs everything it might need into a giant prompt is not just slower; it is more expensive on every step it takes afterward. Managing what lives in the context window is therefore not housekeeping. It is one of the main levers on both latency and cost, which is why later chapters return to it again and again.

5.6.4 Patterns that make models cheaper

Because these costs are so central, a whole toolkit of efficiency techniques has grown up around them, and it helps to recognize the names even before the deployment chapter uses them in earnest.

  • Mixture of experts. Instead of running every one of a model’s parameters on every token, the network is split into many “expert” sub-networks and a router sends each token to just a few of them. The model can hold a huge number of parameters, and so a lot of knowledge, while only paying to run a fraction of them per token [18]. It is how several of the largest models stay affordable to serve.
  • Quantization. The model’s numbers are stored at lower precision, for instance eight or four bits instead of sixteen, which shrinks its memory footprint and speeds it up, usually with only a small loss in quality. This is what lets sizable models run on modest hardware.
  • Distillation. A large, capable “teacher” model is used to train a smaller “student” that mimics it, yielding a compact model that keeps much of the teacher’s skill at a fraction of the cost. Many of the small, fast models you will reach for are distilled.
  • Speculative decoding. A small, cheap model drafts several tokens ahead and the large model checks them in one parallel pass, accepting the run when it agrees, which produces the same output faster [19].
  • Prompt caching. When many requests share a long, unchanging prefix, such as a big system prompt or a fixed document, the provider can cache the processed form of that prefix and skip recomputing it each time, cutting both latency and the input bill for the repeated part [20].

You do not need to implement any of these to build an agent, and most are handled for you by the model provider or serving stack. What matters here is the mental model they share: every one is a way of buying back the time or money that the token-by-token machine would otherwise charge you. Keep that framing and the deployment chapter will feel like applying tools you already understand rather than learning a new subject.

We have now walked the whole machine, from a word entering as a token to an answer leaving as one, through the training that shaped it and the economics of running it. It is time to collect the consequences, because the payoff of all this is a short, practical list of things the machine’s design forces to be true.

5.7 What this means for building agents

We opened this chapter with a promise: that a handful of slogans about language models would stop being things to memorize and become things you could see. This is where it pays off. Every practical rule below is not advice we are asking you to trust. It is a direct consequence of a part of the machine we just assembled, and you can trace each one back to its cause.

  • Context is limited, and it is the only memory there is. The model reads a fixed-size window and remembers nothing between calls (Section 5.4.4). So an agent’s real skill is deciding what to put in front of the model on each turn. This is why memory systems and retrieval exist, and why “just give it more context” eventually stops working.
  • Generation is probabilistic, by design. A reply is sampled token by token from a distribution (Section 5.4.2), so the same prompt can give different answers. Turn the temperature down when you need reliable, structured output; do not expect bit-for-bit repeatability; and test your agent as the statistical system it is, over many runs rather than one.
  • Training shapes behavior, including its blind spots. A model’s manners, refusals, confidence, and honesty are fingerprints of how it was raised (Section 5.5). Alignment optimizes for answers people like, which is not the same as answers that are true, so plan for a fluent, agreeable engine that will sometimes be confidently wrong.
  • Tool calling is learned behavior plus orchestration. The model was trained to emit structured requests for tools (Section 5.5.5), but your code does the calling and feeds the result back. Treat tool use as a handshake you design, not a capability you can assume works perfectly.
  • Exact symbol work belongs in tools, not the model. Because the model reasons over tokens, not letters or digits (Section 5.2.3), counting, spelling, and precise arithmetic are exactly the tasks to hand to a calculator or a few lines of code rather than trust to the model’s intuition.
  • Long prompts cost money and time on every turn. Input tokens are billed each turn and inflate the KV cache (Section 5.6.3), so in a long conversation the growing transcript, not the short replies, is often the dominant cost. Fast, affordable agents come from careful context management, not from stuffing the window.
  • Model choice is an engineering decision. Reasoning-trained, quantized, distilled, and mixture-of-experts models trade accuracy, latency, and price differently (Section 5.6.4). Picking a model is a trade-off to measure against your task, not a search for the single “best” one.

Notice that none of these are really about the model in isolation. Each one is a statement about the system you build around it: what you feed it, what you check, what you hand to a tool, what you store outside the window, and which model you point at which job. That is the quiet thesis of this whole book, arriving early: an agent is not a language model. It is everything you wrap around a language model to turn a brilliant, unreliable next-token machine into something you can depend on. That is the new colleague we met on the first page, finally seen clearly: brilliant, fast, widely read, and unreliable in ways you can now name, predict, and design around.

Which raises the obvious next question. We can see what the machine is. But how far can we actually trust it to reason, to hold an argument together, to plan, to know the edge of its own knowledge? That is exactly where the next chapter goes. Having opened the engine, we are finally ready to ask what it can and cannot think.

5.8 Summary

  • A language model does not read text; it reads tokens, subword chunks from a fixed vocabulary that each map to a number. This is why it struggles to count letters, spell precisely, or do exact arithmetic, and why those jobs belong in tools.
  • The transformer turns tokens into points in a meaning-space, uses attention to let each word absorb its relevant context, and stacks that operation deep so understanding is assembled in layers.
  • Text is generated autoregressively: at each step the model outputs a probability for every token, the system samples one, appends it, and feeds the longer text back in. Sampling is why outputs vary, and temperature is the knob that controls it.
  • The model’s only memory is its context window. It remembers nothing between calls; the illusion of memory comes from resending the transcript, which is why context management is central to agent design.
  • Models are raised in stages: pretraining builds broad competence, supervised fine-tuning teaches it to answer, alignment (RLHF and its successors) shapes it toward preferred responses, and reasoning training with verifiable rewards makes it better at checkable problems. Training, not magic, explains a model’s habits.
  • Aligning to human preference optimizes for answers people like, not answers that are true, which is a root cause of confident, agreeable errors.
  • The KV cache makes token-by-token generation practical but grows with context, so long prompts cost more memory, money, and latency on every turn. Mixture-of-experts, quantization, distillation, speculative decoding, and prompt caching all buy that cost back in different ways.
  • Every practical rule for building agents, limited context, probabilistic output, tools for exact work, careful context management, model choice as a trade-off, follows directly from how the machine is built.

5.9 Exercises

  1. See the tokens. Take a paragraph of your own writing, a line of code, and a long product name or URL, and estimate how many tokens each becomes using the rule of thumb of about four characters per token. Which fragments most, and what does that predict about where the model will be most and least reliable?
  2. Explain the party trick. In your own words, explain to a non-technical colleague why a capable model can miscount the letters in “strawberry” yet write a flawless essay about strawberries. Which part of the machine is responsible?
  3. Turn the knob. Describe two features of an agent you would build, one where you would set temperature near zero and one where you would set it high, and justify each choice in terms of how sampling works.
  4. Follow the money. A support agent holds a twenty-turn conversation, resending the full transcript each turn. Explain, using the KV cache and per-turn input billing, why the cost of turn twenty is much higher than the cost of turn one, even if both replies are the same length. Name one design change that would reduce it.
  5. Match the stage to the behavior. For each of these, name the training stage most responsible and say why: (a) the model knows the boiling point of water, (b) it answers a question instead of continuing it, (c) it politely refuses a harmful request, (d) it works step by step through a competition math problem and checks itself.
  6. Model as a decision. Pick a task you would want an agent to do and argue for a specific trade-off along one axis, a smaller and cheaper model versus a larger and more accurate one, or a reasoning model versus a fast one, framing it as an engineering choice with a measurable consequence rather than a search for the “best” model.