AI Engineering

1Product first, model second

Classic ML engineering started from data to train YOUR model. AI engineering reverses the order: you start from an existing model, trained by someone else, and the whole trade consists of adapting, evaluating and serving it.

This reversal has three consequences that structure the book:

• you no longer touch the model's weights (except late finetuning);
• models are so large that inference optimization becomes a subject of its own;
• outputs are open-ended (free text, not a label), so evaluation becomes the hardest problem.

The method's promise fits in one sentence from the preface: "Tools become outdated quickly, but fundamentals should last longer." Huyen knows what she's talking about: she taught TensorFlow in 2017, and draws the lesson on herself.

2English is the models' native language, and it costs you

In Common Crawl, the web corpus that feeds the models, English weighs 45.88%; Bengali, spoken by 272 million people, 0.093% (ch. 2). That imbalance costs you two ways:

• quality drops: GPT-4 solves three times more math problems in English than in Armenian or Farsi, and zero out of six in Burmese;
• the bill climbs: the same content, tokenized, is more than four times longer in Hindi than in English, ten times longer in Burmese. On an API billed per token, a rare language costs ten times more for a worse service.

For a French-speaking developer the lesson is concrete: test your prompts in French, but know that the model "thinks" in a language where your market is a minority.

3Hallucinations are not bugs, they are the product

A language model generates the probable, not the true: "Anything with a non-zero probability, no matter how farfetched or wrong, can be generated by AI" (ch. 2). Huyen devotes to sampling (temperature, top-p) the section she says she was most excited to write: that's where inconsistency, hallucinations and spectacular gains are all born. The Best-of-N trick is the striking example:

# generate N answers (temperature > 0 → they differ),
# then a small evaluator keeps the best one
answers = [llm(prompt) for _ in range(5)]
best = max(answers, key=score)
# quality ≈ that of a model 30× larger (OpenAI, 2021)

On the origin of hallucinations, the book lays out two complementary hypotheses:

• self-delusion: the model treats its own tokens as facts, and snowballs on its first mistake;
• knowledge mismatch: during supervised finetuning, annotators rely on knowledge the model doesn't have. It is literally taught to assert without knowing.

Neither is solved.

4Evaluation is the bottleneck, not the model

When Huyen asks teams how they evaluate their applications, many answer that they eyeball the results (ch. 3). Meanwhile, public benchmarks reassure you for the wrong reasons:

• OpenAI found 13 benchmarks with at least 40% of their data already inside GPT-3's training set;
• a Stanford PhD student got near-perfect scores from a one-million-parameter model by training it only on the test data. His satirical paper: "Pretraining on the Test Set Is All You Need".

The book's answer: build YOUR own evaluation pipeline, anchored to business criteria defined before you write code. And if you reach for an AI judge, know its documented biases: Claude-v1 prefers itself by a 25% margin, and judges favor longer answers and the first position. The Greg Brockman line Huyen quotes: "Evals are surprisingly often all you need."

5Context is the new feature engineering

A model fails first when it lacks information: Huyen calls context construction "feature engineering for foundation models" (ch. 10). The RAG pattern (Retrieval-Augmented Generation: fetch relevant documents from an external knowledge base and inject them into the prompt before the model answers) is the standard response to that gap, and the book gives rare calibrated benchmarks:

• Start with term-based retrieval (BM25), which Perplexity's CEO describes as genuinely hard to improve upon;
• embeddings come after, and their bill is real: it is not uncommon for a vector database to cost one fifth, even half, of the model API bill (ch. 6);
• the most counterintuitive call-out: if your knowledge base fits under ~200,000 tokens (about 500 pages), put everything in the context and forget RAG (Anthropic note, ch. 6).

The RAG pipeline, link by link, fits in a few lines:

question = "what is your return policy?"

# 1. fetch the 3 closest documents from the external base
docs = vector_db.search(embed(question), top_k=3)

# 2. PASTE them into the prompt, before the question
prompt = f"Answer using ONLY:\n{docs}\n\n{question}"

# 3. the model answers from supplied facts, not its fuzzy memory
answer = llm(prompt)

if contains_personal_data(input):
    block()                         # INPUT filter (PII, secrets, prompt injection)

answer = llm(prompt)

if judged_toxic(answer):
    answer = fallback_message()     # OUTPUT filter (a 2nd moderator model)

Three things I didn't know before reading it

• Grok, X's model, was caught quoting OpenAI's usage policies: the web is so full of ChatGPT outputs that new models train on them without meaning to (ch. 2).
• Repeating 0.1% of the training data 100 times is enough to drop an 800M-parameter model to the level of a 400M one (Anthropic study, ch. 8).
• Asking ChatGPT to repeat the word "poem" forever made it drift until it spat out chunks of its training data (ch. 5).

Still relevant in 2026?

The fundamentals, yes: evaluation first, context before finetuning, compound errors in agents, progressive architecture. The proper nouns turn fast (details in my take): take the principles, re-check the tools. No second edition announced; the author's official GitHub repo keeps the resources up to date.

Who is it for?

For going further

The whole learning section of this site is built by coding with AI, the book's official repo (chiphuyen/aie-book) maintains per-chapter resources, and in this library, Designing Data-Intensive Applications plays the same role for distributed systems.