Beyond Code

1Writing code costs almost nothing, being wrong costs as much as ever

First family: verification, because that is where the cost now sits. The course builds on an experiment run in mid-2025 by METR, a research organization (section 1). Sixteen seasoned open-source developers worked through 246 real tasks, each randomly assigned to be done with AI or without.

Beforehand, they predicted a 24% gain. Afterwards, they felt 20% faster. The measurement said the opposite: 19% slower. Between the feeling (a 20% gain) and reality (a 19% loss) lies a 39-point gap. To the course, that gap is worse than the slowdown: it shows that nobody was measuring.

The reason is simple. Generating code is visible, the lines scroll by. Verifying is not: you have to walk through everything the code touches. "Generation is a stream. Verification is a landscape."

2A mechanical gate answers, a review gives an opinion

If verification is the cost, it had better verify something. "Is this code good?" has no defined answer, the course says (section 6): each reviewer judges by their own idea of quality. Frédéric Brochet, at the University of Bordeaux, served 54 oenology students a white wine dyed red. They described it with red-wine vocabulary. So the course sets two kinds of check against each other:

• Advisory review (review, for short): a person or an agent reads the code and gives an opinion. Two competent reviewers can disagree.
• Mechanical gate: a test or a type checker (the tool that verifies every value has the expected type) holds the code against a specification. The answer is yes or no, the same on every run.

The course's example is Knight Capital, a trading firm, in 2012. An update deployed by hand missed one server out of 8. In 45 minutes, the firm lost 460 million dollars. A mechanical gate would have caught the gap before the market opened: a script comparing the 8 servers was enough. Hence the line: "Advisory review asks whether someone did the work. Mechanical verification asks whether the work succeeded."

The course adds an observation: replacing human reviewers with AI reviewers fixes nothing. Its AI reviewers, with no ego and no career, still nitpicked over details that did not matter. So the flaw lies in review itself, not in the people.

3The tests are the product, the code is disposable

A mechanical gate needs a reference: the contract. For the course, a contract describes what a component must do, written as tests you run (section 6). Documentation can be read without being obeyed, a test cannot. The example is a payment function with an idempotency key. That key is a unique identifier per request, so a request sent twice can be recognized:

• A valid card and an amount produce a successful charge.
• The same key sent twice returns the first charge, without creating a second one.
• An expired card returns the error "card_expired", and nothing is charged.

The course draws a rule for incidents from this, starting from a case. A customer is charged twice, because a network timeout made the client resend the request. The reflex would be to debug the faulty code. The course does the opposite. First you write a test that reproduces the failure: the same key must return the existing charge within 50 milliseconds. Then the agent rewrites the code from scratch. The new version must pass the old tests plus this one, or it gets thrown away. The course puts it with a farmer's image: the code is cattle, not pets. You replace one head of cattle without mourning it.

// Made-up example, not from the book: the test written after the incident
public function testSameKeyReturnsExistingCharge(): void
{
    $first  = $this->payments->charge('tok_visa', 4990, key: 'order-1042');
    $second = $this->payments->charge('tok_visa', 4990, key: 'order-1042'); // the resent request
    $this->assertSame($first->id, $second->id);            // the same charge
    $this->assertCount(1, $this->bank->transactions());    // a single debit
}

The course names the limit itself. A test only checks what someone thought to write. A contract is still a proxy, and Goodhart's law lies in wait: when a measure becomes a target, people optimize the measure instead of the goal. The answer: extend the contract after every incident, so it gets closer to the real need.

4A well-written but off-topic tutorial misleads more than random text

Second family: organizing the work, starting with what you feed the agent. What it has in front of it is called the context: your request, the pasted files, the conversation. The course's example will ring a bell for any web developer (section 5). A PostgreSQL connection times out, and the developer pastes the client library's whole README.

The README is the project's overview file, clear and well written. It also covers the connection pool (connections kept open for reuse), SSL encryption and transactions, each with its own timeouts. The model mixes them up and suggests a bigger pool. Given only the connection settings and the error message, it finds the real culprit: the TCP keepalive setting, which decides when an idle connection is treated as dead.

The course generalizes from this case. A well-written but off-topic tutorial, like that README, degrades the answer more than random text of the same length. Random text looks like nothing, so the model ignores it. A tutorial on a neighboring topic shares the question's vocabulary, so the model blends the two. Hence the line: "The well-written tutorial is a trap."

The course draws three habits from this:

• One task per exchange, instead of six questions in the same message.
• From a 4,000-line file, give the 200 lines that concern the task.
• Ask neutrally, "analyze this code for potential issues" rather than "this code looks correct, right?", because the model tends to confirm the answer you hint at.

5Splitting a system is already designing it

One task per exchange assumes you know how to split a big job into tasks. The course makes that split the main act of design (section 7). In 1999, NASA loses the Mars Climate Orbiter, a 327-million-dollar probe. Lockheed Martin, which built it, delivers thrust data in pound-force-seconds. JPL (the Jet Propulsion Laboratory, NASA's lab), which flies it, expects newton-seconds, a factor of 4.45 apart. Neither team was wrong on its own side. The boundary lacked a typed contract, one that forces every exchanged value to carry its unit. "The orbiter was not lost because of bad engineering. It was lost because of bad decomposition."

To decide whether to split at all, the course asks three questions:

• Capacity: a token is a chunk of a word, the unit the model counts. If the whole specification fits in about 10,000 tokens, do not split. The course gives no source for that threshold.
• Coupling: write the contract between the two pieces. If that contract takes a paragraph to explain, the course says, the boundary is probably in the wrong place.
• Cost of being wrong: if a boundary is expensive to undo, wait until you understand the domain. That is the case for the monolith first, one application before separate services.

The course adds a landmark: look at where the data changes shape, such as the raw request a web server receives (HTTP) becoming a validated domain object.

6A single agent scored 28 out of 28, a pipeline of eleven agents scored zero

Splitting applies to agents too, and the course puts it to the test (section 2). Same model, same task. The pipeline had 50 dollars, and the single agent cost roughly as much as that pipeline's planning alone. The task: the backend of a vacation rental booking platform, made of 7 independent services. Only the way the work is organized changes, across four setups:

• A single agent: 28 out of 28, all 7 services correctly wired together.
• A hierarchy, a coordinator delegating to specialists: 18. The coordinator ended up coding everything itself.
• A swarm of 8 agents working in parallel on shared files, with no lead: 9.
• A pipeline of eleven specialized agents in a row, from diagnosis to review: 0.

The pipeline spent the 50 dollars on five planning stages and never wrote a line of code. The swarm produced 152 files, nearly two and a half times what the single agent needed. Half its agents wrote names in snake_case (customer_name), the other half in camelCase (customerName), and not one connection between services worked.

The result is striking, but it has to be read for what it is. The score out of 28 adds up seven metrics of compliance with the specification, a proxy by the course's own admission (section 11). The text names neither the model, nor the number of runs, nor who ran the experiment. The author warns that these setups do not represent the state of the art. It is one memorable run, not proof. The course also reports a fifth setup, with typed contracts and tests instead of reviews: it avoided the absurd rejections but burned its budget writing ever longer specifications. A test is not a magic wand.

7A red test coordinates better than a meeting

That leaves the case where the task is too big for one agent. Several agents must then coordinate, and the course compares two ways (section 9). The first is dialogue: everyone talks to everyone. The second takes after termites, whose mounds rise four meters high without any termite holding the plan. Each one reacts to the chemical traces the others leave. In 1959, the biologist Pierre-Paul Grassé named this stigmergy: coordination through the traces that work leaves in a shared environment.

The course puts numbers on the difference. With 8 agents in dialogue, each talks to the 7 others: 8 × 7 = 56, and each channel is counted twice, so 28 channels. With 50 people, that becomes 1,225. Through traces, each agent does one read and one write, however many there are.

In software, the traces are the repository, the task board and continuous integration (CI), which reruns the tests on every push and shows green or red. The course's scenario: a developer changes the payment module, and a test of the notification service turns red. A second developer sees the red, reads the failure and fixes the parser. Not a single message passed between them. "A description of a bug is a lossy compression of the bug. The test failure is the bug."

The course does not sell this as a miracle cure: two agents can react to the same trace, or edit the same file in incompatible ways. It keeps a rule with three cases (section 10):

• The task fits in one agent's context: a single agent.
• It splits into loosely coupled pieces: coordination through traces, tests and contracts.
• It calls for an ambiguous trade-off (a compromise across services, a business priority): hierarchy, and only there.