Serverless Architecture Patterns

1An eventually consistent system works like a busy coffee shop

First family: facts, starting with what they replace. In a busy coffee shop in the morning, one person taking orders, making the drinks and taking payment would send the line out the door. So the shop splits the work into stages. You pay at the till, you get a receipt, then you wait for your drink at the counter.

The receipt is the heart of the example: it states the fact that you paid (lesson 29). You know your drink will come, and the shop knows it must serve it or refund you. Nobody locked the till while the coffee was being made. The book draws its rule from this: "Facts are the lifeblood of an eventually consistent system" (lesson 30).

Each stage therefore performs a single atomic action, meaning all or nothing, then publishes a fact once it is done. The next stage reacts to that fact. The whole system moves forward through a chain of short actions.

The coffee shop has its failures too, and the book tells them. The crew loses your coffee and puts a new one at the head of the queue. Then they find the original, whose label was smudged: you now have two coffees. Gilbert's conclusion: "duplicate output is better than no output" (lesson 30). The next stage simply has to cope with duplicates.

The same shop doesn't guarantee order either. A simple drink comes out before a complicated one, and the customer sorts things out at the end. Programs that read events must do the same, because arrival order is never guaranteed (lesson 30). Both failures, duplicates and disorder, come back in idea 5.

2Gilbert's event sourcing keeps history outside the service

Those facts still have to be kept somewhere. Take a bank account. The book compares three versions of it (lesson 34):

• Without event sourcing: one row per account holding the balance, overwritten on every operation. The book admits it: "We are losing information."
• Traditional Event Sourcing: one row per operation, credit or debit. Nothing is lost, but reading the balance means rereading every operation on every request.
• System-wide event sourcing: the service keeps only the current balance, and every change is sent to the other services as a fact.

The third model is the book's own. Its pivot is CDC (Change Data Capture). CDC is the stream of changes a database publishes on every write. A small function in the service reads that stream and forwards each change to the event hub. The hub delivers events to every interested service. The book also calls it "the bus".

So the service itself keeps no history. The book hands that job to an event lake. It is file storage, one per subsystem, that receives every event and keeps it forever. For Gilbert, this lake "acts as the system-wide transaction log" (lesson 34).

The lake also makes replay possible. A bug made a service miss some events? You resend the missing period. A new service shows up? You feed it a few weeks or months of events. The book is firm: "we do not republish events", or every service would receive them again (lesson 35). They go to the one service that needs them.

3A service writes to one place only, then CDC publishes

Keeping the facts isn't enough: you also have to publish them without losing any. Picture a service that saves an order in its database, then sends it to the event hub. The program crashes in between. The database has the order, but nobody downstream will ever hear about it.

Hence the book's rule: "we cannot reliably save data to more than one data store at a time" (lesson 36). The reason lies in distributed transactions. The protocol that commits two writes together, two-phase commit, doesn't scale. So you pick a single place to write first:

• Database-first: the service writes to its database, and that database's CDC publishes the event. The book keeps it for services where the user must read back right away what they just wrote.
• Stream-first: the service publishes the event, and downstream services store it. The book keeps it for services with no direct user, or for a database without CDC.

The database-first case has a bonus (lesson 61). The function reading the CDC first publishes every raw change, with the row before and after. That stream is an audit log: what changed, when, and by whom. Only then does a rule pick out the real business event, for instance a status that turns to "submitted".

4Watertight bulkheads stop a failure from spreading between services

Publishing through the event hub makes each service independent of whoever reads it. The book carries the idea over to failures. A ship is divided into watertight compartments: a leak floods one, not the whole hull. Those partitions are called bulkheads, and the book places two of them, one on each side of the service:

• Outbound bulkhead: the event hub. The service publishes and moves on, without knowing who reads. A downstream failure therefore doesn't block it (lesson 31).
• Inbound bulkhead: the materialized view, a local copy of another service's data, updated by its events. If the source service goes down, the copy still answers with the latest data received (lesson 53).

The counter-example is the direct call. A service that queries another one on every request goes down with it. The book is blunt: "services that have direct dependencies on other services are less resilient" (lesson 53). Its frontend-facing services (BFF, backend for frontend), the ones that serve one specific frontend, "do not wrap other services". They own everything they need to keep running during an outage (lesson 65).

That comfort has a price, which the book names back in the coffee shop: transparency. During an upstream outage, the local copy grows stale. So users must be able to see how old the data they read may be (lesson 29).

5Inverse locking drops stale events instead of retrying

Second family: keeping the system correct when things go wrong, starting with duplicates. The book starts from a plain statement: "exactly-once delivery of messages is unrealistic" (lesson 55). So the same event will sometimes arrive twice. Receiving it twice must change nothing: that is idempotence, like an elevator button you press five times.

The book's example shows where duplicates come from (lesson 41). A program reads events in batches of 100, and event number 60 crashes processing on every attempt. The whole batch then goes round and round, and "the first 59 events will update the target over and over again".

Disorder comes on top of duplicates, and the book handles both with one move. It starts from classic optimistic locking. The row keeps a version number, and if someone wrote in the meantime, your write fails and you retry. Inverse optimistic locking does the opposite: "we drop incoming events if the timestamp is older or equal to the current oplock value in the datastore" (lesson 56).

In practice, each row keeps, in an oplock column (short for optimistic lock), the timestamp of the last event applied. An equal timestamp means a duplicate, an older one means a late event: either way, you ignore it. The book writes it for DynamoDB. In SQL, the same rule boils down to one condition. Here is my translation, not the book's:

INSERT INTO orders (id, status, oplock)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE
    SET status = EXCLUDED.status, oplock = EXCLUDED.oplock
    WHERE orders.oplock < EXCLUDED.oplock;
-- 0 rows touched: duplicate or late event, ignored

ON CONFLICT turns the insert into an update when the order already exists. EXCLUDED is the row you tried to insert, in other words the incoming event. The book's whole rule sits in the final WHERE: you only write if the event is newer.

One last trap remains. A single row may combine several event types, say the order and the driver assignment. With a single lock, a recent order event moves the timestamp forward, and the older assignment is wrongly dropped, fields and all. The fix is one lock per event type, which the book calls an Inverse Oplock Join (lesson 56).

6A control service orchestrates through events and never calls anyone

With duplicates handled, what's left is a process that fails halfway. In the book's food delivery, the order goes to the restaurant, a driver is picked, and every step has its error paths. At first, each service reacts to the others' events. That is choreography: dancers with no leader, adjusting to one another.

As error paths multiply, the logic scatters across every service. The book then finds choreography "harder to understand, trace, debug, test, and maintain" (lesson 83). It gathers that logic into a control service that orchestrates the process. Its definition: "control services react to events by producing more events" (lesson 84).

This conductor never calls another service. It emits an entry event, "task initiated", and waits for the exit event, "task completed" (lesson 87). A participant's contract comes down to that pair of events.

The same contract serves the saga, a long process split into atomic actions, an idea the book traces back to 1987. If a step fails with no way forward, you undo the earlier ones through compensations. Yet, the book warns, "a compensating transaction is not akin to a rollback" (lesson 88). It works like accounting: nothing is erased, mistakes are fixed with adjusting entries.

To know when to compensate, the book tells two failures apart (lesson 89):

• Fault: an unexpected, technical error. You fix it, then resubmit the event to move forward.
• Abort: an expected business condition with no way forward. It triggers the compensations upstream.

The book's example shows an abort, over two steps. The request is submitted, step 1 succeeds, then step 2 aborts. The control service then emits "step 1 cancelled", the compensation for step 1, then "request cancelled". Two rules are enough: any abort cancels the request, and an abort of step 2 also cancels step 1.

7A snapshot becomes an event, and for a balance order stops mattering

Compensations repair a process. What remains is computing a state, such as a balance, when events arrive out of order. The book admits that event sourcing "has a bad reputation for being overly complex for most use cases" (lesson 90). The culprit: you have to rebuild state by replaying events.

The usual fix is the snapshot: a picture of the state at one moment, so you only replay what came after. Gilbert moves it. A control service collects deposits and withdrawals, computes the balance, and publishes a "balance" event (lesson 91). The next calculation starts from the latest published balance and adds whatever arrived since.

This calculation copes with disorder thanks to ACID 2.0, four properties borrowed from Pat Helland: associative, commutative, idempotent, distributed. Commutative means order doesn't change the result, like 2 + 3 and 3 + 2. The book's example comes with numbers. In both orders below, each balance in parentheses is a snapshot: the total published after a group of events. The two orders group the events differently:

• First order: deposits of 100 and 30 (balance 130), a withdrawal of 50 and a deposit of 70 (balance 150), a deposit of 20, final balance 170.
• Second order: a deposit of 100 and a withdrawal of 50 (balance 50), deposits of 30 and 20 (balance 100), a deposit of 70, final balance 170.

The book concludes: "the events arrived in different orders and the ending balance of both scenarios is 170" (lesson 91).

The book admits the limit of its example. It follows the happy path and ignores the case of an older event arriving after a snapshot. Gilbert leaves that case to the reader "as a thought exercise" (lesson 91). Yet that is precisely the hard case.

8Across two regions, replicate the result, not the work

Duplicates and disorder have one last playground: several regions, that is, several of the provider's data centers in different parts of the world. Gilbert lived through the 2017 S3 outage with a single-region system, with nothing he could do (lesson 95). In his experience, a major regional outage happens about every two to three years.

To run in two regions, the temptation is to replicate the event hub: each region receives every event and redoes all the work. Gilbert says no: "I recommend against replicating at the bus level" (lesson 102). Each region would run the same processing, pay twice, and might even get different results.

He replicates the database changes instead, already computed. In his words: "we perform the work once and replicate the output" (lesson 102). A trap follows: the region receiving the copy must not publish the event again. So the book tags each row with its region of origin. An order processed in the East region is copied to the West region. In the West, the function reading the CDC sees the row arrive, reads "East" as its origin, and publishes nothing.

Recovery after an outage relies on the idempotence from idea 5. When a region takes over, it also takes over polling the systems outside the subsystem. It then rewinds about 30 minutes to fill the gaps (lesson 105). Idempotence absorbs the duplicates this creates.

9To erase a person from an immutable lake, destroy their key

Last idea: protecting those facts, kept forever in the lake. The book opens with a true story, Code Spaces in 2014 (lesson 108). An attacker takes the company's cloud account hostage, then deletes everything, backups included, because they lived in the same account. The company goes under.

The book draws a rule from it: each subsystem gets its own cloud account, and backups live in an isolated one. Gilbert also distrusts disk encryption on its own: "Disk-level encryption only secures the data when the disk is removed from the system" (lesson 114). Yet privileged access reads the data in the clear.

So the book encrypts sensitive fields in code, with envelope encryption: a data key encrypts the field, and a master key encrypts the data key. Each event carries, in an eem field (envelope encryption metadata), the encrypted data key and the list of protected fields (lesson 115). A fresh data key is used for every save, so a stolen key opens a single object.

That leaves the GDPR right to erasure, under the EU regulation on personal data. It looks incompatible with a lake where nothing is ever modified. The book's answer is crypto-shredding: one master key per person. When a subject asks for erasure, "we just need to delete the subject-specific key" (lesson 115). Their data stays in the lake, but nobody can decrypt it anymore.