Practical Microservices

1A view counter erases what it counts

The first need of Video Tutorials is mundane: show the total number of videos watched on the home page. The CRUD reflex fits in one line of code. The videos table gets a counter column, and each view raises it by one. For anyone used to CRUD MVC, it is the natural move (p. 20).

Garofolo then asks three questions. What if creators were paid based on these counters? What if we found a mistake in the counting? What if we wanted to track a video's popularity over time? All three hit the same wall: "If all we have is current counts, we’ve lost the information we’d need to recalculate." (p. 20)

A video_views table, with one row per view, would do better. But it must exist from day one. Added later, it only fills for the future, and the earlier views are lost for good (p. 21). Even this table does not last long. Before paying a creator, you must check that a view is legitimate, a check too long for the span of a web request. You must also warn other parts of the system, on every view, that they have work to do. The table would barely cover that, and for this second need it would be a custom rig (p. 21).

The second flaw is sneakier. The book plans to upload, view, count, comment on and rename videos, and asks whether a single videos table could hold all that (p. 24). With this single table, each new feature must know who else reads and writes it (p. 21). The author sums it up: "We’re one stinking model into this project, and MVC CRUD has already coded us into a corner." (p. 21)

Garofolo does not condemn CRUD for all that. If the counter were the only use of this data, he finds it reasonable (p. 22). The problem starts when the same data serves several purposes. He comes back to it in the conclusion. For prototyping, the MVC framework remains in his eyes the fastest tool, especially if you already split the catch-all table (pp. 247-248).

His answer has two parts. First, you store the changes themselves: "we’ll model our data by storing the state transitions themselves." (p. 22) "A video was viewed" becomes a row you never modify again, and the counter is just a computation over these rows. This is the Event Sourcing announced above. Second, the pieces of the system stop coordinating by reading the same table. They exchange messages, small records that one writes and another reads later. Their shape is written in black and white, a contract: "the different parts will communicate through an explicit contract" (p. 22).

To remember: an overwritten column is a decision to forget, made without knowing it. Before creating a counter, ask whether someone will one day want to know where it comes from. The next idea shows why cutting the application into services does not fix the second flaw, the shared table.

2Putting a table behind HTTP does not make a microservice

Faced with a growing monolith, the common reflex is to extract services. The book plays this reflex out on its application. Next to videos, a users table appears, and the two models call each other in the same code. They are split into two web services, each with its own database.

The result is a bill. There are now two servers and two databases. Function calls become HTTP calls, and the SQL join has to be rewritten in code (p. 29). Worse, videos depends on users: if users goes down, videos goes down too. The videos service "now has two reasons to fail" (p. 29). The author's diagnosis: "It is in every aspect still a monolith, only it is distributed now: a distributed monolith." (p. 29)

His definition explains why the split changed nothing: "A monolith is a data model and not a deployment or code organization strategy." (p. 26) The number of servers does not matter. As long as everyone depends on the same data, the system stays in one block.

The book's criterion is therefore autonomy, which defines a microservice in his view (p. 29), and it is strict. The book speaks of a Component, a word it prefers to "service", judged too vague (p. 23). A Component answers no question: what you query is called a database, even behind HTTP (p. 30). It asks none either: "Components don’t ask questions of anything else." (p. 30) If it queried another service, it would fall with it.

One channel remains, the asynchronous message: you write it into a shared store, the Message Store, and whoever wants it reads it later. The book distinguishes two kinds (p. 30):

• Command: a request, in the imperative (PublishVideo). The Component that receives it may refuse it.
• Event: a fact in the past (VideoPublished). "Events are records of things that have happened." (p. 30)

A store that everyone reads, is that not one more shared table? No, and the difference lies in the message. A table gets rewritten, and whoever reads it depends on its shape of the day. An event has already happened: "You don’t have to like it, but you do have to deal with it." (p. 30) Each Component documents the commands it handles and the events it publishes (p. 30). That list forms its contract, and the author wants it frozen: "Message contracts are supposed to be immutable" (p. 247). The other pieces depend on this contract, never on how the Component stores its data. The team commits to communicate "through that contract rather than through each other’s internals" (p. 163).

A message name comes from the business. The book bans create, update and delete from its message names (p. 32). A visitor who signs up produces Registered, not UserCreated. The nuance matters, because an account created by an administrator is not a registration (p. 86). The author insists: "The domain event isn’t an alias for a CRUD action: it’s capturing the user’s intent." (p. 86)

Around the Component, the book sorts the whole system into five pieces (pp. 22-23):

• Application: what the user touches, the HTTP routes, request and response. All the CRUD code you already wrote is one.
• Component: it carries a business process, reads commands and publishes events.
• Message Store: the store where all messages live, at once "a durable state store as well as a transport mechanism" (p. 23).
• Aggregator: it reads events and derives data ready to display.
• View Data: that data, read-only. No decision is made from it (it is always slightly behind the events), "hence why it’s called View Data" (p. 23).

A user action goes around these pieces. The user calls an Application, which writes a message. A Component reads it, works, then writes new messages. An Aggregator turns them into View Data, which the Application reads to answer (p. 24). The author warns: "This structure is going to feel bizarre at first." (p. 24)

Autonomy pays off fast. When the book writes the registration Application, the identity Component does not exist yet. Only its contract exists, a contract.md file that describes its messages (p. 87). The author's verdict: "With a well-defined message contract you were able to build a client of a Component without having actually built the Component itself." (p. 104) Two teams can therefore move in parallel, each on its side of the contract.

To remember: before calling a web service a "microservice", ask whether it answers questions about its data. If it does, the others depend on its model, and you have distributed your monolith. It remains to know where to store these messages: the next idea opens the Message Store.

3The message store is just a PostgreSQL table

The messages of idea 2 must live somewhere, and many teams immediately think of dedicated software. Garofolo first shrinks the requirements. A stream is the ordered sequence of messages about one thing, for example all the events of user 7f3a. The store must keep append-only streams of immutable messages (p. 41). It must also stop two simultaneous writes from stepping on each other (p. 41).

The book picks Message DB, a free project from the Eventide team that installs into PostgreSQL. Everything rests on a single table, message_store.messages, and a few SQL functions you go through to read and write (p. 43). The author enjoys it: "It’s a lot simpler than you might expect a piece of messaging infrastructure to be." (p. 43) Here are four rows of this table, reduced to their useful columns. It is an example of my own making, not a capture from the book:

stream_name            type                   position  global_position
identity:command-7f3a  Register                      0               41
identity-7f3a          Registered                    0               42
viewing-c2d9           VideoViewed                   0               43
identity-7f3a          RegistrationEmailSent         1               47

Four columns carry the essentials. The type column gives the message name (Registered). The stream_name column says who it belongs to. Its name follows a convention: what comes before the first dash is the category (identity), what follows identifies the entity, the thing tracked, here a user (p. 34). Commands live in a separate category: "Note that identity and identity:command are two separate categories." (p. 88) This separation draws the Component's boundary: only it writes to identity and handles identity:command (p. 35). The position is a message's rank within its stream. The global position (global_position) is its rank in the whole table (p. 42). The first detects conflicts, the second reads messages in arrival order, as ideas 4 and 5 show.

The second requirement goes through the write function, write_message, which accepts an expected version (p. 45). "Streams have versions, and their version is equal to the maximum position value of all the messages in that stream." (p. 47) The book's example: two instances of the same Component handle at the same time the command "credit 5 dollars" on an account. Each would write a Deposited event, and the account would be credited twice (p. 47). So you write "only if the stream is still at version 3" (p. 47). By the definition above, the first write brings the stream to version 4, and the second is refused. The book's code turns this refusal into a readable conflict error (p. 47). The mechanism is the one in idea 4 of the note on Implementing DDD, CQRS and Event Sourcing, housed here in the database.

There remains the question the author says he hears often: why not Kafka? Kafka is a very common broker (a message courier). It moves large volumes of events between programs and stores them in named queues, the topics. Garofolo sees a good tool in it and gives two reasons not to use it as a store (p. 49):

• Streams are not declared in advance. Kafka expects topics planned ahead. Yet "In an event-sourced system, we can’t know what streams are needed in advance." (p. 49) With one stream per user, you would have to know every future signup.
• No expected version. Kafka cannot refuse a concurrent write. The author takes up a line from Scott Bellware: "Kafka is a buffer and not a Message Store" (p. 49).

A broker finally adds operations work and new failures. For this precise need, it offers no advantage: "message brokers offer no benefits over PostgreSQL, something we’re already using anyway." (p. 49)

To remember: before adding a broker to do Event Sourcing, look at the database you already have. A table, a naming convention and a conditional write are enough to start. Storing the facts is not enough, you must still get screens out of them: that is the next idea.

4A screen reads one row, and the row knows which message it already counted

A log of facts is perfect for writing and disastrous for displaying. The author says it bluntly: "Message stores are fantastic for writing data, but they are exceptionally bad for rendering screens to users." (p. 51) Adding up all the views each time the home page displays is not an option.

The answer is the Aggregator of idea 2. It reads each VideoViewed event and updates a table shaped for the screen. Garofolo pushes the idea far: "What if we could build our database schema in such a way that every screen only required querying for a single row?" (p. 53) His pages table therefore has one row per screen, with a JSON document. The home page's holds { "videosWatched": 42, "lastViewProcessed": 24 } (p. 53). You write events, you read pages: this is the CQRS announced at the opening (p. 52).

The danger comes with duplicates. A message can be delivered twice (idea 5 shows how), because "it is physically impossible to guarantee exactly-once delivery" (p. 57). A naive counter would then count two views for one.

The remedy is called idempotence: an operation is idempotent when doing it again changes nothing more. The author says it his way: "there are only two states that matter: it having been called zero times, and it having been called one or more times." (p. 57) Here is the update query, summarized from the book (p. 55):

UPDATE pages
SET -- counter + 1, and position of the processed message
WHERE page_name = 'home' AND
  (page_data->>'lastViewProcessed')::int < :globalPosition

The counter and the position of the last applied message change in the same statement. An already seen message has a global position lower than or equal to the one stored in the row. The WHERE condition becomes false, and the query touches nothing. The proposed exercise is telling: remove this condition, click a few times on the button that simulates a view, restart the server. The counters climb with no new view (p. 63). The book does not say why at this point. Idea 5 gives the reason. The read position is only written every 100 messages, so a restart rereads messages already counted. Then "Now imagine that instead of aggregating view counts, this were a service performing seven-figure wire transfers." (p. 63)

The SQL developer then objects that this counter duplicates information. Garofolo accepts that. What you never duplicate is the source of truth, the messages: "do not duplicate the authoritative source of truth" (p. 62). Derived data multiplies freely, and the book draws three different views from the same events: the global total, the count per video and the home page (p. 62). Classic MVC does as much: its joins are also derived data, but rebuilt on every query (p. 62).

To remember: an idempotent view stores, next to its data, the trace of the last applied message, and it writes both in the same query. It remains to know how the Aggregator receives its events. The answer is a very simple loop, the subject of the next idea.

5Subscribing to messages means rereading the table in a loop

An Aggregator must see new messages as they come. A broker pushes them to its readers. Message DB pushes nothing, and the author warns: "Hopefully polling does not offend your software design sensibilities because that’s what the solution is based on." (p. 65) Polling means asking regularly "what's new?" instead of waiting to be told.

The book builds this subscription by hand. Each reader declares handlers, one function per message type to process. The core of the loop fits in five lines, where tick does one round of reading (pp. 72-73):

while (keepGoing) {
  const messagesProcessed = await tick()
  if (messagesProcessed === 0)
    await Bluebird.delay(tickIntervalMs)
}

Bluebird is a JavaScript promise library, which the book uses everywhere. At each round, tick reads at most 100 messages of the category, after the last processed global position. It passes them one by one, in order, to the handler of their type. If the batch was empty, the loop waits 100 milliseconds before asking again (pp. 66-67).

The subscriber stores its position in the Message Store itself. Every 100 messages, it writes a Read event that holds its position, in a stream named after it (subscriberPosition- followed by its identifier). On restart, it rereads the last Read and resumes from there (pp. 66-68). The batch size, the wait and the frequency of Read are changed when the subscription is created.

The detail matters. The position is written after processing, and only from time to time. A hard stop between two writes therefore makes it reread messages already processed. The book accepts this: "we have no guarantees that we’ll only handle a given message once" (p. 67). This is at-least-once delivery: every message arrives, sometimes several times. Each handler must therefore be idempotent, like the view of idea 4. The position is only a shortcut: "recording and retrieving the read position is a performance optimization and has nothing to do with correctness" (p. 73). Without it, you would have to recheck the whole stream at each startup, perhaps a million messages (p. 73).

Finally, an error stops the whole subscription. The choice is deliberate: "We’re not going to bother trying to automatically recover from an error. If something goes wrong at this level, we want human attention on it." (p. 73)

To remember: a subscription is a loop, a position and a rule, any message can come back. With these three pieces, the Aggregator sees the facts go by. A Component, for its part, must decide, and for that it needs the exact state of an entity: that is the next idea.

6To decide, a Component recomputes the entity from its own stream

The identity Component receives a Register command. Before writing Registered, it must know whether this user is already registered. It cannot trust the View Data: it is always a step behind, and idea 2 forbids deciding on it. The author takes a heavier example, a $10 million wire to accept or not (p. 106).

The answer fits in one function of the Message Store, fetch. It reads all the events of the entity's stream, then applies them one by one to a starting object. The recipe that says how to apply each event is called a projection. Here is the identity one, shortened of its email field (p. 108):

const identityProjection = {
  $init () {
    return { id: null, isRegistered: false }
  },
  Registered (identity, registered) {
    identity.id = registered.data.userId
    identity.isRegistered = true
    return identity
  },
}

The $init key gives the starting state, an empty identity. Each other key bears the name of an event type, and its function returns the updated entity. An event with no key is ignored (p. 107). The author sums up the role of fetch: "This is the function that implements the event sourcing in our system." (p. 106) The Register handler loads the identity this way. If isRegistered is already true, the command was already processed and the Component does nothing. Otherwise, it writes Registered into this identity's stream (p. 115). The Component's idempotence therefore goes through here: the effect of an already processed message shows in the projected entity (p. 117).

A rule follows from this definition: only events enter a projection. "Commands are not state. They are requests to change state, so we will never use a command type in a projection." (p. 107) A refused command changed nothing. Counting it would distort the entity.

The book clears up in passing two words that sow confusion. To project, for a Component, is to recompute the entity at each command, in order to decide. An Aggregator does the same operation continuously, to feed a screen. Hence the formula: "Components project; Aggregators aggregate." (p. 116) Replaying also has two meanings. The first is this projection redone at each command. It triggers no side effect, because a projection only returns an object (p. 116). The second concerns a Component or an Aggregator written after the fact. Its subscription starts from the very first message of the Message Store and climbs up to the present (p. 116). This is what happens to the identity Component, written when the Application was already sending it commands (p. 116). This catch-up processes messages, with their effects. It must not redo them, for example rewrite Registered for an identity already registered. The answer "starts with “idem” and ends with “potent.”" (p. 117), in other words idempotent.

The author admits a limit. A read returns at most 1,000 messages by default, so fetch does not project a longer stream (p. 110). He accepts it: "It is very unlikely that any particular stream will ever have more than that number, so we’ll accept this limitation" (p. 110), with monitoring to know when to change. The known remedy is the snapshot: from time to time you store the computed state, then replay only what follows. The book describes it without implementing it (p. 114).

To remember: a Component believes only its own stream, reread at each command, never a view. Computing state from events is detailed in idea 8 of the Implementing DDD note. The principle quickly meets a concrete case, the duplicate email address, which the next idea unfolds.

7A duplicate email address is a question for the business, not a catastrophe

With registration, the book walks into its own trap, and says so. View Data is eventually consistent: it ends up reflecting the events, with a few moments of delay. Yet the signup form must check that the address is not already taken. For that, the Application reads the user_credentials table, View Data filled by an Aggregator (pp. 95-96). The author admits: "Keep in mind that this check for a duplicate email used eventually consistent View Data." (p. 96) It is "a direct contradiction of what we said" in the rule set in idea 2 (p. 99).

The book plays out the resulting race, in eight steps (pp. 100-101). It is the worried reader who tells it, and the author judges it right: "Those points are completely accurate." (p. 101) Two visitors send the address tricksie@example.com at the same time. Both pass the check, because the table does not yet contain this address. The Component writes two Registered events. The Aggregator inserts the first one's row, then hits a second Registered with the same address. Two identities now share one address, exactly what the check was supposed to prevent. The worried reader concludes that we will not recover from this (p. 101).

How long does the danger last? A window "on the order of milliseconds" (p. 101), and the author finds it unlikely that two requests fall into it (p. 101). The cautious reader objects, and the author words the objection himself with a line from Nassim Nicholas Taleb: facing a probability of ruin, you end up ruined (p. 101). His answer: "The thing is, getting these two events written isn’t ruin." (p. 101)

The repair is simple, because the duplicate has its own user identifier, which already tells the two identities apart. The book sketches it without coding it: you could add a needs_to_change_email column to user_credentials, set to false for the first registrant. The Aggregator would write the second one's row with true, and a screen would invite this user to change address (pp. 101-102).

If uniqueness were truly non-negotiable, it would be a fundamental property of identities: two of them would never share an address. It would then be up to the Component that owns identities to guarantee it (p. 102). It would keep its own list of taken addresses, off limits to the others, and reject the command with a RegistrationRejected event (p. 102). For Video Tutorials, the business team ruled: address uniqueness is not a fundamental property of identities (p. 102). The lesson stays whole, because the discussion "does show why we don’t want to rely on View Data to make decisions in our systems" (p. 102).

The book gives a method to settle such cases, four questions to ask (p. 100):

• How likely is what we want to prevent?
• How bad is it if it happens?
• Who owns the constraint, the Application or the Component that owns the data?
• What does "correct" mean here?

For the last one, the author sends you to the business team: "Work with your business team. That’s a good thing to do anyway." (p. 103)

To remember: a uniqueness constraint on eventually consistent data is not a bug to hide. It is a risk to size and then decide with the business, and a simple repair is often enough. The same reflex holds, more seriously, when the effect leaves the system: that is the next idea.

8An email that has left does not come back: zero sends or two, the business chooses

In a Rails project, the author saw the welcome email hooked to the creation of the user row (p. 133). Why would writing a row trigger a send? The book moves the sending into its own Component, send-email. It receives a Send command and answers with a Sent or Failed event (pp. 134-135).

First rule: the caller chooses the email's identifier. "For idempotence to work, the client of an idempotent component must supply the ID." (p. 135) The identity Component computes it from the user's address (p. 148). A command sent again therefore carries the same identifier. send-email then rereads that email's stream, sendEmail- followed by the identifier, and finds Sent there if it already left (pp. 140-141).

The real problem lies elsewhere. SMTP, the email protocol, does not always say whether the send succeeded. With a third-party provider, the request can fail while the email has already left, and a new attempt will send it twice (p. 135). In the book, "sending" means writing a file, then the success event. A stop between the two makes the file be rewritten at restart (p. 136).

For Garofolo, this choice does not belong to the developers. "Which failure case is worse? Not sending an email, or sending it more than once?" (p. 136) Developers know the technical details, the business team knows the effect of each case (p. 136). For registration, the decision is written in the Component's contract: "sending our emails more than once is preferable to not sending them at all." (p. 136)

The same question gives the opposite answer for a bank card, where a double charge is unacceptable. The book would then write the intent before acting: "we’d want to write that we’re attempting to charge the card before charging it, so that we’d never attempt to charge it twice." (p. 153) A forgotten charge is then fixed by an admin tool or by hand (p. 153). The author stays lucid: "There is no way to guarantee that our system will agree with an external system about the state of the world" (p. 153). Keeping a trace of each step at least helps debugging.

It remains to know who triggers the send. The book lays out the path in six steps (p. 146):

1. identity sees its own Registered event go by.
2. It writes a Send command into send-email's command stream.
3. send-email reads the command.
4. It sends the email and writes Sent.
5. identity sees Sent go by.
6. It writes RegistrationEmailSent into the identity's stream.

Step 5 calls for sorting. identity subscribes to send-email's events (category sendEmail). Yet other Components can also ask for emails, and identity wants only its own. Metadata is the service information attached to a message. identity slips in a key originStreamName, the name of its own stream. send-email copies this key into the events it writes. identity's subscription filters on it and hands it only the events whose origin is in the identity category (pp. 142-151).

This pattern is called orchestration: one Component writes an explicit command to another. The book places in this family two names you often meet, the process manager and the saga, two variants of orchestration (p. 154). Choreography would be the opposite: send-email would watch Registered and know on its own what to do, like dancers who each know their part (p. 154). Garofolo picks orchestration: "We’ll favor orchestration because it makes processes more explicit." (p. 154) You read in a single place that a registration sends an email.

To remember: facing an effect you cannot take back, ask the book's question, "zero times or twice?" Let the business decide, then write the answer into the contract. The send setup, a command, a Component and a completion event, has another virtue: it makes a whole infrastructure brick unnecessary.

9Background tasks do not need a job queue

In a classic web application, any long work goes to a background job queue. This tool runs the work outside the request. The book cites DelayedJob, which relies on the database, and node-resque, which needs Redis, an in-memory database (p. 158). The author opens on an image: ordering a pizza by phone and staying on the line until delivery. Some work has no business running within the span of a web request. Networks go down from time to time, and "If you haven’t accounted for that in your system, these external failures will become internal failures." (p. 157)

Garofolo then defines a background job at its simplest: asynchronous work that probably reports its progress along the way (p. 158). A command that starts the work, events that report on it: this definition rings a bell. The author cites Greg Young: "What is a queue but a degenerate form of a stream?" (p. 158) Do you need Kafka or RabbitMQ, another broker, for background jobs? His answer is "no", then "seriously, no", because "you’ve already built every construct you need to perform long-running processes outside of the request/response cycle." (p. 159)

The textbook case is a video upload, which must be transcoded, meaning converted into the formats browsers play. The Application writes a PublishVideo command. A video-publishing Component reads it, transcodes, then writes VideoPublished with the address of the converted file, or VideoPublishingFailed with the reason (pp. 159-161). The video is officially published only when VideoPublished is written (p. 160). The Component protects itself from duplicates like the email one, since its projection remembers that a publication was already attempted (p. 167). The author concludes: "publishing a video isn’t really any different from sending an email." (p. 167)

The question of idea 8 comes back, with another answer. A stop between the end of transcoding and the writing of VideoPublished will make the work be redone. This time, the business accepts the duplicate: "Transcoding videos is cheap, and we absolutely cannot accept failing to transcode the videos" (p. 171). The redone work writes to the same place, so no orphan file lingers (p. 171).

One nuance limits the demonstration: the book simulates the transcoding (pp. 160, 168). You never see a real job of several minutes. The thesis, though, is clear: "Everything already is a “background” job." (p. 163)

To remember: before installing a job queue, check whether your messages already do the work. A command, a subscribed Component and a completion event are enough. What asynchrony simplifies on the server side, it makes you pay for on the screen: that is the next idea.

10An asynchronous screen answers before it knows the answer

The book does not hide that Components cost more to build than a monolith. Its image is the watertight doors of a ship: they take more metal, but a leak stays in one compartment. The author puts numbers on it, as an image and not a measurement. Building the monolith would cost ten points of complexity, the Component system fifteen. But that total is not what exhausts you. What counts is what you must keep in mind while you work. With Components, "you only have to hold two points of complexity in your mind at any given time." (p. 173) The message contract closes the doors between the parts. With the monolith, you need eight out of ten, because nothing separates one part from another (p. 173). The screen is where you pay the difference.

The example is a creator who wants to name their video, called "Untitled" by default. The form sends a NameVideo command, and the Component answers with VideoNamed or VideoNameRejected. The book has the Component validate the name, not the Application, and the reason lies in a distinction. A malformed email address is refused in the Application, because the rule comes from the outside world, not from the system: Video Tutorials "did not invent email." (p. 175) An empty name, by contrast, is a rule of the system itself, so it falls to the Component (p. 175).

The screen changes shape accordingly. Instead of one edit the video form, it offers one per task, one to name and one to describe: "They’re not here to edit a video: they’re here to name a video." (p. 176) A VideoUpdated event, by contrast, would not even say what changed (p. 191).

The catch comes at the end of the HTTP request. The command may not have been processed yet: "Truth be told, we don’t even know if it’s going to get processed." (p. 178) The admission follows: "This type of user interaction is where our architecture is more difficult to work with than a monolith." (p. 178)

The book's answer has three pieces, in the order they are used (pp. 178-181):

1. Each message carries in its metadata a traceId, the identifier of the user action that triggered it. The events written in response to NameVideo reuse its own.
2. A view table, video_operations, keeps for each traceId the operation's success or the reason for its failure. An Aggregator fills it.
3. After the form is sent, the Application redirects to a waiting page that carries this traceId in its address. As long as no row exists, the page shows "Operation pending" and reloads every second. Then it leads to the video or shows the failure.

The book flags a trap. Each HTTP request gets its own traceId (p. 31). The one you look for is the one of the request that wrote the command, not the one of the waiting page (p. 179).

The author does not sell this pattern as universal: "Is this always the right pattern? No. But it works for this use case." (p. 181) He admits the flow "ain’t pretty" (p. 190). Keeping the request open until the result would tie up the server (p. 190). A rich JavaScript interface would do the same waiting in the background, but the book does not build it (p. 191).

To remember: in an asynchronous system, the screen acknowledges, then reads the result in a view by following the action's identifier. It remains to take this system out of a single process, then to troubleshoot it when it breaks. This same traceId becomes the main tool there: that is the next idea.

11Distribute for an observable reason, debug through messages (overview)

For now, the whole system runs in a single process (p. 201). Should the deployment be split? The book cites Tyler Treat: "The first rule of distributed systems is don’t distribute your system until you have an observable reason to." (p. 201) The reason the author retains is availability: a Component outage must not bring down the site users see (p. 202). When he splits the system into two deployments, he touches neither the Applications, nor the Components, nor the Aggregators. Only the configuration and the entry point move: "Autonomy made the deployment strategy independent of the actual system model." (p. 204)

Once the system is deployed, the book shows what the Message Store brings when it breaks down. It stages one: many signups, but few logins. The "registration email sent" column of the admin table answers "no" for every user (p. 219), and the book follows this lead. A successful registration produces five messages, from Register to RegistrationEmailSent. A query that filters messages on the traceId in their metadata finds only three, a sign that the email does not go out. The cause is a commented-out line in the configuration, which keeps the send Component from starting (pp. 220-224). The author notes that his query uses no index, and would not hold up on millions of messages (p. 222).

He finally separates technical logs from business state. The former go in the logs, but business state does not, "because that goes in the Message Store" (p. 223).

To remember: a traceId copied into every message costs one line of code and makes a failure readable in one query. It remains to check, before production, that each piece does its job, and the book shows that autonomy makes these tests simple.

12Testing a Component means writing a message and reading the ones it produces

Many think you need the scale of a web giant and complicated test tooling to do microservices. Garofolo answers: "Bunkum and balderdash!" (p. 232) His argument comes from the autonomy of idea 2. A Component communicates only through the Message Store, and "It has absolutely no idea if a message is coming from a live system or if it’s coming from a test." (p. 232) So you test it exactly as it runs.

The book tests the three writing pieces this way (pp. 233-237):

• An Aggregator: you build a VideoViewed event, empty the view tables, call the handler twice, then check that the counter equals 1.
• A Component: you give it a PublishVideo command and a doctored Message Store. Only its fetch function, which loads the video's state, fails on purpose, the write works. The Component must record the failure: the video's stream then holds a single VideoPublishingFailed.
• An Application: you send the registration form, then read the command stream to find a single Register command.

The Aggregator's test shows idempotence in one line. Here is its sequence, shortened (p. 234):

reset()
  .then(() => aggregator.init())
  .then(() => aggregator.handlers.VideoViewed(event))
  // Call it a second time to verify idempotence
  .then(() => aggregator.handlers.VideoViewed(event))
  // then read the 'home' row: videosWatched equals 1

The same event goes through twice, and the view counts only one view. A tension appears, though. Mocks and stubs are the fake objects that replace a dependency in a test. The author calls them "unholy things like mocks and stubs" (p. 232). Yet his Component test replaces fetch with a function that fails on purpose (p. 236).

The book goes further: monitoring production is sometimes better than predicting every case, and "it’s quite all right to monitor production rather than try to predict every test case" (p. 238). The full history makes it possible: "We can sample message workflows in production and verify that they’re correct after the fact." (p. 238) Example: a Register command left without a RegistrationEmailSent beyond a delay triggers an alert (p. 238).

To remember: when each piece talks only through messages, a test boils down to messages in and messages out. Calling the same handler twice proves idempotence. It is the last piece of the book: a system you understand, change and verify piece by piece.