Event sourcing without an event store: CDC as the log of facts

Table of contents (6 sections)

A bank account, stored in an accounts table with a balance column. A withdrawal of 50 euros runs UPDATE accounts SET balance = balance - 50, and the previous balance is gone. The balance is the account's state. The list of its operations is its history, and the UPDATE erases it.

Event sourcing avoids that loss. You no longer keep the latest balance, but the list of operations: deposit of 100, withdrawal of 50. To know the balance, you add them up. I use this method every day, with an event store, a database that holds only that list, appended to and never modified. I described it in CQRS in Go, part 4: PostgreSQL as an event store.

John Gilbert makes the same observation I do: "We are losing information. We only retain the latest balance." Yet in Software Architecture Patterns for Serverless Systems (Packt, 2024), he builds his event sourcing on that very table, with no event store. The log of facts in the title comes from somewhere else: CDC (change data capture). This component reads the database's internal log, where the database itself records every change it makes.

So the book gives a different answer to a question I thought was settled: where should history live? Here are both answers side by side, what CDC simplifies, and what it costs.

Two places to keep history

In classic event sourcing, the state is computed from the history. A command (an instruction given to the service) comes in, for example withdraw 50. The service reads every event of the account and replays them, meaning it adds them up in order, to rebuild the current balance. It decides, then appends its new events to the event store. The read views, called projections, then feed on that event store.

The account and its events form an aggregate. Each event of the aggregate carries a version number. A UNIQUE (aggregate_id, version) constraint stops two simultaneous writes from taking the same number. History lives inside the service, and the service needs it for every decision.

Gilbert flips the setup. The service keeps a current-state table: one row per account, updated with a plain UPDATE. The service no longer records history, the database does, in its internal log. CDC reads that log and outputs every insert, update and delete, one after the other.

A small function, which the book calls a trigger, turns each change into an event and publishes it to the hub. The hub is a shared channel: each service listens for what it cares about.

History moves downstream, to the services that receive the events. An event lake listens to everything that goes through the hub and files it by date in object storage, with no expiry. In Gilbert's words, "the event lake acts as the system-wide transaction log." Put simply, history no longer belongs to the service that writes, but to the system. Services listen to the hub live, and none of them reads the lake to make a decision.

You cannot write to two data stores at once

CDC is not a convenience in Gilbert's design. It comes from a constraint: "we cannot reliably save data to more than one data store at a time."

A concrete case, made up for the occasion: an online shop service saves a purchase in its database. Then it publishes a purchase created event to tell the other services. That is two writes. A crash between them loses one.

I dealt with the same problem in part 2 of my idempotence series, with a different answer: the outbox. The event is written to an outbox table, in the same PostgreSQL transaction as the purchase. A transaction groups several writes: either all succeed, or none. A worker, a small program, then reads the table, publishes the events and marks each row as sent.

Gilbert offers two variants:

• Stream-first (the stream goes first): the service publishes to the hub first, and a downstream service stores the data.
• Database-first: the service only writes its table, and that table's CDC publishes the event.

Gilbert's rule for choosing is simple. If users need to see what they just created right away, write to the database first, which requires a database with CDC. Services with no user in front of them, such as gateways to external systems, publish first.

The book never uses the word outbox in the transactional sense. Yet as I read it, database-first is an outbox whose table is the database's own internal log. The gain is concrete: no outbox table, no published column to keep up to date, no worker for you to write. The service reads and writes a single model, its table.

The work does not vanish, though: someone has to run the CDC. With PostgreSQL, a tool like Debezium reads the internal log, and it is one more service to keep running. Gilbert instead relies on the CDC of a database managed by the cloud provider, such as DynamoDB Streams.

One detail from the book is worth borrowing. The trigger first publishes each change as it is, with the row before and the row after. Only then does a rule publish a clean domain event, for example thing-submitted when the status field becomes submitted. The raw changes serve as an audit trail and help a young subsystem discover its true domain events.

Idempotence and ordering without a version number

A concrete case, made up for the occasion: the message debit 50 euros arrives twice, because the sender resent it after an outage. If we debit twice, the customer loses 100 euros instead of 50. An operation is idempotent when playing it ten times gives the same result as playing it once. Making the debit idempotent means recognizing the second message and ignoring it.

Without an event store, consumers lose the safety net I know well: the per-aggregate version. A consumer is a service that reads events to update its own view. Yet it receives duplicates, and out-of-order events. Gilbert accepts this from the start: "exactly-once delivery of messages is unrealistic."

Duplicates have three sources:

• a client request resent after a timeout;
• a batch of events that fails halfway and restarts in full;
• a resend from the lake.

Disorder comes from a faulty producer, a traffic spike, or an event set aside and resubmitted later.

First line of defense: deterministic identifiers, meaning identifiers recomputed identically on every retry. The simplest reuses the ID of the change provided by CDC, which stays stable across retries. If one change yields two domain events, add a fixed suffix per type.

Second line of defense: inverse optimistic locking. Classic optimistic locking rejects a write when the data changed since it was read, and the user has to reload. On the stream side, Gilbert does the opposite. The row keeps, in a field called the oplock, the timestamp of the last applied event, meaning its date and time. His rule: "we drop incoming events if the timestamp is older or equal to the current oplock value in the datastore."

To picture it: the row's oplock says 2:00 pm. An event stamped 1:59 pm arrives: older, so we ignore it. An event stamped 2:00 pm: already seen, so we ignore it too. An event stamped 2:01 pm: we apply it.

The book adds a refinement. A view that gathers the data of several event types into one row keeps one oplock per type. Otherwise a late order-created would be dropped because of a newer order-submitted, even though it carries fields the other one lacks. Here it is in Go and PostgreSQL:

// Example written for this article, not taken from the book (its code is JavaScript on DynamoDB).
// An "order" view joined from two event types, received in any order.
// One oplock per type: created_at_ms and submitted_at_ms (business timestamp in milliseconds).

const applyCreated = `
INSERT INTO order_view (id, customer, created_at_ms)
VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE
   SET customer = EXCLUDED.customer, created_at_ms = EXCLUDED.created_at_ms
 WHERE order_view.created_at_ms IS NULL
    OR order_view.created_at_ms < EXCLUDED.created_at_ms`

const applySubmitted = `
INSERT INTO order_view (id, status, submitted_at_ms)
VALUES ($1, 'submitted', $2)
ON CONFLICT (id) DO UPDATE
   SET status = 'submitted', submitted_at_ms = EXCLUDED.submitted_at_ms
 WHERE order_view.submitted_at_ms IS NULL
    OR order_view.submitted_at_ms < EXCLUDED.submitted_at_ms`

func (v *OrderView) Apply(ctx context.Context, e Event) error {
    var res sql.Result
    var err error
    switch e.Type {
    case "order-created":
        res, err = v.db.ExecContext(ctx, applyCreated, e.EntityID, e.Customer, e.Timestamp)
    case "order-submitted":
        res, err = v.db.ExecContext(ctx, applySubmitted, e.EntityID, e.Timestamp)
    default:
        return nil // type this view does not care about
    }
    if err != nil {
        return fmt.Errorf("apply %s: %w", e.ID, err)
    }
    if n, _ := res.RowsAffected(); n == 0 {
        v.dropped.Add(1) // duplicate or late event: dropped, but counted
    }
    return nil
}

Each query is an upsert: it creates the row if it does not exist, otherwise it updates it. In that case, it only does so if the received timestamp is newer. The first event received creates the row's skeleton, and the next ones fill it in, in any order. The dropped counter follows the book's advice: measure what you ignore, so you notice the day you ignore too much.

Rereading my own idempotence article, I found the same idea without the name. My balance projection did an upsert with WHERE account_balances.last_event_id < EXCLUDED.last_event_id, so an old event would not overwrite a newer state. That is an inverse oplock: the write is ignored if it is older than the row. Only the compared value differs. In my case, it is an event's position in an ordered event store. In Gilbert's, it is the time the user acted.

What CDC costs

Gilbert's model takes plumbing away from the writing service. The bill lands on the consumers, and it comes down to three cases, on top of running the CDC itself, as seen above.

Two events in the same millisecond

In the book's event format, the timestamp is a number of milliseconds, taken when the user acted. Picture two address changes on the same purchase, both stamped 1574100988321. The second one arrives with a timestamp equal to the row's oplock. It is dropped as a duplicate, and the view keeps the old address until the next change.

The book does not cover this case in its inverse oplock section. It comes close elsewhere, when choosing how to sort events. The timestamp is "frequently the best option, so long as it does not result in duplicates."

One way out comes from me, not from the book. The service that writes, upstream of the hub, already has a classic optimistic lock for its own writes: a field used to refuse a stale write. Gilbert says it can hold "a timestamp or counter." Pick the counter, publish it in the event, and compare on it downstream. You get a version per entity back, without an event store.

The late event after a snapshot

A snapshot is an already computed state that you reuse as a starting point instead of recomputing from the beginning. Gilbert does keep computations over history, but he moves them out of the writing service.

An intermediate service, which he calls a control service, collects deposits and withdrawals, computes the balance and publishes a balance event. That event acts as a snapshot: the next calculation starts from the latest published balance and adds the newer events. His example shows the same deposits and withdrawals received in two different orders, with the same final balance of 170.

The book admits the limit. Its example "ignores edge cases such as when an older detailed event arrives after a snapshot," and the author leaves them "as a thought exercise." By detailed event, he means a deposit or a withdrawal, as opposed to the balance. The code assumes older events are "already counted in the latest snapshot."

As I read it, here is what can happen:

1. the control service publishes a balance, so a snapshot;
2. a deposit older than that snapshot arrives after it;
3. the next calculation starts from the snapshot and never sees that deposit.

Gilbert offers a lead without coding it: each snapshot lists the IDs of the events that produced it. You can then check afterwards and emit an adjustment.

In an event store, this case does not exist for the aggregate. The version constraint forces writes to go one after the other. A snapshot taken at version 100 is completed with events from version 101 onward. None of them arrives late. Disorder only affects projections, where the conditional upsert is enough.

Soft deletes and row expiry

In a system where events arrive late, a DELETE breaks idempotence. If an updated event arrives after the deleted one, the row comes back to life. So Gilbert deletes logically, without erasing the row. The command sets deleted = true. CDC sees a plain update, and the trigger publishes thing-deleted. Queries filter on the deleted field.

Consumers stay lean thanks to a TTL: a time to live set on each row, computed from the event's timestamp. The book suggests 33 days by default. The TTL has a side effect worth knowing: it bounds how long a duplicate is recognized. An event resent after expiry is processed a second time. Hence the book's warning to "ensure that ttl is not too short."

Resending means sending old events again to a consumer that needs to catch up. You never republish them to the hub: Gilbert notes that "this would broadcast the events to all consumers." You send them from the lake to that single consumer. It must therefore absorb duplicates and disorder, which brings us back to the inverse oplock.

Event store or CDC: how to choose

The two approaches answer different needs. My criterion: what does the command need to decide, the history or the present state? Deciding means, for example, accepting or refusing a withdrawal.

• The event store, when the decision depends on the aggregate's history. A balance that must never go below zero is one case. Another: a state machine, meaning steps to go through in an exact order. The per-aggregate version gives you that order for free, and the computation of the state stays inside the service.
• Current state with CDC, when the decision only looks at the present state. History then mostly serves audit, resending and bootstrapping new services. That covers most data-entry screens.
• Both in the same system, which is what Gilbert does himself. At the edges of the system, ordinary tables with CDC. In the middle, services that compute over history. His event sourcing does not disappear, it moves.

One condition comes before everything else: without a database that exposes its internal log, database-first does not exist. You are left with stream-first, or a hand-written outbox.

Conclusion

Gilbert tells the story of his first cloud e-commerce system, kept in sync by streams and functions. Here is his admission: "I didn't realize it at the time, but I was implementing the Event Sourcing and CQRS patterns." I took the opposite path: I was writing an inverse oplock in a WHERE clause without knowing the term existed.

So the mechanisms of the two schools look more alike than people say: outbox or CDC, conditional upsert, stable identifiers. The book taught me I was looking for the difference in the wrong place. It lies not in the tool but in the question the service asks before writing.

Going further. John Gilbert's book has its own book review of Software Architecture Patterns for Serverless Systems. For per-aggregate event sourcing, see my review of Understanding Eventsourcing and the order in which to read Event Sourcing and CQRS books. The basics of HTTP idempotence are in part 1 of the idempotence series.