Observability Engineering

1Observability versus monitoring: the test of never-seen states

Take a new kind of outage. A service's p99, the time under which 99% of requests answer, is excellent. Yet customers complain about timeouts. The book turns this into a test (pp. 5-6). Can you find hidden timeouts when the p99, p99.9 and p99.99 are fast? Can you isolate the requests of the 142nd slowest user? The rank is arbitrary: a p99 never points at a given user (p. 5). If the answer is no, you are doing monitoring, not observability.

Monitoring rests on the metric, a single number with optional tags: "A metric is a single number, with tags optionally appended for grouping and searching those numbers." (p. 9). A machine compares this number with a threshold (for example, CPU above 80%) and decides whether to alert (p. 20). That works for what you planned for: "Monitoring is for the known-unknowns, but observability is for the unknown-unknowns." (p. 13). A known-unknown is a failure you know is possible without knowing when it will come, such as a saturating database. An unknown-unknown is a failure nobody imagined.

The authors' definition rests on one condition: "If you can understand any bizarre or novel state without needing to ship new code, you have observability." (p. 6). Shipping code to add a log assumes you already knew what to look for. The authors admit the bar is high (p. 6). The word comes from control theory (Rudolf Kálmán, 1960, p. 4).

Why are metrics not enough? A matter of cardinality, the number of distinct values of a field (pp. 13-14). A user ID has as many as there are users, an HTTP status code has a few. High-cardinality fields (user, cart or request ID, host name, deployed version) are almost always the most useful for debugging: a unique ID is the needle in the haystack (p. 14). Yet "Unfortunately, metrics-based tooling systems can deal with only low-cardinality dimensions at any reasonable scale." (p. 14). A dimension is a field of the data, such as the user ID or the host name. The reason, outside the book: each combination of tag values creates a series of measurements that the tool stores separately. A "user" tag creates one per user.

The way from one to the other is one-way: "You can always downsample a high-cardinality value into something with lower cardinality" (p. 14), never the reverse (idea 2). Dimensionality, for its part, is the number of dimensions, so of fields, in the data (p. 14).

On dashboards (pages of graphs), the book is blunt: "However, dashboards are a poor choice for discovering new problems with debugging." (p. 20). The typical case is tool hopping (pp. 23-24). An error spike shows on a graph. You open other graphs, search the logs, then copy the request ID into the tracing tool. That tool follows a request from service to service (idea 3). If the request wasn't traced, you start over. Context is lost at each hop.

The human consequence remains: is the team's best debugger always the one who has been there longest? With good tools, it would rather be the most curious engineer (pp. 25-26).

2One wide event per request, instead of scattered metrics

If you want to query afterwards with high-cardinality fields, you need a format that keeps them together. That is the event: "an event is a record of everything that occurred while one particular request interacted with your service." (p. 52). The request is the unit of work: downloading a file or answering an HTTP request (p. 56).

The mechanism is simple. When the request comes in, you create an empty map. During the request, you add every interesting detail: IDs, variable values, headers, parameters, duration of remote calls. On exit or error, the whole map leaves as a single key-value record (p. 52), toward a data backend. This is the observability tool's store, which must accept arbitrarily wide events (p. 57). You then query this store by filtering and grouping on any field (p. 52). With mature instrumentation (the code that fills the event, idea 4), "It is common for events generated with mature instrumentation to contain 300 to 400 dimensions per event." (p. 58).

$start = microtime(true);
$event = ['route' => $_SERVER['REQUEST_URI'], 'build' => BUILD_ID];
register_shutdown_function(function () use (&$event, $start) {
    $event['duration_ms'] = (int) ((microtime(true) - $start) * 1000);
    $event['status'] = http_response_code();
    error_log(json_encode($event)); // one JSON line per request
});
// anywhere in the code: $event['user_id'] = $user->id;

Why not metrics? A metric is a pre-aggregated measurement, for example the average load time over the last five seconds. Once the average is computed, the detail cannot be recovered: it is the finest you can look at (pp. 53-54). Every new question therefore needs a new metric, defined before the bug (p. 14). A request takes part in hundreds of disconnected metrics, with no thread saying which come from the same request. Conversely, take 1,000 events each carrying its load time. You recover the same average, and also the one per second or per user (p. 54).

Unstructured logs fail differently. They are lines of text for humans. A single unit of work is scattered over several lines, with a shared ID only sometimes. One login produces five log lines between 6:01:00 and 6:01:19. The book merges them into a single event of eleven fields, with duration_ms=123 and a trace ID (pp. 55-57). Structured logs (same content as key-value) are not enough either: "Many logs are only portions of events, regardless of whether those logs are structured." (p. 56). The thesis is therefore not "logs against events": structured logs are recommended (p. 55), provided you have one line per unit of work, with all the context.

Last point, the schema: the authors refuse to impose one, since predefining a schema forces you to foresee the dimensions to capture (p. 58). That is exactly what observability tries to avoid.

3A trace: the same ID, copied from service to service

An event describes one request in one service. When the request crosses three services, you must stitch them together, otherwise you cannot tell which one is slow. The answer is the trace: "Distributed traces are simply an interrelated series of events." (p. 61). Without this link, a slow database slows the services that call it. Three or four layers up, dozens of services look slow and you cannot find the source (p. 62). With a shared ID, you climb the chain up to the truly slow link.

Here is the vocabulary. The trace is drawn as a waterfall. "Each chunk of this waterfall is called a trace span, or span for short." (p. 63). A span is thus an event from idea 2, that of one step of the request. A span is either the root or nested in another: if A calls B which calls C, A is the parent of B and B the parent of C (p. 63). Each span carries five mandatory pieces of data (pp. 63-64):

• trace_id: the ID of the whole request, created by the root span and propagated at each step (p. 64).
• span_id: the span's unique ID.
• parent_id: the calling span, absent from the root span: that is how you recognize the root (p. 64).
• a start timestamp.
• a duration.

What remains is passing from one service to the next, the propagation. The calling service writes these IDs into the headers of its outgoing request, and the called service reads them. It is the most common way to share the information (p. 67). The book's example uses the B3 format, a set of tracing headers. X-B3-TraceId carries the trace ID. In X-B3-ParentSpanId, the caller writes its own span_id, which the callee reads as its parent's ID (p. 67). The example has at least three spans: the original request, the call to an authentication service and the call to a name service (p. 66).

The trace is also useful outside RPC (remote function calls), because any set of related events can be stitched into traces (p. 71). Three uses. You can isolate costly pieces of code (the "hot blocks"), such as JSON deserialization, in their own spans, even in a monolith. You can create one span per piece of a batch job. You can stitch multi-line logs into a waterfall, a stopgap useful to start but not advised long term (pp. 70-71).

The example code (ch. 6) is pseudo-Go that does not compile, and the authors themselves find it a bit artificial (p. 69). Example of a flaw: time.Now().Sub(startTime) returns a time.Duration, in nanoseconds, that the code stores in a field named duration_ms. The code is there to show the five fields and the propagation, not to be run.

4OpenTelemetry: instrument once, send wherever you want

Each tool vendor ships its own instrumentation library. If you switch, you must re-instrument everything, and two tools in parallel double the measurement cost (p. 74). The remedy is OpenTelemetry (OTel), an open standard born in 2019 from the merger of OpenTracing (2016) and OpenCensus (2017) under the CNCF (pp. 74-75). It has become the only open source standard for application instrumentation (p. 75). The promise: "you can instrument your application code only once and send your telemetry data to any backend system of your choice" (p. 75). Eleven languages are listed, PHP included, but all the book's code is in Go (p. 75).

Four components are enough to follow (the book's glossary has seven, pp. 75-76). The API is the specification: you instrument without caring about the implementation. The SDK is the implementation, which keeps state and batches data before sending. The exporter is the SDK module that translates in-memory objects into the destination's format, OTLP by default (OpenTelemetry's protocol). The Collector is a standalone program that receives the data, processes it, then sends it to one or more destinations. Outside the book, per OTel's documentation: once the application is wired to the Collector, changing destination is done in its configuration.

You proceed in two steps. First automatic instrumentation: OTel generates spans for HTTP, gRPC and database calls, incoming and outgoing. You get at least the skeleton of who calls whom between the microservices and their dependencies (p. 77). Then custom instrumentation, where the value lies: "the real power of observability comes from custom attributes that add context to help you debug how your intended business logic is actually working" (p. 73). An attribute is a field of the event from idea 2: customer ID, shard ID, errors. You set it on the current span, or on a span opened around a costly internal step (pp. 78-79).

ctx, span := tracer.Start(ctx, "export")
defer span.End()
span.SetAttributes(attribute.String("user.id", userID), attribute.Int("rows", n))

Metrics, for their part, are recorded as span attributes: "In general, most metrics such as measures and counters should be recorded as attributes on the trace span to which they pertain." (p. 79). Stored this way, the metric stays a value of the event, groupable later like any other field. Two exceptions: an exact count (without sampling, idea 10), or a process-specific value read periodically, such as the number of goroutines (pp. 79-80). For sending, the most sensible default is the OTLP gRPC exporter, toward your provider or a Collector (p. 81).

5The analysis loop: compare the abnormal zone with the rest

An alert fires, with no known explanation. What do you do? The classic method is to start from what you know: you consult a runbook (a written procedure for a known failure), browse dashboards and guess a cause. Yet "that time spent creating runbooks and dashboards is largely wasted, because modern systems rarely fail in precisely the same way twice" (p. 84).

The authors propose instead to debug from first principles: assume nothing, start from what is proven, form a hypothesis, validate or invalidate it by observation.

The method is called the core analysis loop, in four steps (p. 87):

1. start from the overview of what triggered the investigation, the alert or the complaint.
2. check that what you believe you know is true: is there a notable performance change? It shows as a drop on a curve.
3. look for the dimensions that explain the change: look at examples from the abnormal zone (the part of the graph where the curve drops), group by dimension, filter on a value.
4. ask whether you know enough, otherwise filter the view to isolate the zone and go back to step 3.

The computation can be automated by comparing the abnormal zone with the baseline, the rest of the system. For each dimension, you compare its distribution inside and outside, then sort by gap (p. 88). Worked example (p. 88):

The first two rows jump out, the third says nothing. That is the sorting a machine does by brute force, knowing nothing about the application. Honeycomb's tool, BubbleUp, does it on a heatmap (a chart of colored cells: time on the x axis, duration on the y axis, color counts requests): you circle the zone, and it ranks the dimensions. In a real incident of the vendor, it surfaces 63 interesting dimensions. Among them, global.availability_zone = us-east-1a appears in 98% of the zone's events against 17% of the baseline (pp. 89-90). The real cause was a network problem in that zone at the cloud provider, confirmed by affected customers (p. 90). No code was at fault, which only a wide comparison could show.

The loop works only with the basic building blocks of observability (p. 90), so the wide events of idea 2. With logs, you need to have added request IDs and a lot of post-processing (pp. 90-91).

The authors also rule out AIOps, a term Gartner launched in 2017 for AI applied to operations (pp. 91-92). The argument: the AI must draw the line between "anomaly" and "baseline". In an environment that changes all the time, it either tightens the line too much and flags normal variations (false positives), or loosens it too much and takes anomalies for normal. Every deployment is itself an anomaly, besides. The authors conclude: "any computer can crunch the numbers and detect a spike, but only a human can attach meaning to that spike" (p. 92).

6Who looks at what: monitoring for systems, observability for code

The method raises a practical question: do you throw away metrics? No, with a split rule: "observability is best suited to understanding issues at the application level and that monitoring is best for understanding issues at the system level" (p. 95). The book defines software this way: "software is the code you are actively developing that runs a production service delivering value to your customers" (p. 97). The system covers everything else: infrastructure and runtime environment (p. 98).

Table 9-1 (p. 98) compares the two on nine factors. I keep three:

• rate of change: the system changes little (monthly package updates), the software every day, with commits.
• predictability: the system is predictable, the software hardly, because new features keep arriving.
• point of view: the team that runs it asks whether the system is healthy, while the customer wants every request to get its resources in time.

The last row of the table gives the method: the system is understood through monitoring, the software through observability. A case for a PHP or Go developer: the server CPU is at 90%, that is system, a metric and an alert are enough. A curl call to the payment API takes 8 seconds for a single customer, that is software: your code and your requests, which no CPU graph shows. You need that request's event, with the customer ID (my example, outside the book).

Aggregated metrics therefore suit infrastructure (p. 99). The boundary follows operational responsibility, not hosting. A component bought as a service (Kafka, Memcached) is not your infrastructure: its vendor runs it. The one your team installs, configures and troubleshoots is (p. 97).

The more you outsource, the less infrastructure monitoring you have to do. On a PaaS (a platform that manages servers for you), few infrastructure alerts are enough if the provider is robust (p. 100). Two safeguards remain: watch high-level metrics (CPU, memory, disk) as early alarms, and see system metrics and application data side by side (p. 101).

7Start with the biggest pain point, and instrument along with the feature

How do you adopt these ideas without a six-month project? The book gives six pieces of advice (ch. 10). The first is to join a community of peers, for example Honeycomb's Slack group (p. 108). The most counterintuitive is the second: "Counterintuitively, for observability, starting small is one of the bigger mistakes we often see people make." (p. 109). Otherwise the team bears all the work without any benefit (p. 109). The advice: "Don’t pick an easy problem to start with." (p. 109). Take the unstable service that wakes the on-call, the unexplained database congestion or the load of an unknown user, then spread the success (p. 109). For the export that spins forever from the start of this note, you therefore start with it (my example). You set the span of idea 4 with the customer and the number of rows, instead of instrumenting the home page that works fine.

The third, buy rather than build, is discussed in the box. The last three boil down to this:

• iterate through on-call: on-call is the rotating duty. When an engineer is paged, their first action is to instrument the faulty area with the new tool, then use it. After the second or third time, people understand it is simpler (p. 112).
• reuse what exists: duplicate the log stream to the new tool, add a unique ID to structured logs, or recreate the best dashboards as queries (pp. 112-113).
• plan the last effort, the hardest: iterating through on-call gets you about half or two thirds of the work. The rarely touched parts of the stack stay painless, so without a driver: you need a completion plan (p. 114).

The book extends the idea in three points (ch. 11). A question to ask on every change: "How will I know if this change is working as intended?" (p. 121). A limit: "Observability is not for debugging your code logic." (p. 119). It shows which region of the system to look at, then a debugger takes over, like the microscope after the telescope (p. 120). A gesture: route production alerts for 30 minutes to 1 hour to the engineer who just merged their code. This tightens the loop between shipping and its consequences (p. 122).

8Alert on the user symptom, not on the possible cause

Take an alert that fires as soon as CPU exceeds 80%. It mostly triggers false positives, hence alert fatigue. After every incident you add more. This is normalization of deviance, a term from the investigation of the Challenger shuttle accident: you get used to the noise until you ignore the real signal (p. 128). The verdict: "Alerts for conditions that aren’t tied directly to customer experience will quickly become nothing more than background noise." (p. 130). Failures fixed automatically should not set off an alarm: you review them during business hours (p. 130).

The authors give two criteria for a good alert. First, "it must be a reliable indicator that the user experience of your service is in a degraded state" (p. 131). Then it must be solvable: there is a systematic way to debug and act (p. 131). Otherwise, delete it (p. 138).

The tool to get there is the SLO (service level objective), an internal availability target, stricter than the commitment sent to the customer. It is measured with an SLI (service level indicator), the rule that classes a request as good or bad. An SLI per event beats an SLI per time window: each request is judged on its own, instead of a whole minute being classed good or bad (p. 133). Example: for requests on /home, a successful response in under 100 ms is good. Slower, it counts as an error, even with a success code (p. 133). Each bad event consumes the error budget, the share of bad requests the objective tolerates.

The SLO alert says there is pain, not why. This is where observability takes over: when anything can fail at any time, you need data on everything (p. 134).

The textbook case is a Honeycomb service, Shepherd, which receives customers' data (pp. 135-137). In late 2019, a 1.5% brownout (a partial outage, where only a fraction of requests fail) lasts 20 minutes. According to the caption of fig. 12-2, the bad events occur at fairly regular intervals until the fix. An SLO alert wakes the on-call engineer at 1:29. Seeing the service running, they conclude it was a blip. A fourth alert arrives at 9:55: the incident is declared, then resolved at 10:32. The cause is a memory leak after a deployment: all machines go down together, out of memory, then restart.

The probes, those periodic tests that ask the service whether it answers, saw nothing. They require several consecutive failures, and out of 50 probes only one fails then recovers (p. 136). At the end, the budget shows -566% for a 99.995% target (fig. 12-2, p. 137). A few incidents showed that the SLO alerted well before the classic alerts. The team deleted the alerts on the error percentage of the last five minutes, on the error count and on system behavior. It makes SLO alerts its first line of defense, because SLOs proved their ability to detect brownouts (p. 137).

The story is fuzzy on one point. The team did not trust its SLOs, and their alerts went to a low-priority channel (p. 135). Yet one of them woke the on-call, and we are not told which ones woke people. The lesson stays clear: the 1:29 engineer ignored a good alert, and a culture that does not follow up cancels the tool.

9Burn alerts: forecast the budget running out rather than wait for zero

A 99.9% SLO tolerates 8 h 45 min 57 s of downtime per year, that is 43 min 50 s per month (pp. 139-140). The month here is a twelfth of a year. In event mode, the budget is the number of requests in the period multiplied by (1 minus the target). The remaining budget is that budget minus the failed requests (p. 140). The higher the target, the less time you have to react:

Forecasting (see below) works well up to 99.95%. That is 21 min 54 s of total outage tolerated per average month. Beyond that, it serves more to report than to prevent (p. 140), and the book points to the Site Reliability Workbook (p. 141).

The measurement window is rolling, that is the last 30 days at any moment: "For most SLOs, a 30-day window is the most pragmatic period to use." (p. 141). A fixed calendar window resets the budget to zero all at once on the 1st of the month and lacks data just after (p. 142). The rolling one forgets faults gradually, like human memory: "The better SLO is one that accounts for human emotions, memories, and recency bias." (p. 142).

Two ways to alert before zero. A non-zero threshold (for example, remaining budget under 30%) is crude: "A challenge with this model is that it effectively just moves the goalpost by setting a different empty threshold." (p. 143). Those 30% can last ten days or one hour: the threshold does not say how fast you are heading there (my explanation).

The predictive alert projects the future with two windows (pp. 143-144). The lookahead window says how far you look. The baseline window says how much of the recent past is used to forecast. Do not confuse it with the baseline of idea 5. Not everything justifies waking someone. A drift toward 99.88% over a month can wait for a working day. An SLO heading to 98% within an hour calls the on-call (p. 144).

The authors' rule of thumb: "a given baseline window can linearly predict forward by a factor of four at most without needing to add compensation for seasonality" (p. 145). Seasonality means the cycles of the day, week or year: night lows, sales peaks (p. 145). Hence the heuristic: "a 24-hour alarm is based on the last 6 hours of data, a 4-hour alarm is based on the last 1 hour of data" (p. 152). Too short, the alert becomes "flappy" (unstable). Too long, you see the outage too late (pp. 145, 151).

Here is the book's example for a 99% target over 30 days (pp. 147-148). It counts in units: the basic building block of the calculation, here one measurement point per minute, good or bad (p. 147). A typical month has 43,800 units, the error budget is 1%, that is 438 failed units. Over the last 24 hours, the service saw 1,440 units, 50 of them failed. Over the last 6 hours, it saw 50, 25 of them failed. That 50 is an inconsistency: 6 hours make 360 units, as in the first example (p. 147). The second looks like a rework of the first. I keep its figures to follow the calculation.

The goal is to forecast the next 24 hours, four times the 6-hour window. The linear extrapolation extends the number of failures: 25 × 4 = 100, far below 438. The proportional extrapolation, more realistic, extends the failure rate: 25 units out of 50, that is 50%. Over 24 hours, 50% of 1,440 units makes 720 failures, well above 438. The alert fires, and the budget would be exhausted in about half a day.

The book adds a "context-aware" example (pp. 148-151), which keeps the count of the whole 30-day window. You forecast over four days from one reference day, so you only look back 26 days: 26 + 4 = 30 (p. 149). Those 26 days hold 285 failures, the last day 130. That leaves 438 - 285 = 153 allowed failures. The projection repeats the last day four times: 285 + 4 × 130 = 805, against 438 allowed. Exhaustion would come in 153 / 130, about 1.2 days, which figure 13-7 also says (less than two days, p. 151).

This context-aware calculation is expensive: at Honeycomb, small SLO datasets suddenly cost more than $5,000 a day in AWS Lambda (p. 148).

One last argument holds for per-event SLOs rather than per-minute ones. Take a four-nines SLO where each whole minute is classed good or bad. A single bad minute is enough to burn "25% of your 4.32-minute monthly error budget" (p. 155). My calculation gives rather 23% (1 / 4.32). More generally, "a 1% brownout of a 99.99% reliability system is similar to measuring a 100% outage of a 99% reliability system" (p. 155). In both cases, the failure rate is 100 times the tolerated rate: 1% for 0.01%, and 100% for 1%. The book draws the consequence: more than seven hours to remediate before exhausting the monthly budget (p. 155). Partial brownouts are more frequent than total outages, and an event-based SLO only removes the share of budget that was really bad.

10Sample without distorting the counts

At scale, keeping every event costs more than it brings: "The reality of most applications is that many of their events are virtually identical and successful." (p. 208). Sampling means keeping only part of them. Unlike a pre-aggregated metric, the kept event keeps its full cardinality (p. 208). Four families of strategies (pp. 209-211):

• constant probability: you keep 1 request in 10, for example. Ineffective if you care more about errors than successes, or if some customers send orders of magnitude more traffic (p. 209).
• by recent volume: you keep more events when recent traffic is low, fewer when a burst threatens to overwhelm the analysis tool (p. 210).
• by content: you choose a rate per key, a combination of fields such as the HTTP code, the request type or the paying customer. It works if the set of possible keys is small and stable. You must check the assumption (errors can become the majority again, p. 211).
• key plus history: the rate depends on the volume seen for a field combination in the last 30 seconds. The more frequent a combination, the less it matters (p. 211).

The central point is not to distort the counts. If you keep 1 request in 1,000, each kept event represents 1,000: out of 100,000 requests, you receive about 100 events (p. 209). The rule: "for calculating a count, your system must add together the number of represented events, not just the number of collected events" (p. 210). With a constant rate, the median and the p99 need no adjustment, since sampling does not distort them (p. 209).

With a variable rate, you must weight. Take three (value, rate) pairs: {1, 5}, {3, 2}, {7, 9}. The value is a latency for example, and a rate of 5 means a kept event represents five. The naive median of the three values is 3. Expanding (five times 1, twice 3, nine times 7), you get 16 values. Their median is 7 (p. 210, recounted by me).

Hence a practice: record the rate in the event itself. Otherwise the rate may change later, and the server would no longer know what each event represents (p. 213). Counting errors then becomes the sum of the sampleRate of the error events (p. 214).

For traces, two decision moments (pp. 211-212). Head sampling decides at the head of the trace, so at the start, and propagates the decision to the children in an HTTP header. It suits properties known at entry, such as the endpoint or the customer. Tail sampling decides at the end, for properties known only then (status, latency): it is the only way to decide correctly on those values (p. 212). You then must buffer all spans before ruling, which is costly and impractical from the instrumented code itself: it is done on the collector side (p. 212).

Another trap: if each service samples independently, you rarely keep all the spans of one trace. Consistent sampling fixes this. A random number (Sampling-ID) is drawn once at the root, then propagated, and each service compares it with its own rate. The guarantee has a condition: "Consistent sampling ensures that when the sample rate is held constant, traces are either kept or sampled away in their entirety." (p. 215). If rates differ, the trace may lose links. Take Redis, sampled at 1 in 1,000 under a parent kept at 1 in 10. The kept child always has its parent, but the kept parent loses its discarded children (p. 215).

Outside the book, per the OpenTelemetry specification, common samplers derive the decision from the trace ID. They use a hash or random bits, instead of propagating a separate number.

To target a number of events per second, the service counts the previous minute's requests and deduces its rate. With 5 events per second wanted, that is 300 per minute, 30,000 requests per minute give a rate of 1 in 100 (my calculation).

11One process per request breaks Prometheus's model: the PHP gem

The book's only PHP case hides in a guest chapter on telemetry at Slack (pp. 225-240). Prometheus is a metrics tool that works by pull: at regular intervals, it queries an endpoint that each program exposes with its counters. That assumes a long-lived program that keeps its counters in memory.

Slack's backend runs on PHP then Hack, Meta's PHP dialect, with a process per request: each request starts a process that disappears afterwards: "Since PHP/Hack use a process per request model, the Prometheus pull model wouldn’t work because it does not have the process context" (p. 235). The Prometheus server would query a process already gone, and no counter survives from one request to the next.

Slack's solution: "Slack uses a custom Prometheus library to emit metrics per request to a local daemon written in Go." (p. 235). A daemon is a program that runs in the background. This one aggregates the metrics over a time window and exposes the endpoint that Prometheus comes to read (fig. 18-3).

The same chapter describes a telemetry pipeline: a path between the application and the analysis tool, with receiver, buffer, processor and exporter, which routes, filters and enriches the data (p. 226). Building the base of this pipeline in-house makes no economic sense, given the many modular open source alternatives (p. 239). Running them still costs. This guest chapter thus trusts open source to carry the data, where idea 7 judged it insufficient to analyze them.