System Programming Essentials with Go

1The Go program asks, the kernel decides

First family: how a program talks to the system, starting with the disk. When your program writes a file, it never touches the disk itself. It asks the kernel, the core of the operating system and the only part allowed to talk to the hardware. That request is called a system call, or syscall: write to write, open to open.

Rios turns this into an image. The kernel "plays the role of a strict border control officer" (lesson 10). Every system call is a passport it checks before letting anything through.

That passport has to say what it is about. The system call names its target by a number, the file descriptor, which the system hands out to each open resource. A shell gives every program three of them: 0 for input, 1 for output, 2 for errors. The strace tool makes these exchanges visible, because it prints a program's system calls:

strace -e write ./app
write(1, "Hello, World!", 13) = 13

In that line, the 1 is the output descriptor. The 13 is the number of bytes written.

That leaves the question of which Go code issues these system calls. The book presents three packages, from highest to lowest:

• os: it wraps system calls in a portable way. For example, os.Setenv(key, value) works on every system.
• syscall: one level down, this package is frozen, with no more changes. The reason: operating systems evolve faster than Go can follow.
• golang.org/x/sys: the replacement for syscall. It is a separate module, easier to maintain. It keeps the calls specific to each system, Linux or Windows.

The book's rule follows: go through os, and call x/sys directly "only in exceptional cases" (lesson 10).

2Everything is a file, even the pipe that links two programs

After the disk, the program talks to other programs. A program while it runs is called a process. The simplest way to link two processes is the pipe: a one-way, in-memory conduit. One process writes at one end, the other reads at the other end.

The shell's | is one. In cat log.txt | grep ERROR, cat writes the lines of the file into the pipe. grep reads them and keeps those containing ERROR. The book tells two kinds of pipe apart.

The diagram contrasts the two kinds:

• The anonymous pipe links a parent process to the processes it launched, its children. It disappears with them.
• The named pipe, or FIFO (first in, first out), is a special file. It lives in the filesystem, so any two processes can open it by its name.

The named pipe shows that everything is a file: it links two programs, yet it has a name in the filesystem. The book explains it with an office task inbox. Creating one takes a single line:

syscall.Mkfifo("/tmp/inbox", 0600) // 0600: only the owner can access it

That line brings back the rule from idea 1. Go has no os.Mkfifo: creation goes through syscall or golang.org/x/sys/unix. Creating a named pipe is therefore one of those exceptional cases. Yet the book switches packages from one example to the next. It writes unix.Mkfifo in lesson 28, then syscall.Mkfifo, the frozen package, in lesson 29. Its log tool even calls os.Mkfifo, which does not exist: that code does not compile as written.

The book adds two pieces of advice on pipes:

• Random names. An intruder can create in advance the named pipe your program expects: that is the "name squatting" attack. The book's rule: "use randomized or unpredictable names to prevent name squatting attacks for named pipes" (lesson 29).
• Pipe or channel? A goroutine is a lightweight task running inside a Go program. A channel links two goroutines, the way a pipe links two processes. The rule is clear-cut: pipes between processes, channels between goroutines.

3The Unix socket, the machine's internal phone

The pipe has a limit: it only goes one way. The socket, a communication endpoint, lets two processes answer each other. The Unix domain socket links two processes on the same machine, like an internal phone line.

Here is how the image decodes. A classic network call dials a public number: an IP address and a port. The internal phone only needs an extension name, here a file path. The book draws three advantages from it:

• no network protocol overhead
• a file path as the address
• access rights that are the file's: "Access to UNIX sockets can be controlled using filesystem permissions" (lesson 31).

l, _ := net.Listen("unix", "/tmp/svc.sock")  // server
c, _ := net.Dial("unix", "/tmp/svc.sock")    // client

The server listens with Listen, the client calls with Dial. That file path has a downside. Listen first creates the socket in the kernel, then binds it to the path, which becomes a file. That file outlives the program, since sockets "persist in the filesystem until explicitly removed" (lesson 31). Hence the book's os.Remove before listening: it deletes the file left by a previous run.

The same file is also deleted on shutdown, thanks to signals. A signal is a short message the system sends to a process. On SIGINT (Ctrl+C) or SIGTERM (the system's shutdown request), the book's server closes the listener and deletes the file (lesson 31).

4The ballast: inflate memory to clean it less often

Second family: memory. In Go, you do not free memory by hand. The GC (garbage collector) does it: it spots the memory nothing uses anymore and frees it. It is part of the runtime, the code shipped in every Go binary, which manages memory and goroutines.

The GC triggers when the heap has doubled since the last collection. The heap is the memory the program reserves while it runs. On a small heap, that doubling comes fast, so the GC runs constantly. It is a bin too small, one you have to empty all the time.

That doubling is not set in stone. It depends on GOGC, an environment variable: a setting passed to the program when you launch it. Its default value, 100, lets the heap grow by 100% before the next collection. The book compares GOGC to a home thermostat (lesson 37). At 50, the GC runs more often, and the heap stays small. At 200, it runs less often, and memory climbs. The value off turns the GC off.

The ballast makes the bin bigger without touching GOGC. It is "a large allocation of memory that is never used" (lesson 39), for example make([]byte, 10<<30), that is ten gibibytes. The ballast inflates the heap, so the doubling threshold moves back. The bin is bigger: you empty it less often. The book advises trying GOGC before the ballast, as the less invasive route (lesson 39).

Ten useless gibibytes still do not cost ten gibibytes of RAM. The book notes that the ballast "resides mostly in virtual memory" (lesson 39). Virtual memory is a reservation: the system only supplies real memory if the program touches it. And the ballast is never touched. So it raises the GC threshold without weighing on real memory.

The book backs the technique with a true story, at Twitch, the live video site. An API gateway service there, the front door that receives the requests, was hit by "refresh storms". When a streamer's live feed drops, viewers reload the page in a burst. With the ballast, the book reports, at peak load (lesson 39):

• about 99% fewer GC cycles
• 30% less CPU
• 45% lower latency at the 99th percentile, the latency that 99% of requests stay under.

The book also lists six cases where you should avoid the ballast. The most important one: the ballast can hide a defect in the code.

5GOMEMLIMIT: a soft memory ceiling that retires the ballast

The ballast has an expiry date, and the book gives it right away. The technique is "only relevant up to Go version 1.19" (lesson 39). Its successor is GOMEMLIMIT, another environment variable, like GOGC.

GOMEMLIMIT=512MiB ./app

This variable sets a soft ceiling for the memory managed by Go. Soft, because the GC grows more aggressive as it nears the limit, without forbidding the program to go past it. The book compares that ceiling to a speed sign that "suggests a safe speed but can't physically slow down your car" (lesson 40).

Should you keep the ballast as well? The book sees no point in stacking them: using both together "can be redundant, like wearing two watches to tell the same time" (lesson 40).

The chapter ends on arenas, memory zones freed in one go, instead of object by object. The book warns that they "may not exist in future releases" (lesson 40). Finally, it asks three common-sense questions. One of them sets up what comes next: without measurements, "you're guessing" (lesson 40).

6Measure before optimizing: four tools, from the simplest to the finest

Third family: measure so you stop guessing. The book gives four tools, which I order from the simplest to the finest:

1. watch the GC at work, without writing any code
2. ask the compiler where variables live
3. time a function, then compare two versions
4. find where the time goes, with a profile.

The first tool goes back to the GC of idea 4. Launched with GODEBUG=gctrace=1, the program prints one line per collection. Each line gives the collection's duration, the share of time spent in the GC, the heap size before and after (lesson 38). The book's rule: a heap that grows "without similar decreases after GC cycles might point to memory leaks" (lesson 38).

The second tool questions the compiler. A variable lives in one of two places. The stack is the memory that belongs to one function call: it frees itself when the function returns. The heap, on the other hand, waits for the GC. The book's image: on the stack, the variable is a friend crashing on your couch for the night. On the heap, it is "more like signing a lease" (lesson 44).

The compiler picks the place through escape analysis. A variable whose address leaves its function "escapes": it has to outlive the call, so it goes to the heap. The book's example is a createPerson function, which creates a person and returns its address. The command go build -gcflags "-m -m" then prints moved to heap: p (lesson 44). In plain terms: the object created will give the GC work to do.

The third tool times. A benchmark measures how long a function takes. In Go, it is written func BenchmarkX(b *testing.B) and run with go test -bench. To compare two versions of the code, the book saves two series of measurements, old.txt and new.txt. It then hands them to benchstat (lesson 46). That tool shows the gap as a percentage, with a p-value. Below 0.05, the gap is unlikely to come from chance.

A benchmark can also mislead. The book lists six benchmark pitfalls (lesson 47). The sneakiest comes from the compiler: it removes a computation whose result nobody uses. The benchmark then times nothing. The book's fix: store the result in a package-level variable, so the compiler keeps the computation.

The fourth tool answers the question the benchmark leaves open. A benchmark tells you how long a function takes, not where that time goes. The CPU profile reveals that, thanks to pprof, the profiling tool shipped with Go:

f, _ := os.Create("cpu.out")
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()

This code records the profile in cpu.out. go tool pprof then reads that file, as text or as a flame graph. In that graph, each function is a bar, and a wider bar weighs more CPU. The memory profile follows the same motion, with pprof.WriteHeapProfile.

7Logs, metrics, traces: three ways to watch a service run

A profile runs for the length of a test. A service, on the other hand, runs continuously, and you observe it while it runs. One part of the course, about a tenth of the text, gives three tools for that:

• Structured logs, with log/slog. They write JSON you can query. The book also says what not to put in them: "Avoid logging sensitive information such as passwords" (lesson 60), personal data, large binaries.
• Metrics, with Prometheus, a server that polls your service and stores its counters.
• Traces, with runtime/trace, opened with go tool trace. They record what goroutines and the GC are doing.

Logs come first, and the book makes two comparisons there. The first one sets two libraries against each other. The book redoes the same example with zap, an external logging library, then settles it this way (lesson 59):

• zap when speed comes first and you want to tune every detail.
• slog for simplicity, and because it fits with context, the package that carries a request's deadlines and values.

The second comparison sets two uses of logs against each other:

• Debugging logs are very detailed and temporary. They serve developers, while they hunt down a bug. Example: the list of functions running at the moment of an error.
• Monitoring logs are leaner and permanent. Tools aggregate them, and the operations team reads them too. Example: one line per HTTP request, with the method, the URL and the status code.

For the book, the difference comes down to "the objective, detail level, audience, and life span" (lesson 59).

After logs, metrics. They come in four kinds:

• the counter, which only goes up (for example, the number of requests received)
• the gauge, which goes up and down (for example, the memory in use)
• the histogram, which shows the distribution, for example the 95th percentile of latencies
• the summary, which computes quantiles on the fly (the percentile is one special case).

OTel (OpenTelemetry) brings the three tools together. It is the open instrumentation standard, the way a program emits its logs, metrics and traces. A collector, a relay that receives this data and forwards it, is launched with docker-compose. In OTel, a span is a trace segment that follows a request.

The book admits a limit: "Logs are not ready yet" (lesson 63) in OTel's Go SDK, its development toolkit. The book's example, for its part, stays minimal. It only opens one span, in one handler, the function that processes an HTTP request:

_, span := otel.Tracer("example-tracer").Start(r.Context(), "handleRequest")
defer span.End() // opens a trace segment, closes it when the function returns

8The capstone: a well-weighed distributed cache that never checks its copies

Last idea: the capstone leaves behind the single machine of the first seven. The book builds a distributed cache there. A cache keeps answers in memory to serve them again quickly, without redoing the work. The book's image: "a mini fridge next to your desk", so you do not walk to the kitchen every time you are thirsty (lesson 70). Distributed, this cache runs on several machines at once, which the book calls nodes.

The book's test tells the project in two requests. You start two nodes, on ports 8080 and 8081. You write the key foo, with the value bar, on the first node. You read it back on the second, which answers bar: the write has travelled from one node to the other (lesson 75).

To get there, the book adds three mechanisms:

• Eviction removes entries, because memory runs out. Two rules work together (lesson 73). The LRU rule (least recently used) removes the entry read the longest time ago, when the cache is full. The TTL rule (time to live) removes an entry that is too old.
• Replication copies each write onto the other nodes. It works peer to peer: each node sends its writes to all the others itself, with no leader node (lesson 75).
• Sharding gives each key an owner node, chosen by consistent hashing. A hash function turns a text into a number, and that number places nodes and keys on a ring. A key goes "to the first node it encounters while moving clockwise" (lesson 75). A node that receives a key owned by another node forwards it there (lesson 76).

This project is worth reading mostly for its trade-offs. Before each mechanism, the book compares the options and says why it picks one. For replication, it weighs four families on five criteria, among them fault tolerance and agreement between copies (lesson 74):

• Primary and copies: a single node receives the writes, and its failure blocks them all.
• Peer to peer: no single point of failure, but the copies often agree only after a delay. That is eventual consistency.
• Publish-subscribe: a relay broadcasts the writes, and becomes the fragile point.
• Consensus (Raft, Paxos): the copies stay in agreement, at the cost of latency and complexity.

The book keeps peer to peer, without selling it as perfect: "it is a reasonable approach to get things going" (lesson 74).

The code, though, lacks the rigor of those comparisons. The node answers "OK" to the client without waiting for the copies to go out. Each copy then leaves on its own, with no acknowledgment and no retry. In this code, a node that is down therefore loses for good the writes made in its absence. Here is the end of a send, in the replicateSet function (lesson 75):

_, err = client.Do(req)
if err != nil {
    log.Printf("Failed to replicate to peer %s: %v", peer, err)
}
log.Println("replication successful to", peer) // reached even after the failure

These five lines hide two defects. After the failure, no return stops the function: the log prints "replication successful" right after the error. The _ on the first line throws away the other node's response, so the body of that response, its content, is never closed. Yet the book's chapter on good practices teaches exactly that: an unclosed body "can keep underlying sockets open longer than necessary" (lesson 84).

Sharding, for its part, cannot be judged on the evidence: the text shows the ring's signatures, not their code (lesson 76). In plain terms: read this project for how it chooses, not for its code.