AI agents on a production bug: two wrong leads, then 8 crashes out of 8

Table of contents (8 sections)

I run a server shared by about thirty members. Each of them runs their own rtorrent there, a command-line BitTorrent client. This Thursday, I handled a production bug with AI agents, from the outage to the fix.

The bug first. After an rtorrent upgrade, instances died one to three minutes after starting. They left no message, no crash file, nothing in the logs.

The bug is not the subject of this article, it is where it comes from. The subject is how to work with agents on a machine where other people keep their files. Who plans, who executes, who verifies? How do you prove before touching the members? Where the AI was wrong, where I was wrong, and what settled it in the end.

How "clean" became a plan

Before the outage, there was a plan, and that plan came out of a conversation. To follow it, you first need to know who I was talking to.

I did this work with Claude Code, Anthropic's tool where an AI runs commands by itself. An agent is an AI that works this way: it runs something, reads the result, then decides what comes next. The main session, the one I talk to, can launch subagents. Each one gets a written brief, works on its own and sends back a report.

That day, two models shared the roles:

  • Fable, the more expensive of the two, writes the plans and runs the hard analyses.
  • Sonnet executes: it writes the code and the tests from the plan. The main session also ran on Sonnet.

An instance, here, is one member's rtorrent, a process of their own. The server runs about thirty of them, all on the same program, which I wanted to move from version 0.9.8 to 0.16.28. My instruction to the AI fit in one sentence: "as fast as possible while doing it clean".

"Clean" means nothing to a machine. Instead of rushing in, the AI first sent questions back: how many instances, who owns them, what must never be lost, and how long a member can stay offline.

Those answers, it could not read on the server. The instances belong to the members, not to me. What must never happen is a member losing their list of torrents, or rtorrent rechecking all their files, hours of disk work for nothing. A few minutes of downtime, on the other hand, nobody will notice.

It turned those answers into mechanics. One instance at a time, so a mistake only hits one member. A backup before each switch, the moment an instance moves to the new version. A way back ready in a single command. A check script after each switch. It added an idea I didn't have: try the way back once, cold, before needing it. The split was clear. I knew what must not break, it knew how to check it.

About twenty instances went through this way, all green. Yet one question was left that neither of us had asked: green, but for how long? The instances answered for us: some of them started to die, without a word.

Two wrong leads, then one stubborn sentence

When a program dies without a message, there is nothing to read, so you guess. The AI guessed twice. It took a sentence with no technique in it to bring the hunt back to facts.

First lead: my disk regulator, a home-made service that slows instances down when the disk can't keep up. The AI checked before concluding. The regulator had applied nothing at that moment, so the lead fell.

Second lead: my account's login session, the group of programs the system ties to my connection to the server. The instances still alive all ran inside it. Its idea: when a session closes, the system takes down the programs started from it, and the dead instances came from another session, closed in the meantime.

This time, it didn't check. It had written in plain words "a serious hypothesis, not a certainty". It still restarted the dead instances in a separate session and changed the launch script in production. An unchecked idea had just modified a server shared by about thirty people. That is when I typed the sentence that changed everything:

"There has to be an error message going somewhere, in the rtorrent logs, the system logs or elsewhere. It doesn't just stop without saying anything."

This sentence is a method, the one an experienced sysadmin applies without thinking, and I interrupted the AI to put it back on that track. It comes in three steps:

  • A program never dies in silence. It always leaves a trace: an error message, an exit code, a signal recorded by the system.
  • If you see nothing, the way you are looking is broken. Before any theory, fix the observation.
  • Hypotheses come after the message, not before. A coincidence like "the survivors all live in my session" stays a lead until a trace confirms it.

The AI was doing the opposite: it started from coincidences to build an explanation. Once redirected, it asked the right question: why can't we see anything?

Every program has a channel set aside for its failure messages, the error output. But the instances ran under screen, a program that keeps a terminal open in the background, and screen swallows that channel. The message existed, but nobody could read it.

To fix the observation, it restarted four instances under systemd, the program that starts and watches the server's services. With systemd-run, each one becomes a temporary service, and systemd records everything it says in its journal, error output included. A little over a minute later, the journal showed the sentence that screen was hiding:

terminate called after throwing an instance of 'torrent::internal_error'
  what():  PollInternal::modify(): epoll_ctl(DEL) error for event: name:curl_socket fd:69 : Bad file descriptor
rt-diag3-<member>.service: Main process exited, code=killed, status=6/ABRT

The message was finally there. It still had to be understood.

A bug outside my field

That message meant nothing to me. This bug is not my field: I build applications, I never go down to the kernel or the network libraries. The AI is what taught me the problem. The analysis subagent of the next part took the message apart, then the session explained it to me step by step. Here is that explanation, as I ended up understanding it.

You need three notions, all resting on the kernel, the core of the system that handles files and network for every program. A descriptor is the number the kernel gives to each open file or network connection, here number 69. epoll is a watch list kept by the kernel. rtorrent registers its connections there so it gets woken up when one of them has data.

The third notion is libcurl, the library that speaks HTTP for rtorrent. A tracker is the server that puts in touch the people sharing the same file. rtorrent regularly sends it an announce, a message that says "I share this file, who else does?", over HTTP, through libcurl. To keep its epoll list exact, rtorrent asks libcurl to warn it at every close. To do so, it hands libcurl its own close function, through the CURLOPT_CLOSESOCKETFUNCTION option. libcurl must call it every time it closes a connection.

The trigger is a dead tracker, one that does not even answer the SYN, the first packet of a connection, the equivalent of a "hello". Here is the chain, in order:

Six steps. rtorrent opens connection 69 through libcurl and registers it in epoll. The tracker does not answer the SYN. Sixty seconds later, libcurl closes the connection without calling rtorrent's function. The kernel removes it from epoll. libcurl then asks rtorrent to remove it. rtorrent asks the kernel to remove a number that is already closed, gets an error and stops. rtorrent ① opens connection no. 69through libcurl, registers it in epoll tracker ② does not answerno reply to the SYN libcurl ③ gives up after 60 scloses 69 without calling rtorrent kernel ④ removes 69 from epollthe number is closed, rtorrent can't tell libcurl ⑤ then asks for the removal"remove 69 from your list" rtorrent ⑥ removes 69 from epoll: errornumber already closed, rtorrent stops Diagram made for this article, from the analysis report.
Step ③ is the bug: libcurl closes a connection that never got established without telling rtorrent.

rtorrent, for its part, does nothing wrong. It trusts what libcurl's documentation promises: to be warned of every close. And its code refuses any epoll error, because it would rather stop than carry on with a wrong list. The crash comes 60 seconds after the announce, the time it takes libcurl to give up: that timeout is set in rtorrent's code.

So an instance whose trackers all answer never crashes. The mystery from the start finally made sense: some instances held and others did not, depending on their trackers. The one I later used as a guinea pig had six dead ones.

With this mechanism in mind, I could finally follow the discussion. The AI did not hand me a solution. It put four options on the table, with what each one costs:

  • Go back to the old version, 0.9.8. Safe and immediate, but we give up the upgrade.
  • Tune the kernel so it gives up sooner on unanswered connections (tcp_syn_retries). No code to write, but nothing guarantees it is enough.
  • Rebuild libcurl with the line the curl authors have already fixed. It cures the cause, but it touches a library the whole system depends on.
  • Fix libtorrent, rtorrent's library, so it absorbs the silent close of step ③. The change stays inside rtorrent, and we remove it the day Debian, the server's system, ships a fixed libcurl. That was its recommendation.

I did not need to become an epoll expert. I needed to understand enough to judge what each option touches, what it risks and how we roll it back. That much, the AI taught me in one conversation. I chose to fix libtorrent. It was the AI's recommendation, but the choice was mine, made knowingly, and on one condition: that the fix be proven before touching the members' instances.

Two things remained to prove: that the explanation was right, and that the fix held.

Search the code, then prove it

That explanation and those options did not fall from the sky. They came out of a single request, typed as soon as I had the error message:

"Go search the internet and the source code for what's going on, and analyse it with Fable."

The session turned it into a four-point brief for a Fable subagent: read the code, search what the curl and rtorrent authors know, write a fix, prove it on an isolated test bench. It worked in a separate folder, away from the members' instances. Less than an hour later, its report said three things: what it had found, what it had proven, and what it had not proven.

What it found

The bug comes from libcurl, and it is known. The curl authors fixed it with one line in July 2023 (PR #11439, shipped in curl 8.2.0). But Debian 12 still ships a libcurl from before that fix, 7.88.1. On the rtorrent side, an open issue reports our exact message, and nobody has found the cause.

The report also listed what it had not found: no newer rtorrent version that solves it, no published fix, no configuration option. That list is worth as much as the cause, because it saves me from hunting for a magic option that does not exist.

What it proved

A convincing explanation was not enough: the session had just given a wrong one. So the test bench rebuilt the trap at small scale, on a private network: a fake tracker with two addresses that never answer, and 300 dummy torrents calling it every 15 seconds. Each run answers a single question:

RunThe questionResult
original rtorrentDoes the bug reproduce?8 crashes out of 8, with the production message
fixed rtorrent (libtorrent option)Does the fix hold?30 min 36 s with no error, 30,397 announces
original rtorrent, libcurl fixed by one lineIs the cause the right one?16 min without a crash
original rtorrent, tuned kernel (tcp_syn_retries=3)Is the tuning enough?3 crashes out of 3

The third row is the most important. It does not test the fix, it tests the cause: rtorrent unchanged, one single libcurl line changed, and the crash disappears. The fourth closes the door on kernel tuning, the option that needed no code.

The chosen fix is about a hundred lines. libtorrent records each connection it hands to libcurl. If libcurl closed it on the sly, libtorrent forgets it instead of crashing, and keeps its strict check for everything else.

What it did not prove

The report ended with its own limits. I copy them here, because they say how far it can be trusted:

  • no run with real trackers or real peers;
  • one of the fix's safety checks never came into play during the runs;
  • half an hour of uptime, not days.

So the report drew the line: proven on a test bench, not yet in production. The next step had to reach production without betting the whole server.

A guinea pig before everyone else

The bench proved the fix at small scale. For production, we took back the upgrade's rule: one instance at a time, and an immediate way back. A simple list decides which instances run the fixed version, and removing one line is enough to roll back.

The first one on the list was a guinea pig picked on purpose: the instance with six dead trackers, the most exposed to the bug. My rule: the others move to the fixed version only if that one holds. As I write, on Thursday November 19 at 12:32, it has been running for more than ten minutes, well past the 60 to 90 seconds where it used to die. The rest waits.

This time, the question forgotten at the start, green but for how long, had its answer: ten minutes, on the most exposed instance, before touching the others. It should have been asked from the upgrade on.

The instruction I should have given

Looking back, our "clean" had three holes. The bug had nothing to do with them: we could have filled them before starting anything.

  • "Clean" did not say how to tell an instance that works. We had defined what must not be lost, nothing more. The check took the simplest test, the torrent list reloaded at startup. But the bug kills one to three minutes later, so its green light proved nothing.
  • Nothing said when to stop. "As fast as possible" asked neither to wait between two switches nor to stop at the first dead instance.
  • Nothing said how to see. Under screen, a dead instance left no trace.

The corrected plan is only a few lines longer:

"Switch a single instance first, the most exposed one: lots of torrents, lots of trackers. Run it under systemd so its error output is kept. It went through cleanly if it is still running after ten minutes and its journal holds no error. Only then, switch the others in small batches, with the same check. At the first dead instance, stop everything and put the old version back."

Each hole now has its answer. "Clean" has a test the AI can check on its own: ten minutes with no error in the journal. The stop is written down: at the first dead instance, roll back. And systemd keeps the error output, so we can see why an instance dies.

With this instruction, only one instance would have died, and its error message would have been there from that first crash. Slowing down on the first instance would have been the fastest path.

Conclusion

Technique did not find the cause. A sysadmin's sentence did, "it doesn't stop without saying anything", along with systemd, which kept the error output. As soon as the AI had that message, a fact instead of an idea, it did everything else, fast and well.

Before that message, it was guessing. That is where it is dangerous, because it guesses in the same calm voice as when it knows: its "serious hypothesis, not a certainty" had already changed a production script. Where being wrong is cheap, on the other hand, it shines: reading unfamiliar code, digging through an issue, setting up a test bench.

So my job was not to type the commands. It was to refuse the explanation that did not fit, to demand the error message, to pick the fix knowing what I was choosing, then to demand proof before deploying. That part does not show in the result. I wrote about it in "The AI did it".

And the proof is worth more than the fix. The fix will be thrown away the day Debian ships a fixed libcurl. The proof stays: one line changed in libcurl, and the crash disappears. An explanation, even a right one, can be argued with. A run that goes from red to green cannot.

Sources

Documentation consulted on November 19, 2026:

In the Developer's Great Book

A free book where each idea is told once, in its place, from silicon to human judgment.

Comments (0)