AI code review: papers and tools compared to my skill

Table of contents (7 sections)

When you change code, a colleague reads your changes before they go live. That is code review. I have AIs do it, with a tool I wrote for Claude Code, Anthropic's programming assistant. That tool is a skill: a folder of instructions that Claude Code reads before a specific task.

My skill, senior-review, launches several reviewers. A reviewer is an instance of Claude launched to read the code (you can also call it an "agent"). It does not know the author of the code, the person who wrote it (I explained why the reviewer must never be the author). Each defect a reviewer flags is called a "finding". I use this skill almost every working day: 236 reviews since August 14.

The skill has three settings, called tiers. "quick" is light, "standard" is the default, "deep" is supposed to be more rigorous, with more reviewers. It keeps a journal: for each review, it records how many findings were raised and how many pass verification. Verification demands proof for each defect and tries to refute the most serious ones, meaning to show that they are false.

The journal says otherwise. On "deep", 74% of the findings raised survive verification. On "quick", it is 89%. Apparently, I had paid more for reviewers that miss more often.

Those numbers made me want to check whether my skill is solid or a hack I happen to like. I launched two research agents: one in the scientific literature, the other among existing tools.

AI code review: my skill against the other tools

I compare my skill to other AI review tools. There is Anthropic's, commercial tools (CodeRabbit, Greptile, Qodo, Bugbot) and community skills. I kept nine criteria.

• Context: the whole repository (the project folder), its history, the code that calls the changed part.
• Independence: the reviewers do not know the author of the code.
• Proof: the reviewer shows how it knows the defect exists.
• Refutation: someone tries to prove the defect is false.
• Noise: few false alarms (defects flagged wrongly).
• Memory: the tool keeps track of its mistakes.
• Loop: fix then re-review, with a maximum number of rounds.
• Read-only: the tool cannot change anything in the code.
• Ticket: the tool reads the description of the work requested (the "ticket").

I added three criteria that were missing: externally validated measurement, integration and cost. Externally validated measurement: has someone other than the vendor tested the tool? Integration: does the tool work with no setup in "pull requests", the requests you post to have your changes reviewed?

Each column of the table is a tool or a group of tools, each cell a score from 0 to 5 (5 is the best). It is my judgment, based on the documentation I read. "Not documented" in a source does not mean "absent", and I scored accordingly. "1-2" covers the tools of the group. Look at proof (5 for my skill) and external measurement (2).

"Official Claude Code" covers the managed review service from Anthropic and its code-review extension. That extension launches four agents at the same time. Other agents then check the defects found, one per defect: an unconfirmed defect is dropped. Its documentation cites a confidence threshold of 80, which is no longer in the code.

My strengths are the required proof, the ticket, the loop with a limited number of rounds and the memory. My weaknesses: high cost, no native pull request integration, and above all weak external measurement. The commercial tools display benchmarks (numbered tests), but each vendor wins on its own measure, such as "comments resolved before merge". Those numbers cannot be compared. I cite none of them.

What the research says, and what it does not

The research agent opened the page of about thirty papers. The figures come from their abstracts, except one, taken from the body of the Cihan study. Several are "preprints", texts not yet reviewed by other researchers. Six results matter for a review skill.

• Give the reviewer the intent of the change. A study tested GPT-4o and Gemini 2.0 Flash. They classify 68.5% and 63.9% of code cases correctly (good or faulty) when given the problem description. Without it, they do worse: up to 24.8% of correct pieces of code receive wrong suggestions (Cihan et al., arXiv 2505.20206). My skill reads the ticket before the code.
• More instructions do not mean fewer false alarms. A "prompt" is the text of instructions given to the model. More elaborate prompts, which ask for explanations and fixes, make correct code get judged faulty more often (Jin and Chen, arXiv 2508.12358).
• A model does not correct itself. Without external feedback, self-correction does not help and sometimes hurts (Huang et al., ICLR 2024). It works with a reliable signal, such as an executed test (Kamoi et al., 2024). Hence my rule: no finding without proof. A fix is proven when undoing it turns a test red, meaning it fails.
• Several reviewers are not as many independent votes. A model family is a vendor and its models (Claude, GPT, Gemini). On a language task, nine models from seven families were made to vote. Their answers look so alike that they count for only about two independent votes. The study is a single-author preprint (Kohli, arXiv 2605.29800). My reviewers are all Claude models, so the same family: the effect should be worse for me.
• Debate is not better than voting. Making models debate does not reliably beat simple majority voting (Smit et al., arXiv 2311.17371). It can even degrade the accuracy of answers (Wynn et al., ICML 2025 workshop). My refutation looks like a debate: its value comes from my proofs, from no study.
• At equal cost, a single agent matches or beats a team. That holds on reasoning tasks, not code (Tran and Kiela, arXiv 2604.02460, preprint). Cost is counted in tokens. A model reads and writes text in small pieces, often a word or part of a word. Each piece is a token.

One last figure comes from a real tool, CodeRabbit. Out of 31,073 reactions to its reviews, across 239 repositories, 56.3% of the comments were rejected. They were mostly false alarms, repeats or off-topic remarks (Lin et al., arXiv 2607.03316). That is a single tool, but the order of magnitude is a warning: more than one comment in two ends up rejected.

What my journal actually contains

Here is the output of the learn command, which analyses the journal. It keeps only the reviews whose counts add up: 189 out of 236, of which 182 fall in the three tiers of the table. The last column gives the cost of a finding, in tokens.

The cost depends on the number of agents. A finding costs about 66,000 tokens with 1 agent, 50,000 with 4 to 5, then 87,000 from 6. Nothing shows that adding reviewers helps beyond five.

Be wary of these readings, though: the reviews are not about the same changes. A "diff" is the list of lines changed by a change. Its median size (the middle value) is 158 lines in quick against more than 1,000 in deep.

Single-agent reviews mostly re-read a small change, round after round. The comparison "one big reviewer against several", at an equal budget, remains to be done.

My real gap is elsewhere: the way to read that "74%". Here is the path of a finding.

To picture it, think of a dictation. A first corrector underlines words: the reviewers. A second one strikes out those it judges correct: the verification. The words still underlined look like mistakes. Yet the pupil, who is me, may refuse one: "it is a character's name, it is on purpose". Counting them measures the second corrector, not what the pupil accepts.

My journal did the same. Its "precision", the share of findings that survive verification, measured what my own filter lets through, not what I accept. A confirmed finding may well be rejected by me ("that is intended"). The 236 lines of the journal did not carry what became of each finding. A tool that grades itself with its own filter measures only that filter.

What I changed

The journal and the research made me change three things, published in the skill's public repository (version 1.26).

• What became of each finding. The journal now records how many confirmed findings were fixed, rejected, or moved to another ticket. The meaning of "precision" does not change. An "acceptance" figure is added: fixed divided by fixed plus rejected. An imagined example: 8 fixed and 2 rejected give 8 out of 10. No reviewer decides what becomes of a finding: the counts come from me or from the review loop.
• The reviewers' model is recorded, so I can later compare precision by model.
• The 5-agent cap becomes verifiable. The skill must not launch more than 5 agents per review. The learn command now flags the reviews that exceed it (11 since the cap exists, 28 over the whole journal). It also computes how many findings come out per 100 lines of diff.

A fourth change, version 1.25, comes from a book on agent architectures (how to organise several agents) that I was reading in parallel. A reviewer can crash or return an empty answer. The skill then relaunches it once. On the second failure, the report writes "not covered" next to the aspect of the review it was meant to check. The final verdict can no longer be "approved". Before, a reviewer that dropped out left a hole the report did not show.

A separate agent read the code for these changes. It found six defects, including a case where two simultaneous writes to the journal could make a line disappear. Each got a failing test before the fix.

What I did not do

I did not plug the skill into pull requests. It is interactive, built for my workstation: I launch it myself. Plugging it in would need a program that launches itself on every code push. It would need setup, passwords and a charge on every push. For a solo use, it is not worth it.

I did not add a reviewer from another model family either. It would need a second account, with the vendor of that other model. The study that justifies it is about a language task, not code.

What remains is the experiment the research asks for: one big reviewer against the group of reviewers, on the same diffs, at an equal budget. By my estimate it would cost about 300,000 extra tokens per trial review. I am not running it right away, because the journal first needs a few weeks of recorded outcomes so that the comparison speaks about acceptance and not about the filter.

What I do not know

My recall, meaning the share of defects a reviewer finds, is measured on only seven cases with injected bugs. An injected bug is a defect planted on purpose to test the reviewer. Depending on the bug, it ranges from 3 out of 3 to 1 out of 3. Seven cases settle only a coarse gap.

No verified paper gives the ideal diff size either. None gives the point from which several agents earn more than they cost. The token ratios I had read about teams of agents are found in none of the papers the research agent opened.

Conclusion

The most useful part of this research was neither a paper nor a tool. It was discovering that my own measurement graded itself with the filter it was supposed to judge. The only score that matters for a review is what the reader of the review does with the findings: fix them, or dismiss them. Everything else, the architecture included, is a hypothesis as long as that number is missing.

If you have an AI review your code, note what you do with each defect (fixed, rejected, moved to another ticket). Ten lines of log are enough to see whether your review is worth anything.

The book on agent architectures mentioned above has its page in my library. The skill is public: github.com/ohugonnot/claude-skills, senior-review folder, with the history of every version.