How to delegate code review to agents
How to feed an AI reviewer, test it on past PRs, and decide when a human should look.
Translated from French with AI. Read the original.
Imagine a CTO whose team of developers grows every day, and who plans to read every line of code themselves to stay in control. Nobody would believe that will hold up for long. Yet that is what many teams do with agents: they produce more and more code, and humans review all of it.
I think our role can no longer be to read every line. In the first post, I described this imbalance. In the second, I looked at what keeps us from delegating review. This one is about what comes next: how to go about it in practice.
Delegating means giving context
Delegating review is a real act of delegation, as with a colleague: you have to give the agent what it needs to succeed.
The agent starts with solid general knowledge. It has probably read, during training, most of the books on good design, architecture, security, readability and code smells. That knowledge exists, and you have to bring it out with the right questions. What it can’t guess is your project, your context, your team’s tastes.
Where to find material for a code reviewer
Three techniques complement each other:
- Past PRs. Let the agent go through all the old PRs, their comments and instructions, and draw applicable rules from them, with examples. The humans on the team have already written down what matters to them.
- Yourself. Sit down and write out everything that’s important, with your priorities. What you know about the project is often written nowhere, and the agent won’t find it on its own.
- Ongoing reviews. In every review you see what the agent missed, what it lacks. Add it to its knowledge base.
Testing a reviewer automatically
Testing a prompt means taking a case, running it, and analyzing the result. For a reviewer, the case already exists in the history.
Run it on past PRs. Pick old PRs that were reviewed in depth and replay the reviewer on them. Does it find at least what was flagged back then? That score can be computed. It’s what’s called a verifiable reward, and it’s what makes evaluation simple. You also look at what it finds in addition and what it misses, and judge how relevant the extra findings are. You can go further and let an agent adapt the prompt to correct itself on those cases. These PRs become a test base for improving the reviewer.
Run it alongside the human. For a while, the human and the agent review the same PRs, and you add to the reviewer whatever it’s missing.
Keep doing manual reviews until they add nothing more. It’s a way to reassure yourself that you have little left to say. The limit: as the level rises, you notice other details, and you can find them endlessly.
Several specialized reviewers that agree, and a fixer in a loop
For a deeper review, I split my single reviewer into several specialized reviewers, each with its own angle (complexity, tests, architecture, security, etc.), coordinated by a main agent. The sweep becomes much deeper. I also had to instruct it to look beyond the git diff: the context, the method, the whole file. It also needs fairly strict instructions so that it doesn’t stop after a few findings and looks for absolutely everything.
Then I introduced a fixer that loops with the reviewers until a high level of quality is reached. The reviewers see each other’s work and agree, through the main agent, on what needs fixing. This loop raises a few implementation problems, and I’ll probably write a post about that experience.
In my library, everything starts with review-all, the orchestrator. It reviews nothing itself: it pins the commit, launches five reviewers in parallel (complexity, tests, architecture, security, slop), then publishes their summary on the PR, with one comment and one status per reviewer. They all follow the same protocol: each finding is classified Critical, Major or Minor, with its file and line. If there are things to fix, the same reviewers approve the fix plan and then verify the result, without redoing a full review. Then the merge happens automatically if CI passes.
If you want to start from something that works, you can help yourself to the Unlockers Skills Library, which is open source. To try it, ask your own agent to read this set of skills and apply it to one of your PRs.
A risk score to decide where a human should look
Another technique: have the review produce a risk score, then let an approver decide whether to bring in a human. This scoring mechanism isn’t published in my library yet.
The question is how to score. Rather than a numeric grade, I recommend three or four levels, for example minor, major, critical. It’s more reliable with an LLM, and it’s already the vocabulary humans use in meetings to rank problems. Several scores can be aggregated to give an overall picture.
To catch errors in the scoring itself, a human also reviews a few randomly picked PRs, three for example.
A PR without bugs can be merged, design gets reviewed later
In a project like the Linux kernel, only the maintainers who know the project can merge. In my view, this met two needs: avoiding bugs and inconsistencies, and making sure the important details of the project are known to the person who decides.
Those two needs are now covered differently:
- Functional correctness is settled upstream. On some projects, it has become easy to reach near-total test coverage, protected by mutation testing.
- Knowledge of the project is something you supply to the agents. A reviewer that has it goes far beyond what a human can keep in their head.
A PR that passes these two filters probably has no bugs, so it can be merged. What a human eye still brings is design: architecture, bad patterns, code that turns sloppy. The SlopCodeBench benchmark has 15 agents extend their own code, iteration after iteration: structural erosion increases in 77% of trajectories, and their code is 2.3 times more verbose than that of 473 open-source repositories. It isn’t necessarily representative, because those agents have no quality harness there. In any case, the PR isn’t always the right place to deal with it. Fixing a detected drift can turn a small feature increment into a global refactoring of the codebase.
This design review can therefore happen on a delay, across several PRs, by looking at recent changes. I have an agent that does this kind of longer-term review: it flags bad smells, problems that appeared recently, and opportunistic refactorings to do given the work done.
A friend tried the experiment: review no longer blocks the merge. The developers had to ask themselves how to get quality another way, which called on their creativity.
My view: the role of humans remains important, but it is no longer played out mainly in a PR.
The cost of refactoring has become very low
There’s also an economic reason. Before, letting untidy code in was expensive: without complete tests, you couldn’t refactor it safely, so you had to block it at the door. With tests in place, changing an implementation costs almost nothing.
The other day, I took a 9,580-line codebase, entirely vibe-coded and without a single test. With a single prompt and 1h45 of autonomous agent work, I had it covered by tests, split into modules and cleaned of every bad pattern that automated tools detect (the details). Clinging by reflex to the old review rules no longer has any justification.
Going further
- Unlockers Skills Library: my review agents and skills, with their shared protocol.
Johan Martinsson
