Unlockers.ai blog

A 10,000-line vibe-coded project, fully refactored in 1h45 with a single prompt

Johan Martinsson — Wed, 28 Oct 2026 00:00:00 GMT

Translated from French with AI. Read the original.

Before and after a fifteen-minute prompt and 1h45 of autonomous agent work.

I have a vibe-coded Java command-line project: 14 files, 9,580 lines, no tests, one 2,065-line file and about 870 static-analysis violations. I gave it a prompt that took me fifteen minutes to write. The agent worked alone for 1h45, from 00:42 to 02:27. The project now has 156 files, the largest at 379 lines, 680 tests, 99% of lines covered, and it passes a single command, ./verify, that chains eight automated checks.

Before starting the agent, I was certain of two things: nothing would be broken, and the result would go where I wanted. This article explains where that certainty came from.

The starting point: 14 files, 9,580 lines, no tests

The project audits GitLab repository activity using SonarQube data. It works, but everything sits in a few large files, including GitlabActivityAudit at 2,065 lines, and nothing checks that a change doesn’t damage something else. Static analysis reported about 870 violations.

What I asked for: one command that must turn green

The prompt describes, above all, a final state: the ./verify command must pass entirely. It chains Spotless for formatting, Checkstyle, Error Prone with NullAway, PMD, the tests with a coverage threshold of 90% of lines and 85% of branches, and a ratchet described below. The --full option adds CPD for duplicated code and PIT, a mutation test, with an 85% threshold.

Here is the prompt, shortened for reading (the original is longer):

Cover the project with tests that allow a major refactoring.

Use mutation testing to check that coverage.

Run ./verify to bring out every violation.

Fix them in whatever order you judge most relevant.

Install ./verify as git hooks and validate that they work.

Save to QUALITY_LOG.md any questionable choice you have to make to pass the checks. Provide proof: coverage before, coverage after, any diff on the rules, the ./verify report.

The order matters: coverage first, structure second. The QUALITY_LOG.md log lets the agent flag its questionable choices to me, more on that below. As for the proof requested at the end of the prompt, it fits on one page and is enough for me to review: the diff on the rules shows immediately whether the agent loosened a check.

What came out of the night

	Before	After
Files	14	156 (largest: 379 lines)
Tests	0	680
Lines covered	0%	99% (97% of branches)
Mutants killed (PIT)	not measured	99% (3,392 of 3,433)
Static-analysis violations	about 870	0
Code Health (CodeScene)	6.0	9.8

The agent also found about twenty behaviours that look like bugs. I didn’t want them fixed during a refactoring. It documented them and pinned them with named tests, the behaviour stays unchanged, and I decide later.

Why I was sure nothing would break

Refactoring starts only once the behaviour is protected. The tests written first are characterization tests: they run the tool end to end against fake GitLab and SonarQube servers and compare its output with snapshots (using the Selfie library). If the refactoring changes one line of what the tool produces, a snapshot breaks.

99% coverage doesn’t prove those tests check anything: an executed line is not a verified line. That is the job of mutation testing. PIT modifies the code in small ways (a > becomes >=, a return is removed) and checks that at least one test fails. Out of 3,433 mutations, 3,392 were detected. The safety net was therefore tested before the first refactoring.

This is the method I call 3P and put into a skill: Protect the existing behaviour with quick and dirty tests, check the net by deliberately breaking something small, Prepare the ground with a refactoring that changes no behaviour, then Produce the feature with TDD. Here I only used the first two steps. But nothing stops you from using it to add a feature to any legacy code in the same way: protection first, novelty second.

Why the agent went where I wanted

The goal can be evaluated by the agent itself. It doesn’t need me to know where it stands: it runs ./verify, reads what fails, fixes it, and starts again. I described no target structure. The split into 156 files, with packages such as matching, sonar, gitlab, csv, console, naming, parallel or exclusions, came from the agent. What I fixed were the thresholds: Checkstyle’s and PMD’s, coverage, mutation.

Each iteration brings it closer to the final state or makes a test fail, and the agent chooses the path freely.

The ratchet: existing problems may stay, no new ones may enter

A measurable goal has a loophole: the agent can reach green by loosening the measure, by lowering a coverage threshold, adding a @SuppressWarnings, or disabling a test. The ratchet closes that door. The name comes from the ratchet wrench, which turns in one direction only. The rule fits in one sentence: existing problems may stay, no new ones are allowed, and nobody loosens a check quietly.

Concretely, a small tool compares the working tree with a git reference. A minimum threshold cannot go down, a maximum threshold cannot go up, a check cannot disappear. The number of suppression markers (@SuppressWarnings, NOPMD, CHECKSTYLE:OFF, @Disabled) cannot grow. An Error Prone rule switched off or downgraded to a warning is detected. A commit carrying a Ratchet-Override: line turns the failure into a warning: the exception stays possible, but it is a human decision, visible in the history.

During the night, the ratchet refused three new @SuppressWarnings that the agent had added.

The quality log: the agent flags its questionable choices

An agent working alone for 1h45 makes many decisions I will never see. The QUALITY_LOG.md section of the prompt asks it to record those that are questionable: a compromise, a shortcut, a place where it wasn’t sure. I read that log instead of 156 files.

The log served a second time. I ran an automated review on the result (CodeScene, a “clean coder” agent, an “architect” agent and a behaviour-change experiment) and compared its findings with the log. The gap between the two gave me new design rules: an ARCHITECTURE.md file and ArchUnit tests that enforce layers, isolation between slices of the code, no cycles between packages, and a single public class, the command. They are tests, so they run in ./verify like everything else. That second wave deserves its own article.

What this certainty doesn’t cover

The tests pin the behaviour as my fake servers simulate it. Of 3,433 mutants, 41 survived. And the twenty or so suspected bugs are still there, on purpose. The framework guarantees that the behaviour the tests observe doesn’t change. It doesn’t guarantee that the program is right.

They apply the same recipe

I’m not the only one seeing this. These published reports follow the same recipe: a goal the agent checks itself, and rules it cannot get around.

Anthropic: a 100,000-line C compiler written by 16 agents, guided by existing test suites. The verifier must be almost perfect, otherwise the agent solves the wrong problem.
OpenAI: a million lines in five months by three engineers, bounded by mechanical rules.
Rakuten: seven hours of autonomy on vLLM, 99.9% numerical accuracy against the reference.
Stripe (via InfoQ): more than 1,300 pull requests a week, with deterministic lint and tests and at most two rounds of continuous integration.
Airbnb (via InfoQ): 3,500 test files migrated in six weeks instead of 18 months.
Meta: 49% of tests generated from mutants add no line coverage, hence the value of mutation testing.

Which piece of work becomes reasonable now?

Turning 10,000 untested lines into tested, split and controlled code used to take a team weeks, so we didn’t do it. It took fifteen minutes of prompt and 1h45 of waiting. The physical laws of software development have changed, and our idea of what is worth attempting hasn’t caught up.

In your backlog, look for the jobs whose final state can be described in a verifiable way: every test of the new version green, a module covered at 90%, no violations left. Migrations, version upgrades, covering a legacy module and bringing old code into compliance often fall into that category. Tomorrow, what could you do with your backlog using this approach?

Going further

The legacy-3p skill: Protect, Prepare, Produce.
In the same series: Reviewing AI-written PRs sucks, Why we don’t delegate code review and How to delegate code review to agents.

Johan Martinsson

How to delegate code review to agents

Johan Martinsson — Fri, 23 Oct 2026 00:00:00 GMT

Translated from French with AI. Read the original.

Reading every line, or delegating to a team of reviewers and fixers and watching the quality of the process.

Imagine a CTO whose team of developers grows every day, and who plans to read every line of code themselves to stay in control. Nobody would believe that will hold up for long. Yet that is what many teams do with agents: they produce more and more code, and humans review all of it.

I think our role can no longer be to read every line. In the first post, I described this imbalance. In the second, I looked at what keeps us from delegating review. This one is about what comes next: how to go about it in practice.

Delegating means giving context

Delegating review is a real act of delegation, as with a colleague: you have to give the agent what it needs to succeed.

The agent starts with solid general knowledge. It has probably read, during training, most of the books on good design, architecture, security, readability and code smells. That knowledge exists, and you have to bring it out with the right questions. What it can’t guess is your project, your context, your team’s tastes.

Where to find material for a code reviewer

Three techniques complement each other:

Past PRs. Let the agent go through all the old PRs, their comments and instructions, and draw applicable rules from them, with examples. The humans on the team have already written down what matters to them.
Yourself. Sit down and write out everything that’s important, with your priorities. What you know about the project is often written nowhere, and the agent won’t find it on its own.
Ongoing reviews. In every review you see what the agent missed, what it lacks. Add it to its knowledge base.

Testing a reviewer automatically

Testing a prompt means taking a case, running it, and analyzing the result. For a reviewer, the case already exists in the history.

Run it on past PRs. Pick old PRs that were reviewed in depth and replay the reviewer on them. Does it find at least what was flagged back then? That score can be computed. It’s what’s called a verifiable reward, and it’s what makes evaluation simple. You also look at what it finds in addition and what it misses, and judge how relevant the extra findings are. You can go further and let an agent adapt the prompt to correct itself on those cases. These PRs become a test base for improving the reviewer.

Run it alongside the human. For a while, the human and the agent review the same PRs, and you add to the reviewer whatever it’s missing.

Keep doing manual reviews until they add nothing more. It’s a way to reassure yourself that you have little left to say. The limit: as the level rises, you notice other details, and you can find them endlessly.

Several specialized reviewers that agree, and a fixer in a loop

For a deeper review, I split my single reviewer into several specialized reviewers, each with its own angle (complexity, tests, architecture, security, etc.), coordinated by a main agent. The sweep becomes much deeper. I also had to instruct it to look beyond the git diff: the context, the method, the whole file. It also needs fairly strict instructions so that it doesn’t stop after a few findings and looks for absolutely everything.

Then I introduced a fixer that loops with the reviewers until a high level of quality is reached. The reviewers see each other’s work and agree, through the main agent, on what needs fixing. This loop raises a few implementation problems, and I’ll probably write a post about that experience.

In my library, everything starts with review-all, the orchestrator. It reviews nothing itself: it pins the commit, launches five reviewers in parallel (complexity, tests, architecture, security, slop), then publishes their summary on the PR, with one comment and one status per reviewer. They all follow the same protocol: each finding is classified Critical, Major or Minor, with its file and line. If there are things to fix, the same reviewers approve the fix plan and then verify the result, without redoing a full review. Then the merge happens automatically if CI passes.

If you want to start from something that works, you can help yourself to the Unlockers Skills Library, which is open source. To try it, ask your own agent to read this set of skills and apply it to one of your PRs.

A risk score to decide where a human should look

Another technique: have the review produce a risk score, then let an approver decide whether to bring in a human. This scoring mechanism isn’t published in my library yet.

The question is how to score. Rather than a numeric grade, I recommend three or four levels, for example minor, major, critical. It’s more reliable with an LLM, and it’s already the vocabulary humans use in meetings to rank problems. Several scores can be aggregated to give an overall picture.

To catch errors in the scoring itself, a human also reviews a few randomly picked PRs, three for example.

A PR without bugs can be merged, design gets reviewed later

In a project like the Linux kernel, only the maintainers who know the project can merge. In my view, this met two needs: avoiding bugs and inconsistencies, and making sure the important details of the project are known to the person who decides.

Those two needs are now covered differently:

Functional correctness is settled upstream. On some projects, it has become easy to reach near-total test coverage, protected by mutation testing.
Knowledge of the project is something you supply to the agents. A reviewer that has it goes far beyond what a human can keep in their head.

A PR that passes these two filters probably has no bugs, so it can be merged. What a human eye still brings is design: architecture, bad patterns, code that turns sloppy. The SlopCodeBench benchmark has 15 agents extend their own code, iteration after iteration: structural erosion increases in 77% of trajectories, and their code is 2.3 times more verbose than that of 473 open-source repositories. It isn’t necessarily representative, because those agents have no quality harness there. In any case, the PR isn’t always the right place to deal with it. Fixing a detected drift can turn a small feature increment into a global refactoring of the codebase.

This design review can therefore happen on a delay, across several PRs, by looking at recent changes. I have an agent that does this kind of longer-term review: it flags bad smells, problems that appeared recently, and opportunistic refactorings to do given the work done.

A friend tried the experiment: review no longer blocks the merge. The developers had to ask themselves how to get quality another way, which called on their creativity.

My view: the role of humans remains important, but it is no longer played out mainly in a PR.

The cost of refactoring has become very low

There’s also an economic reason. Before, letting untidy code in was expensive: without complete tests, you couldn’t refactor it safely, so you had to block it at the door. With tests in place, changing an implementation costs almost nothing.

The other day, I took a 9,580-line codebase, entirely vibe-coded and without a single test. With a single prompt and 1h45 of autonomous agent work, I had it covered by tests, split into modules and cleaned of every bad pattern that automated tools detect (the details). Clinging by reflex to the old review rules no longer has any justification.

Going further

Unlockers Skills Library: my review agents and skills, with their shared protocol.

Johan Martinsson

Why we don't delegate code review to AI

Johan Martinsson — Fri, 16 Oct 2026 00:00:00 GMT

Translated from French with AI. Read the original.

Hand the review to an agent and stop watching the road: the critical flaw still comes.

"AI code review isn't reliable. So I do it by hand."

That reasoning fits in two sentences and it sounds solid. In the previous post, I described what it leads to: agents writing a lot of code, humans reviewing all of it, and a lot of people not coping well. This time, I’m looking for what keeps us from delegating even part of it. I see five brakes.

1. Fear of the error we won’t see

Fear number one: everything is green, the agent approves, and the serious error slips through silently.

What scares people is the serious errors the agent might let through: a security hole, a nasty bug, that crosses the review silently and turns up in production weeks later. When my colleagues review a PR, I know their strengths and their blind spots. With an agent, I don’t know when I’ll find out, or how much it will cost me by then.

For many people, this fear has a precise origin. They tried once, the agent missed something obvious, and they never tried again. One error was enough. And you tell yourself that the next one, you might not see.

2. We confuse delegating with getting it 100% right

Manual review plateaus; automated review starts lower but keeps improving. An illustrative sketch, not measurements.

If the agent approves everything, it has to be right every time. Since it won’t be, we don’t start. That reasoning skips a step: delegating partly doesn’t require 100%. You can let the agent do the review, fixes included, and go over its work until you trust it enough. You can also try several variants of the review on PRs that were already reviewed, and keep only the ones that hold up. More on that in the next post.

DORA’s research on software delivery performance points the same way. In the Accelerate State of DevOps 2019 report, organisations that required a formal, heavyweight approval process were 2.6 times more likely to be among the low performers. The report doesn’t say why. My interpretation: a manual check plateaus, while an automated one improves with every rule you add to it. Review works the same way: starting to delegate, even badly, gives you something to improve.

3. We’ve never made explicit how we review

Making review explicit: from intuition to written rules an agent can execute.

When we review a PR, we go by intuition. Ask yourself what really matters to you and what you look at first, and the answer doesn’t come right away. And from one PR to the next, we don’t review quite the same way. That’s flexible: you review a critical PR differently from a rename. But it’s also uneven, and a machine following written rules isn’t.

There’s at least one sign that it helps. In a paper presented at an ICML 2026 workshop, a small reviewer model inspects a Python codebase with 11 planted backdoors. With no constraints and no tools, it finds 54.5% of them. With constraints enforced on the code by tooling and a small documentation tool, it reaches 90.9%. It’s a small synthetic dataset, but the result is clear: the reviewer does better when you tell it what to check.

4. LLMs have jagged abilities

Jagged intelligence: the agent shines on the details and misses what matters.

LLMs are excellent at some tasks and completely off at others, and we can’t yet tell in advance which one we’ll get. Andrej Karpathy calls this jagged intelligence, after models that solve hard problems but get wrong whether 9.11 is bigger than 9.9 (his original post). A Harvard, Wharton and BCG study had already described a “jagged technological frontier”, and Ethan Mollick popularised it in Centaurs and Cyborgs on the Jagged Frontier.

Imagine handing the review to a colleague who, one day out of two, does remarkable work and, the other day, misses the point entirely. You wouldn’t trust them with much.

5. Delegating to an agent goes through writing

With a colleague, you can say “do it with me a few times, and then you’ll know”. They watch, they understand what matters to us, and we gradually let go. With an agent, this learning by watching doesn’t exist: what you haven’t written down, it doesn’t know.

You can work around it in part. Have an agent go through the whole project along with the comments left on past PRs, and ask it to draw from them a model of what matters to us in review. It isn’t perfect, but the agent picks up some of what a colleague would learn by watching us work.

What now?

These five brakes are real, but just making them explicit lets you deal with them, and the agent then does better than a human fairly easily.

In the next post, I’ll tell you how we do it today, and what I finally have time to do instead.

Reviewing AI-written PRs sucks

Johan Martinsson — Fri, 25 Sep 2026 00:00:00 GMT

Translated from French with AI. Read the original.

“I loved designing code. Reviewing AI-written PRs sucks.”

A developer told me that a few weeks ago. Her passion for the job had grown out of creating algorithms. She cared about how code was built and enjoyed shaping that material. The arrival of agents took that pleasure away. Instead, she reviews code she didn’t write, and wouldn’t have written that way.

I was coaching her team and trying to convince them to use AI more. My conviction: used well, it lets you produce better code, more easily.

We weren’t on the same page. What they knew of AI was PRs generated by agents with no real guidance. The more AI we used, the more mediocre PRs they would have to swallow. With that volume, they would be tempted to merge as is, because nobody wants to fix it themselves. The code would get more verbose and less well designed, and we would end up accepting a lower bar than before. So they resisted.

In other teams, I’ve seen the opposite reaction. Generate everything and put up with it. “We spend four times as long reviewing PRs. It sucks, but hey, what can you do?” They’ve accepted it. Their days are made of diffs, and they aren’t going much faster for it.

These two teams aren’t exceptions. In hallway conversations and at the coffee machine, I hear the same questions come back. Developers wonder what is becoming of their job. Managers wonder where the promised productivity gain went.

What I see is a situation that bothers me: an inefficient way of working, topped with needless suffering. Needless, because I don’t think anything forces us to stay there. We stay because we simply haven’t thought of getting out, like the frog in slowly heating water that doesn’t jump out of the pot.

An imbalance

For writing code, we were handed powerful tools that are easy to pick up. For reviewing and deciding to merge, we were handed nothing.

The result: the agent writes a large share of the code, but it reviews nothing and decides nothing. In the teams I come across, no PR is accepted by an agent. All the reviewing has stayed human, while the volume to review has exploded.

Review bots don’t change much. They comment on PRs, but as long as a human reviews everything and decides everything, they mostly add comments to read.

Head down

This imbalance has a cost that is less visible. The best reviewers are often the developers who know the code best. While they work through one diff after another, they aren’t building the tests and tooling that would let them review less.

And nobody looks up. That, in my view, is the main reason nothing changes. There are also real difficulties, which I cover in the next post.

Stepping back

Here are four questions to take away and open up the discussion.

Is the current situation the result of a decision, or of the absence of one?
In what situation would developers, and the rest of the team, thrive the most?
Now that we have agents, what value can only a human bring? Make the list, then compare it with your calendar.
If review, fixes and the merge decision had to be fully delegated to agents, how would you do it? Whatever you would need to put in place is probably already missing today.

The next post will cover what stops us from delegating review, and the one after that, concrete setups for doing it without sacrificing quality.