Making AI code review care about correctness

First version of an AI pull request reviewer is easy to imagine. Take the pull request diff, give it to a model and ask for a review.

That works well enough for a demo/test, it can summarize the change, point at obvious mistakes and produce something that looks like a review. But it does not work well enough for the kind of pull requests where you actually want help with.

The problem is not that the model cannot read a diff, but a common problem is that many pull request issues are not in the actual diff by themselves. They quite often appear between the changed code and the code around it, for example a helper below the changed function, an existing repository pattern, a platform edge case or an abstraction that is only visible if you check more than the change.

That is why It’s more about the AI review being less as a prompting problem and more as a context problem.

The project is a Python-based Azure DevOps PR reviewer. It runs from a pipeline, resolves pull request range, collects repository context, applies repository-specific review guidance, asks a model for a review, runs a second verification pass and posts the verified review back to the PR.

The important part is not the actual LLM at the end but rather everything that happens before the model is asked to review anything.

What I wanted from the tool

I did not want a bot that comments on every possible style preference, but a reviewer that was useful enough to read and actually saves time in the review phase.

That pushed the design towards a few constraints:

prefer correctness over comment count
use repository context instead of raw diffs only
make the review advisory, not merge-blocking
show when context was skipped or truncated
keep the pipeline thin and put review behavior in code
separate review generation from review verification

Those choices make the tool less impressive in screenshots. It might say there are no high-confidence issues instead of generating a long review-shaped answer. But that is the point, since in a pull request review a weak AI comment is not free. Someone has to read it and then decide whether it is real and then remember whether the tool is worth trusting next time.

Diff-only review was the baseline

The obvious baseline looks like this:

pull request diff
-> model
-> review comment

And this baseline can be already somewhat useful in some cases. It can catch the simple mistakes, suspicious conditions, copy-paste bugs and obvious test gaps. And it can also be cheap to build.

But the true weakness is that the model is forced to infer too much. If it does not know the repository, it has to guess whether a pattern or a choice is intentional. If it cannot see the lower level abstraction, it may complain about something that is handled elsewhere. If it cannot see similar code, it may miss that the new change violates a repo specific convention.

This creates two bad outcomes:

false positives that waste reviewer attention
missed issues that were only obvious with repository context

Both matter in the end, a tool that complains too easy will get ignored in the long run. Also a tool that only reviews the visible patch misses exactly the kind of issue I needed it to catch.

Some of the recent benchmark work points in the same direction. SWR-Bench is built around pull request level review with project context, not only isolated chunks. That was close to the problem I kept seeing in practice. Reviewing a chunk is not the same as reviewing the change.

The shape of the reviewer

The idea was to make the reviewer as modular and easily integratable as possible, so the easy answer was a CLI tool rather than a long-running service. In this case, integrated into Azure DevOps repos where some service hooks can trigger the pipeline when a pull request changes and the pipeline then runs the reviewer command.

The pipeline is intentionally, very simple. it just checks out the reviewer code, loads the runtime configuration and starts the CLI. The actual scripts and CLI parts own the actual review logic.

The flow looks like this:

pull request event
-> pipeline run
-> resolve PR metadata from Azure DevOps
-> clone or refresh the repository
-> calculate the real Git diff range
-> collect repository context
-> apply review profile
-> build bounded evidence packet
-> reviewer model pass
-> verifier model pass
-> post Markdown comment to the PR

A detail I cared about early was treating the webhook payload as an identifier and not as the full source of truth. The event says that a pull request changed, then the reviewer calls Azure DevOps to resolve the repository, source branch, target branch, PR status and commit metadata.

That distinction makes the rest of the tool easier to reason about. The review starts from the real pull request state and not from whatever happened to be inside the event payload.

Real Git range first

Before collecting context or calling a model the reviewer needs to know what actually changed.

For an active pull request the useful range is usually from the merge base to the source tip. For a completed pull request the merge metadata is better when it is available, because branches can move, disappear or be force-pushed after the PR is completed.

The simplified flow is:

resolve PR metadata
-> fetch source and target refs
-> compute merge base when needed
-> diff base to head
-> collect changed files and statuses

This part is not really special, but it matters a lot if the Git range is wrong, the model is reviewing the wrong change. Once this was handled properly the reviewer was no longer looking at whatever the checkout happened to contain. It was looking at the actual pull request range.

Shallow history was one of the annoying practical details. A shallow fetch is fast in a pipeline but it may not contain enough parent history to calculate a merge base. The tool uses a bounded fetch first and falls back to deeper history when the merge base cannot be resolved.

That gives the normal case a fast path without making the review “silently” wrong.

The evidence packet

After the Git range the harder question is what the model should see.

Instead of handing the model only the patch the reviewer builds a bounded evidence packet:

pull request title and description
real Git range
changed files and statuses
diffs
surrounding code around changed hunks
related files from the same area
likely tests
symbol references when cheap to find
warnings about skipped or truncated files
repository-specific review guidance

This changed the review quality more than prompt tweaking did.

Repository context helps the model answer questions that are not possible from the diff alone:

Is this a new pattern or an existing one?
Is the behavior handled in a helper below the changed code?
Is there a test file that usually changes with this source file?
Is this repository sensitive around a specific boundary?
Was a file skipped or truncated because of size limits?

The context builder is still heuristic and not a whole application analysis. Being an LLM, it can’t always prove that the change is absolutely correct, but even heuristic repository context is a large improvement over raw diff review, because it gives the model local facts to reason from instead of asking it to fill gaps with guesses.

The evidence packet is also bounded. Large files, generated files, lockfiles and oversized diffs can waste the entire prompt budget. So the tool skips or truncates those deliberately and the review output says when that happened.

A partial review should look partial and if the model did not see a file, the final comment should not pretend that it did.

There is also a “trap” here: more context is not automatically better. SWE-PRBench found that tested models still missed most human-flagged pull request issues and that larger context setups could perform worse when the extra material diluted attention. That matches the practical shape of the tool. The packet should be selected evidence, not a repository dump.

Review profiles made it less generic

The next useful piece was repository-specific review guidance.

Different repositories need different review pressure. One repository may have a fragile public API, another may have generated files that should almost always be ignored and then one could have authorization-sensitive paths, pipeline conventions or test patterns that are easy to miss from the changed file alone.

I put that steering into review profiles instead of baking it directly into the main prompt.

A profile can be customized per repo, for example these mock review profiles:

{
  "default": {
    "focus": [
      "Prefer correctness, security and reliability findings over style comments.",
      "Put missing tests under test gaps unless a concrete defect is proven.",
      "Call out skipped or truncated context instead of guessing."
    ],
    "max_review_files": 70,
    "max_changed_files": 150,
    "max_total_diff_size": 200000,
    "max_prompt_size": 250000,
    "skip": [
      "**/generated/**",
      "**/dist/**",
      "**/coverage/**",
      "**/*.snap"
    ]
  },
  "repositories": [
    {
      "match": ["backend"],
      "focus": [
        "Focus on request handling, authorization boundaries, data validation and error handling.",
        "Check whether changes affect API behavior, persistence logic or background job execution.",
        "Look for risky changes that should have focused tests or migration coverage."
      ],
      "skip": [
        "**/generated/**",
        "**/bin/**",
        "**/obj/**"
      ]
    },
    {
      "match": ["frontend"],
      "focus": [
        "Focus on state handling, user-visible behavior, form validation and API integration.",
        "Check whether loading, empty, error and permission states are still handled correctly.",
        "Look for changes that may break routing, caching or shared component behavior."
      ],
      "skip": [
        "**/generated/**",
        "**/storybook-static/**",
        "**/coverage/**"
      ]
    },
    ...
  ]
}

The point is not that a JSON file automatically makes the model obey local rules, but that a human reviewer does not approach every repository with the same mental checklist. The profile gives the model a small amount of that local review posture before it writes anything.

That made the tool feel much less generic. The comments became more relevant to the repository being reviewed instead of sounding like the same checklist pasted onto every pull request.

Correctness mattered more than comment count

The prompt is made deliberately defensive.

The reviewer is told to separate high-confidence potential issues from warnings, open questions, refactor suggestions and test gaps and missing tests are not allowed to become bugs by themselves. Plausible concerns without enough evidence are pushed down instead of being promoted into confident findings.

The most important rules are short:

Only put high-confidence defects under Potential issues.
Put plausible but unproven concerns under Warnings or Open Questions.
Put missing or incomplete tests under Test Gaps unless a concrete defect is shown.
Lower confidence when the conclusion depends on external behavior not in the evidence.
Say when there are no high-confidence potential issues.

That sounds strict, but it is necessary, the failure mode of AI code review is not silence. The failure mode is confident noise that will just hurt the tool in the long run.

A smaller review with one valid finding is better than a long review that makes developers argue with the tool.

That concern also shows up in SWR-Bench. The paper treats false positives as a practical adoption problem for automated review, because invalid alerts overwhelm developers. That is the behavior I wanted to avoid, so the tool is biased toward fewer comments that survive the evidence so that when it finds an high possibility issue, it most likely will be one.

The verifier pass

The biggest quality improvement came after the first review was generated.

The reviewer runs a second model pass. The first pass receives the evidence packet and writes a draft review, then the second pass receives the same evidence packet plus the draft review, then checks whether the findings are actually supported.

The verifier is not supposed to add findings, but only to remove, downgrade or reword weak ones.

The flow looks like this:

evidence packet
-> first model pass
-> draft review
-> second model pass with the same evidence
-> remove or downgrade unsupported claims
-> final review

One useful way to think about it is this:

reviewer: What issues are present?
verifier: Are these proposed issues actually supported?

Those are different jobs.

This reduced the kind of comment I dislike most: a finding that sounds plausible but is not really proven by the provided code. The verifier does not make the system formally correct. It is still a model checking model output. But it changes the pressure from “produce a review” to “keep only what survives the evidence.”

There is some research support for this general shape, although none of it maps exactly to this implementation. So I would not treat the verifier as exactly proven just because the pattern looks reasonable. I will keep measuring what it actually removes, what it misses and whether it ever makes the review worse.

Self-Refine shows that generation, feedback and refinement can improve model outputs and Chain-of-Verification uses a draft and a later verification step to reduce hallucinations. But there is also work showing that models can fail to correct their own reasoning without external feedback. So the verifier should not be an “are you sure?” prompt. It needs evidence and a stricter “job”.

The closest pull request review result I found was SWR-Bench. It reports that aggregating multiple independent review attempts improved issue-detection F1 by up to 43.67%. Also it also tested same-model aggregation and cross-model aggregation and both showed improvement. So I would not treat a different second model as automatically better, but more as a setting to measure. The more important part is that the second pass is doing a different job.

The limitation is obvious: the verifier sees the same evidence packet as the reviewer, so if the context builder missed the file that proves or disproves a claim, the verifier has the same blind spot.

That is why the evidence packet still matters more than trying to create good prompt wording.

Posting the review

After verification the tool posts a Markdown comment back to the pull request.

I kept the first version simple: one top-level comment, not inline annotations for every finding. Inline comments can be useful later, but they also create more noise and require better line mapping. I wanted the review to be correct enough before making the comment placement more sophisticated.

The comment format is intentionally predictable:

summary
potential issues
test gaps
warnings or open questions
notes about skipped context

That structure matters because it makes the review easier to scan and it also makes future improvements simpler. For example the tool can later add a stable marker to its own comment and update the previous bot comment instead of posting a new one on every run.

The first priority was not comment lifecycle management. It was making the review worth posting.

Review and exploration are different modes

I also separated pull request review from local repository exploration.

For PR review, the model does not get live tools. It receives the evidence packet that the reviewer already built. That makes the pipeline run more repeatable: the same pull request state should produce roughly the same review input.

For local exploration, read-only repository tools are useful. It can be useful to ask questions like “where is this helper used?” or “which tests cover this area?” But that is a different trust model from automated PR review.

Keeping those modes separate made the PR reviewer easier to reason about. The pipeline path is bounded and evidence-based. The exploration path can be more interactive, but it still needs file access limits so it cannot wander outside the checked-out repository.

What made it useful

The useful comments were usually not dramatic. Most were small correctness, maintainability or test-coverage observations that were easy to act on.

That is a good outcome. A PR reviewer in my opinion does not need to produce surprising architectural insights every run, but it really needs to catch enough real issues, with low enough noise, that developers keep reading it.

The main win was not that the model knew more than a human reviewer, more so that the tool gathered the boring context before asking the model to reason.

That made it better than a conventional diff-only reviewer in exactly the cases I cared about: where the relevant evidence was next to the diff, below the diff, in a nearby test, or in a repository convention that the raw patch did not show.

What still limits it

The tool is still bounded by its context builder.

If the relevant behavior is outside the gathered files, the model may not see it. If platform semantics are not included as evidence, the model should treat them as unverified. If the PR is too large, the tool has to skip the model call or reduce the amount of full-context code.

The verifier helps with false positives, but it cannot prove correctness. It can only check the draft review against the evidence it was given.

That means the most important future work is still context quality:

better related-file discovery
better test discovery
better repository profile guidance
better measurement of accepted findings
better handling for repeated comments
targeted extra context for verifier decisions

The most interesting improvement would be targeted verifier context. If a draft finding mentions a specific function, configuration value or behavior, the tool could gather extra evidence around that claim before the verifier decides whether it should survive.

That seems more valuable than simply increasing the token budget. Since as stated before more context is not automatically better, the right context is better.

The other missing piece is measurement. SWR-Bench evaluates generated reviews by whether they cover real review issues, not by whether the prose looks like a review. For this tool the equivalent metrics are valid findings per PR, false positives per PR, findings accepted or fixed by developers and cost per valid finding. Comment count is not really a good quality metric.

Conclusion

The most useful version of the reviewer did not come from one better prompt.

It came from changing the order of operations:

resolve the real Git range
-> gather repository context
-> apply repository profile
-> enforce size limits
-> generate draft review
-> verify review against the same evidence
-> post only the surviving findings

That order made the reviewer more honest.

The model still writes the review, but it is no longer being asked to review from a thin patch and vibes. It is asked to reason from bounded evidence and then a second pass checks whether the comments survive that evidence.

The research I found supports that shape more than any specific model pairing, so a different verifier model may help but the stronger idea is simpler: generate candidate claims, then make those claims survive evidence before posting them.

For this kind of tool that is the difference between an interesting demo and something I can actually use in pull requests.