Open Code Review: inspect review scope, filters and model work
Measure Open Code Review cost alongside missed defects
Design a controlled review comparison that records tokens, latency and human acceptance without treating an upstream benchmark as a guaranteed saving.
What you will learn
- Read the benchmark trade-off
- Separate prompt limits from total spending
- Include the human review work
Before you start
- Git changes and merge-base comparisons
- Basic CLI use and model API credentials
Choose a review scope, explain exclusions, prepare a controlled pilot and distinguish source inspection from runtime evidence.
Key takeaways
- The upstream benchmark includes a recall trade-off.
- Prompt size is not the same as total spending.
- Human review time belongs in the evaluation.
Read the benchmark trade-off
The README reports better precision and F1 with substantially lower token use in its benchmark, while explicitly acknowledging lower recall than the compared general-purpose agents. These are project-reported results from a particular evaluation. We have not reproduced them on the pinned revision.
A team that only counts fewer comments may reward a reviewer for missing bugs. Build a small labeled change set, preserve the same model and scope across conditions, and measure false positives and missed known defects alongside accepted findings. Report the sample and its limitations.
Separate prompt limits from total spending
The architecture page distinguishes the prompt ceiling from the model’s output limit. It also describes review rounds and an aggregate token budget. These controls address different parts of a run, so changing a context-window value should not be described as setting a complete monetary spending cap.
Grouping, planning, retries and later review rounds can add model work. Delegation moves reasoning to the host account rather than removing its cost. Capture the actual usage reported by the provider and host, and record any fields that your chosen integration does not expose.
Include the human review work
For each trial, record elapsed time, model usage, accepted findings, missed labeled defects and minutes spent checking comments. Repeat runs when output variability matters. Use the same acceptance policy throughout, including how duplicated findings and uncertain reports are counted.
A useful rollout decision states the observed trade-off for that sample. If usage falls but a serious labeled defect is missed, the lower bill alone does not settle adoption. The example below is a proposed experiment record; it contains no invented measurements.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Prepare a fixed set of small changes with independently labeled defects.
- 2
Hold model, scope and acceptance policy constant while comparing configurations.
- 3
Record usage, elapsed time, missed defects and human checking time.
Copy-ready example
{
"proposal": true,
"sample": "fixed labeled changes",
"model": "record exact choice",
"tokens": null,
"elapsedSeconds": null,
"falsePositives": null,
"missedDefects": null,
"humanMinutes": null,
"benchmarkExecuted": false
}Frequently asked questions
Can I assume the README saving for my repository?
No. Reproduce a controlled comparison with your changes, model and configuration.
Is delegation free?
It uses the host agent’s model access and quota. Account terms and measured usage still matter.
Sources
- Open Code Review / README.mdSource checked 2026-09-18
- Open Code Review / pages/src/content/docs/en/architecture.mdSource checked 2026-09-18
- Open Code Review / pages/src/content/docs/en/integrations/delegate.mdSource checked 2026-09-18
- Open Code Review / pages/src/content/docs/en/configuration.mdSource checked 2026-09-18