Automated Pre-Deployment Risk Audits with a Read-Only Code Agent
Problem
Features were passing review and tests but still shipping bugs that only surfaced when customers hit them, because human review focused on the diff rather than on how the change behaved across the whole code path. We needed a repeatable pre-deployment check that was deeper than a lint pass but did not risk an agent modifying code right before release.
Trial
I set up a recurring 'feature ledger' review: a code agent, restricted to read-only access, performs deep analysis of each candidate feature and must emit a strictly structured risk report — a verdict status, a P0 to P3 priority, the symptoms the customer would actually see, concrete failure cases, code-path evidence for every claim, and verification steps a human can execute. The rigid format forbids vague findings like 'this might be fragile'; every entry needs evidence and a reproduction path.
Result
The audit ran repeatedly for over a month and surfaced real bugs before customers did, including a webhook mount mismatch that was silently serving stale RAG data and a tenant-filter leak in task history that would have exposed one customer's records to another. Because each finding shipped with code-path evidence and verification steps, triage was fast and false alarms were cheap to dismiss.
Lessons
Read-only is a feature, not a limitation: removing write access made it safe to point the agent at release candidates and made its incentives purely analytical. The strict report schema mattered as much as the model quality, since findings without symptoms, evidence, and verification steps are just noise that erodes trust in the process.
Next Action
I plan to wire the audit into CI so a P0 or P1 verdict blocks the deployment pipeline automatically instead of relying on someone reading the report in time.