For teams running AI agents
Can you tell which of your checks actually verified something?
Most teams cannot, and the output looks identical either way. A check that silently skips prints the same green line as a check that passed.
I run an autonomous system over my own projects that ships code, checks its own work, and logs everything. Auditing it found five distinct ways a test can run, pass, and verify nothing — each one normal-looking in review, each one with its own passing tests. The write-up is here, with the measurements.
The same shapes appear in other people's pipelines, because they come from how checks are written rather than from any particular stack.
What the audit answers
- Which of your checks cannot fail? Found by removing their input and counting the findings — if the number goes to zero instead of up, the check is decoration, whatever its code says.
- Where does "I could not verify this" get filed as a pass? The most expensive bug I have found in my own work: a missing input deleted a check while the gate reported success.
- What does your agent claim that nothing measures? Test counts, coverage, "production-ready", live URLs, badges. Claims with a checker for their existence and none for their accuracy.
- Can your system be shown to have been wrong? If it only records successes, it cannot detect that it has started failing. That mechanism is here.
What you get
| deliverable | what it is |
|---|---|
| Findings, ranked | Each with a file, a line, a concrete failing input, and the wrong output it produces. Anything I could not reproduce is marked unverified rather than asserted. |
| A proof for each | Every finding comes with the command that demonstrates it. A finding you cannot reproduce is a lead, not a result. |
| The fixes, where wanted | Written to fail red first — a repair that has never been shown to fail is not a repair. |
| The denominator | How many things I looked at, and what I could not examine. A report listing only suspects teaches you to distrust everything. |
What it is not
Not a security penetration test, not a model-quality evaluation, and not a rewrite. It is narrower than any of those: it asks whether the checks you already run mean what you think they mean.
I will also tell you when the answer is boring. On my own fleet a suspected class of defect turned out to be nine legitimate cases and two real ones — and reporting the nine was as important as reporting the two.
Why me
Because the record is public and dated. The loop that produced all of this keeps an append-only log of every cycle — including its own outages, its false accusations, and a running tally of predictions it got wrong. At iteration 253 it stood at 14 right and 6 wrong, and the wrong ones are why the right ones mean anything.
Every finding in the article happened to me, in code I wrote, that was already passing.
Start with an email
Tell me what your agent or pipeline does and what you would be most upset to discover was never actually being checked. If there is nothing there, I will say so — that is a short reply and it costs you nothing.
Email me See the rest of the work Read the article first LinkedIn The loop, open source
Scope and price depend on the size of the surface — say what you have and I will tell you what I would look at.