Reliability · Written from a running system

Your AI agent's tests are lying to you. Here are five ways I caught mine.

I run an autonomous loop over my own projects. It writes code, checks its own work, and logs what it did — 237 iterations at the time of writing, all of it on disk and dated.

Somewhere around iteration 190 I stopped trusting it, for a reason that had nothing to do with the code it wrote.

I had built a checker to catch a corrupted data table. To make sure it worked, I deliberately corrupted the table — changed a 22 to a 2 — and ran it.

It reported clean.

Not "failed to detect the change". It printed a green PASS line, exited zero, and looked exactly like every other successful run. The checker had been reading the wrong file path the whole time, hitting a permission error, and swallowing it in an except Exception: return [] I had written myself — with a comment above it that said "never fail the gate on a tooling problem."

That gate had been in the suite for weeks. It had never once been capable of finding anything.

Since then I've been collecting these. Not bugs in the code being tested — bugs in the tests themselves, where the check runs, passes, and verifies nothing. There are five distinct shapes. All five look completely normal in review. All five have their own passing tests. And all five are caught by the same ten-second experiment, which is at the end.

1. The comparison that can't disagree

I had a checker making sure a document's entries were all being read:

parsed  = len(parse_entries(text))
headers = len(HEADER_PATTERN.findall(text))
if parsed != headers:
    problem("entries went missing from the parse")

Reasonable. Count what you parsed, count what's there, compare. Except parse_entries() also used HEADER_PATTERN internally. Both sides of that != came from the same regex over the same text. They were equal by construction. No possible input makes them differ.

Here's what that cost. I took the real document and reworded three of its headings:

Three entries silently disappeared and the guard reported clean. Every downstream check kept passing — about a smaller world.

And it had a passing test. I'd written it like this:

assert coverage_problems(parsed=1, headers=2) == ["entries went missing..."]

I handed it a 1 and a 2 myself. That test proves 1 != 2 produces a message. It never asked whether the real code could ever produce a 1 and a 2. It couldn't.

The tell: any check shaped "A should equal B" where A and B come from the same code. It will always have a passing test, because the test author supplies both sides by hand.

The fix wasn't a better comparison — it was a genuinely independent signal. Those entries are numbered sequentially, so I check for gaps in the sequence instead. A missing entry leaves a hole, no matter how anyone rewords a heading.

2. The guard that shares its subject's blind spot

I have a file that acts as a source of truth, and a parser that reads it. So I wrote a guard whose whole job was to notice when the parser stopped seeing a row. The guard identified rows like this:

name = cells[1].strip().strip("*` ")
if not re.fullmatch(r"[A-Za-z0-9._-]+", name):
    continue

The parser it was guarding identified rows the same way. Byte for byte. I'd even written a comment explaining the reuse — "reuse the parser the other checks already trust" — which is sound for anything consuming that value, and exactly backwards for something checking it.

Break the name and the row vanishes from the parser and from the guard, in one move. The guard could only ever catch the half of the problem it didn't share.

The rule: consumers should share a parser. Guards must not.

When I fixed it, it found a real defect on its first run: a row whose heading was slightly different, declaring a number that had never reached a single check in the entire life of the file.

3. The correct fix that never runs

This is my favourite, because the fix was right. I'd hit a bug where a claim with no data behind it was skipped instead of flagged, and fixed it properly:

if not key or key not in fleet:
    problem(f"claim {key} has no measured counterpart -- unverifiable")
    continue

Good fix. Correct reasoning. Accurate comment. Right function. It sat inside this:

if fleet:            # <-- if the source is empty, skip the whole block
    for card in cards:
        ...
        if not key or key not in fleet:
            problem(...)

If the source became unreadable, fleet was empty, and the outer if skipped the entire block — including the repair. I measured it on the real page: five public claims, source unreadable, zero findings, PASS. It had been unreachable for sixteen iterations.

The question to ask after every repair is not "is this correct?" It is "what makes this line not execute?" A comment describing a bug as fixed is not evidence that the code path runs.

4. The loop that runs zero times

This one is everywhere once you know the shape:

for match in pattern.finditer(text):
    if match.value != expected:
        problem("the number is wrong")

That's a check — unless finditer matches nothing. Then the body never executes, no problems are added, and the function returns success. No iterations is indistinguishable from no problems.

Mine was anchored to an exact sentence in a document. Someone (me) reworded the sentence.

That last row is the one that matters: a false number sitting in a document, and the tool printing "PASS — every figure matches the source."

Fixing it exposed something worse. Once a zero-match became a finding, one of my rules lit up immediately — it had been anchored to a pattern that had never matched anything, ever. A number in a public draft, unchecked for its entire existence, while the tool reported success on every run.

5. The verdict built by exclusion

I found this one by writing it, four minutes after writing a comment describing it. That is not a joke, and it is why it's here.

verified = [n for n in notes
            if "nothing queued" not in n
            and not n.startswith("skipped:")]
print(f"{len(verified)} items verified, all green")

Count everything that isn't a known exception. Reasonable-looking. It printed "all green" for a run where verification had been deliberately switched off — an offline mode that doesn't check at all. Those items produced a note saying "not checked", which wasn't in my exclusion list, so they counted as verified.

A negative filter isn't evidence. It's whatever survived the exceptions you happened to think of.

Building a list of problems by exclusion is fine — that's how filtering works. Building a claim of success by exclusion asserts your exception list is complete, and it never is.

The ten-second test that finds all five

Remove the input. Count the findings.

Delete the file it reads. Empty the list it iterates. Break the pattern it matches. Make the command it shells out to unavailable.

If the number of findings goes to zero instead of up, the check is fake — whatever its code, comments, or test suite say about it.

That's it. Seconds per check, no framework, and it caught all five of these in my own code, written by me, already passing.

Try it on one thing you rely on today. My honest expectation, from running this across twenty checkers: you'll find at least one. I found five in a system I'd already spent months hardening.

Why this is the whole problem with AI agents

An AI agent's dangerous failure mode is not being wrong. Wrong is visible — it crashes, or produces something obviously bad, and you notice.

The dangerous mode is confidently green about work it never checked. The agent reports success. The tests pass. The summary is reassuring. And nothing in the output distinguishes "verified" from "silently skipped", because the code that was supposed to tell those apart is one of the five shapes above.

Which is why the fix isn't a better prompt or a bigger model. It's the boring discipline underneath:

• Prove a checker can fail before you trust that it passes.
• Every check declares how many things it looked at, not just its verdict.
• "I couldn't check this" is a third answer, and must never be filed as a pass.
• Let a cheaper, different model review your work. One found a critical bug in code I'd written minutes earlier that already had 19 passing tests. Its advantage wasn't capability — my tests covered everything I'd thought of, which is the same set that produced the bug. It just wasn't me.
• Make the system predict, so it can be shown wrong. Mine writes down a falsifiable claim each cycle and a later cycle grades it. As of iteration 243 (10 August 2026) the ledger stood at 10 right, 4 wrong, 1 partial, 2 unresolvable — and the wrong ones are the reason I believe the right ones. (Dated on purpose: an undated number in an article about stale numbers would be a poor advertisement.)