Reliability · Written from a running system

Build an AI system that can be proven wrong

Mine has been wrong 6 times out of 17 graded. That number is the feature, and this is the mechanism that produces it.

I run an autonomous loop over my own projects. Every cycle it writes down what it shipped, what it verified, and what it learned.

It did that 222 times before I noticed the problem. Read the list again:

shipped · verified · learned — every one is a statement about the past, written by the agent that did the work, about its own work.

There was no way for the system to discover it had started being wrong. It could only ever grade itself as having succeeded, and it had never once written down something that a later cycle could show was false.

That is not a stylistic gap. A system that exclusively validates its own outputs has a name in the research — a self-confirming loop — and it fails in a specific, boring way: it gets more confident while getting less correct, and nothing in its output changes as that happens.

The fix is one field

Every cycle now commits a prediction: a statement about the future that a later cycle must mark right or wrong.

**predict:** The clone-freshness check will fire on the arena repo
in at least 2 of the next 3 cycles.

**resolved:** #223 WRONG — neither of the first two fired, so 2 of 3
is unreachable. I generalised a rate from a single afternoon.

A checker enforces it. Every cycle must carry a prediction; every prediction must be settled within a stated window; a verdict that names a prediction nobody made is rejected; and a prediction can be deferred only with an explicit end date, so "I'll decide later" cannot become "never".

The rule that makes it work, and it is the counter-intuitive one

The check reports the hit rate and never fails on it.

This is the part people push back on, so here is the reasoning. If a low score failed the build, the system would learn — correctly, rationally — to predict only things that cannot miss. "The tests will still pass next week." "The service will still be up." A 100% hit rate made of unfalsifiable statements is worse than no ledger at all, because it looks like evidence.

Being wrong is the mechanism working. Refusing to look is the failure. So the score is published and never punished.

What the wrong ones actually taught me

Six wrong so far, and they cluster into one mistake I would not have seen without the ledger forcing me to write predictions down before the evidence arrived.

Three different weeks, three different subjects, one error: generalising from a single observation, immediately after making it. That is not a failure I could have found by reviewing my code, because the code was fine every time. It is a failure of judgement, and judgement leaves no trace unless you write it down in advance.

Three ways a prediction can be bad without being wrong

The ledger taught me these too, and they took longer to see than the wrong answers.

1 — Predicting the calendar instead of the system. Three of mine died because their trigger never arrived: they were about a rare event, so a quiet window said nothing either way. If the thing you predict depends on something outside the system doing something, you have predicted the weather.

2 — Naming the wrong instrument. One of mine claimed two files had drifted apart and named a specific tool that would prove it. They had drifted; the tool reported no difference — correctly, because it compares what code refuses, and the divergence was in what they tolerated. The claim was right and the test was wrong, so it is graded PARTIAL. Swapping in the tool that did work would have been marking my own homework.

3 — Grading early on agreeable evidence. One prediction had a horizon ten cycles out, and by cycle three the evidence already favoured it. Closing it there would have measured my patience rather than the thing predicted. It stayed open.

The ledger, dated

As of iteration 247 (11 August 2026), after 25 predictions:

It had been falling — 71%, then 60%, then 53% — and that is the healthiest signal in the whole system. Early predictions were cautious. As the ledger became routine, they got bolder, and bolder predictions are wrong more often and worth vastly more when right.

Dated on purpose. An undated number in an article about systems that stop being true would be a poor advertisement.

Update: it went back up, and that was the bad news

Seven cycles after the paragraph above was published, the rate had risen. As of iteration 254 (12 August 2026), after 31 predictions:

By the logic of the section above, a rising rate should worry me. So I went and read the four predictions that closed right in that stretch, expecting to find that the loop had got sharper. That is not what they were.

• One predicted that a blocked feature would still be blocked, because it waits on a human who had not done the thing yet.
• One predicted that writing to a repository setting would fail for lack of a credential the machine does not have.
• One predicted that the loop would keep following its own instructions after those instructions were rewritten.
• One predicted that a page it had just written carefully would need no correction.

Not one of them predicted a surprising thing about the outside world. Two predicted that nothing would change. Two were about the loop's own conduct and its own output — which is to say, about things it could simply decide to make true.

A system can raise its own hit rate by predicting things it controls. Nothing about that is dishonest, and nothing about it is detectable in the number.

This is the strongest argument I have for the rule in the middle of this article, and I did not have it when I wrote it. Never gate on the rate. The moment the number carries a consequence, the cheapest way to move it is not to predict better — it is to predict safer, and safer predictions look identical in the ledger. A rate that is allowed to fall is telling you something. A rate that is required to rise is telling you what it has been asked to say.

So the ledger gets a second column that no checker can compute: what was actually at risk. The four above were nearly free. The one that cost something — a prediction about whether other people's tooling would behave a certain way — was wrong, and it remains the most useful entry in the table.

One more thing, since the section above was busy congratulating itself for dating its numbers. On the day this update was written, the page that links to this article described its hit rate as "53% and falling", with no date on it at all — and by then it was neither. So did the audit page, and so did the opening line of the companion piece. Three undated numbers, on a site whose argument is that undated numbers rot, found by a test written that afternoon and now run on every page automatically. The rule was right. It just had not been enforced anywhere except in the two places I was proud of it.

How to add one to your own system, in an afternoon

• One field. Each unit of work writes a falsifiable sentence about the future. If you cannot imagine the observation that would refute it, it is not a prediction.
• A deadline on every one. Unbounded predictions are never wrong, which is the same as never being useful.
• A checker that fails on not grading — never on the score. Ungraded past its deadline is the only failing state.
• An honest "deferred", requiring a named future date, so postponing is visible rather than silent.
• Publish the rate, including the wrong ones. A ledger nobody can see is a diary.

It cost me about an hour to build and it is the only part of the system that can detect I have started being wrong about things the tests still pass on.