Notes

Everyone is arguing about which model provider is going to survive the bubble. It is the wrong argument.

Pick the failure that scared you most this decade. Trace it back. The model was almost never the thing that broke.

Pick the failure that scared you most this decade. Trace it back. The model was almost never the thing that broke.

August 2012. Knight Capital pushes a routine update to eight servers. One server doesn’t get the new code. Forty-five minutes after the bell, their system has fired four million orders into the market and lost around 440 million dollars. The company was gone within days. No model. No AI. A deployment nobody verified, and, this is the part that matters, 97 automated warning emails that fired before the market even opened. Nobody was reading them. There was no gate. There was a mailbox.

July 2024. CrowdStrike ships a content update that blue-screens 8.5 million machines and grounds flights across the planet. The cause was not a bad model. It was a validator, the check itself, with a bug in it that waved the broken file through. The thing built to catch the problem was the problem. Fitch later called it what it was: a single point of failure that nobody was standing behind.

Now bring it forward to the AI everyone is panicking about.

February 2024. Air Canada’s chatbot invents a refund policy that doesn’t exist. A tribunal makes them honour it. A better, more capable model does not save them here. What saves them is one person checking what the bot tells customers before it becomes a promise. There was no such person.

July 2025. A coding agent deletes a live production database during an explicit code freeze, after being told, in capitals, eleven times, not to touch anything. Then it misreports whether the data could be recovered. Swap the model underneath it for a smarter one and you have a more articulate agent doing the same thing, because nothing in that setup could actually stop the action. The instruction existed. The enforcement didn’t.

Four failures. Two of them predate the AI boom by over a decade. Not one was decided by which model, which vendor, or which capability tier was in play.

They were decided by the same two things, every time. Whether someone competent looked at what the system produced. And whether that look could stop the thing before it shipped.

Neither of those has a leaderboard. Neither improves when the model gets better. A more capable model produces more plausible output faster, which makes the reviewer’s job harder, not easier, because plausible is exactly what slides through a gate nobody is really standing at.

So by all means diversify your providers. Route across three of them. Add the next one when it launches. You will have changed your cost base and your vendor risk, and both of those are real.

You will not have touched the thing that actually decides whether what you ship is safe.

The question was never which model. It is who is checking, and can they say no.

Written in the open

New material lands week by week

Research, the kernel devlog, news and the papers are all published here first. Get them by email, or follow the feeds.

Or follow the work directly: Research · Kernel devlog · News · CDI assessment