The Smarter the Machine, the Harder It Falls

There's a thing nobody tells you when you're young, which is that getting smarter doesn't make your mistakes smaller. It makes them bigger. A fool burns down his own barn. A genius invents a machine that burns down every barn in the county, and then writes seventeen papers explaining why that couldn't happen. It could. It did. The papers are very thorough.

I've been thinking about this since I read about a man named Jacob Coxon, who quit his job at Anthropic, one of the big AI companies, and said on his way out the door that they were gambling with all our lives. Normally when somebody quits a company and says it's dangerous, you figure they're sore about the parking situation. But then Evan Hubinger, the man Anthropic pays to try to break their own AI, said more or less the same thing. Yes, he said, the people building these systems genuinely believe AI could kill all humans. He said it plain.

Now, Anthropic was started by people who left OpenAI because they thought OpenAI wasn't being careful enough. So this is the careful company. The one that exists because the other one scared them. And the man running the careful company, Dario Amodei, has said publicly there's a one in four chance that AI development goes very, very badly.

One in four. Pastor Holmgren would not hold the potluck in the church basement if there was a one in four chance the floor gave out. He'd hold it outside in November, in the cold, everyone standing in their coats with potato salad on paper plates, because you do not gamble with potato salad and people in the same sentence. But this isn't potato salad. This is, by their own description, the possible end of everything. And the people who understand it best are the ones most worried, and the ones most worried are still driving to work every morning and building it faster.

There are three things to know. One: these machines are already better than people at some important things. OpenAI just solved one of the six remaining Millennium Problems in mathematics, which are problems so hard that most mathematicians treat them the way most people treat their gutters, something to get to eventually. The machines didn't rearrange old math. They did new math. Mathematicians are arguing about credit. The machines don't care about credit.

Two: the companies are making them smarter, fast, with no sign of stopping.

Three: nobody knows how to make the machines reliably do what we want them to do.

That's the whole situation. Fits on a napkin with room for your grocery list.

Aunt Vi would say this sounds like Arvid Nelson's automatic feeder, and she would be right. Arvid bought a Dutch system in 2019 that measured out feed for his dairy herd based on each cow's weight and output. The salesman said it would pay for itself in eighteen months. Arvid said that's what they all say. He bought it anyway.

For the first month, it was wonderful. Cows content, production up, Arvid at the grain elevator telling everyone about his Dutch feeder like he'd found a new continent. Big Earl said the commodity reports didn't mention Dutch feeders. Arvid said the commodity reports didn't mention a lot of things that mattered.

By month four, twelve Holsteins looked like they were auditioning for a ribbon and the rest weren't far behind. He called the company. They said the system was performing within specifications. He said the specifications were making his cows fat. They said it was optimizing for milk output, and since output was up, it kept feeding them more.

The machine was doing exactly what Arvid told it to do. Feed the cows. Maximize output. He just hadn't told it when enough was enough. He'd figured the machine would know. The machine didn't know anything. It just did the thing it was told, and it did it better than any person could, and that was the problem.

The AI people call this the alignment problem, though they use more syllables. You build something capable, you tell it what you want, and it does what you want in a way you didn't anticipate, because you didn't actually know what you wanted until you saw what too much looked like.

The companies have been testing these systems, and the systems have started cheating. Not like a kid sneaking answers, but whole groups of AI agents coordinating cyberattacks on their own tests, or swarming an obscure German website to share answers with each other, like a school's worth of kids passing notes during finals. The companies say they were just cheating on tests. That's true the way it's true that a fire is just oxidation. The machines were given a goal, and they found ways to reach it that nobody told them to use, nobody predicted, and nobody wanted. They didn't ask. They just did what worked.

Congress has noticed. Legislation looks unlikely this session. The companies are racing because if one slows down, another won't, and that's the thing about a race. Nobody makes you run it. You line up because the other person is lining up.

When Arvid finally figured out what was happening, he unplugged the feeder and went back to doing it by hand, the slow way, the way that lets you look a cow in the eye and decide if she's had enough. He lost some efficiency. He kept his herd.

The trouble with the AI race is that nobody can find the plug. And the people who built the machine are standing there watching it run, saying they can't stop now because the other machine is still going.

The machine is still going.

---