Ten Percent Chance of Everything

There's a number floating around AI safety circles right now, and it goes like this: ten percent chance of human extinction. Evan Hubinger said it. He works at Anthropic, which calls itself a safety company, which is a bit like a tobacco company calling itself a health organization, because while they may be working on making their products less lethal, they're still producing the same products, still making money off the same things, still waking up every morning and going back to the lab to make the thing smarter, and the thing they're making smarter is the thing they just told you might kill everyone, and they said it with a probability attached, like a weather forecast, like there's some comfort in the precision of it, like "ten percent chance of human extinction" is a number you can file somewhere and revisit at the next quarterly review.

I want to sit with that for a second, because I think we're not sitting with it enough. What does it even mean to assign a probability to the end of the world? Not the end of a company, not the collapse of a market, not even the fall of a civilization in the distant abstract, but the extinction of our species, the thing that makes every other question moot, and we're treating it like a risk assessment, a calculation to be made and then managed, because that's what we do with numbers, we manage them, we put them in a framework, we build a mitigation strategy around them, and then we go back to building the thing that generated the number in the first place, because the alternative is stopping, and stopping is the one thing the machine is not designed to do.

I've sat in rooms (not AI labs, to be clear, but rooms where people with power over harmful systems are the same people perpetuating them) where everyone in the room knows the thing they're doing is causing harm, and they know it, and they continue, because stopping would require dismantling the entire structure, and the structure is the job, the structure is the identity, the structure is the funding and the salary and the career trajectory and the sense of purpose that gets you out of bed in the morning, and you can't just walk away from that, not easily, not without losing something that feels essential to who you are, and so you don't walk away, you publish a warning instead, you say "ten percent chance of human extinction" in a public forum, and then you go back to work, and somehow the warning becomes the justification, somehow the act of naming the danger is treated as equivalent to addressing it, and the machine keeps moving.

Anthropic is not uniquely villainous here. That's almost the worse part. The debate has genuinely shifted from whether AI poses a risk to how large that risk is, which sounds like progress, sounds like the field is maturing, sounds like we're being responsible, and maybe we are, a little, but it also means we're debating the size of the problem while continuing to grow it, and the people doing the debating are the same people doing the growing, and that's not a conflict of interest that gets resolved by publishing a paper, that's a conflict of interest that's structural, that's baked into the incentive architecture of the whole enterprise, because the research is funded by the product, and the product is the thing the research says is dangerous, and there's no clean way out of that loop, not without someone deciding to actually stop, and nobody is stopping, because stopping means losing, and we don't have a framework for what winning looks like if it means walking away from the machine.

What gets me, and I mean this in the way that keeps me up at night, the way that sits in my chest like a stone, is the intertwining of the warning and the work. The people closest to the danger are the ones building it. The people who understand it best are the ones who are least able to stop, because they've organized their entire professional existence around it, their identity, their community, their sense of what matters, and so the warning comes out, and it's real, and it's honest, and it's terrifying, and then they go back to making the thing smarter, because that's who they are, and the warning gets absorbed into the culture as evidence of seriousness, evidence of responsibility, evidence that the right people are thinking about the right things, and maybe they are, but thinking about the right things and doing the right things are not the same thing, and we keep confusing them, we keep treating the acknowledgment of danger as a substitute for addressing it, and I don't know how we stop doing that, I genuinely don't, but I know that "ten percent chance of human extinction" is not a number that should let anyone sleep well, and I know that the people who said it are sleeping, because the work continues, and the work is the whole thing, and the whole thing is what's at stake.