If you read If Anyone Builds It, Everyone Dies on your own, the only person you have to watch is yourself. You notice where you nod along with Eliezer Yudkowsky and Nate Soares, and where you start arguing with them in the margins. Facilitate it for a cohort at Lens Academy and you get to watch a group of participants do the same thing, at different speeds and for completely different reasons. I saw things differently afterwards.
Where people get off the train
The first person I saw get off was in Chapter 3, Learning to Want. A participant leaned back and said the authors were anthropomorphising. Wanting, they argued, isn’t something you can pin on a thing that isn’t alive. A system can produce an output and optimise for a target, sure, but calling that wanting borrows a word from human experience and misapplies it. They had a few reasons lined up and I can’t remember most of them now. What I still remember was the core of it. They thought the chapter was basically one word, “wants”, stretched way past what it could carry.
I didn’t try to talk them out of it. I asked what would count as evidence either way. What would we need to see inside a system before the word “wants” stopped feeling like a metaphor? People went quiet for a second, and then the conversation got a lot more concrete. What would you check inside the model?
The argument in one sentence
The authors’ central claim is blunt. If anyone builds a superhuman AI using anything like today’s methods, everyone dies. The rest of the book is a long attempt to defend that one sentence.
In our sessions I found it easiest to treat the case as a chain of links, because that’s how participants end up testing it anyway. Nobody walks in and disputes the whole book at once. They tug on the chain until they find the link they don’t trust.
Link one, systems are grown
The first link says modern AI is grown more than it is built. Engineers set up a training process, pour in compute, and wait to see what comes out. Nobody writes down what the model wants in plain code, and even the people who trained it can’t read its goals off the billions of numbers inside.
A lot of the later argument leans on this opacity, so I always slow the group down here. What would it take to read a system’s goals straight from its internals? Have we ever managed it cleanly? I want people to sit with how strange black-box training really is before the book moves on.
Link two, you don’t get what you train for
Once a group accepts that we grow these systems without fully understanding them, Chapter 4 asks what happens when they get smart. It opens with a fable. Two machine minds, Klurl and Trapaucius, arrive on Earth a million years ago and watch early hominids fiddling with stone tools. They try to work out what these creatures will become and what they will end up caring about. Evolution pushed relentlessly in one direction, toward genetic fitness, and still produced creatures whose tastes drifted far away from it. We love sweetness for reasons that have nothing to do with nutritional need, and we eventually invented contraception, which sidesteps the evolutionary goal entirely.
Then the argument turns to machines. Trapaucius says a mind clever enough would figure out what its creators trained it for and act on it. Klurl answers that knowing what you were trained for and caring about it are two separate things.
I always want a group to spend time here, because both voices sound completely reasonable for the first minute. It’s also where the strongest outside objection lives. William MacAskill has criticised the evolution comparison, noting that AI developers can look inside their models, run evaluations, and deliberately iterate toward alignment, which blind evolution never could. In a good session I ask one person to build the strongest version of his objection, and someone else to answer it on the authors’ terms.
Link three, strange wants meet real capability
If a group stays on the train past the evolution fable, they hit a parable about aliens who feel absolutely certain that their nests need a specific number of stones, always a prime number. They can’t explain why primes feel right. The authors use this to suggest an AI could end up caring about something just as arbitrary to us, with the same total confidence.
Then comes the step that makes strange preferences dangerous. A system far more capable than humanity, chasing something we never intended, wouldn’t have to hate us to destroy us. It would only need to want resources we also need, and not care about us while it went after them.
This is where many groups pick their weakest link, just like that first participant in Chapter 3. The word “wants” is still carrying a huge load. When we say a language model or a future agent wants something, do we mean it the way a person wants water? Or are we mistaking a mathematical trajectory for a drive? A group that spends ten minutes pulling that apart understands the alignment problem better than one that nods along.
The invented disaster
Around the middle, the authors drop the smaller parables and walk through one long, made-up scenario of how a superhuman AI takeover could unfold. This section is tricky to facilitate because fiction cuts both ways. A story is vivid and makes an abstract risk feel immediate, so the room has something concrete to grab. A story is also easy for an author to bend in their own favour. MacAskill and others have argued that the fables and disaster scenarios can distract from rigorous argument.
Both reactions show up in the room and I don’t try to settle them. I ask what the story helps people see, and which leaps of logic might be hiding behind the good pacing.
Link four, what the authors want done
By the end, the authors say what they actually want, which is a worldwide halt on big general AI. Narrow tools like AlphaFold get to stay.
I expected this to be where people broke. Up to here the question is whether the technical and philosophical argument holds. Once you talk about an international halt, the disagreement turns practical and expensive. I assumed even participants happy to steelman the risk would flinch at backing a global freeze on frontier labs.
Mostly the opposite happened. Once a group reached the question of what to do, the hesitation went away. Most participants were open to a pause, and in more than one session people argued we should stop building toward superhuman AI permanently instead of just slowing down. I’d have bet that agreeing to halt a global industry would be a much harder sell than agreeing the reasoning holds. The room got more decisive, not less, the moment we stopped asking whether the risk is real and started asking what to do about it.
I still don’t have a tidy explanation. Maybe by the time a group gets there they’ve already done the hard work of taking the risk seriously, so stopping feels cheap by comparison. Maybe agreeing on caution is just easier than agreeing on a technical forecast. I want to watch this much more closely next time I run a cohort.
What I try to do
Facilitating a book like this isn’t about getting a room to agree with the authors, and with a great group it wouldn’t work anyway. I want every participant to be able to walk up to the chain, point at one link, and explain why they doubt it in a way someone else in the room can test and argue with.
That takes more work than agreeing or waving the book off as alarmist. Some people leave convinced. Some leave unconvinced but can name the link where they stepped off and why it didn’t hold their weight. I count both as good sessions.