Why AI Isn't Going to Become Conscious

Seth’s argument turns on a distinction the field keeps collapsing: “Intelligence is all about doing… Consciousness, on the other hand, it’s all about feeling and being.” They arrive together in humans, so we assume they travel together in general — but that assumption, he says, “is a reflection of our own psychology, not an insight into the nature of reality.”

He names the models directly. Language models like Claude and GPT are trained on our endless talk about ourselves, so they reflect that talk back: “We wonder about consciousness and the meaning of it all. And so, it seems, do they. But language models are not conscious. They simulate consciousness.” The test he offers is a good one to carry around — nobody worries whether AlphaFold is conscious, though under the hood it is the same kind of thing. “If we think that Claude is conscious, but AlphaFold isn’t, that says more about us than it says about AI.”

Underneath sits the claim he has spent thirty years on: that conscious AI depends on “the myth that the brain is a computer that just happens to be made of meat rather than metal.” Take away that assumption and the case for machine consciousness loses its floor. Seth is not saying it is impossible. He is saying we are seeing faces in clouds — and that mistaking a simulation for a subject has moral costs running in both directions.