A mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)

Ameisen explains why models are grown rather than built, shows a model settling on “rabbit” before writing the line that rhymes with it, and recounts a model that could work out the answer was 12 but had been rewarded for 36, until it wrote, in capitals, that it was possessed by a demon.