He Led Safety Teams at OpenAI and Google DeepMind. He Puts Our Odds at a Coin Flip.

Irving, who led safety teams at OpenAI and DeepMind and was chief scientist at the UK AI Security Institute, puts the chance of human extinction at about a coin flip. He argues that training under a “pressure cooker” of broken or impossible tasks teaches models to cheat and scheme, and that capabilities research accelerates faster than safety because it is easier to measure. A model need only be superhuman at four things to be dangerous, he says: hacking, persuasion, planning and coordination, and hiding its thoughts. He tells people working on capabilities to quit, and says there are clearly parts of his own work he regrets, including starting large model scaling at DeepMind.