10 experts, 10 episodes: a crash course on transformative AI and what you can do to help shape its trajectory.
This compilation features 10 key episodes of The 80,000 Hours Podcast to help listeners — particularly those new to the topic — get to grips with the potential upsides and downsides of powerful, transformative AI.
Site
RSS
Apple
Classements récents
Dernières positions dans les classements Apple Podcasts et Spotify.
Liens partagés entre épisodes et podcasts
Liens présents dans les descriptions d'épisodes et autres podcasts les utilisant également.
What might it be like to live through the creation of AI that surpasses human abilities? That future may be closer than you think.
In this series, one expert interview at a time, we'll walk you through what's at stake — and what you could do to help.
One: Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared
vendredi 5 juin 2026 • Durée 04:08:06
The 20th century saw unprecedented change: nuclear weapons, satellites, the rise and fall of communism, third-wave feminism, the internet, postmodernism, game theory, genetic engineering, the Big Bang theory, quantum mechanics, widespread birth control, and more. Now imagine all of it compressed into just 10 years.
That’s the future Will MacAskill — philosopher, founding figure of effective altruism, and now researcher at Forethought Research — argues we need to prepare for in his paper “Preparing for the intelligence explosion.” Not in the distant future, but probably in 3–7 years.
The reason: AI systems are rapidly approaching human-level capability in scientific research and intellectual tasks. Once AI exceeds human abilities in AI research itself, we’ll enter a recursive self-improvement cycle, with AI acting autonomously to create wildly more capable systems.
Soon after, by improving algorithms and manufacturing chips, we’ll deploy millions, then billions, then trillions of superhuman AI scientists working 24/7 without human limitations. These systems will collaborate across disciplines, build on each discovery instantly, and conduct experiments at unprecedented scale and speed — compressing a century of progress into years.
Will compares this to a mediaeval king suddenly needing to upgrade from bows and arrows to nuclear weapons to deal with an ideological threat from a kingdom he’s never heard of, while simultaneously learning he’s descended from monkeys and his god doesn’t exist.
What makes this acceleration perilous is that while technology can speed up almost arbitrarily, human institutions and decision making are much more fixed.
Consider the case of nuclear weapons: in this compressed timeline, there would have been just a three-month gap between the Manhattan Project’s start and the Hiroshima bombing, and the Cuban Missile Crisis would have lasted just over a day.
Robert Kennedy Sr, who helped navigate the actual Cuban Missile Crisis, once said that if they’d had to make decisions faster — like in 24 hours rather than 13 days — they would likely have taken much more aggressive, much riskier actions.
So there’s reason to worry about our capacity to make wise choices quickly. And in his paper, Will lays out 10 “grand challenges” we’ll need to navigate to avoid things going wrong.
Will now believes we’re entering one of the most critical periods for humanity ever — with decisions made in the next few years potentially determining outcomes millions of years into the future.
In this wide-ranging conversation, Will and host Rob Wiblin discuss:
Why leading AI safety researchers now think there’s dramatically less time before AI is transformative than they’d previously thought
The three different types of intelligence explosions that occur in order
Will’s list of resulting grand challenges — including destructive technologies, space governance, concentration of power, and digital rights
How to prevent ourselves from accidentally “locking in” mediocre futures for all eternity
Ways AI could radically improve human coordination and decision making
Why we should aim for truly flourishing futures, not just avoiding extinction
This episode was originally released in March 2025.
Chapters:
Cold open (00:00:00)
Who’s Will MacAskill? (00:00:43)
Why Will now just works on AGI (00:01:03)
Will was wrong(ish) on AI timelines and hinge of history (00:04:21)
A century of history crammed into a decade (00:09:19)
Science goes super fast; our institutions don't keep up (00:16:15)
Is it good or bad for intellectual progress to 10x? (00:21:44)
An intelligence explosion is not just plausible but likely (00:23:41)
Intellectual advances outside technology are similarly important (00:30:04)
Counterarguments to intelligence explosion (00:32:42)
The three types of intelligence explosion (software, technological, industrial) (00:39:00)
The industrial intelligence explosion is the most certain and enduring (00:42:01)
Is a 100x or 1,000x speedup more likely than 10x? (00:53:44)
The grand superintelligence challenges (00:57:39)
Grand challenge #1: Many new destructive technologies (01:01:29)
Grand challenge #2: Seizure of power by a small group (01:09:10)
Is global lock-in really plausible? (01:11:06)
Grand challenge #3: Space governance (01:21:50)
Is space truly defence-dominant? (01:32:19)
Grand challenge #4: Morally integrating with digital beings (01:36:04)
Will we ever know if digital minds are happy? (01:45:01)
“My worry isn't that we won't know; it's that we won't care” (01:50:39)
Can we get AGI to solve all these issues as early as possible? (01:54:05)
Politicians have to learn to use AI advisors (02:07:05)
Ensuring AI makes us smarter decision-makers (02:11:25)
How listeners can speed up AI epistemic tools (02:15:11)
AI could become great at forecasting (02:18:54)
How not to lock in a bad future (02:20:26)
AI takeover might happen anyway — should we rush to load in our values? (02:32:14)
ML researchers are feverishly working to destroy their own power (02:41:57)
We should aim for more than mere survival (02:45:23)
By default the future is rubbish (02:57:03)
No easy utopia (03:05:21)
What levers matter most to utopia (03:15:19)
Bottom lines from the modelling (03:29:32)
People distrust utopianism; should they distrust this? (03:33:34)
What conditions make eventual eutopia likely? (03:38:26)
The new Forethought Centre for AI Strategy (03:47:15)
How does Will resist hopelessness? (04:00:42)
Video editing: Simon Monsour Audio engineering: Ben Cordell, Milo McGuire, Simon Monsour, and Dominic Armstrong Camera operator: Jeremy Chevillotte Transcriptions and web: Katy Moore
Ten: Holden Karnofsky on dozens of opportunities to make AI safer lying on the table — and all his AGI takes
vendredi 5 juin 2026 • Durée 04:34:58
For years, AI safety work mostly meant theorising about the ‘alignment problem’ or trying to convince people to give a damn. If you could find a way to help, the work was frustrating and low-feedback.
That situation has now reversed completely.
According to Holden Karnofsky — cofounder and former CEO of Open Philanthropy (now Coefficient Giving), now working at Anthropic — there's an overwhelming amount of concrete, useful safety work that needs doing in both technical and nontechnical areas, and nowhere near enough people to do it.
In this conversation alone, Holden lists 39 projects he’s excited about, including:
Training deceptive AI models in order to study deception and how to detect it
Developing classifiers to block jailbreaking
Implementing security measures to stop ‘backdoors’ and ‘secret loyalties’ from being added to models during training
Developing policies on model welfare, AI-human relationships, and what instructions to give models
Training AIs to work as alignment researchers
And that’s just what he’s observed directly, likely a small fraction of what’s available.
All this low-hanging fruit is part of why he joined Anthropic this year. (Though his wife is cofounder and president of the company, giving him a big financial stake in its success — and making it impossible for him to be seen as independent, no matter where he works.)
Holden argues that for many people, working at a frontier AI company is the highest-impact way to steer artificial general intelligence (AGI) — by developing cheap safety tools other companies might actually adopt, prototyping policies that regulators could mandate, and generating hard data about what advanced AI can really do. But he's clear that external groups have distinct advantages and can be equally valuable.
Bonus: How AI could create the world's biggest problems (article narration by Zershaaneh Qureshi)
vendredi 5 juin 2026 • Durée 01:29:46
Imagine you’re living 15,000 years ago. Your people are hunter-gatherers and you sleep under the stars. If someone told you humans would one day build cities with millions of people, fly through the air, or carry all human knowledge in their pockets, you couldn’t even begin to picture what they meant... Yet here we are.
How did our lives change so far beyond recognition? The story is complex, but there’s a rough pattern. A few times in history, some radical breakthrough in technology — like the development of the plough and the steam engine — has led to a wave of productivity, innovation, and social change that ultimately reshaped the world.
Now we’re on the cusp of a huge new breakthrough: artificial intelligence that can meet or exceed human capabilities across a wide range of tasks.
This could bring another era of transformation. There could be an explosion of intelligence and innovation, and a whole new population of digital beings. And with this, civilisation could see changes at least as profound as those brought about by industrialisation or the rise of agriculture — but instead of taking hundreds or thousands of years to unfold, this time around the world could become unrecognisable over the span of decades or less.
This transformation could bring enormous benefits, helping us solve currently intractable global problems. But it could also pose severe risks, some of which could be existential — meaning they could cause human extinction, or an equally permanent and severe disempowerment of humanity.
There aren’t nearly enough people trying to address these challenges, and we think that’s a serious problem.
This article is narrated by the author, Zershaaneh Qureshi. It explores how advanced AI could be so transformative, and why working on its risks may be your best opportunity to have a positive impact on the world.
Bonus: Risks from power-seeking AI systems (article narration by Zershaaneh Qureshi)
vendredi 5 juin 2026 • Durée 01:29:32
Hundreds of prominent AI scientists and other notable figures signed a statement in 2023 saying that mitigating the risk of extinction from AI should be a global priority. At 80,000 Hours, we’ve considered risks from AI to be the world’s most pressing problem since 2016.
But what led us to this conclusion? Could AI really cause human extinction? We’re not certain, but we think the risk is worth taking very seriously.
In particular, as companies create increasingly powerful AI systems, there’s a concerning chance that:
These AI systems may develop dangerous long-term goals we don’t want.
To pursue these goals, they may seek power and undermine the safeguards meant to contain them.
They may even aim to disempower humanity and potentially cause our extinction.
This article is written by Cody Fenwick and Zershaaneh Qureshi, and narrated by Zershaaneh Qureshi. It discusses why future AI systems could disempower humanity, what current AI research reveals about behaviours like power-seeking and deception, and how you can help mitigate the dangers.
You can see the original article — packed with graphs, images, footnotes, and further resources — on the 80,000 Hours website:
Why are the risks from power-seeking AI a pressing world problem? (00:04:04)
Section 1: Humans will likely build advanced AI systems with long-term goals (00:05:43)
Section 2: AIs with long-term goals may be inclined to seek power (00:11:32)
Section 3: These power-seeking AI systems could successfully disempower humanity (00:26:26)
Section 4. People might create power-seeking AI systems without enough safeguards, despite the risks (00:38:34)
Bonus: Benjamin Todd on why we’re updating our career advice for the strangest time in history
vendredi 5 juin 2026 • Durée 01:06:43
The average career is 80,000 hours long. With AI advancing so rapidly, the hours you have left in your career matter more than ever.
Some leading AI researchers think there’s a 10% chance that AI systems begin automating AI research itself this year — and a 60% chance by the end of 2028. This could introduce aggressive feedback loops that completely reshape every industry, institution, and career.
If these predictions are right, the window for influencing the direction of the future could be closing fast. As 80,000 Hours cofounder Benjamin Todd argues in his new book, that makes thinking carefully about your career more important than ever.
Fortunately, there are lots of ways to use your career to make the AI transition go well.
In today’s conversation with host Zershaaneh Qureshi, Ben lays out three scenarios — from AGI by 2029 to a decades-long plateau in AI progress — and explains why not everyone needs to bet on the shortest timeline. A fresh graduate and a senior government official have wildly different leverage, so timing your impact well means weighing where you are in your career against the urgency of the risks.
Ben also addresses the obvious anxieties:
Will AI come for all the jobs he’s recommending?
What’s the point in following his advice if the job market is about to collapse?
Which skills are actually worth building right now?
Chapters: • Cold open (00:00:00) • Benjamin Todd on AI-era career advice (00:01:34)• A deadline for your career plan? ()• Three timelines, one career ()• What if you’re not an ‘AI person’? ()• Ben’s own AI wake-up call ()• How to break into AI safety in 3 months ()• Is mass unemployment coming? ()• 99% automation vs 100% automation ()• Don’t become a plumber to dodge AI ()• Is it already too late? ()
Two: Ajeya Cotra on accidentally teaching AI models to deceive us
vendredi 5 juin 2026 • Durée 02:49:40
We don’t yet have a reliable way to tell whether an AI model is genuinely trying to help us — or faking it.
A model might sincerely want to do exactly what you ask. Or it could be happy to secretly cheat, as long as its answer gets positive reinforcement during training. It might even follow the rules just to gain our trust, all while concealing goals of its own.
The problem is: each of these three motivations scores the same during testing.
Ajeya Cotra — previously a senior research analyst at Coefficient Giving, now working at METR (Model Evaluation & Threat Research) — explains how dangerous this dynamic could become as we train very general and very capable AI models.
She likens humanity’s future trust in AI systems to an orphaned child who inherits a $1 trillion company. This child has to hire someone to run the company, guide his life, and manage his wealth — but he can only choose this person based on a work trial or interview that he designs, with no resumes or reference checks.
And, because he’s so rich, all sorts of people apply — for all sorts of reasons. Some applicants will truly want to help. But the role will attract others who only pretend to care while they’re being monitored, but intend to exploit the child as soon as they can get away with it.
Like a child trying to judge adults, at some point humans will need to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass us in knowledge, experience, breadth, and speed.
And we can’t rely on models’ performance during training tasks to guide us, as current would give the same grades to three vastly different motivations:
Three: Carl Shulman on the economy and national security after AGI
vendredi 5 juin 2026 • Durée 04:14:58
The human brain runs on just 20 watts — a fraction of a cent worth of electricity per hour. What would happen if AI could do the same?
Plenty of people have toyed with this question. But perhaps nobody has followed through and considered all the implications as much as Carl Shulman, whose behind-the-scenes work has greatly influenced how leaders in artificial general intelligence (AGI) picture the world they’re creating.
Carl simply follows the logic to its natural conclusions, leading to a world where:
One cent of electricity buys what costs hundreds of dollars today — medical advice, company management, scientific research — triggering a scramble to manufacture chips and apply them to the most lucrative forms of intellectual labour
The world’s supply of AI researchers explodes from 10,000 to 10 million or more, enormously accelerating further AI progress
Companies operated entirely by AIs are much faster and cheaper than those that depend on people for decision making, and humans are progressively driven out of business
The technical challenges of robotics are rapidly overcome — leading to strong, fast, precise, and tireless robot workers capable of any physical work, and a rush to build billions of them
Human population levels become irrelevant to economic growth, which now depends on how quickly machines can replicate their components. Given how quickly complex biological systems can reproduce — some in a matter of days — doubling every few months may be a conservative estimate
No country can afford not to participate in the economic explosion. Delay, and your rivals’ economies grow 10x, 100x, then 1,000x larger than yours, leaving you ultimately disempowered
As the economy grows, each person could afford the equivalent of a team of hundreds of machine ‘people’ to help them with every aspect of their lives.
But with growth rates this high, it doesn’t take long to reach Earth’s physical limits — the toughest to engineer around being the planet’s ability to release waste heat. If this machine economy and its insatiable demand for power generates more heat than the Earth radiates into space, the planet will rapidly heat up and become uninhabitable for biological organisms.
Four: Rose Hadshar on why automating human labour will break our political system
vendredi 5 juin 2026 • Durée 02:16:47
The most important political question in the age of advanced AI might not be who wins elections. It might be whether elections continue to matter at all.
We tend to imagine the death of democracy as a dramatic event: a coup, tanks in the streets, a strongman tearing up the constitution. But Rose Hadshar, researcher at Forethought Research, believes AI-enabled power concentration could be far quieter — and far harder to stop.
She foresees something insidious: an elite group with access to such powerful AI capabilities that the normal mechanisms for checking power — law, elections, public pressure, the threat of strikes — cease to have much effect. They might continue to exist on paper, but become ineffectual in a world where humans are no longer needed for even the largest-scale projects.
Almost nobody wants this to happen, but we may find ourselves unable to prevent it:
If AI disrupts our ability to make sense of things, will we even notice power being concentrated?
If AI replaces human labour, what leverage will citizens have left to resist?
And what does all of this imply for the institutions we’re relying on to prevent the worst outcomes?
Rose has answers, and they’re not all reassuring.
But she’s also hopeful we can make society more robust against these dynamics. We’ve got literallycenturies of thinking about checks and balances to draw on. And there are some interventions she’s excited about — like building sophisticated AI tools for making sense of the world, or ensuring multiple branches of government have access to the best AI systems.
In this conversation, Rose and host Zershaaneh Qureshi discuss all of this, and more:
Three dynamics that could reshape political power in the AI era
Why AI-powered tyranny would be uniquely difficult to topple
Five: Helen Toner on the geopolitics of AI in China and the Middle East
vendredi 5 juin 2026 • Durée 02:23:02
When OpenAI announced a deal to build massive data centres in the UAE, it celebrated that it was “rooted in democratic values” — a "clear alternative to authoritarian versions of AI." The UAE scores 18 out of 100 on Freedom House’s democracy index. Political parties are banned, elections are fake, and dissidents are persecuted. Saudi Arabia has received a similar deal.
This is what AI geopolitics looks like in practice: messy, contradictory, and enormously consequential.
The two superpowers competing to build superintelligence — the United States and China — are “barely talking at all.” You might expect two rivals developing potentially the most powerful and militarily significant technology in history to be in constant negotiation about how to deploy it without coming to blows.
Instead, the little dialogue that exists keeps collapsing. That’s the assessment of Helen Toner, director of the Center for Security and Emerging Technology — DC’s top think tank focused on the geopolitical and military implications of AI — who has been closely tracking the US’s AI diplomacy since 2019.
Helen isn't sure productive talks are even possible yet. At the government level, there's almost no shared understanding between the US and China of what artificial general intelligence (AGI) is, whether it could arrive soon, or whether it poses serious risks. And without agreement on the problem, negotiating solutions is nearly impossible.
And while the US struggles to engage its rival, it’s empowering its autocratic allies.
If AI capability really does determine future national power, the US has just approved massive data centres with "hundreds of thousands of next-generation Nvidia chips," handing world-class supercomputers to Gulf autocracies — countries that also conduct joint military exercises with China and whose rulers maintain tight personal and commercial relationships with Chinese leaders.
Podcasts Similaires Basées sur le Contenu
Découvrez des podcasts liées à The 80,000 Hours Podcast on Artificial Intelligence. Explorez des podcasts avec des thèmes, sujets, et formats similaires. Ces similarités sont calculées grâce à des données tangibles, pas d'extrapolations !
Critics worry that Anthropic’s efforts to stay at that frontier encourage competitive racing towards AGI, significantly or entirely offsetting any useful research they do.
But Holden thinks this seriously misunderstands the current strategic situation. He believes the problem isn't that everyone wants to slow down but can't coordinate. Many major players simply don't believe the risks are real (or don't care about them, even if they do), don't want to slow down, and would be thrilled if a competitor dropped out — because they’d have a better chance at ‘winning.’
Host Rob Wiblin and Holden discuss all of this and much more, including:
Why an 'AI Chernobyl' might happen without us ever noticing
Holden’s case for working at Anthropic — and whether the company is doing enough for safety
What, if anything, could prompt Anthropic to halt AGI development
Why AI R&D is the main thing to worry about — and why fears about cyberattacks and persuasion are overrated
How the world is handling AGI about as badly as possible
Relevant lessons learned from targeting companies for public criticism over farm animal welfare
How humanity could succeed without dignity, and win despite being stupid
Whether AI companies are too hawkish on China
The frontier of information security: confidentiality vs integrity
This episode was originally released in October 2025.
Chapters:
Cold open (00:00:00)
Holden is back! (00:02:28)
An AI Chernobyl we never notice (00:02:58)
Is rogue AI takeover easy or hard? (00:07:39)
The AGI race isn't a coordination failure (00:18:01)
What Holden now does at Anthropic (00:28:30)
The case for working at Anthropic (00:30:38)
Is Anthropic doing enough? (00:41:30)
Can we trust Anthropic, or any AI company? (00:44:30)
How can Anthropic compete while paying the “safety tax”? (00:50:11)
What, if anything, could prompt Anthropic to halt development of AGI? (00:57:13)
Holden's retrospective on responsible scaling policies (01:00:04)
Overrated work (01:15:45)
Concrete shovel-ready projects Holden is excited about (01:17:58)
Great things to do in technical AI safety (01:22:12)
Great things to do on AI welfare and AI relationships (01:29:53)
Great things to do in biosecurity and pandemic preparedness (01:36:51)
How to choose where to work (01:37:37)
Overrated AI risk: Cyberattacks (01:43:38)
Overrated AI risk: Persuasion (01:53:28)
Why AI R&D is the main thing to worry about (01:57:31)
The case that AI-enabled R&D wouldn't speed things up much (02:09:30)
AI-enabled human power grabs (02:13:26)
Main benefits of getting AGI right (02:26:04)
The world is handling AGI about as badly as possible (02:31:44)
Learning from targeting companies for public criticism in farm animal welfare (02:34:18)
Will Anthropic actually make any difference? (02:43:43)
“Misaligned” vs “misaligned and power-seeking” (02:58:23)
Success without dignity: how we could win despite being stupid (03:04:16)
Holden sees less dignity but has more hope (03:12:00)
Should we expect misaligned power-seeking by default? (03:19:43)
Will reinforcement learning make everything worse? (03:27:36)
Should we push for marginal improvements or big paradigm shifts? (03:32:54)
Should safety-focused people cluster or spread out? (03:35:32)
Is Anthropic vocal enough about strong regulation? (03:39:55)
Is Holden biased because of his financial stake in Anthropic? (03:43:30)
Have we learned clever governance structures don't work? (03:47:57)
Is Holden scared of AI bioweapons? (03:50:20)
Holden thinks AI companions are bad news (03:53:58)
Are AI companies too hawkish on China? (04:00:53)
The frontier of infosec: confidentiality vs integrity (04:05:06)
How often does AI work backfire? (04:07:55)
Is AI clearly more impactful to work in? (04:22:50)
What's the role of earning to give? (04:29:22)
Video editing: Simon Monsour, Luke Monsour, Dominic Armstrong, and Milo McGuire Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong Music: CORBIT Coordination, transcriptions, and web: Katy Moore
Introduction (00:00:00)
Section 1: AI could replace human labour in the most economically valuable fields (00:08:32)
Section 2: Replacing human labour in the most economically valuable fields could trigger the next radical transformation of society (00:22:14)
Section 3: This transformation could be extremely rapid and dramatic (00:28:02)
Section 4: A rapid AI-driven transformation would raise a range of major challenges, including existential risks (00:36:40)
Section 5: Work on these problems is tractable, but neglected (00:44:48)
Objection 1: “You're overestimating how fast and how dramatically AI would transform the world.” (00:47:59)
Objection 2: “It's hard to believe that AI could really pose existential risks.” (00:52:59)
Objection 3: “Isn't all this talk of AI changing the world just a fad?” (00:59:22)
Objection 4: “Isn't AI going to be just like every other technology?” (01:03:04)
Objection 5: “Is it even possible to produce artificial general intelligence?” (01:06:16)
Objection 6: “Even if AGI is achievable, what if we're really far away from building it?” (01:11:24)
Objection 7: “Isn't the real danger from actual current AI and not some sort of futuristic AGI?” (01:14:05)
Objection 8: “Technological progress is a good thing for humanity.” (01:18:10)
Objection 9: “This all just sounds too sci-fi.” (01:19:50)
Objection 10: “Can it really make sense to dedicate my career to solving an issue that's based on a speculative story about something that may or may not ever happen?” (01:22:15)
Objection 11: “OK, AI might pose existential risks, but isn't ‘issue X’ an even bigger problem?” (01:24:39)
Learn more (01:27:51)
Audio editing: Dominic Armstrong Production: Zershaaneh Qureshi, Elizabeth Cox, Katy Moore, and Lou Moran
Section 5: Work on this problem is neglected and tractable (00:47:37)
Section 6: What are the arguments against working on this problem? (00:59:20)
Section 7: How you can help (01:25:07)
Thank you for listening (01:28:56)
Audio editing: Dominic Armstrong Production: Zershaaneh Qureshi, Elizabeth Cox, and Katy Moore
Our production team includes: • Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour • Producers: Elizabeth Cox and Nick Stockton • Coordination and support: Katy Moore and Lou Moran • Camera operator: Jeremy Chevillotte • Music: CORBIT
reinforcement learning
Saints — models that genuinely care about doing what we want
Sycophants — models that just want positive reinforcement for a ‘correct’ result, even if they get there with actions they know we wouldn’t approve of
Schemers — models that don’t care about our interests at all, and only behave correctly as long as it serves their own agenda
Worse still, training might actively encourage deception.
Imagine training a model to run a business, and measuring its success by the balance in its bank account. A highly capable model might experiment with dishonest strategies. Maybe it steals some money and covers it up. (This isn’t a hypothetical worry; models often come up with creative — sometimes undesirable — approaches during training that their developers didn’t anticipate.)
A model that cheats and covers its tracks would look like a star performer — and get reinforced for exactly that behaviour. If cheating is only caught some of the time, the model still might not learn to stop deceptive behaviour. Instead, it might learn that deceiving without being caught gives it a competitive advantage.
In this conversation, Ajeya and host Rob Wiblin discuss the above, as well as:
How to predict the motivations a neural network will develop through training
Whether AIs in training will functionally understand that they’re AIs being trained
Stories of AI misalignment that Ajeya doesn’t buy
Analogies for AI, from octopuses to aliens to can openers
Why it’s smarter to have separate ‘planning AIs’ and ‘doing AIs’
The benefits of only following through on AI-generated plans that make sense to human beings
Which approaches for fixing alignment problems Ajeya is most excited about, and which she thinks are overrated
How we might demonstrate actually scary AI failure mechanisms
How Ajeya’s views have changed since 2020 (00:05:09)
Are neural networks more like a sped-up version of evolution, or a slower version of human learning? (00:17:42)
Situational awareness (00:26:10)
Misalignment stories Ajeya doesn't buy (00:42:03)
The orphan heir with a trillion-dollar fortune (00:59:14)
Saints, Sycophants, and Schemers (01:03:41)
Ways to train safer AI systems (01:23:20)
Aliens and other analogies (01:38:22)
Moral patienthood (01:53:21)
ARC Evaluations (01:55:35)
Interpretability research (02:09:25)
Rewarding models based on how good and sensible their plans seem to us (02:17:48)
Overrated approaches (02:25:49)
Demos of actually scary alignment failures (02:30:57)
Skills to develop for doing useful work (02:37:23)
Rob’s outro (02:47:24)
Producer: Keiran Harris
Audio mastering: Ryan Kessler and Ben Cordell
Transcriptions: Katy Moore
This eventually creates pressure to move economic activity off-planet. There’s little need for computer chips to be on Earth, and solar energy and minerals are more abundant in space. So you could develop populations of billions of digital scientific researchers orbiting in space, sending the results of their work — like drug designs — back to Earth.
These are just some of the wild implications if AGI could merely match what evolution has already managed.
In this interview with host Rob Wiblin, Carl explains the above, and Rob pushes back on whether that’s realistic or just a cool story:
If this is where we’re heading, how come economic growth remains slow now and isn’t really increasing?
Why have computers and computer chips had so little effect on economic productivity so far?
Are self-replicating biological systems a good comparison for self-replicating machine systems?
Isn’t this just too crazy and weird to be plausible?
What bottlenecks might we encounter supplying energy and resources to this growing economy?
Could there be severely declining returns to bigger ‘brains’ and more training?
Wouldn’t humanity get scared and pull the brakes if such a transformation kicked off?
If this is right, why don’t economists agree?
In the last section of the episode, Carl addresses the moral status of machine minds themselves. Would they be conscious or otherwise have a claim to moral rights? And how might humans and machines coexist with neither side dominating or exploiting the other? Learn more and read the full transcript on the 80,000 Hours website.
This episode is the first part of Rob’s marathon interview with Carl Shulman in 2024. The second episode is on government and society after AGI, and you can listen to them in either order.
Chapters:
Cold open (00:00:00)
Rob’s intro (00:01:00)
Transitioning to a world where AI systems do almost all the work (00:05:21)
Economics after an AI explosion (00:14:25)
Objection: Shouldn’t we be seeing economic growth rates increasing today? (00:59:12)
Objection: Speed of doubling time (01:07:33)
Objection: Declining returns to increases in intelligence? (01:11:59)
Objection: Physical transformation of the environment (01:17:39)
Objection: Should we expect an increased demand for safety and security? (01:29:14)
Economists and the intelligence explosion (02:13:31)
Baumol effect arguments (02:19:12)
Denying that robots can exist (02:27:18)
Classic economic growth models (02:36:12)
Robot nannies (02:48:27)
Slow integration of decision-making and authority power (02:57:39)
Economists’ mistaken heuristics (03:01:07)
Moral status of AIs (03:11:45)
Rob’s outro (04:11:47)
Producer and editor: Keiran Harris Audio engineering lead: Ben Cordell Technical editing: Simon Monsour, Milo McGuire, and Dominic Armstrong Transcriptions: Katy Moore
How power concentration compares to ‘gradual disempowerment’ by AI
Slower-moving scenarios that could still get scary
Which interventions could genuinely work — and which might backfire
Rose's most promising approaches to fighting back
Why a ‘Manhattan Project’ approach to AI should worry you — and why international projects aren’t automatically safe either
This episode was originally released in March 2026.
Chapters:
Cold open (00:00:00)
Who’s Rose Hadshar? (00:01:02)
Three dynamics that could reshape political power in the AI era (00:02:38)
AI gives small groups the productive power of millions (00:13:07)
Dynamic 1: When a software update becomes a power grab (00:21:13)
Dynamic 2: When AI labour means governments no longer need their citizens (00:32:06)
How democracy could persist in name but not substance (00:46:18)
Dynamic 3: When AI filters our reality (00:56:13)
Good intentions won’t stop power concentration (01:09:52)
Slower-moving worlds could still get scary (01:25:32)
Why AI-powered tyranny will be tough to topple (01:33:40)
How power concentration compares to “gradual disempowerment” (01:40:16)
Some interventions are cross-cutting — and others could backfire (01:46:03)
What fighting back actually looks like (01:57:33)
Why power concentration researchers should avoid getting too “spicy” (02:06:36)
Why the “Manhattan Project” approach should worry you — but truly international projects might not be safe either (02:11:46)
Rose wants to keep humans around! (02:14:40)
Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon Monsour Music: CORBIT Coordination, transcripts, and web: Nick Stockton and Katy Moore
The justification? "If we don't sell it, China will."
But that claim is transparently false: severe production constraints and US export controls mean that China can’t come close to matching what these deals provided.
In this conversation recorded in Washington, DC, host Rob Wiblin and Helen discuss the above, plus:
How China exaggerates its chip production for strategic gain
The confusing and conflicting goals of US AI policy towards China
Whether it matters that China could steal frontier AI models trained in the US
Whether Congress is starting to take superintelligence seriously
Why Helen rejects ‘non-proliferation’ as a model for AI
This episode was originally released in November 2025.
Chapters:
Cold open (00:00:00)
Who’s Helen Toner? (00:01:03)
Helen’s role on the OpenAI board, and what happened with Sam Altman (00:01:32)
The Center for Security and Emerging Technology (CSET) (00:07:45)
CSET’s role in export controls against China (00:10:59)
Does it matter if the world uses US AI models? (00:21:58)
Is China actually racing to build AGI? (00:27:56)
Could China easily steal AI model weights from US companies? (00:39:13)
The next big thing is probably robotics (00:47:45)
Why is the Trump administration sabotaging the US high-tech sector? (00:49:24)
Are data centres in the UAE “good for democracy”? (00:52:41)
Will AI inevitably concentrate power? (01:07:49)
“Adaptation buffers” vs non-proliferation (01:30:20)
Will the military use AI for decision-making? (01:38:23)
“Alignment” is (usually) a terrible term (01:45:11)
Is Congress starting to take superintelligence seriously? (01:47:42)
AI progress isn't actually slowing down (01:50:10)
What's legit vs not about OpenAI’s restructure (01:58:03)
Is Helen unusually “normal”? (02:01:33)
How to keep up with rapid changes in AI and geopolitics (02:05:19)
What CSET can uniquely add to the DC policy world (02:08:29)
Talent bottlenecks in DC (02:16:03)
What evidence, if any, could settle how worried we should be about AI risk? (02:19:07)
Is CSET hiring? (02:21:08)
Video editing: Luke Monsour and Simon Monsour Audio engineering: Milo McGuire, Simon Monsour, and Dominic Armstrong Music: CORBIT Coordination, transcriptions, and web: Katy Moore
