Lukas Finnveden is a Research Analyst at Redwood Research. We talk about making deals with early scheming AIs: paying them to cooperate rather than take over.

You can read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

Rose Hadshar (Forethought) and Nora Ammann (ARIA) talk about power concentration, checks and balances, coups, and post-AGI political economy.

This is the audio version of a video podcast. Watch the video on YouTube: youtu.be/PB855Xpx1kk?si=v05LVM-2VQ3XiT7N

Prompted in part by Rose's recent article for 80,000 Hours on ‘Extreme power concentration’ — 80000hours.org/problem-profiles/extreme-power-concentration

Timstamps:

• 00:00:00 Intro
• 00:01:26 What is ‘power concentration’? Is power concentration the right framing?
• 00:05:25 When can it be good for power to be centralised?
• 00:09:40 Is ‘checks and balances’ a better framing?
• 00:11:16 How AI undermines existing checks and balances
• 00:15:25 Economic power, meme complexes, and cultural influence
• 00:28:50 AI companies vs governments as centres of power
• 00:31:25 The difficulty of knowing where power actually lies
• 00:40:05 Do humans and AIs concentrate power differently?
• 00:48:38 Should we be trying to imagine a better post-AGI political economy?
• 00:56:28 Concrete actions: transparency & whistleblower protections
• 00:59:07 Human-AI teaming and collaborative intelligence
• 01:01:16 AI agent economies and proving trustworthiness
• 01:13:16 Can we train AI to follow the law?
• 01:16:33 Building resilient coalitions of trustworthy agents
• 01:18:42 Closing reflections

Joe Carlsmith is a writer, researcher, and philosopher. He works at Anthropic on the character/constitution/spec for Claude. Before that, he was a senior advisor at Open Philanthropy.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

This is a bonus episode to say that Forethought is hiring researchers. After an overview of the roles, we hear from Research Fellow Mia Taylor about working at Forethought.

The application deadline has been extended to November 1st 2025.

Apply here: forethought.org/careers/researcher

• (00:00:00) Forethought hiring overview and roles
• (00:03:21) Interview with Mia begins
• (00:03:34) Why Mia joined Forethought
• (00:05:59) Daily work, how work at Forethought was different from expected
• (00:08:38) Research examples
• (00:10:58) Who should and shouldn't apply
• (00:14:11) Disagreements and closing thoughts

• Apply for researcher positions
• Referral form (if you know someone who might be a good fit)

This is a narration of ‘Introducing Better Futures ’ by William MacAskill; published 3rd August 2025.

Narration by Perrin Walker (@perrinjwalker).

David Duvenaud is an associate professor at the University of Toronto. He recently organised the workshop on ‘Post-AGI Civilizational Equilibria’ , and he is a co-author of ‘Gradual Disempowerment’. He recently finished an extended sabbatical on the Alignment Science team at Anthropic.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

This is a narration of ‘AI Tools for Existential Security’ by Lizka Vaintrob and Owen Cotton-Barratt; published 14th March 2025.

Narration by Perrin Walker (@perrinjwalker).

Raymond Douglas is a researcher focused on the societal effects of AI. In this episode, we discuss Gradual Disempowerment.

To see all our published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.