Lukas Finnveden is a Research Analyst at Redwood Research. We talk about making deals with early scheming AIs: paying them to cooperate rather than take over.

You can read a full transcript here.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.

Rose Hadshar (Forethought) and Nora Ammann (ARIA) talk about power concentration, checks and balances, coups, and post-AGI political economy.

This is the audio version of a video podcast. Watch the video on YouTube: youtu.be/PB855Xpx1kk?si=v05LVM-2VQ3XiT7N

Prompted in part by Rose's recent article for 80,000 Hours on ‘Extreme power concentration’ — 80000hours.org/problem-profiles/extreme-power-concentration

Timstamps:

• 00:00:00 Intro
• 00:01:26 What is ‘power concentration’? Is power concentration the right framing?
• 00:05:25 When can it be good for power to be centralised?
• 00:09:40 Is ‘checks and balances’ a better framing?
• 00:11:16 How AI undermines existing checks and balances
• 00:15:25 Economic power, meme complexes, and cultural influence
• 00:28:50 AI companies vs governments as centres of power
• 00:31:25 The difficulty of knowing where power actually lies
• 00:40:05 Do humans and AIs concentrate power differently?
• 00:48:38 Should we be trying to imagine a better post-AGI political economy?
• 00:56:28 Concrete actions: transparency & whistleblower protections
• 00:59:07 Human-AI teaming and collaborative intelligence
• 01:01:16 AI agent economies and proving trustworthiness
• 01:13:16 Can we train AI to follow the law?
• 01:16:33 Building resilient coalitions of trustworthy agents
• 01:18:42 Closing reflections

Joe Carlsmith is a writer, researcher, and philosopher. He works at Anthropic on the character/constitution/spec for Claude. Before that, he was a senior advisor at Open Philanthropy.

To see all Forethought’s published research, visit forethought.org/research. To subscribe to our newsletter, visit forethought.org/subscribe.