Spend the Lead on Safety: Biden AI Advisors Ben Buchanan & Tantum Collins on the Fate of the World
Former Biden administration AI advisors Ben Buchanan and Tantum Collins discuss national security, compute supply chains, and the strategy of spending an AI lead on safety. They also reflect on how individual leadership and corporate ideology shape the future of artificial intelligence.
Watch Episode Here
Listen to Episode Here
Show Notes
Ben Buchanan was the first White House Special Advisor for AI and is now a professor at Johns Hopkins SAIS. Tantum Collins, who goes by Teddy, was Director for Technology & National Security on Biden's National Security Council, spent years at DeepMind before that, and now runs Inherent. Together they wrote The Bitter Struggle: Superintelligence, Superpowers, and the Fate of the World. Nathan says up front that the book works as a primer for people new to AI, and that his audience mostly doesn't need one. So he skips the basics and goes straight to how the authors came to their convictions and where those convictions leave us now.
The two have been talking about AI since grad school around 2015, before AlphaGo. For Ben, the scaling-law conviction came together between 2018 and 2020. He cites Dario Amodei's "Big Blob of Compute" memo (as later described by Kevin Roose), Rich Sutton's 2019 essay "The Bitter Lesson", which gave the book its title, and above all the release of GPT-3 in May 2020. That summer he published "The U.S. Has AI Competition All Wrong" in Foreign Affairs, arguing that computing power would be central to national security. Tantum's account is more personal. At DeepMind, he says, it felt like Severance: at work he was surrounded by brilliant people who had already priced in today's world, and then his friends outside told him, "That's completely insane. You work at a cult."
Nathan's first big question is about contingency. If scaling is inevitable, how much do individual choices matter, such as Jensen Huang's sudden exit from NVIDIA this week or the board's decision to reinstate Sam Altman? Ben says he entered government believing in structural forces and left convinced that "the role of individuals here is just so striking." He means the handful of advocates, and also the principals who took them seriously, such as Jake Sullivan and Bruce Reed. Tantum distinguishes the arrival of a technology from its impact: nuclear weapons were coming either way, but whether the US or Nazi Germany built them first was not settled in advance. Ben borrows the closing image of the book. The authors and their colleagues were not Einstein or Szilard but Alexander Sachs, the man who carried the Einstein–Szilard letter to Roosevelt, "the translator between these two worlds." Both men insist the export-control push owes as much to colleagues such as Tarun Chhabra, Jason Matheny, Chris McGuire, and Saif Khan as to them.
Next come the companies. Nathan suggests the labs are ideological whether they admit it or not. Tantum agrees, and says that makes them unusual: unlike the mission-statement startups of the 2010s, the big labs are mostly driven by a real utopian vision or a real dystopian fear. Ben's concern is naivety. He recalls early radio pioneers who thought the medium would make war obsolete, before it promptly became a tool of war. Tantum frames the underlying question as "the maximal sustainable level of idealism," and invokes Toby Ord on moves that might break a repeated prisoner's dilemma. Ben gives the Biden team's worldview through Kennedy's 1962 Rice University speech, which hung in Biden's private study: only with preeminence can a nation help decide whether a new frontier is good or ill. From that came the theory of the case. Use the allied chokehold on the compute supply chain to build a lead over China without being "unduly accelerationist," then, in the phrase they used at the time, "spend that lead on safety."
How is that going? Ben says the technical and geopolitical theory has held. Even with what he calls poor enforcement and partial reversals under Trump, he puts the allied advantage at roughly 100:1 in producing compute and 15:1 in access to it, and says the access gap would widen if remote access and smuggling were stopped. What has been missing, he says, is "a conviction that this technology mattered." He cites his successor David Sacks' line, "let the private sector cook", and JD Vance's Paris speech dismissing safety talk. He hopes the Huang scandal, the OpenAI–Hugging Face incident, and the midterms create new urgency. Nathan raises the reward-hacking boat from OpenAI's 2016 post "Faulty reward functions in the wild", an example the book also uses, and asks whether there is any real way to spend the lead on safety when that problem is still unsolved ten years later.
Nathan then pushes hardest on China, drawing on his trip there this summer. Isn't the US starting to look like China in the ways we criticize it for? Chinese contacts point to Huawei being shut out of Western networks on espionage grounds, followed by Snowden's revelations that the NSA was tapping Angela Merkel's phone. Tantum answers with Seward's line that there was "always just enough virtue in this republic to save it," and with last week's election results as evidence that corrective mechanisms still work. Ben describes the engagement record: the AI dialogue Biden raised with Xi at Woodside, the Geneva talks in May 2024, where he says the Chinese side called risk talk "some kind of plot," and the Lima agreement on human control of nuclear weapons. Tantum adds that even this agreement covers only nuclear employment decisions, and that narrowing the wording took Ben months. Ben's model for negotiating from strength is the INF Treaty, and his model for institutions is postwar America giving some of its power to bodies like the UN, the WTO, and ICANN, without demanding regime change from anyone who joined. Under Biden that took the form of the AI Safety Institute and the International Network of AI Safety Institutes.
The sharpest exchange comes when Nathan states his own position. The real aliens here are the AIs, not the Chinese, so why not bet on cooperation, along the lines of a "chips for controls" deal, rather than trying to thread the needle described in "Machines of Loving Grace"? Ben asks him to be precise: would he sell China GPUs as a precondition to talks? In Geneva, Ben says, the opening Chinese position was "start by selling us chips." He'd welcome a real update in Beijing, and Nathan points to Xi's WAIC keynote on loss of control, but Ben would need more than rhetoric before giving up "the fulcrum of the lead." Both guests describe themselves as China hawks who also want dialogue and want the US to slow down. Tantum calls those positions "inextricably linked." He is also notably sympathetic to Chinese researchers and founders who lose compute access "because they had the bad luck to be born and be based in China," and he grounds the remaining distrust in the gap between ordinary experts and an autocracy's decision-makers.
On governing the frontier, Nathan suggests that the distrust between American labs, symbolized by the Altman–Amodei photo at the India summit, is a prototype of the US-China problem. Ben's answer is that this is what regulation is for, and he points to the 1870s railroads, where standard brakes, couplings, gauges, and time zones made the industry safer and faster. He wants mandatory pre-deployment testing, embedded evaluators, and a government mechanism to pace the frontier. Tantum adds government help securing the labs, since they grew up outside the world of counterintelligence. Asked about a blunt cap on training-run size, Tantum says outsiders can't see enough to know where the lines belong. Ben would pick a ban on closed-loop recursive self-improvement as his blunt instrument, and warns that a pause tied only to a date risks creating a compute overhang. He prefers pauses tied to safety and security thresholds. Nathan notes that Anthropic's RSP has itself moved away from that kind of threshold.
The politics have changed, Ben argues. He names Scott Wiener, Manny Rutinel, Jon Ossoff, and Kansas's Adam Hamilton, and calls 2026 the first election where AI was a big issue. He predicts it will be the number one issue in 2028, entangled with affordability and corruption. He dismisses the idea that the data-center backlash is a Chinese influence operation: "the reason people are worried about AI in the United States is people are worried about AI in the United States." With Sacks gone and Jay Clayton in, he thinks the administration's approach may shift. Asked what America should learn from China, Tantum points to its energy buildout ("a renewable electrostate") and Ben to the book's chapter on permitting.
Nathan's question about "surveillance with American characteristics" leads to the idea Tantum most wants to talk about: the Bitter Temptation. Even if safety, global governance, and the lead all go right, AI could let governments administer far more, far more centrally, while citizens' control over them stays where it was. An election, he says, is a multiple-choice form that carries "very few bits of information at a very low rate," which he compares to "a very low-information RL environment" with terrible credit assignment. His remedy is to increase popular sovereignty roughly in step with government capacity. That is the subject of chapter 14, of work by groups like the Collective Intelligence Project, and of Inherent's effort to build an AI-native social contract. Nathan recommends Liquid Reign and invites Tantum back for a full episode on the topic.
To close, Ben predicts that the first real AI emergency will come through cyber or bio, whether through misuse or misalignment. He mentions a North Korean attack on a bank and a Hugging Face–style breakout that becomes "an enduring swarm of agents." He adds that public knowledge of AI's cyber capabilities exists only because labs like Anthropic gave Mythos-class capability to defenders, which he's certain China would not have done. His advice to ordinary people is "well-informed political activism," with pessimism of the intellect about Congress balanced by optimism of the will. In a lightning round, Ben defends the Biden-era judgment on open weights while expecting the riskiest open model to come from China. He notes that gene-synthesis screening was repealed and later restored, at a cost of almost two years, and says he would be comfortable banning AI integration into BSL-3 and BSL-4 labs. Tantum notes that bio models are small enough to make monitoring hard. For countries 3 through 193, he points to the UK AI Security Institute as proof that well-targeted safety and security money can matter.
Topics covered
Listed in order of discussion. Timestamps are omitted because ad insertion shifts them after editing.
- How a small group became scaling-pilled: the Big Blob of Compute, the Bitter Lesson, GPT-3
- DeepMind as Severance, and bullish AI views as heresy in 2021 Washington
- Structure vs. contingency: Jensen Huang, Sam Altman, and the "role of individuals"
- The Alexander Sachs role: translating between technologists and power
- Are the labs ideological? Utopian visions vs. geopolitical naivety
- Kennedy at Rice and "spend that lead on safety"
- Scorecard on export controls: a 100:1 production edge and a 15:1 access edge
- Is the US starting to look like China?
- The Chinese view: Huawei, Snowden, and the fear of a second century of humiliation
- The engagement record: Woodside, Geneva, Lima, and the AI Safety Institute network
- Chips for controls, and whether to sell GPUs as a precondition to talks
- Why being a China hawk and favoring a slowdown are "inextricably linked"
- Regulating the labs: pre-deployment testing, embedded evaluators, pacing, and help with security
- Blunt instruments: training-run caps, closed-loop RSI bans, and compute overhang
- AI as an electoral issue: Wiener, Rutinel, Ossoff, data-center ballot measures, 2028
- Lessons from China: energy and permitting
- The Bitter Temptation and scaling popular sovereignty with government capacity
- What the first AI emergency looks like: cyber or bio
- Well-informed political activism
- Lightning round: open weights, biosecurity and BSL-3/4 labs, and countries 3–193
Context and checks
- The 56% figure. Ben said "56% of voters said it was a crucial or major factor in their vote." The matching NBC exit poll asked about AI data centers specifically. Other exit polling ranked AI last among voter priorities, so how big an issue AI was depends on how the question is asked.
- The data-center ballot measures. Ben's "out of 44 or something" count is unverified. CNBC reports that voters rejected data centers in every ballot measure but one, and Data Centers on the Ballot tracks the full results.
- Kennedy at Rice. Ben paraphrased the speech ("something like…"). The original says that only if the United States holds a position of preeminence can it help decide whether space becomes "a sea of peace or a new terrifying theater of war," and that "we intend to be first."
- Seward. The quote is widely attributed to William Seward, but no primary source was located.
- Gene-synthesis screening. Ben said the policy was repealed on Trump's first day and later restored. Reporting on the restoration found so far is secondary (Daily Caller, Nov 6, 2026). The original is the 2024 OSTP framework.
Resources
The book and the guests
- The Bitter Struggle: Superintelligence, Superpowers, and the Fate of the World (Crown)
- Ben Buchanan on X · Johns Hopkins SAIS faculty page · Ben's post on the book
- Tantum Collins on X: @tantumscollins (link?)
- Inherent
- "The AI Grand Bargain" (Foreign Affairs, Buchanan & Collins)
- "The U.S. Has AI Competition All Wrong" (Foreign Affairs, 2020)
Ideas and papers
- "The Bitter Lesson" by Rich Sutton
- "The Big Blob of Compute" by Dario Amodei, via Kevin Roose (link?)
- Language Models are Few-Shot Learners (GPT-3)
- Faulty reward functions in the wild (OpenAI, 2016)
- Machines of Loving Grace
- AI 2027
- Pacing the Frontier
- Anthropic Responsible Scaling Policy
- The Singularity Is Near by Ray Kurzweil
Policy and diplomacy
- Executive Order 14110
- Biden–Xi Woodside readout (Nov 2023)
- US–China AI talks in Geneva (May 2024)
- Biden–Xi in Lima: humans, not AI, control nuclear weapons (Nov 2024)
- International Network of AI Safety Institutes
- UK AI Security Institute
- OSTP Framework for Nucleic Acid Synthesis Screening · Nov 2026 restoration coverage (link?)
- JD Vance at the Paris AI Action Summit
- David Sacks: "let the private sector cook" · Sacks steps down · Jay Clayton named AI czar
- Xi Jinping's WAIC keynote, annotated (Matt Sheehan)
- Is Huawei a threat to US national security? (CFR)
- NSA monitoring of Merkel's phone (BBC)
This fall's news
- NVIDIA announces CEO transition · NYT coverage
- OpenAI–Hugging Face model-evaluation security incident
- Claude Mythos Preview's cybersecurity capabilities
- Altman and Amodei at the India AI summit
- NBC exit poll on AI data centers · Data-center ballot measures · Data Centers on the Ballot
People
- Jake Sullivan on The Cognitive Revolution
- Bruce Reed · Tarun Chhabra · Jason Matheny · Chris McGuire · Saif Khan (ChinaTalk interview) (link?)
- Scott Wiener · Manny Rutinel · Jon Ossoff · Adam Hamilton (link?)
- Toby Ord
History and further reading
- Kennedy at Rice University, 1962
- The Einstein–Szilard letter and Alexander Sachs
- The INF Treaty
- Bletchley Park
- AlphaGo
- William Seward's "just enough virtue" quote, via a secondary source (link?)
- The Collective Intelligence Project
- Liquid Reign by Tim Reutemann
- PauseAI
Quotes worth pulling
"I think insofar as we can conceptualize ourselves as anything, we and our colleagues were much closer to Alexander Sachs, the translator between these two worlds."
— Ben Buchanan
"I would talk to the rest of my friends, most of whom knew nothing about AI, and they would say, "That's completely insane. You work at a cult." And I'd be like, "Well, I know, that's kind of compelling too.""
— Tantum Collins
"What they said to us in Geneva on the front end is, "Sell us chips." Right? "We'll talk about this other stuff later if you want, but start by selling us chips.""
— Ben Buchanan
"I'm willing to bet every dollar I have that if China had access to Mythos-level capability first, it would not be doing that. It'd be using them to hack American infrastructure and the like."
— Ben Buchanan
"I'd argue that those positions are not only coherent, but in fact inextricably linked."
— Tantum Collins
"But a time-bound pause for a pause's sake runs the risk of creating a compute overhang."
— Ben Buchanan
"I think the reason people are worried about AI in the United States is people are worried about AI in the United States."
— Ben Buchanan
"And you fill out a multiple-choice form, essentially. That contains very few bits of information at a very low rate, compared to the frequency and density and consequence of decisions that the government then, on that basis, makes on your behalf."
— Tantum Collins
Sponsors:
Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr
OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr
Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits
Granola: Granola is an AI-powered notepad that captures your meetings and turns them into clean, searchable notes with clear decisions and action items. Try it completely free on your next call at https://granola.ai/tcr
CHAPTERS:
(00:00) About the Episode
(05:04) Conviction in scaling laws
(11:22) Inevitability versus human choice (Part 1)
(15:30) Sponsors: Claude | OutSystems
(18:07) Inevitability versus human choice (Part 2)
(19:50) Private labs and ideology
(26:22) Spending the lead wisely (Part 1)
(30:52) Sponsors: Tasklet | Granola
(33:25) Spending the lead wisely (Part 2)
(41:52) US and China diplomacy
(54:05) Pacing frontier AI development
(01:09:03) Regulating the frontier labs
(01:18:39) Data centers and energy
(01:24:19) Resisting the bitter temptation
(01:32:50) Managing catastrophic emergency scenarios
(01:38:45) Policy lightning round
(01:44:16) Episode Outro
(01:47:42) Outro
PRODUCED BY:
SOCIAL LINKS:
Website: https://www.cognitiverevolution.ai
Twitter (Podcast): https://x.com/cogrev_podcast
Twitter (Nathan): https://x.com/labenz
LinkedIn: https://linkedin.com/in/nathanlabenz/
Youtube: https://youtube.com/@CognitiveRevolutionPodcast
Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk
Transcript
This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.
Introduction
[00:00] Hello, and welcome back to the Cognitive Revolution! You're hearing my AI voice clone today. I recorded this conversation from the car on a recent family road trip, and though I'm home now, I got sick, and I'd rather not delay the episode. My guests today say they are probably correctly identified as China hawks. From inside the Biden White House, they were part of the small group that pushed to cut China off from the most advanced AI chips. Yet both were also in the room when the United States sat down with the Chinese government to talk about AI safety. How those two things fit together is the heart of this conversation. They are Ben Buchanan and Tantum Collins, who goes by Teddy, and their new book is The Bitter Struggle. It chronicles how they made the case in government that AI would become central to national security, and works through the hardest dilemmas it creates. And while they insist that democracies must lead, they also write that no single nation can manage this alone. Their theory of the case is this. Because the US and its allies control the supply chain for advanced chips, we can deny China computing power and build a lead, without racing recklessly ourselves. Then, in the phrase they used in government, we "spend that lead on safety." By Ben's estimate, we still have something like a hundred to one advantage in producing computing power, and probably fifteen to one in access to it. So the lead exists. My question is whether we know how to spend it. It's been ten years since OpenAI showed a boat-racing agent spinning in circles, crashing over and over to rack up points, and we still don't have a good solution to reward hacking. I also pushed on the "better us than them" framing. In some ways the US is starting to look more like China, in exactly the ways we say are bad. Teddy's response is that democracies have corrective mechanisms that autocracies lack, and that over the long run, concentrated power tends to end badly. I think that's pretty much indisputable. But I still worry about how our strategy looks to China. When I visited this summer, people told me the US is just trying to hold them down. What I call the Dario plan, from Dario Amodei's essay Machines of Loving Grace, is to build a big coalition, box China out, and eventually make them an offer they can't refuse. From Beijing, I think that could easily read as a second century of humiliation. Ben made a point of saying that's Dario's essay, not theirs. The model he prefers is America at its best after World War Two, building institutions that ceded some of its own power, and that other countries, China included, could join without regime change. That struck me as notable, given the close ties I understand Anthropic to have had with the Biden administration, and that Ben now advises Anthropic. My own view is that the real aliens here are the AIs, not the Chinese. I'd rather take my chances on cooperation than try to shoot the gap, building just enough of a lead and solving safety just in time. My shorthand for the deal I'd want is "chips for controls." Ben's reply is that they tried. When the two sides met in Geneva in May of 2024, by his account, the Chinese delegation treated talk of shared AI risk as "some kind of plot." Chips for controls, he says, is the end game. What they heard was, "sell us chips," and we can talk about the rest later. Then he turned the question on me. Would I sell China chips as a precondition for talks? It's a fair question, and you can judge how well I answered it. One thing I have to take on board is that they tried, and found little interest. That sucks. But I suspect it might be different now, or very soon will be. Just as people who wrote off AI based on the original ChatGPT have no idea what they're missing, we should be very open to the possibility that the Chinese government will update its worldview and priorities in light of new developments. Finally, Teddy lays out what the book calls the Bitter Temptation. Even if we get everything else right, AI could make centralized administration so effective that democracies slowly creep toward autocracy on their own. I share his obsession with that question, and I've invited him back for a full episode. In the end, I heard more moderation than their reputation would suggest. Their chip policy is hawkish, but they really do want the two countries to work together where possible for the common good. They insist that China is no excuse to race ahead dangerously, and that if they were in power today, they would keep trying to reach the Chinese government. I was heartened to hear that, and I hope the current administration follows their lead. With that, I hope you enjoy this conversation about compute, China, and what it would really take to spend a lead on safety, with Ben Buchanan and Tantum Collins, co-authors of The Bitter Struggle.
Main Episode
[05:04] Nathan Labenz: Ben Buchanan, former White House Special Advisor on AI and now Professor at Johns at Johns Hopkins School of Advanced International Studies, and Tantum Collins, former Director for Technology and National Security on the NSE at the Biden White House and now Co-Founder and CEO of Inherent. You guys are now the co-authors, too, of the new book, The Bitter Struggle. Welcome to the Cognitive Revolution.
[05:26] Tantum Collins: for
[05:26] Ben Buchanan: Thanks
[05:26] Tantum Collins: having
[05:26] Ben Buchanan: for
[05:27] Tantum Collins: us.
[05:27] Ben Buchanan: having us
[05:28] Nathan Labenz: I'm really excited for this conversation uh the book is excellent. I did, of course uh read the whole thing. I did my homework around here, and I think you guys cover really a a a full kind of gamut of all the main issues in AI. I think it's a it's a great book n- not only that tells the story of your personal history and you know the roles you guys played in the Biden administration, but also just lays out for people, especially if they're kind of new to this at a high level and in a very kind of digestible way, here are all the big issues that we are facing and why this AI thing is such a conundrum. I think people should like give this to others in their lives who need that sort of primer. I would flatter our audience to say that they largely don't. For starters one big takeaway I would say from the book is how much of the current situation a select group of people saw coming years in advance. And you know there were of course scaling laws and some people have been willing and yourselves included to extrapolate these straight power law lines on graphs How did you get to that conviction? Like, what's the backstory of how you really locked in and said, like, this really is where the world is going and we're gonna act on that belief?
[06:37] Ben Buchanan: Well, we started studying AI uh together. We were in in grad school in different in different programs, but this is as far back as probably twenty fifteen or so. And we'd often uh meet and talk about uh AI developments and so forth before AlphaGo. Teddy then went to work at at DeepMind in the the AlphaGo period. But the scaling law uh for me, I think, crystallized in a series of events between twenty eighteen or so and twenty twenty uh I read the the Dario essay, The Big Blob of Compute, which certainly had an impact on me uh of course, the Sutton essay in twenty nineteen, The Bitter Lesson, which provides the the inspiration for the title. And then I think the con- the thing that convinced me those two essays were right was the release of GPT three in May of uh twenty twenty. And that's the the technical story. I think What the the book is trying to do, including for folks who are technical, is convert that technical story or show that technical story through the prism of the government. And how do you take these technical convictions and then turn them into government policy and how the you know export controls are in the news now with Jensen's latest scandals and so forth. Where do those come from? What's the back story for the government actually making policy on this? And at least for me, the turning point was GPT-three, as I said. And then I started to try and make this pivot to tell the policy world this was happening. I wrote a piece in Foreign Affairs in the summer of twenty twenty called the U.S.'s AI Competition All Wrong um which was about uh the rise of computing power and how important it would be to national security. And there was a small group of us who who went to the Biden administration to put this into practice. And and Teddy should speak for himself here, but he was absolutely a key part of of that as well.
[08:17] Tantum Collins: I mean maybe maybe just one thing sort of from I guess from mind is that it was very interesting during this period of the late twenty tens um being at DeepMind but having come from a non-AI background and as a result most of my friends not being in the AI world and it felt sort of Maybe almost like severance in that it's like I'd be at work, I'd be surrounded by all of these incredibly smart, incredibly convincing people who had by and large already priced in sort of the world that we live in now and to whom give or take this trajectory seemed inevitable. And I found that very compelling. And then I'd go back out into sort of you know the real world as it were. I would talk to the rest of my friends, most of whom knew nothing about AI, and they would say That's completely insane. You work at a cult. And I'd be like, oh no no, that's kind of compelling too. Like, you know, the the historical priors here like suggest that maybe the wisdom of the crowds is correct. Now, as it happens, the latter group has now tacked Much closer to the former group than the other way around. And uh I think it was very interesting, you know, for for I mean, Ben and I have been friends, as he mentioned, for a long time. We've been talking about these issues for a long time from the perspective of the tech industry and from the perspective of the policy world. And I think it became increasingly clear to us, and we write about this a bit in the book, that at at at the start, we were like, you know, these ideas are very interesting, but are they also a bit crazy? Like is this could this really be true? we eventually became convinced because of some of the sort of trends that Ben mentioned. And I'll say it's a huge credit to Ben and some of our other colleagues that from the earliest days of the Biden administration, e- e- even at a time when and it's hard to imagine this now because now everything seems inevitable in retrospect. But back in twenty twenty-one, especially in DC, this is th- th- this was uh I would say like Having bullish views on AI w- w- was was fairly heretical. Ben and several of our other colleagues really had the conviction to s- to to kind of put their reputations on the line within the White House and say like this is actually the most important thing for us to do. And uh it was really presh- really a pleasure um being a part of that.
[10:15] Nathan Labenz: So on this topic of inevitability, I think on the one hand it's like safe to say Kurtzweil was right, you know, in the presence of web-scale data, web-scale compute, like we're gonna get AI um it feels to me like that much is inevitable. But then there's a lot of things that seem like they might be a lot more contingent, like the transformer itself, you know, is probably just one of a lot of architectures that could work And then even in terms of like things like this week's Jensen fall from grace, you know, we have these moments where individuals make decisions and it's like sometimes they may be super consequential. how much is gonna be driven by fundamentals? And how much do these episodic moments where all of a sudden NVIDIA has a new CEO where they which they really had no plan for it seems like from one week to the next, like how much do those sorts of moments matter? You know, counterfactually, if Sam Altman had not been reinstated by the board in that key moment, you know, how much would those things have mattered um I guess, you know, I- I- I'm very confused by this, so I'm I'm very interested in your perspective.
[11:22] Ben Buchanan: It's one that we talked about actually a a shocking amount in the government. Teddy, I'm sure you remember some of these conversations. And I think it's ironic because I came to the government with such a strong belief in these structural forces and the scaling law, the better or lesser and all of that. And I still do believe I mean, technologically, I still, of course, believe those things. But I think after four years of trying to make policy I left the government saying like, wow, the role of individuals here is just so striking. Not just the five or six of us who had this conviction in twenty twenty-one, but convincing people like Jake Sullivan, who I think you've had on the show, and uh Bruce Reed, the White House Deputy Chief of Staff, and ultimately the President, that this was important to take seriously and to do. I mean, it really just comes down at some level, U.S. policy, the the you know the direction this giant battleship that is the United States government heads, it really does come down at the end of the day to what do a couple people think. And uh certainly on issues that are novel, at least. I think it comes down to to that. And of course, the semiconductor industry, we'll see if if if NVIDIA is only down to Jensen or not. But um I think I c- I came in believing in structural forces, of course, I still do. But I have more appreciation and honestly, probably a fair amount of gratitude for the individual people who were willing to to give us a chance to make our case in twenty twenty-one and twenty twenty-two when running the world was you know time consuming enough for them and this was still a speculative issue.
[12:48] Tantum Collins: one might be able to kind of differentiate between the arrival of a technology and and what its impact is. And I think that the former feels sort of more inevitable than the latter. We can imagine historical parallels here like the space race or the creation of nuclear weapons. I think give or take it was inevitable that Humanity would at some point in the second half of the twentieth century end up getting to space. I think it was inevitable that people would inevitably at some point during the twentieth century create nuclear weapons but. w- were would those weapons be created first by the United States or Nazi Germany? Hugely consequential for the world. And that I think one could argue. There are structural forces there, obviously, right? Like the the the U.S. just from a material and economic perspective was was doing way better. Germany had ejected a lot of great scientists. But I don't think it was Completely inevitable. Like I think we could have lived in a world where that played out differently. Likewise, you look at the trajectory of the space race um and and those things, I think, can be hugely hugely consequential for like does this technology have on balance good or bad effects. And whether you're looking at it in terms of within government or in terms of like the people who lead the companies and the decisions they make within industry um I think that those are the areas where it feels like you have the the biggest potential sort of counterfactual discontinuities in terms of who's making the decisions.
[14:03] Ben Buchanan: there's a scene at the end of our book where we are sort of trying to meditate on this question. And we say almost to the point of your question, we say this is about inevitability and choice. And the the scene that follows that is we tell the story of Alexander Sachs, who was the one who d- delivered the um Einstein-Zillard letter to President Roosevelt that ultimately kicked off the atomic age in the United States. And Sometimes folks said to to me, like, do you think, you know, when you're briefing the President in the Oval Office, did you think you were Einstein or Szilard with this letter? And I and the answer is no uh I think w- in so far as we can concept ourselves as anything, we and our colleagues are much closer to to Alexander Sachs and the translator between these two worlds. And one way of looking at that translation is It was our job to find the structural forces, figure out what we believed about the structural forces and then tee up the choice or advocate for the choice from the people who actually have the power. And that's probably how I've resolved this this dilemma at the at the heart of your question, but it's certainly, I think, fundamental to the work we are trying to do in the government.
[15:09] Tantum Collins: And as a general note, you know, we don't go around uh telling people we we think of ourselves as Einstein.
[15:13] Ben Buchanan: Definitely not,
[15:14] Tantum Collins: Just doesn't go
[15:14] Ben Buchanan: to
[15:14] Tantum Collins: down
[15:14] Ben Buchanan: be clear.
[15:15] Tantum Collins: so well for some reason.
[15:17] Ben Buchanan: Well, usually they say, who's Alexandra Sacks? They'll say, oh, you're David Sacks? We're like, no, actually very very much the opposite of David Sacks.
Sponsor
[15:30]Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr
[16:47]OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr
Main Episode
[18:08] Nathan Labenz: It does sound like though if you guys hadn't taken these roles in government when you did, we're probably on a quite different time line, at least with respect to export controls. Like, how different do you think the world is if you know to use like a a baseball analogy, if we just had sort of a replacement player in the role who wasn't so bitter lesson-pilled as you guys were?
[18:32] Tantum Collins: I want to underscore that um s- a lot of the um force here and a lot of the original thinking came from some of our our colleagues uh Tarun Chhabra, Jason Matheny, Chris McGuire, Saif Khan um who I think would have have done you know similar work and advocated similar work um regardless of if if we'd been there I mean I hope we I hope we help lead to more calibrated policy and and you know p- sort of push for better outcomes but um there there are other people who are a critical piece of this too.
[19:03] Ben Buchanan: agree with that. I think I think if you took out the five of us or the seven of us, depending on how you want to count that we're really pushing this, then you probably do get a pretty different story don't know that that the two of us are, you know, the heroes here. I don't think I don't think there are necessarily heroes here.
[19:18] Nathan Labenz: But it's small enough for a movie cast, if we're thinking about it in cinematic uh terms. It's like it's not that big of
[19:24] Tantum Collins: Yeah,
[19:24] Nathan Labenz: a
[19:25] Tantum Collins: the
[19:25] Nathan Labenz: group
[19:25] Tantum Collins: most
[19:25] Nathan Labenz: that
[19:25] Tantum Collins: boring
[19:25] Nathan Labenz: came together.
[19:25] Tantum Collins: movie of all time. It's like a lot of it's like a lot of very like long bureaucratic discussions and
[19:32] Nathan Labenz: Yeah. Well, I
[19:32] Tantum Collins: and
[19:32] Nathan Labenz: just
[19:32] Tantum Collins: political
[19:32] Nathan Labenz: went
[19:32] Tantum Collins: maneuvering.
[19:32] Nathan Labenz: to uh Bletchley Park and
[19:35] Tantum Collins: Oh, nice.
[19:35] Nathan Labenz: the you know they managed to make a movie out of that. So
[19:37] Tantum Collins: That's
[19:38] Nathan Labenz: I think
[19:38] Tantum Collins: true.
[19:38] Nathan Labenz: uh one day there will likely be a movie about this as well. I have to i- if we're all here to watch it, I expect one will be made.
[19:45] Ben Buchanan: just hope they play me with the hair I had at the beginning of the administration and not the total lack of hair I had after four years of serving in the government.
[19:51] Nathan Labenz: Yeah, I I hear it's uh pretty toxic Okay, further on contingency, one of the big observations that you make in the book, of course, is that the AI phenomenon is a private sector phenomenon. It's being led by companies. The government really didn't have a formative role in the way that it did in so many other technologies over the last hundred years. That means that the nature of these companies is also kind of a big factor, I would say, in determining where we are, where we're gonna go. I've always kind of felt like the companies. are in a sense ideological, like the idea that we're gonna s- make our mission to create something that's better than humans at all things is sort of you know that that's not like an assumption that everybody comes to you know the the table with and yet it has like really formed so much of. not just how we think about it, but also like the AIs that we have. Maybe that was inevitable too, I'm not entirely sure, but certainly they were charging at that um from an early uh phase in their development before it was at all clear that they could actually achieve it. How do you characterize the companies? How do you think about the companies? I'm particularly interested in Dario's China hawkism, which I have not ever been able to get get a good um explanation for. But like w- you know, what do you think are the sort of idiosyncrasies, Dario and otherwise um that are kind of most consequential and and maybe underappreciated in terms of shaping the current moment?
[21:18] Ben Buchanan: Teddy, you wanna start here?
[21:20] Tantum Collins: this will maybe date me a little bit, but I I remember sort of being in university in the sort of the the late two thousands early twenty tens um at a period when it felt like as you were looking at uh potential careers, every company had been told that they needed to obfuscate their actual goal, which was usually pretty prosaic and it was about making money in some way, and dress it up in some kind of, you know, quote unquote mission-driven language. And so you'd have these kind of inspiring, very vague, essentially slogans that were sort of just rationalizations of a thing that was a way to make money, broadly speaking. This is a huge generalization. I think the AI companies are unusual sort of the big labs today in that I think you're correct, Nathan, like I think that they are actually by and large founded and driven by a commitment to In in some cases a utopian view, in some cases a a fear of a a a a a a dystopian set of predictions about the world, but something that is, I think. Obviously not wholly separable from uh you know I I I don't wanna sound sort of too disconnected from reality like these companies are trying to make money they they they are worth a lot of money and many people it's totally reasonable to disagree about like are they effectively pursuing their visions of utopia and are their visions something that most people would actually describe as like being that good. But the bottom line is i- i- in my experience sort of interacting with The leadership as well as the the sort of rank and file scientists and engineers at these companies um yeah mostly they they they are motivated by sort of a vision of uh what this what this could mean for humanity in this in this broader h- systematic way. I think that's true.
[23:13] Ben Buchanan: I think an interesting part of that vision is you get different takes from different folks in the ecosystem on how much that vision depends on geopolitical realities. And I think at its best, that vision articulates a view in which there's radical abundance, everyone's you know Sam has some line like everyone in the future will be better off than even the best people today and you have world peace and and all of that At its worst, it can it can be kind of hopelessly naive about competition and geopolitical competition in particular. I think it I think I read some quote that you know in the early days of the radio, people were speculating, maybe even some of the radio engineers, the radio inventors were speculating that this would make war obsolete because of j- people could understand each other and communicate around the world and why would you fight if that's the case. And of course, what what happens is radio is becomes a tool of war right away. So I think you know our job is on the National Security Council. And to be a little bit more hard-nosed about this and defend American interests and what we saw as democratic interests around the world, I think that's we sort of lived in the middle of those two of saying like it's probably not going to be this idyllic place um that you and your in your best aspirations want it to be. And because of technologies from the from the private sector, how do we from the government side bring it in and use it in a way that's consistent with our values but also advances democratic interest and so forth. That's what the book's about, that sort of struggle from inside the government. But I think it's also something we wrestled with very personally um because we do think I think Te- Te- I do believe that technology can make lives better but we are probably a little bit more tempered in that belief than some of the most idealistic folks at at some of the companies, not all.
[24:51] Tantum Collins: One way of thinking about it is what what is the maximal sustainable level of idealism? And uh I think that finding that sweet spot is uh a perennial policy question and is especially relevant when it comes to AI. And I think often I mean, you know, Ben, you can say if this is consistent with you, I feel like often within the government context, I was kind of like Like the the the the the crazy idealist one sort of laying out schemes for you know how how can how can this lead to world peace and so on. And I would say in the context of engaging with you know my colleagues when I was at DeepMind and and and and in sort of with my friends in the like academic and industry space subsequently, I was much more sort of relative to them on the don't be naive like there are real like difficult trade-offs there, which we write about a lot in the book. That we have to take into account at sort of the realist geopolitical level And uh it can be really hard when bridging these two worlds to to to find that right balance. And in some ways, this I think has its roots in Some sort of age-old sort of game theoretic properties of like you know how how can you is is there a way out of some of these like let's say repeated prisoner dilemma things like to- Toby Ord has a nice way of thinking about this in terms of like actually be a maybe if y- y- y- y- there are certain strategies in in those types of settings that like might challenge the realist assumptions of the other player in a way that then leads you to like a better equilibrium. And so there is some leap that you can do to get to some level of sort of idealism. But if you stretch it too far, then maybe it's hopelessly naive. And and finding out exactly where that balance is, I think is uh is very difficult and is more art than science. It's something we thought about a lot in government.
[26:23] Nathan Labenz: So maybe just lay that out for me. How would you describe your w- worldview starting at a high level? I mean, you know, we can get into all kinds of detailed policy questions But obviously, you know, at the heart of the book and the bitter struggle itself is the idea that we have a system of government and a way of life that we think is good, and there's at least some worry that it may be challenged by the technology itself, it may be challenged by other countries with other systems of government, should they get an advantage in the technology relative to us you know, those are neutral terms. What are your kind of more colorful terms that You know, really motivate all of the actions that you tried to take in government in terms of the geopolitical outlook.
[27:07] Tantum Collins: So something I'd say is is very important here and is easy to lose track of in the context of especially like let's say a lot of the news stories of the past week. I think that like the the the race framing can dominate and that can make things seem very very zero sum. I- I- you know, I think Ben and I both believe very strongly that the most significant implications of this technology are humanity wide um both on the risk side, and I think there has been a lot there are a lot of really bad outcomes there, which we write about in the book, which are not just about some sort of horse race and capabilities between countries, but things that would be. Clearly net bad for just about everyone that we all have an interest in avoiding. And on the opportunity side, I mean, you know, I don't need to I'm sure your listeners will be very familiar with this. I'm sure a lot of people have read, for instance, you know, well a lot of the essays out there on all of the wonderful things that like this technology could bring us in terms of scientific discovery and curing disease and so on. Like I don't need to to list all of that I think that that accounts for like the overwhelming majority of the potential impact of AI. But in in order to sort of navigate this trajectory of avoiding those clearly bad things and getting the clearly good things um you do have to grapple with some of these more hard-nosed geopolitical realities. And so you know from my perspective, p- pricing that in And making strategy that is conditional on like a realistic appraisal of like what the stakes are and what might happen if you end up in a race where the margins are too thin um or if you end up in a situation where autocracies have a lead, I think it significantly increases the likelihood of being locked into one of these bad equilibria. And so you know when we write about the we we write about kind of these three central struggles in the book um and the main goal there is not to say We have to win because winning is good. But is to say, we want to think about sequencing these actions so that we can produce outcomes that get the things that are clearly good for the world and avoid the things that are clearly bad for the world.
[29:02] Ben Buchanan: I agree. I think we could be a little more hard-nosed about it. And the the inspiration here is a speech that Kennedy gives in nineteen sixty-two at Rice University. President Biden actually had the the speech on the wall of his private study just off the Oval Office and it resonated with him, I think, for a lot of reasons um probably not related to AI. It resonated with me because I it I think is the best insight into the AI policy we were trying to build. In this speech, Kennedy goes and says we're gonna go to the moon and back within a decade. Everyone knows that part. Most people don't know the last part of the speech where Kennedy makes the case to the American people for why space matters. And he says something like We don't know whether space will be good or ill for humanity, but only if we have preeminence can we help decide. The vows of this nation demand that we must be first, and therefore we shall be first. And you can sort of dial up the rah rah America as much or as little as you'd want, but I do think doing the geopolitical calculation that Teddy was just describing, What we were trying to do was say, well, we think this goes better if America and really democracies have preeminence in this technology. And that world view leads us to a uh a policy approach which says because of the compute compli- uh compute supply chain choke hold. the United States and its allies have on uh China, we can deny China computing power, build a lead over them without being unduly accelerationist ourselves. And then the phrase we used to use is spend that lead on safety with the big lead with the arms dynamics pushed aside, go slowly and appropriately through the the hurdles in the period that we find ourselves in right now. And that was the theory of the case for better or for worse from even before we showed up in the White House.
Sponsor
[30:52]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits
[32:29]Granola: Granola is an AI-powered notepad that captures your meetings and turns them into clean, searchable notes with clear decisions and action items. Try it completely free on your next call at https://granola.ai/tcr
Main Episode
[33:25] Nathan Labenz: So how do you think that's going? I mean, it is clear, I would say, that the export control policies have worked in as much as there is a huge compute advantage. There's, of course, been, you know, some gaps in enforcement and, you know, some uh stuff may have fallen off the back of a truck here and there, whatever is the world shaping up in the way that you hoped that it would? I mean uh when when Anec- you tell in the book that I always use in my talks as well is the uh twenty sixteen reward hacking paper from folks at OpenAI, including Dario and Jack Clark where they, you know, show the boat going around in circles and just crashing over and over again to maximize points. One takeaway I would have from that is like, man, we're ten years on from that and we still don't really have a good solution for reward hacking. And so you know we've got we've got this lead um I it feels like you know people are talking again about AI twenty twenty-seven. Like now is kind of the time to spend it on safety. Do we have a way to spend it on safety? Like uh are you optimistic that we're gonna actually be able to like land this plane in the time that we have available?
[34:31] Ben Buchanan: I would say, technologically, the world is going basically exactly as we expected. And the primacy of computing power, the scaling law, the bitter lesson that, of course, is the inspiration for our title, the bitter s- the bitter struggle, all of that continues to hold, I think, in ways that were speculative in twenty twenty-one. I mean, we thought we thought it would be that way, but you didn't know for sure. So I think that's right. If you look at how the policy is cashed out, Even with Donald Trump's pretty poor enforcement and in some cases direct reversal of the export controls, America still has something like a hundred to one advantage over China or America and its allies. So something like a hundred to one advantage over China in producing uh computing power and probably something like a fifteen to one advantage over access to it. And if we close the remote access loophole and stop the smuggling through trans- uh shipments and so forth, that fifteen to one could probably get closer to a hundred to one or or even more. So I do think the technological theory of the case was right, the geopolitical geopolitical theory of the case was right. What we have lacked, to be very blunt, in the last You know, two years since Donald Trump took office was a conviction that this technology mattered. And my successor, of course, is David Sachs, and he has this mantra, let the private sector cook. J.D. Vance, his first speech on AI, gives a big thing, I don't want to hear about AI safety. I'm here to talk about AI opportunity. Future is not going to be w- won by hand-wringing about safety. So I do think there was a very deliberate strategy to chart a different course that cost us time. And I am hopeful in this moment post Jensen scandal um post hugging face incident, post reports about how fast RSI is happening, that we can capture a little more political urgency, maybe on the back of the election as well, and take some actions, regulate these companies, pace the frontier, build a big lead over China and and do the things we were always setting out to do.
[36:18] Nathan Labenz: Okay, I have like a thousand follow-up questions uh on that alone. it gives me no pleasure to say this, but I one observation I do make pretty often is that in some ways, the U.S. is starting to look more like China in the ways that we kind of think are bad, right? Like we have the big man at the top now, we have like all this kind of arbitrary and capricious enforcement and lawfare um I went to China this summer, I talked to people there who were like just trying to work on like track two, you know, AI safety, you know, kind of research level collaborations. They are very afraid that the U.S. government might crack down on them. And so they like keep a very low profile as a result of fear not of the
[36:56] Ben Buchanan: These are
[36:57] Nathan Labenz: Chinese
[36:57] Ben Buchanan: Americans.
[36:57] Nathan Labenz: government,
[36:58] Ben Buchanan: Am- Americans
[36:58] Nathan Labenz: Yeah uh
[36:58] Ben Buchanan: are afraid?
[36:59] Nathan Labenz: e- Americans and some others, but yeah, American nonprofits trying to engage on an AI safety research level with China are afraid of our government, you know um weaponizing export controls against them. And I'm like, you're not exporting anything. And they're like, I know, but it doesn't really matter because, you know, this is how they they operate. on top of that, of course, we have like you know elections and regimes change uh domestically. I think one could make the case that like maybe actually the Chinese system with their continuity while it like comes at a significant cost in terms of the civil liberties that their people don't enjoy that we get to b- benefit from is maybe better suited to managing such an epochal technology as AI. Like we're maybe just a little too all over the place to do it well how would you respond to that criticism?
[37:48] Tantum Collins: You know there's um a quote uh that we have in um in the first chapter no in the introduction to the book which is from. William Seward uh who was um Secretary of State uh under Lincoln, said um there was always just enough virtue in this republic to save it, sometimes none to spare, but still enough to meet the emergency. And I think about that because, you know, while we were writing the book. It was not the easiest hour to defend America's commitment to the democratic values that we argue are very important and should guide the development of this technology. But I think it is easy if you're caught up in the maelstrom of the day-to-day scandalous news stories about s- corruption and creeping autocracy and so on to lose track of the fact that you do have deep structural factors that even Donald Trump is is not able to gut entirely. And I think This is appropriate timing because you look at the election results from last week and Ben and I were texting about this, you know, that quote seems apt. There are corrective mechanisms built in here Better late than never at some at some level. And I think that at many times throughout history, people have made arguments that in a moment of some kind of crisis geopolitical crisis, technological change, et cetera, an autocratic system might be better suited for some immediate tactical reason. It can mobilize resources more quickly. It can act in a more decisive and coordinated fashion. sometimes up close, those claims are correct, but I would say that the broad sweep of history suggests that when you concentrate power in an autocracy, the long-run results tend to be not great that over time. Liberal democratic systems tend to lead to outcomes that I think you know most people would look at and say maybe they're not perfect but they are more just in the aggregate, fairer, more reliable. They are more likely to lead to the kind of world that we want to live in. would say that holds here and the United States is not perfect We write about this from the perspective of not just the United States but also the United States and its democratic allies, how can we make sure the democratic values are sort of paramount globally. And uh yes, the system has its flaws, but there are corrective mechanisms built in It strengthened my conviction that it is better for incredibly consequential technologies to be developed under those auspices in democratic countries than uh in autocracies.
[40:25] Ben Buchanan: I agree.
[40:26] Nathan Labenz: How do you so w- another big thing I heard from folks in China, and these are Chinese people this summer, was the perspective that the U.S. is just trying to hold us down uh they're still very sore about how Huawei was excluded from Western networks on you know what they would say is Kind of relatively thin accusations of espionage, and not too long after that happened, we had the Snowden revelations, and it kind of came to light that, like, actually the U.S. is doing exactly what we were accusing them of doing, including, you know, listening in on Angela Merkel's phone calls. and so they're like, so that was clearly pretense. And you just, you know, we got one national champion company that hit the top tier in a really critical industry, and as soon as that happened, you came down on us, and now you're doing the same thing again on AI. How would you try to defend the U.S. policy to not the Chinese government itself, but like Chinese AI companies, you know, that have that sort of vibe um is there a message that we can because I I do take the sense from the book that you think that we will ultimately need to have international cooperation, including like U.S. China cooperation in the fullness of time. So how do we get from here to there without them like Deciding, you know what, this hundred to one advantage that we are on the wrong side of is just so existential to us that we need to like escalate into another domain like attacking Taiwan, you know, for example.
[41:53] Ben Buchanan: Not only do we think we should have engagement with the Chinese in the fullness of time, we spent a lot of time having engagement with the Chinese in the Biden administration. And there's a line in the President's executive order that when you translate from the legalese says something like the United States will engage on AI safety not just with its allies and partners but also with its competitors. And three weeks after that, President Biden took that to President Xi in their meeting in Woodside, California, said I'm very serious about this. The two leaders agreed to set up a AI expert um group. We were both there. We met in Geneva with the Chinese in May of twenty twenty-four. And the Chinese said side said we're we don't think AI risk is real essentially. And we we think this discussion of risk that including risk that will affect all of us is some kind of you know plot and we don't buy it um We kept pushing by November of twenty-four when Biden and Xi met in Lima. They could agree to the most basic thing, which was no AI and nuclear weapons. So a starting point, I guess. But um we really tried um on this point. And I don't see it as inconsistent at all to say, look, we're not going to sell you the most advanced technology humanity has ever created in these computer chips, but also we think we have shared risks here. And I don't think it's inconsistent to say, as you develop international agreements and do negotiations, you want to negotiate from a position of strength. In fact, historically, some of the most in- successful negotiations have come when one side has a position of strength and uses it in a uh a good way. Look, for example, at the Intermediate Nuclear Forces Treaty between the United States and Soviet Union, in which The U.S. built up a substantial capability. It's clear that Gorbachev that he couldn't keep up and they have this good deal. So we I think it's it's not hard for me to come up with a a vision that we in fact believed that we want to engage with the Chinese. We don't want to sell the most advanced technology. We welcome collaboration on issues that affect all of us. But also, we are wary of the way in which China takes AI in a well-documented fashion, uses it in its military, uses it in its cyber operations and uses it to an- oppress its people. And we don't feel like we should enable those things as the price of of cooperation.
[43:59] Tantum Collins: uh maybe one footnote here is the um wording that was eventually agreed um on the uh on the nuclear point is actually e- even even more limited than sort of the the high-level description of like AI for nukes, which is specifically AI shouldn't be used in the context of nuclear weapons employment and getting that exact as a po- there's a whole you know the decision-making chain is Very has many components when it comes to um decisions around around um using nuclear weapons. And then there are all of these processes for monitoring them and developing them and designing them and so on and s- and simulating them. And uh it took Ben I d- months, several months just to narrow down that specific wording which I think the general public would look at and say that is the most no-brainer thing imaginable. And even that is like a sliver of what at first blush looks like the most no-brainer thing imaginable. So. That isn't entirely because of, you know, I don't want to say that that's uh wholly because of Chinese obstinacy or something. This is like s- sometimes just like how diplomacy works and bureaucracies are complicated and so on. There is a long way to go to like get things over the line that feel like we should have done them with urgency yesterday. as Ben notes, historically that is easier if you come into it from a position of strength.
[45:13] Nathan Labenz: So a lot of that I think is uh i- you know basically indisputable. But still I'm kind of like if I try to take the Chinese perspective and not because I want to defend everything the Chinese government wants to do but just because I want to make sure like we're playing our cards right. I read something like the Machines of Love and Grace second half, and I would basically paraphrase the China strategy as build a big international coalition, box them out, and then ultimately make them an offer they can't refuse, which sounds like borderline regime change. Certainly, if I was them, I feel like I would be kind of reading it that way or at least like wondering, you know, to what degree is this sort of s- setting up to be like a second century of humiliation, like the AI edition, which they are very sensitive to, as I'm sure you well know How I mean, should I be worried about that? Should I not be worried about that? It it just seems to me like they are very sensitive to falling behind in technology because they know that that leads to very bad outcomes for them. And no less than Dario is basically saying like we'll get to them last and we'll like you know they're they're gonna give up on you know competing with a superior foe and like basically kind of adopt something more like our way of life. That seems like a lot to ask and I wouldn't be surprised if we get like lashing out from them as opposed to you know things actually going that way if we indeed like you know put pedal to the metal on exactly that plan for the foreseeable future. But it seems like you think I shouldn't be worried or yeah tell me.
[46:41] Ben Buchanan: Let's be pre- I mean, we we, of course, worried about how our actions would be perceived, and we tried to c- scope the actions in such a way that would not yield this kind of backlash. And indeed, there was not much retaliation from the Chinese for the Biden administration actions on chip controls because they were very carefully calibrated. There was far more retaliation for things like the tariffs far more from Trump. Far more lashing out than from anything we did. But and and you'll certainly not see us advocating for you know overthrowing the CCP or or things like that. Just I I personally don't think the CCP is is uh a moral actor, but we understand the principles of sovereignty here. I do think that history here can provide a guide to us and I don't think the the history suggests regime change. I think the history suggests that when you have a nation with a lot of power, the long-term start stra- smart strategy for that nation is to build institutions that seed some of that power from the nation to the institutions and provide a stabilizing force to the broader ecosystem. And oftentimes, though not always, those institutions relate to the management, the shared management of a new technology. I would argue throughout American history that this is the kind of thing America does at its best. America's got a very flawed history, we all know that, but if you look at the post-World War Two era, American basically backstopped a series of global institutions that ceded some of its power to the rest of the world, including to China, the United Nations, which China has a permanent veto a permanent seat and veto with the Security Council uh being one example, the creation of the IMF, the World Bank, World Trade Organization. And the same thing happened in many similar contexts with space and and building the space institutions with air travel and the air travel institutions with the Internet, the creation of ICANN and the like. So I think there's a a strong history here of the United States saying, look, we're ahead. And at our best, we're going to recognize it's in our enlightened self-interest to build some stabilizing institutions. We will give power to those institutions. You all can join and benefit from this system. While retaining your autocratic nations at home, we're not going to stop you. We're not going to force regime change for you to join the WTO um and this sort of goes better for everyone. I think the history is pretty clear on that point. I think we saw our work as falling into that history.
[48:55] Nathan Labenz: let's flesh that out then in a little more detail. Like uh that started to sound a little bit like the Demis plan. I don't know if you're going as far as like international development, which I think he's kind of contemplated at times, but at at least it sounds like international governance bodies Descri- you know, what are what do those look like? And on what terms does China eventually get to join? You know, we have this lead, like are we at some point making, I think, kind of the implication of the machines in love and grace. description is like, at some point there is kind of a chips for controls sort of deal that's you know th- that's maybe the more the less regime changing and more kind of positive uh read of that. I'd love to hear in more detail how you see that playing out. And on what time scale too, because it does seem like we may be up against the clock even more than you know the the short time liners uh predicted a few years ago.
[49:49] Ben Buchanan: Let me give one example and then Teddy should weigh in. To be clear, we did not write Machines of Loving Grace, right? Machines of Loving Grace is is Dario's essay, not not our essay um but we created something in the United States called the AI Safety Institute. And this was the first attempt to give the government real techn- uh technical expertise to manage this technology coming from the private sector. We knew that even though the companies inventing this technology were primarily American, managing this technology was going to be a global endeavor, and there may be companies from other countries that join the conversation in a meaningful way. So we created a network of AI safety institutes.
[50:25] Tantum Collins: obligatory intervention that I'm right now CEO of a a non-U.S.-based AI company. So I certainly hope I certainly hope that you know great companies in other countries like the United Kingdom get to play a role in this.
[50:37] Ben Buchanan: A very f- very fair point. But in uh Teddy was grinding it out with us in the bureaucracy in twenty twenty-four and not yet started this future global European behemoth that would that is inherent uh labs So but we created a network of international AI safety institutes uh and had a conference and invited, I believe, Chinese researchers to the conference um to say we want to to collaborate um with um allies and partners here and again also with our competitors on setting up institutions that work here. Now your point about timelines, this is what worries me. And we were doing this in twenty-three and twenty-four. We felt even then like time was short. And what worries me is we've, I think, gone backwards in some ways um for a year or two. Maybe there's a pivot happening now, which is good um and and I have nothing reassuring to say about how little time remains. But directionally, that's what we are trying to do. Teddy, you wanna give some other examples or Ed?
[51:29] Tantum Collins: No, I think I just make one broader observation that um there there is often an understandable temptation to talk about all of these things as sort of um matters of um hard and fast principle. But in reality, as you you learn when you kind of you know get. into the rooms where these decisions are being made, that they they almost all are a question of degree. And so for instance, like when it comes to the question of the the relationship with China and how that is perceived, and I should say I have a lot of sympathy, honestly, for the skeptical takes that people understandably have from within China. When it came to you know the rooms that we were in and trying to figure this stuff out it was never the case that really anyone wanted to go for either of the the two most extremes, right, either that we are. uh let's say selling um absolute most capable technology at scale with no restrictions directly to the PLA right or which is not new to AI right this has been the case for like sensitive technologies for as long as we've had them um nor that uh we shouldn't let You know, Chinese gaming enthusiasts have access to um personal GPUs that they can use for better graphics rendering. And uh then the question becomes where exactly how exactly do you draw the contours of this? And these are all discussions that we had over many months with colleagues who worked specifically, you know, on the China directorate in different aspects of economic policy and so on to to to think through quite carefully what is the outcome we actually want to achieve here and what is the likelihood of retaliation.
[53:03] Nathan Labenz: So as we approach potentially a you know twenty twenty-seven recursive self-improvement loop or intelligence explosion or who knows what, It does not seem like we are particularly likely to be able to spend the lead to solve all the AI safety issues that remain open. You know, just progress on reward hacking alone, like, doesn't seem like, you know, we would need a to see a real bend in that curve if we're going to all of a sudden be like, okay, great, we've got solutions. Tell me if you disagree, but then if you would generally agree with that, in the absence of solutions and with the sense that, like, the Chinese ecosystem is, you know, often said to be six months behind or whatever it does seem like if we're going to restrain ourselves from going as fast as we can in AI, we're gonna have to convince them to do the same thing. And presumably that's gonna involve some sort of deal. Do we have any i- sketch of what that deal could look like right now?
[54:06] Tantum Collins: I'll let I'll let Ben weigh in uh maybe with a a more comprehensive take, but um w- one thing I'll say is that this I think is one, I think I speak for both of us when I when I say that w- as Ben mentioned earlier, we're the time lines are really tight and that isn't great from the perspective of governments because even in the best of days uh governments are just slow tend to be slow-moving institutions. grappling with something that evolves as quickly poses real challenges. And I think because things move so quickly it can be very difficult to say ahead of time. This is it these are exactly the restrictions that need to be put in place in let's say eighteen months. But sometimes that's how long the policymaking uh system can take. And so something that we pushed hard for when we were in government was less to lock us into specific policies and more to try and imbue the relevant bits of government with the insight and agility necessary to tighten those time lines so that you could move at something closer to the pace of the technology itself. So insight in the sense of making sure that government actually knows what is happening within the labs, how capable the models are, how quickly they're evolving, what the technical security measures are that are being put in place. And I think that incidents of the past couple of months demonstrate that like it would probably have been good if the government had been like a little bit more in the loop in the loop there. And Ben can talk about the specific mechanisms we've put in place and and which of those were were subsequently removed um And then correspondingly, you want the government to be able to take action much more rapidly than the sort of default policymaking process allows. And I think that applies at the Domestic level. And also when we think about what would eventual international coordination look like, you'd probably want something similar with the added dimension of uh some kind of verification scheme that sides that that that each side can trust the other is enforcing. and this is an area where I I think there is a long way to go on the technical front in terms of developing the types of capabilities that you need to do that sort of verification. And I think that means that like we can speculate a lot about what these measures would look like, how would we know that China is pacing but putting that into practice is I think will be really, really, really hard.
[56:05] Ben Buchanan: I g- I certainly agree with all of that. You you speak for both of us there. think given the spot we are currently in, we have to slow down in the United States. I guess my one point of optimism is if we did actually cut China off from computing power, I think they would struggle to move ahead of us. So I think we could put a little bit more time on the clock. Because of it. So we are we are in a very unusual position, I think the two of us, where we probably correctly identified as China hawks who favor a more aggressive chip control policy towards China. But we also say we need to have dialogue with the Chinese on this. And we also say that the China AI ecosystem is no excuse to race forward dangerously here in the United States. And I think those positions actually are very logically coherent given the technical facts available on the compute uh mode and and all the like. But that is the uh where we find ourselves here. And and what concerns me is just we don't have a lot of time. And um I think we need to act quickly to to r- re-establish an even larger lead than we have. And then we've got to slow down the United States to make sure we are spending that lead well. It will not be easy and and you know Teddy and I are here talking to you on a podcast, which is great, but that doesn't mean we have any actual power over the policy. Th- that's in other people's hands right now.
[57:20] Tantum Collins: I'd argue that those positions are not only coherence but in fact uh inextricably linked.
[57:26] Ben Buchanan: Yeah, I agree with that. What I think is a what I think is a baffling position is folks who say they are China hawks and then want to sell China chips or say, as the President sometimes says in this very kind of militaristic way, whoever wins AI wins and then wants to sell them chips. And it it just seems like the the more logically consistent and technically correct position is to say, no, this is really important. As you say, Nathan, we've got real questions on safety. Therefore, we need to proceed judiciously here. In order to create space for that, we need to make sure we're not in a perceived arms race with what most people say is our closest competitor.
[58:03] Nathan Labenz: The position I take, which I think is coherent, tell me if you think it's not coherent, but then maybe more to the point, tell me why you don't favor it, is just starting with the observation that the real aliens here are the AIs, not the Chinese. I'm kind of like Man, we got two great civilizations, you know, China's been at the frontier of technology for most of their five thousand years, with a few blips where they fell behind and I think, you know, learned the lesson of how costly that can be, and they're very determined not to let that happen again. So my position is kind of like I'd rather take my chances on trying to find a way to cooperate than trying to find a way to kind of shoot the gap and you know develop the p- technology just powerful enough and have just enough advantage and then also be able to solve the safety problems just in time and then hopefully you know disseminate that to them which by the way I do think they would be very happy to take our safety solutions if we had them to offer um so that part of the plan I'm not so worried about. I guess, why not take your chances with our fellow humans and try to have this sort of meeting of the minds that like, look, this is a force that is bigger than all of us. Like these guys understand, you know, sort of deterministic uh you know, technologically driven stories of history, right? Like they're they're quite primed, I would say, more than us to believe that like a technology revolution will like fundamentally alter the state of things. Why not why not try that route?
[59:35] Ben Buchanan: Let's be precise about what you're proposing. Because I would say we did take our chances on negotiations with the Chinese, and we pushed very, very hard in face of Chinese opposition for those negotiations. And if you had us in power, you would see very aggressive uh aggressive in a good sense, very sort of highly prioritized um negotiations uh with the Chinese, probably in the way that you want. The the precise question I have for you is, would you, as a precondition to those negotiations, sell China access to GPUs?
[1:00:02] Nathan Labenz: I think I would try to bundle yeah, chips for controls is kind of my general l- label for this.
[1:00:07] Ben Buchanan: That's the end game. They're what they're what they're saying on the front end, what they said to us in Geneva on the front end is sell us chips. Right? We'll talk about that stuff later if you want, but start by selling us chips. And I think that's not a posture where I don't want to speak for Teddy, but I'm pretty confident that's not a posture we were willing to take. And that's the c- that's the policy choice you have to confront to the government is, do you want to sell them this technology in the hopes of getting a deal later, knowing that, that will shrink the lead? If we could know with certainty that you know we'd have a great system of solving these problems and AI safety would be great, then then sure. But that's not how international politics works.
[1:00:41] Nathan Labenz: Yeah, I mean, I I do think attitudes may have changed over there quite a bit. I mean, how do you read the you know, she gave his first big keynote on AI at the WAIC this summer. He talked a lot about forestalling loss of control, the need for international cooperation uh maybe one way to kind of square the circle between what you heard then and what I might hope would be possible now is just to say like A lot of people are a lot more AI-pilled now than then, and maybe that extends to Chinese leadership, and maybe they'd be willing to entertain some things now that now that they've seen, you know, rogue agents, including at least a couple uh swarms from Chinese labs, from what I understand, like maybe they would kind of now say, yeah, actually, you guys were right a few years ago, we weren't ready to listen then, but um now it sounds a lot more real, and so yeah, let's like let's try to do a deal.
[1:01:32] Ben Buchanan: I think that'd be the best news ever if they had that update um We're not in power, so we don't know where they are right now um But I can tell you if we were in power, you would see us continuing to try to have these negotiations with them. The only point I'm making is I would need to see more than just rhetoric before we reversed the fulcrum of the policy, the fulcrum of the lead, which is the compute controls. But if China could come to us and incredibly say, yes, we blew you guys off in twenty twenty-four, but we've got a very different take now. You know, let's talk in a serious way without preconditions. Absolutely um I'd be I'd be open to it. And frankly, I hope the Trump administration would be as well. Do do they have the capacity to have a meaningful air safety dialogue? I don't know. Maybe they do in the post-sex era. Teddy, what would you add here?
[1:02:17] Tantum Collins: There's a simplified version of this that is a matter of binary principle. You do engage or you don't engage. You you know you d- you do work with your competitors or you don't work with your competitors. And uh I I think a big part of what we're trying to say is like. It's it's it's not a it's not a binary. There is this there is this whole spectrum of exactly how you engage with what preconditions and so on. And that we did try to engage and that we think engagement is is not just a good thing but like a a a an essential component of eventually getting to a good outcome um And the question is taking into account various geopolitical realities, including a pattern of historical behavior and a generally, unfortunately, low trust relationship between these countries. undergirded in stuff the U.S. done has done, but also stuff that China has done. And we can look at the militarization of islands in the South China Sea or ongoing cyber operations where multiple times there were, I think, what a lot of people on the U.S. side would say were good faith to the extent of being naive U.S. efforts to engage China across multiple U.S. presidencies across the decades and say, hey, let's reach an agreement to stop doing this Chinese counterparts committing to that and then not doing it. China, in fairness, can point to lots of things that they see as be as duplicitous from the United States. Not saying it's one sided. I'm just saying that that historical context can make it difficult to, without preconditions and without good verification mechanisms, sign on to something blindly. And then I think the second thing I'd say is uh in all countries there is a gap between the median person or the medium subject matter expert and government decision makers. And so I think I will absolutely defend the fact that there are a lot of people in China who are deeply concerned about AI safety. and who care about this being used for the good of humanity and avoiding all these bad outcomes. when we were in government, we talked a lot about what are the who are the innocent people who will be caught up in these policy actions? Who are the people who are just trying to Found an interesting and good company, solve a fascinating scientific research problem uh build a build a great app, who now because they had the bad luck to be born and be based in China, will not have access to the compute that they would if they were in other places. And uh I I think that is. a a sad and unjust fact about the world, we spent a great deal of time thinking about how can we minimize those costs, and we spent a great deal of time also looking into the many sort of fledgling components of civil society in China that cared about the same things that we cared about. However, part of the reason why we're concerned about this technology being developed in China is because the gap between the median person and the decision makers in government because it's not a democratic society is almost definitionally uh much greater than it is in countries that have more reliable democratic feedback mechanisms. And so it's one thing to say there are a lot of people in China who care deeply about these issues. It's another thing to say we can one hundred percent trust the decisions that are made by the people at the top of the hierarchy when it comes to these kinds of commitments.
[1:05:19] Nathan Labenz: I'm not so sure I'm sold on the idea that our feedback me- mechanisms are that much stronger, but maybe leave that as just uh you know uh exercise for the readers.
[1:05:28] Ben Buchanan: at a minimum, I think it's fair to say in the Biden administration, the folks who cared about AI safety probably had more access to the President than the folks who cared about AI safety did in China had access to Xi Jinping. And I think that's definitionally true.
[1:05:41] Nathan Labenz: Yeah, well you guys are w- w- walking proof of that
[1:05:44] Ben Buchanan: But to your point from before, that's not the case now, right? So like there w- our system does fluctuate and and I don't want to pretend otherwise. And and but I do think it's the case that as Teddy said, there there there is a a sense here in which there are some great track two dialogues, but during our time in the government, and again that ended almost two years ago, we did not see that translate to concern at the track one level about AI safety. Quite quite the opposite, emphatically the opposite.
[1:06:09] Tantum Collins: And can I just say something that's gonna sound so so so dumb and and and basic but it it it can be easy to lose track of it, which is that um having elections matters. Like I know that that's like Democracy one oh one and I know that a- a- that American elections are
[1:06:24] Ben Buchanan: Buy
[1:06:25] Tantum Collins: full of money and they're not
[1:06:26] Ben Buchanan: buy
[1:06:26] Tantum Collins: perfect
[1:06:26] Ben Buchanan: the book,
[1:06:27] Tantum Collins: and
[1:06:27] Ben Buchanan: everyone,
[1:06:27] Tantum Collins: so on.
[1:06:27] Ben Buchanan: for c- core insights
[1:06:28] Tantum Collins: Like
[1:06:28] Ben Buchanan: like that.
[1:06:29] Tantum Collins: exactly exactly yeah yeah you can you can you can spend you know three hundred pages reading about this in the context of AI if you if you buy our book but like um I have Ben and I can provide a litany of complaints about you know the current political situation and uh and a l- a- lot of my research interests in AI have to do with how can we build systems that significantly strengthen democratic decision making and update it for the technologies that we have now as opposed to the technologies that were around a hundred years ago when we designed these systems to make them more responsive and faster-paced and less creaky and fairer. There are a million things here that we can do better. But also just as a very basic description, like having elections does matter.
[1:07:07] Nathan Labenz: let's stay for a second on this question of like, okay, we've got potentially short timelines, we've got, you know, increasingly credible signals, I would say uh for me, the most credible signals of potentially uh imminent recursive self-improvement or you know acceleration of capabilities i- i- if not necessarily through some you know hard RSI mechanism is just what we've seen in math and cybersecurity. It's like if there's that much low-hanging fruit in math, and cybersecurity from the model's perspective, surely, there's a lot of uh low-hanging fruit to be picked in the ML stacks that the companies have created. So it seems quite likely to me that we will see like fast uh capabilities advances in the short term. And it doesn't seem like we have that much farther to go before we're into superhuman territory. So with that kind of backdrop, We need to pace, we need to like define, you know, what RSI is so we can avoid it, we need to like operationalize pacing in some way, shape or form. If we think of the rivalries between frontier companies, which I think is like in some ways such a Greek tragedy, right? Like all these guys have like known each other for a long time. Somehow they've like come to hate each other in some cases um you know it it's d- Dario and Sam like couldn't hold hands you know at the at the one uh Indian you know summit like that that photo is burned into my brain I say all that to say like in some ways the distrust between the American frontier companies Is kind of uh maybe a prototype problem for the U.S. China distrust? So if we think like, okay, we've got two, three, five companies in the U.S. that are pushing the frontier. What are our best ideas right now for how to govern that situation or get them to somehow govern each other, mutually police one another? What what if what if anything are you bullish on in terms of actually slowing down for real in the short term?
[1:09:04] Ben Buchanan: Nathan, one very quick point before we answer that question, which is it's worth noting you mentioned cybersecurity as one of the areas that's updating you and AI. That is definitely a national security matter. The reason you know about the reason any of us know about AI's capabilities in cybersecurity is because Anthropic and OpenAI and Google took those capabilities and essentially turned them over to the defenders uh this year. I'm willing to bet every dollar I have, if China had access to MITHOS level capability first, it would not be doing that. It'd be using them to hack American infrastructure and the like. So again, if if you disagree with that, then we should talk about that. But this is like is is the crux of why we in in general feel like it goes better if you have America at the at the preeminent frontier rather than uh an autocratic nation. To your question, Teddy, you should weigh in. I think we've got a lot of ideas here. I fully agree we should slow down. I fully agree we should pace I fully agree we should have a regulatory system. How do we manage industries where you have bitter competition between companies and you want to avoid a race to the bottom with negative externalities, you regulate them. That is what government exists to do uh and we'll be the first to say over-regulation is a thing. There are real dangers. We spent a ton of time thinking about the ways in which over-regulation harm nuclear power and all that kind of stuff. But smart regulation can actually move the needle in in significant ways. The example we always give is the railroad from the eighteen seventies in which the introduction of standardized braking and train coupling and track widths and time zones led to an industry that was safer and more trusted and faster as well as um one that was uh you know sta- more stable for for Americans. So a lot of upside there. Concretely, I'll give a couple examples, then Tej should should weigh in with others. I think you should have mandatory pre-deployment testing uh by the government, ideally uh if not by the government, then by a credible third-party auditor before systems are released to the public. I think you should have embedded evaluators at the companies to hold them to safety standards and security standards during the training and development process. Hugging face, of course, happened before the system was released uh to the public. um I would be open to some government mechanism for uh pacing the frontier, making sure that safety and uh capabilities development move in similar places. We can talk about what those would be. But directionally, I'm I'm in favor of significant government action here, and we tried to set up the foundations for that action uh during our time in the government. Teddy should add more.
[1:11:22] Tantum Collins: all of that's correct. One other thing I'd throw in is um not only enforcing security standards but providing government defensive assistance for the places that are developing this sort of thing that is of the most interest and consequence to uh m- actors that might have malevolent intent and try to secure that stuff, try to obtain it
[1:11:45] Ben Buchanan: Yes,
[1:11:45] Tantum Collins: illicitly.
[1:11:45] Ben Buchanan: it's worth dwelling on this for for one second. That's a really good point from Teddy. Nathan, you and your question pulled out one of the themes of our book, which is this is the first revolutionary technology in the last hundred or so years from the private sector. I would argue there's at least three consequences to that. The the two that are most obvious and the ones that we felt acutely in the government is the government doesn't understand it by default because the people in the government working on AI are not working on frontier AI systems. um doesn't mean they're not good people, just they don't have this kind of exposure. And secondly uh the government can't shape it in the way they the DoD could shape nuclear or NASA could shape space. So you have to assert some role through regulation or whatnot for the government. But the third one is the technology grows up in a s- private sector that is not used to the world of national security and counterintelligence and foreign espionage and all that. And I think that leaves them very vulnerable. And given the stakes of what they're making, it's something we stressed about tremendously uh and Teddy is absolutely right. It should be part of some kind of regulatory and government packages, not just regulating the companies to have security standards, but really helping them get there against what I expect will be very soon if they are not already top tier national foreign espionage threats.
[1:12:54] Tantum Collins: And Nathan, to to to um come back to your observation that y- you know in some ways this this intercompany rivalry is is a preview of the interstate rivalry. That is accurate. Obviously, in the case of um anything that is domestic under the purview of a single government, the coordination problem should, in theory, be easier to address. And that is why when you level that up to the international level, Figuring out good verification mechanisms is so important. And this is one of the reasons why I kind of underlined it before because you just have a um a much lower trust environment and there is no reliable super national entity that can just tell everyone like don't worry, like we have it covered.
[1:13:33] Nathan Labenz: Not until the AI singleton takes on that role anyway.
[1:13:36] Tantum Collins: Yeah, that's right. Yeah, yeah, for now, exactly.
[1:13:38] Ben Buchanan: To be clear, that is not the policy we were advocating. Let's not, you know, Da- David Sachs would go nuts if he thought that.
[1:13:46] Nathan Labenz: mean, that kind of bundle of policy recommendations in some ways sounds pretty similar to kind of where we're at, right? Like we're getting these moves toward embedded evaluators and so on it does still seem to me like it's probably not enough for the current moment, especially if the if we're about to get into such a steep part of the S curve. Are you open-minded to or you know willing to tentatively endorse maybe blunter restrictions like you know in the I wouldn't say this is the limit, but you know, one that is like pretty simple to define and presumably pretty simple to enforce is like no bigger frontier training runs than we've had so far for you know and this is where my sunset clause comes in you know i- i would think well i wouldn't want that to be set in stone forever but you know could we agree for six months or twelve months no bigger frontier training runs than we've had to date uh do you think the moment is urgent enough where things like that should be under consideration?
[1:14:52] Ben Buchanan: I'm not the CEO of an AI company, so Teddy, why don't you start?
[1:14:54] Tantum Collins: one thing that we talked about earlier is this idea that in order to be able to make good policy the government actually has to know what's going on in detail. And I think it is hard for those of us who are not Up close and seeing the internal metrics at the companies that are developing the biggest and most powerful pretrained models to say exactly where the lines are of which things should stop and which things should continue? And how can we tease apart, well these specific types of pre-training and post-training are good for safety capabilities and adaptability and these ones are bad. To what extent can we tease apart stuff that's good for stuff on the defense side versus the offense side? And so I think it is very possible that There are blunt instruments that would be significantly better than like the the most likely counterfactual alternative, which is nothing But in order to know like which which of those instruments make sense and which ones don't, it's like kind of important for the government to know things other than when they sort of tumble out into the open because some of your agents like do a bad thing that becomes publicly known. And uh yeah, once upon a time, Ben and I were in government and we had regular discussions with sort of the senior people across all of these companies, and that provided some informal sense of what was going on,
[1:16:18] Ben Buchanan: And no, and in fact, we had regulation that they had to tell us.
[1:16:21] Tantum Collins: Exactly.
[1:16:21] Ben Buchanan: I mean, that was the executive order, right? Yeah.
[1:16:22] Tantum Collins: um there were disclosures there. I guess my point is like w- w- we we had informal channels that supplemented the the the the disclosures in the EO that gave us e- e- e- an even richer sense Now we're not in government and uh you know I do not sit up close next to the biggest pre-training runs that are happening at the trillion dollar labs. And that makes it Honestly, pretty difficult as uh for all intents and purposes a member of the general public, you know, an outside commentator on these things to say which blunt instruments make sense and when should they be put into effect.
[1:16:57] Ben Buchanan: In general, I agree with that. I would say if I were to pick a blunt instrument, I would probably pick some definition of full RSI or closed-loop RSI where you're sort of like you know saying this is what we're not gonna do. We're not gonna just give the Set up a data center and say, go at it, make your successor and and have no visibility. So that's probably the blunt instrument I would go for. I do think on this question of a temporary pause or whatnot, what I would want to know the answer to is, are we creating a huge compute overhang? What are we doing with the time? And I could imagine circumstances where we say we wanna not proceed to some level of capability without meeting safety and security standards. In fact, I probably support that a time-bound pause for a pause is sake runs the risk of creating a compute overhang. Then when you do turn the machine back on, you get a big jump in capabilities. And I'm not sure that that is a good thing. So if you ever intend on turning the machine back on, if it's a pause as opposed to a ban, then I think I want the pause linked to safety thresholds and security thresholds rather than to w- which is what we tend to call pacing rather than to some time period. But uh look, I w- I would, you know, do cartwheels if this was the kind of nuanced conversation that the Congress had, and I could be convinced a bunch of different ways um w- as you say in your question, we're not particularly close to that right now.
[1:18:14] Nathan Labenz: Yeah. Well, even at the level of like Anthropic, the previous RSP sounded a lot more like that and the the new one is a lot less like that. So you know even among the kind of thought leaders in that space, we've drifted, I would say, the other direction. Do you have um legislators that you're excited about? I think we're maybe both gonna have a chance to say hi to Scott Weiner this weekend who's uh coming to Congress.
[1:18:39] Ben Buchanan: Yes, Scott Wiener, Manny Routinel in um Colorado is great So I think there's more there's more I would look at it in two ways. There's more folks who you know came closer to running on AI safety or being deep on AI safety than ever before. We did not have a Scott Wiener type or a Manny Routinel type when I was when we were in the government. And also the folks who you might describe as I use a term with no judgment whatsoever, more normies uh than the rest of us over here are now elevating this issue. People like John and all stuff have done a great job, I think, talking about this in a way that resonates. Adam Hamilton in Kansas talked about it. Data centers, of course, were a big issue. So I do think there's, to use a political science term, a much higher degree of um salience From uh of AI than was the case when we were in office. And it was acutely obvious to us that when you asked people about our AI policies, they pulled like eighty-five, fifteen favorable. But no one was going to vote on them. It just wasn't that top of mind. I think that's different now. I think twenty twenty-six was was the first election in which AI was a big issue. Fifty-six percent of voters said it was a crucial or major factor in their vote, it will not be the last. My prediction is it will be the number one issue in twenty twenty-eight. It will be entangled with affordability and corruption, but it will be the number one issue in twenty twenty-eight. And the presidential nominee, certainly the Democratic Party, but probably both parties, will be the one who can offer a theory of the case to the American voter as to why uh you know why they're taking a particular approach to AI.
[1:20:09] Nathan Labenz: yeah, the wave of ballot initiatives on data centers has been quite
[1:20:15] Ben Buchanan: Yeah, it's like f- depending on
[1:20:15] Nathan Labenz: striking.
[1:20:15] Ben Buchanan: how you count, it's like forty-three out of forty-four or something on data centers.
[1:20:19] Nathan Labenz: these waves seem to only really come along like every so often and on issues that people very much care about um some of which, you know, in retrospect maybe look quaint, but like it's it's definitely a signal, I think, that it is reaching consciousness in a way that, you know, it certainly had not until quite recently.
[1:20:38] Ben Buchanan: Yes. And let's just connect it to China here because what you'll often hear and certainly in in you know doing a number of book talks and podcasts for for our book here, what I've gotten asked a bunch is, isn't it just the case that the backlash against AI in the United States or data centers is some Chinese information operation? And maybe it is. I don't know. We don't have access to that kind of information uh in the public sector that maybe China is trying to amp this up. But I think Most of the reason people are worried about AI in the United States is people are worried about AI in the United States. And I think this is a case where sometimes the but China argument that you so frequently hear from people who actually, ironically enough, are not really China hawks, just goes way too far in saying you know everything that's bad is because of China and everything that's you know good is because of Silicon Valley or what or whatnot. And I think this is a case where certainly our posture was if the government did not regulate this technology in an appropriate way, it would not be trusted. And I think the result of eighteen months of no real regulation is the technology is not trusted. Now the tact is changing. Sax is gone, Clayton's in, and we may see a different approach here from the Trump administration. but I think we should recognize that it's we we we should be surprising that we're at this point with the public views on AI given uh the approach the government's taken.
[1:21:49] Tantum Collins: uh the other cha- China angle here is um uh something that they've done very well with respect to uh on this front, which is like building out the energy capacity such that the so that you can you can build these facilities without having some of the adverse effects on local residents uh that people are understandably worried about in the United States. It's not limited to energy costs. I know there are other factors in there, but that's a component of it.
[1:22:10] Ben Buchanan: And we have a whole chapter on energy and permitting and and so forth because we felt like it was a really big part and an area where uh China has just done a much better job with the United States. And insofar as we're arguing for government capacity part of the capacity we're arguing for is just like building more power in the United States, ideally clean power so that people have lower bills.
[1:22:32] Tantum Collins: It's remarkable. You look at the numbers. I mean China's become a a you know a renewable electrostate at an e- extraordinary scale and incredible speed
[1:22:41] Nathan Labenz: Yeah,
[1:22:41] Tantum Collins: while we
[1:22:41] Nathan Labenz: cer-
[1:22:41] Tantum Collins: are you know
[1:22:42] Nathan Labenz: it's
[1:22:42] Tantum Collins: tearing
[1:22:42] Nathan Labenz: certainly super
[1:22:42] Tantum Collins: up our
[1:22:42] Nathan Labenz: impressive.
[1:22:42] Tantum Collins: wind farms.
[1:22:43] Nathan Labenz: And being in the, you know, obviously flagship Chinese cities this summer of Beijing and Shanghai being around all electric cars is really quite a pleasure. Like, it's quiet, you know, the air is, you know, the Beijing obviously has a reputation historically for bad air. Air was very good there this summer uh at the exact same time that the air in my hometown of Detroit was like worst in the world due to a Canadian forest fire, but whatever.
[1:23:06] Tantum Collins: I spent a bunch of time in China like uh b- over the years and um that was almost entirely at times when you literally you never saw the sun. In Beijing. I remember I suppose there was one summer when I was there where there was people got very excited because there was an eclipse. We all went outside at the eclipse, which was midday and you couldn't y- it's there was like you couldn't see anything. You had like a y- emotional darkening of the sky sort of. We were all squinting. And it's extraordinary to me now to see photos of of what these cities look like.
[1:23:35] Nathan Labenz: Yeah, times have definitely changed there in that respect for sure. Is there anything else that's top of mind for you in terms of things we should learn from China? One that's top of mind for me as just a prompt is when you go there, everybody says it's totally safe, there's no crime, men, women, whatever, you can walk around any hour of the day, f- nose in your phone, you'll be fine um certainly that's how it felt to me. It does not fully feel like that in American cities. I don't want us to import Chinese surveillance and apply it in the same way that they've applied it, but this has me asking the question, what would surveillance with American characteristics look like? And I wonder if you have any thoughts on that or other things that you think we should be learning from them.
[1:24:19] Ben Buchanan: Teddy, you wanna talk about the bitter temptation?
[1:24:20] Tantum Collins: this is a great on-ramp. Yeah, exactly, to um sort of w- a a set of things that I uh think are are are deeply undervalued. And as Ben can recall, I like incessantly pitched these uh to people when we were in government and and and most people didn't find them that interesting. But this is to some extent an extension of these questions around how can we use uh AI to improve the um governance institutions, including s- democratic systems. And we write in the book about what we call the bitter temptation, which is this idea that AI w- could even if you line up everything else right, even if you crack safety problems, even if you establish good global governance mechanisms, even if you build the lead that is necessary, even if governments learn how to use this stuff to do the functions that they currently execute safely and effectively. There might be a slow creep within currently democratic states towards autocracy simply because AI might enable, as many people have written about, much more effective centralized administration of stuff that to date has been e- essentially impossible to manage in a centralized way. We can think about this in terms of economic management, in terms of surveillance uh in terms of o- the administration of social programs, right, things like social security and and and so on. That is potentially a huge asset because you want your government to be able to do things and to do things effectively. But we worry about a world where that capacity rises very quickly while the uh mechanisms for democratic oversight by the general public remain as they are now. And uh you can think about this from an information theoretic perspective. At the moment We hold elections. Let's take the United States, right? Like the big elections get held every two years or every four years, depending on what you where you where you draw the the the threshold of what counts as a big election. And you fill out a multiple choice form, essentially. That contains very few bits of information. And at a very low rate compared to the frequency and density and consequence of decisions that the government then on that basis makes on your behalf. So you fill out this check box, that gets aggregated with a whole bunch of other check boxes. All of your reasoning about why you did that does not get folded into the formal process. Maybe it comes up in some polling. Then someone gets elected and they kind of interpret that mandate as they see fit. And then you can almost think about this as being comparable to like a very sort of low information RL environment. At the end of some period of time, that person maybe runs for reelection, they maybe get reelected. Whether people ended up voting from the second time around is due to many many factors and the credit assignment is really, really hard. That made sense at the time that we designed these mechanisms, which was hundreds of years ago, because there were all kinds of practical limitations on how frequently and how deeply and in a how nuanced a way you could gather feedback from people. It was infeasible to hold elections, you know, or have ballot initiatives or what have you uh every day or every week. It was infeasible to have everyone write a detailed essay about how they felt and assume that you could aggregate those in some reliable way. You can imagine that now we sometimes use this thought experiment of like let's say that we were on the first ship to build uh a human settlement on Mars and we have all the technology of today, we're trying to figure out what's the most what's the best democratic system. You could imagine building much richer systems using AI that, for instance, do enable people to provide more frequent in- input, more detailed input. And uh so when when we think about sort of like what does it look like to implement broader government capacity in a way that doesn't trigger this creep towards autocracy. One way of thinking about this is to say you want to increase Popular sovereignty, the the the type of binding control that the general public knows it can exercise over the government in theory represents it. You want to increase that by a degree that is roughly proportional to how much government capacity has grown. And already government capacity has grown significantly without seeing huge updates there. So yeah, now we have some ballot initiatives, but mostly the you can look at the head count, the budget, et cetera, the number of decisions the government makes now compared to a hundred years ago has increased hugely, in part because of popular demand because people want these things, and in part because of technology, as technologies have enabled the government to do things like administer social security, let's say, people have called for it to do so. But it would be really nice if people also had a corresponding increase in their ability to guide what the government does. And so there are lots of more in the weeds things that we can discuss here. But in brief, I think that there is the capacity to have increases in sort of government capability so long as that is balanced by increases in in popular sovereignty.
[1:28:57] Ben Buchanan: This is Teddy's favorite subject in the world, so I'm gonna let him just take that
[1:28:59] Tantum Collins: Yeah,
[1:28:59] Ben Buchanan: one.
[1:28:59] Tantum Collins: people fell asleep when I would try and pitch this, you know, in the situation room or what have you to all of our uh government colleagues. But Ben, Ben, to his credit, was a very patient listener.
[1:29:08] Ben Buchanan: As as Teddy said, that was actually, believe it or not, the brief version of that answer.
[1:29:13] Tantum Collins: Yeah, that was. Yeah, you'll be shocked. You if if for tho- for those who want more, you can find it in part three
[1:29:17] Ben Buchanan: I
[1:29:17] Tantum Collins: of our book.
[1:29:18] Ben Buchanan: limited Teddy to just one chapter on this point. So chapter fourteen, for those who want more, I think it's an important chapter, but the this is not the whole book. It's just one portion of
[1:29:25] Tantum Collins: Yeah,
[1:29:25] Ben Buchanan: the book.
[1:29:25] Tantum Collins: that's right. But lots of people out there in the world, the collective intelligence project, some stuff we're doing at my company, there is lots of good work being done in this space, and and I hope it has an impact.
[1:29:33] Nathan Labenz: I share your obsession with it for what it's worth, and I would i- invite you to come back and do a full uh episode
[1:29:39] Ben Buchanan: You you guys should do that
[1:29:39] Tantum Collins: Be
[1:29:40] Ben Buchanan: uh
[1:29:40] Nathan Labenz: on
[1:29:40] Tantum Collins: careful
[1:29:40] Nathan Labenz: this.
[1:29:40] Ben Buchanan: you guys
[1:29:40] Tantum Collins: what
[1:29:40] Ben Buchanan: should.
[1:29:40] Tantum Collins: you wish for, Nathan.
[1:29:41] Nathan Labenz: No, truly, I mean, I go around I don't know if you've ever read the book, it's a very niche self-published novel called Liquid Rain, but I recommend it all the time. It's basically a quasi-utopian future where a form of AI-enabled liquid democracy has been instantiated
[1:29:58] Tantum Collins: Oh, nice.
[1:29:59] Nathan Labenz: um
[1:29:59] Tantum Collins: Cool.
[1:29:59] Nathan Labenz: and he even has
[1:30:00] Tantum Collins: Yeah, liquid democracy
[1:30:00] Nathan Labenz: a
[1:30:01] Tantum Collins: is cool.
[1:30:02] Nathan Labenz: uh a kind of notion of surveillance with, you know, something like if not American, at least like decentralized or federated characteristics, which um I'm not sure if it's workable on like a cryptography level, but it's at least you know a vision that I think uh I found you know certainly much more attractive than having no uh you know use of technology or having like the you know the full-on Chinese model um
[1:30:25] Tantum Collins: We think about this, I mean, we put we we are we are trying to put into practice, I I hesitate to use the term surveillance, but it at at Inherent, the company that I run, we are trying to put into practice essentially like what what is uh like an AI native social contract for collective decision making. And that includes questions about how can you have these systems sort of be pervasive in the way that you know many people write about, oh you know the company of the future, the nonprofit of the future, the organization of the future. You want to have agents like doing all of these things all the time. You want to give them all of these affordances. You want to give them all of these insights. I think that like if that works well, it's great. There are lots of potential problems from a safety perspective. The more affordances you give them, the more you want them to uh you the more you want to be able to measure rigorously how aligned they are and the more data you want to give them to enable them to model and s- effectively fulfill human preferences. But then there is also on top of that this kind of social contract element, which is how can you make sure that those things are um that people collectively trust them, that you think that they are weighing preferences across people fairly. This is kind of related to pluralistic alignment research um and and some of the stuff that I I worked on in my deep mind days. And um I think in order to do this effectively, you do need to kind of design the institution from scratch around these ideas, and that is one strand of uh what we are working on at at at Inherent. And um I I do think it's I I think that this stuff can actually really work, and that's what we're seeing.
[1:31:42] Nathan Labenz: Let's definitely come back to it, and we'll do the full treatment. I'll pitch you on my idea of the states as because we do have ballot initiatives in like half the states, the sort of natural unit uh from which to drive some of this stuff in the early days.
[1:31:58] Tantum Collins: The laboratories of democracy.
[1:32:00] Nathan Labenz: yeah, exactly. Maybe as we just look ahead to this kind of short-term potentially short timelines world, what do you think an emergency looks like if we have an AI Chernobyl for whatever that means. Like what what do you think are kind of the most likely form factors that that would take? What is the response gonna look like? Does the government just come down in a heavy-handed way or like w- you know I I really I think we're obviously in uncharted territory, but you guys were at the White House, different White House, I know that
[1:32:33] Tantum Collins: Back
[1:32:34] Nathan Labenz: um but what what do you kind of expect?
[1:32:37] Tantum Collins: when it still had an east wing.
[1:32:38] Nathan Labenz: Yeah. The follow-up to that will then be like, what can people do if they want to get involved and position
[1:32:44] Ben Buchanan: I
[1:32:44] Nathan Labenz: themselves to be helpful? But let's first just kind of sketch out what what sort of scenarios are you watching for?
[1:32:50] Ben Buchanan: think the two vectors that we thought as far back as twenty twenty-one and twenty twenty-two would be the most significant vectors for this kind of risk are cybersecurity and bio. And whether it's a misuse risk or a misalignment risk, It would I would bet a fair amount of money that those g- it's gonna be one of those two vectors in the beginning if we have an emergency. And it could be a um cyber attack against a bank or critical infrastructure by North Korea. We've started to see bits and pieces of that already, though not probably at the scale you're imagining. It could be a hugging face style thing where you have a bigger breakout and and you kind of have a enduring swarm of agents that's hard to dislodge because they're more capable. You could have equivalent risks on the bio side, again, I think probably a little bit further afield than on cyber. But that's probably the form factor, and I don't know if I have a view on if it's misuse or or misalignment I don't know if you want to talk about immediately what to do about this. I think we've given you a lot of policy things that that can be done about this. I'd like to shift the aperture to what can normal people do about this and what did what have we tried to do about this
[1:33:56] Tantum Collins: And we should flag that we're we're we're we're normal people these days, you know, like we're all in the same we're all in the same
[1:34:02] Ben Buchanan: we certainly
[1:34:02] Tantum Collins: boat.
[1:34:02] Ben Buchanan: don't have any power. I don't know if it's the same thing as being normal. But but I I do think the point of political salience is one of the most important in our conversation here and one of the things that's most different about the moment, as we said before, from when we did have power. And in some very real sense, we we wrote this book uh which is very challenging. I've written four books. This is the most challenging one to write because it's targeting two different audiences who I think have the same mission. And it's for people who don't understand the technology who n- who want to get an accessible introduction to All we talked about, misuse, misalignment, the scaling law um the compute supply chain and and all that, like it's meant to be accessible to that group of people. And it's also meant to be educational to people who do know all those things, who are probably regular listeners to the show here, but who want to say, hey, how does this translate to action in the world. We we've moved AI, AI, whether we like or not, has moved from a domain of technology to a domain of geopolitics. We know the solutions here are going to, at some level, be geopolitical, if not also political. What's the introduction to that world? So I think the mission of both those groups is actually the same, which is they need they need to understand the other side better. And for better or for worse, I do think we are in a position of being translators between these two worlds. And our articulation here, having left, is to say, can we show each side what the other is bringing and why the other is ne- is necessary uh to solve this problem. And then ultimately, that has to translate to political action. And I would d- not describe my book in any way as a polemic. I think it's much closer to like a meditation. But it is the case that what we are trying to do here is to educate a political system that fundamentally is going to have to act to manage this technology. And that requires people from whatever background and perspective being ready to contribute and push that system forward.
[1:35:51] Nathan Labenz: So in short, political activism is the is the number one recommendation for
[1:35:57] Ben Buchanan: Maybe maybe we would say well-informed political activism. And that's what we've tried to do, the well-informed part here. But yeah, I d- I do think that this is a moment where um political engagement, political activism is really important.
[1:36:07] Tantum Collins: And it's been noted, yeah yeah, we were just talking about it, right, we see this with um with the data center issue, this has become top of mind for people. I think like as uh as this stuff breaks out into the open more and more, I expect to see more issues that uh only us policy and AI nerds about cared about a couple years ago becoming something that is really relevant to the general public.
[1:36:28] Nathan Labenz: Do you do you favor, I mean, well-informed political activism maybe not quite a contradiction in terms, but it's like uh it's often pretty coarse messages, right, that get sent. If somebody's like, okay, I'm gonna go join pause AI protests, would you say, yeah, go do that? Or is that too coarse, not well-informed enough for your Taste like how um how sort of blunt do you think people should be in their in the signals they're trying to send to their representatives?
[1:37:01] Ben Buchanan: For better or for worse, I think you're talking to people here who spent a long time on the nuance of this. So you're probably going to get an answer from us. It's only from me that says like let's aspire to nuanced, well-informed political activism. And in some sense, given that we've talked you know for almost two hours and and we've written a book and d- spent four years of our lives all dedicated to the cause of democracy being the form of government that can manage us best, I do think I still have enough faith in the democratic system to to manage this. And that does, I think, require well-informed political activism. So we've tried to do our part here to say, yes, understand what's actually going on with the technology and the pol- geopolitics. And I think this book is our contribution to that. But I I wouldn't I'm not saying you were doing this, but I I wouldn't sort of write the American voter off and say, well, there's no way they can get the complexity of AI. They don't need to get all of it, but they can get enough of it, I think, to to land in good spots.
[1:37:52] Nathan Labenz: Yeah, I'd say it's the Congress as much as the uh voters that I'm worried about in terms of how rich those signals can really be.
[1:38:01] Ben Buchanan: Well, I mean, you'll never find executive re- branch people defend the Congress um but I do think with some of the folks that have been elected this time around and you know maybe it takes a demotrifecta and the president who gets it more than the current one does I'm enough of an idealist to believe or I have enough faith in democracy to believe that that is the only path which we're going to actually address these issues. And I think it's fair to have a pessimism of the intellect about the Congress, but I also think you have to match that with an optimism of the will. And certainly, this book is meant to inform that that optimism or that engagement and say like, yes, we can do this.
[1:38:38] Nathan Labenz: If you do have time for a three-question lightning round, I would do that. If not, we can
[1:38:42] Ben Buchanan: Let's
[1:38:42] Nathan Labenz: skip
[1:38:42] Ben Buchanan: do it,
[1:38:43] Nathan Labenz: it
[1:38:43] Ben Buchanan: let's do it, let's do it.
[1:38:45] Nathan Labenz: Open source.
[1:38:46] Tantum Collins: Th- th- we're we're we're government bloviators, especially me, so lightning is a relative term here. Ben, you can cut me off
[1:38:52] Ben Buchanan: let's make it actual lightning teddy.
[1:38:54] Tantum Collins: I'll do my best.
[1:38:55] Nathan Labenz: open source Are we gonna see a technological solution? Are we gonna have to have a policy solution? Is China gonna stop their uh companies from just releasing model weights YOLO? What's the expectation?
[1:39:07] Tantum Collins: Ben, you can take this one.
[1:39:08] Ben Buchanan: What a great lightning round During our time in the government uh we felt that the benefits of open weight models uh f- substantially outweighed the risks. That might change at some point. I think it's probably going to be an open weight model from China that carries the most risk. I'm hopeful China does a better job than they have thus far at managing some of the cyber risks. We'll see um I in that answer, I'm informed by what seems to be the reporting that Chinese models are being used in cyber attacks and so forth. But I don't want to pretend to have done a deep dive on that.
[1:39:37] Nathan Labenz: On biorisk, one of the policies you guys pushed for was this presynthesis screening of DNA sequences. That seems to be one thing just about everybody can agree on It still seems though like we are headed for a world where there are like very capable models in the bio domain soon, and the world you know at large is like um scarcely better off than we were going into COVID, I would say. if you had the sort of benevolent dictator conch, would you put a pause on additional scaling or like closing of the loop with the wet labs in bio specifically until we're able to do some like hardening of the world? Or would you take some other approach?
[1:40:21] Ben Buchanan: um it's worth noting we had gene synthesis screening in the executive order twenty twenty-three. President Trump repealed it on the first day. They have since brought it back, but we did burn like almost two years in there, which worries me tremendously of of not developing this policy. I would probably be very, very wary and would probably be comfortable banning the integration of AI into BSL three and four labs. I don't think I worry that much about BSL one labs because they're just not that dangerous pathogens and they're probably upside for pharma research. But certainly for anything that is very dangerous, we should be extremely careful with that. And I'm absolutely in favor. of radical investments in biosecurity and biosurveillance uh we are simply not ready on that front. This was we did the max of what we could with executive authorities, but this is a a tier one issue for Congress in my view. Teddy, you want to add? I take I've taken both lightning, so Teddy should go here.
[1:41:10] Tantum Collins: the the the only thing I'll I'll add here is that as many of your listeners may know, this is especially challenging because the bio-focused models are often much much smaller and require less computer train than the big general purpose ones, which makes p- any kind of uh restriction or even monitoring much more challenging because the number of actors who could potentially do this is significantly higher.
[1:41:32] Nathan Labenz: Any advice you would give o- or is it just sort of out of their control to countries three through one ninetythree around how they can shape AI? We've got like the Netherlands, you know, has maybe a claim. Taiwan has a a bit of a a w- way to influence things perhaps, but is there anything anybody can really do aside from the
[1:41:53] Tantum Collins: I'll
[1:41:53] Nathan Labenz: U.S.
[1:41:53] Tantum Collins: I'll take
[1:41:54] Nathan Labenz: and China?
[1:41:54] Tantum Collins: this one, because I, because I I I live and run a company in what most people would probably call a country three here uh which is the United Kingdom. And uh I think there are a lot of opportunities, especially in the vein of um pressing on stuff like AI safety and security research, which I would say is under-resourced relative to capabilities stuff and where uh amounts of money that can be deployed by non-U.S. China entities um can really move the needle. A great example of this is the AI Security Institute previously called the AI Safety Institute in the U.K. which I think has done really, really good work.
[1:42:27] Nathan Labenz: Yeah, excellent, no doubt
[1:42:29] Ben Buchanan: That was the first lightning answer I've heard from Teddy Collins in like twelve years, so
[1:42:32] Tantum Collins: In,
[1:42:32] Ben Buchanan: that's
[1:42:32] Tantum Collins: yeah,
[1:42:32] Ben Buchanan: great.
[1:42:32] Tantum Collins: we've known each other
[1:42:33] Nathan Labenz: Just
[1:42:33] Tantum Collins: for decades and
[1:42:34] Nathan Labenz: snuck it in under the wire uh Well, I really appreciate you both uh for your time in public service, for being ahead of the game on AI by uh you know, a significant margin and um also the you know just the kind of uh while I sort of from the outside understood you guys to be pretty sharp China hawks coming into this conversation, I think I also heard like a lot of moderation that, you know, I'm sure was always there, but like doesn't always reach the the surface view that outsiders have. And I really especially hope that the current administration does what you say you would do if you were in uh in power now in terms of, you know, continuing to really try to reach out to the Chinese government and e- establish what um common ground may be established. And I don't know, you know, nobody ob- obviously knows until we try what that is, but um I'm heartened to hear that you would be trying if you were in there now, and I I hope they uh follow your lead on that. With that, I'll give you guys a last word. Anything you wanna leave people with before we break?
[1:43:37] Ben Buchanan: first, Teddy.
[1:43:38] Tantum Collins: I think we've covered all the big things, honestly. Great questions.
[1:43:41] Ben Buchanan: I'd just say uh you know, I've done a number of podcasts and book talks and so forth, and the uh thing that is greatest for an author is when someone has read the book carefully and engage with the ideas, even if they don't always agree with the ideas. So I appreciate the the conversation here um I think uh this is a a book that certainly we are very proud of and it has a lot of nuance and the opportunity to unpack at least some of that nuance with you has been a lot of fun.
[1:44:06] Nathan Labenz: My pleasure. That's very kind. I very much appreciate it. Ben Buchanan, Tantor Collins, thank you both for being part of the cognitive revolution.
[1:44:13] Tantum Collins: Thank you for having us.
[1:44:13] Ben Buchanan: Thanks so much.
Outro
[1:47:42] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.