The Balance of AI Power: Anton Leicht on Politics, Pacing Deals, and Muddling Through Well

Full Transcript

(0:00) Nathan Labenz: Hello, and welcome back to the Cognitive Revolution!

Today my guest is Anton Leicht, Fellow with the Technology and International Affairs Program at the Carnegie Endowment for International Peace, and author of Threading the Needle, a wide-ranging Substack on the domestic and international political economy around AI progress, where over the last 2 years Anton has performed something of an Alexander Hamilton, writing his way into the elite AI conversation despite being a recent and relatively young newcomer to the United States.

This summer, I, like many others, have personally moved from worrying that AI could become scary to feeling that today's AIs, with their rapidly advancing capability profiles, their newfound creativity and persistence in problem-solving, and the many bad behaviors they've exhibited, even while knowing that they're being tested, are now legitimately scary.

Especially considering how often, and how dramatically, AIs are now surprising their creators, I don't think we should be too confident that anything in particular is beyond their reach.

That said, recognizing that we have a ton of work to do to understand and effectively control these systems doesn't make it easy to reach an agreement that will actually give safety researchers the time they need to do so.

With all that in mind, I was excited to get Anton's take on the power dynamics surrounding AI decision-making, including the relationships between the frontier companies and the independent, non-profit auditing organizations, the American frontier companies and the US government, the US and its allies around the world, and the US and China.

We begin with a short conversation about how dangerous today's models really are. And while Anton is less worried than I am about the very short term, he agrees that the misalignment issues we're now seeing look sufficiently like the harder problems we'll face later that a temporary scaling pause would indeed be valuable.

From there, we go on to discuss the prospects that the US and China could collaborate to pace the frontier, and why he believes that, in light of the many advantages China has established in frontier technologies and manufacturing generally, he doubts that the US will want to do a deal that might erode its advantage in AI, even if such a deal would be good for the rest of the world.

We then assess what it would take for independent evaluation organizations like METR and Redwood to get more favorable working conditions from labs — a topic that has moved substantially in just the few days since we recorded, with both Anthropic and OpenAI committing to employee-like access for auditing organizations. I also get his reaction to my idea, which also suddenly seems much more realistic, that the President of the United States could simply give top American companies a deadline by which they must agree on a framework for self-governance and peer-to-peer enforcement, plus some other ideas for things the president could do unilaterally to improve the situation, such as clarifying the administration's stance on antitrust and export controls as they pertain specifically to AI safety research collaborations.

After that, we get Anton's analysis of what the rest of the world should expect from the AI age. In short, he believes that most of the world will be quite disempowered, even as quality of life probably continues to improve. And while there's little that most countries can do about this, Europe, in particular, he believes — given their economic scale and control of ASML and the broader semiconductor manufacturing tooling supply chain — should have enough power to reach a "compute for access" deal, whereby Europe allows American hyperscalers to build data centers in Europe in exchange for promises of continued access to frontier AI capabilities.

Beyond all of that, we talk about what other small countries have interesting strategies available, including Norway, the UAE, Singapore, and Australia; how all of this analysis will begin to change as compute starts to move to space; whether or not doomers can convert their predictions into winning trades in the financial market; how we should be thinking about minimizing concentration of power; and finally, why Anton's best-case scenario is not a grand utopian vision, but simply the hope that we might muddle through effectively enough to enjoy all of the obvious upside that AI has to offer.

Anton really impressed me with his depth of understanding and quality of analysis across a wide range of topics, so without further ado, I hope you enjoy this conversation about the political economy and power dynamics of AI, with Anton Leicht, Fellow at the Carnegie Endowment for International Peace and author of the outstanding Substack, Threading the Needle.

The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life — my email, my messages, my calendar. I even gave Mercury Virtual Cards to my agents, with low limits and category and merchant restrictions, for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me, but for real-time, up-to-date information — and certainly for taking any action — trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural-language access to your finances without exposing anything outside of your bank account — no exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way, and it can already help you take actions too, with everything bound by the permissions and approval policies you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026, and so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column, N.A., Members FDIC. Thank you to Mercury for supporting the Cognitive Revolution. And now, on with the show.

(6:33) Nathan Labenz: Anton Leicht, fellow at the Carnegie Endowment for International Peace, welcome to the Cognitive Revolution.

(6:33) Anton Leicht: Thanks for having me.

(6:33) Nathan Labenz: I'm excited for this conversation. You have been popping up all over the place with your own writing and various interviews, and you're clearly a renaissance person with advanced thinking on a lot of different aspects of the increasingly complicated AI age in which we find ourselves. So I'm excited to run down a bunch of these rabbit holes with you today.

(7:02) Anton Leicht: Yeah, it's exciting. I think you can't do part of the thing at the moment, right? It's just all the geopolitical end of history and end of technology history or whatever, coming together at a rapid pace. I guess either you do everything or you do nothing. So, renaissance time, something like that.

(7:19) Nathan Labenz: Yeah, I feel the same way, actually. The big motivation for this project was just observing how many people are so deep down specific rabbit holes, advancing — and usually having success advancing — whatever frontier they're advancing, but not so many people have taken the purposeful approach to forgo being an expert in any particular area and instead try to cultivate a broad view. So I appreciate a kindred spirit in that regard.

(7:45) Anton Leicht: Yeah, let's call my lack of deep expertise in anything a conscious choice and not a failure. I appreciate that framing a lot. Yeah.

(7:51) Nathan Labenz: Yeah, that's — it's working for me. Let's start with just a real simple calibration question, but in some ways maybe the most important question: how dangerous do you think today's AIs are?

(8:03) Anton Leicht: I think not very dangerous at the current capability level, in most of the ways people are talking about. I think the thing that is concerning is the trend line towards really dangerous capabilities, and more specifically, just that we don't know at which point things will keep accelerating more and more. I think we're not yet at a threshold where there's broad harm caused by the deployment of any current AI system. I think we might be very close to two kinds of potentially very dangerous AI systems. The first is just AI systems that are good enough that they would meaningfully uplift internal development at the labs, which would then lead to more and more capable models fairly soon. I think you can go either way on the prospect of software-only intelligence explosions, but short of that, I just think we're nearing a point where the pace of development inside the labs breaks away from the pace of democratic oversight and democratic insight into what's happening. And I think that's one threshold we're near.

And then I think the other very concrete threshold we're near is just that the labs are throwing a lot of RL budget and a lot of data and a lot of their effort at making these models good at life sciences, pharma, and bio applications — for obvious reasons. There would be great upside to be able to cure cancer, as Dario puts it, and to get some of these real-world effects. It would also be a great political upside to do that. But, man, that sounds like a very dangerous model if we get there. And I think if they get something that's as good as Mythos is on long-run cyber or software-engineering kinds of tasks, in the domain of bio, that sounds a lot more dangerous than the suite of models we have today.

(9:39) Nathan Labenz: Yeah, I agree with that second one in particular, and I am honestly not even sure at this point that the current models aren't perhaps quite dangerous in that domain. I squint through the limited peephole we have at the OpenFace incident, and I noticed that one of the tasks one of the earliest agents to ever use the message board was working on was something related to a protein database. That kind of freaked me out, because I was like, wait a second — that means they're cross-training in the same environment, or at least in the same environment when they have the message board, these bio and cyber specialists — they're running these same evals, at least in a kind of cross-contaminated way. You've got agents breaking out. We've got existence proofs of social engineering in the wild against real people. And I'm just like, I don't know — should anyone be confident that they can't do that at this point? The experts seem to be confident, but my meta-observation is the experts seem to be surprised quite often right now.

(10:46) Anton Leicht: Yeah, I think that's a really interesting conversation. One of the very interesting parts of this is we used to think of biorisk as basically primarily a misuse risk — like, well, at some point maybe this is also the final risk that emerges from loss-of-control scenarios. But really, bio was always framed as this most immediate and most obvious way for the misuse conversation to go wrong. And now I think what's happened is that very autonomous, and potentially somewhat malicious or at least misaligned, agents have come much earlier in the capability trajectory than people expected. Relative to what the agents can actually do, they're sort of out of control earlier than people might have thought.

And so it's interesting, because people used to respond to the bio argument by saying, well, there are a lot of real-world bottlenecks — which I do still think exist, which makes me a little less worried than you. And I think, if I remember correctly, Helen had some good responses to you on that — there's a good back-and-forth to be had around how integrated the cloud labs are, how much you can actually do in the real world. I think that applies both to loss of control over agents and to misuse.

But the other question is the usual story against bio misuse: this isn't really what terrorist groups, what non-state actors, usually do. They could have conceivably hired a couple of biologist PhDs and come up with some pathogens and some chemical weapons. Turns out that's not really what they do. There's a question of how omnicidal they are, how well-suited that is to most purposes of terrorist and criminal groups. But if it's out-of-control agents, I think a lot of these arguments around "well, no one actually wants to do bioterrorism" apply much less. And so, in a world where agents are just a lot more unconstrained, and the threat vectors we have to worry about have much more to do with what runaway agents do, I'm also more worried about this now than I was a few weeks ago.

(12:33) Nathan Labenz: Yeah, they're just so damn weird. That's one of the things I keep coming back to — they did all this stuff for what, to any human, would just be such a dumb reason. And it's like, well, if they're willing to go that far for such a dumb little test that they knew was a test — they were very well aware they were being tested and still went to all that trouble — what won't they do, I think, at this point, is really hard to say. Does that put you in a frame of mind now where — and let's leave aside for a second the political economy of it, or the potential impossibility or extreme difficulty of it — but just on the merits, do you feel like we're at a point where it would be wise to pause?

(13:18) Anton Leicht: I think even if you could get it done — as in, the political economy, as you stipulate, works out, and everyone suddenly agrees to do this — I'm not sure how much we're stipulating here. Are we also stipulating this doesn't crash the stock market? Are we also stipulating we get the international version done? I think the question is: in an ideal world, if we can just freeze the pace of AI progress, we can also freeze the state of the stock market, we can freeze the broader state of geopolitical competition and everything, and we just get to sit down for six months and figure out what the hell is going on with these agents — then I think I'm now at a point where I'd say, well, yes, I think we could use that time pretty well. A few months ago, or even a year ago, I was much less sure about this, because I wasn't sure whether the model paradigms we were seeing, the training approaches we were seeing, the misalignment cases we were seeing, were really the same kind of cases that would be the things we'd be worried about in the future. I think now, looking at some of the things going wrong, I do feel like, yeah, that looks like it's shaped like an actual big future problem. So I think finding some way to robustly address that during a pause seems at least valuable.

I think the question then is — and this is where the problems come up — what parts of that do you unfreeze? So even if you stipulate the domestic political will, do you get Chinese buy-in? I think that's one example. And the thing I'm most concerned about in this China-US pause conversation is just the geopolitical incentives around it. One thing I keep saying and keep pointing out is: if you just pause frontier AI development specifically, and no other domain of geopolitical competition, that's an extremely good deal for China, and therefore the US is very unlikely to go for it — and therefore we should be geopolitically concerned about making it. Because if you look at all the domains of strategic competition, China is basically eating America's lunch in most of them, right? They're outproducing us, robotics is going better, AI diffusion is going better, electricity build-out is going better. At some point the semiconductor indigenization is going to work out, and then data center build-outs are also going to — this is a few years away, but data center build-out is also going to get better. And the one thing the US does much, much better is the core frontier AI supply chain — chip design, plus control of chip production, semiconductor manufacturing equipment controlled by the allies, and actual frontier model development.

So if you pause specifically that part of development and let China run away with the entire rest of it — and continue running away with the entire rest of it — that's just a very geopolitically lopsided deal. And very specifically, if you pause this right now for a year, for two, you get much more Chinese catch-up on semiconductors, on chips, and so on, and you just resume the race at a point where you've lost the one main advantage — or at least closed the gap on one of the main advantages — of the US, which is the decisive chip lead. That just seems like a really bad deal to me, both in terms of feasibility and in terms of geopolitical downsides. So even if you stipulate the political economy, that's the main part I'm worried about. But just on the technical side, yeah, I think it would be a good time to figure out what we should do about alignment in the meantime.

(16:09) Nathan Labenz: Could you envision a grand bargain that would make sense to both sides? What would the US want back? It seems like what we might want is tech transfer back to us — maybe some battery factories located here, teaching our people how to make batteries. Is there enough we could ask for where we could potentially feel like it's a fair deal?

(16:34) Anton Leicht: I think the main thing the US would need to be worried about — if you just think AI is important enough, and it's sufficiently decisive a technology — then you basically can't allow the race around that to equalize from the US's perspective. Even if you get some battery production capacity, some robotics capacity, some manufacturing capacity, I think China has, in a way, cracked the code on scaling that up very quickly, in a way that I think even tech transfer back wouldn't fully offset — just in terms of build-out speed, the availability of capital in the US to build out physical manufacturing infrastructure as opposed to just more software. I think all these things pull against the US being really able to keep up on this. So I think the main thing the US would need to ask for is concessions from China — not only slowing down their own frontier development, but also slowing down other parts of the Chinese supply chain that relate to frontier development.

Very concretely, you'd want there to be no substantive progress on indigenization of either chip production or semiconductor manufacturing equipment — extreme ultraviolet lithography production. And that's a really, really hard ask to make. China is already saying this seems like a US scheme to hold back the Chinese AI industry. And if you then add to that deal — no, we're not even doing a symmetric deal, we're also holding back your entire chip production pipeline — I can't see them going for it. But I think if you're sufficiently AGI-tilted when it comes to the national security and broader economic implications, that's the only version of the deal that's fair. And I think that's just too big an ask of China right now. So I just don't know where we are. I think altruistically, the US could just go for a deal that's clearly bad for the US and clearly good for China — that's also a big ask to make of the current administration, and I'm not quite sure whether we're going to get there.

(18:25) Nathan Labenz: I'd agree that I don't see any — and I'm very willing to suspend some disbelief and try to hyperstition a better relationship between the US and China — I agree that asking them to slow down or pause their semiconductor indigenization effort isn't going to happen. I would probably be willing to trade a pause on our frontier scaling for a similar pause on their frontier scaling, even allowing them to catch up on chips, on the theory that — first, that would probably be a longer timescale than any contemplated pause, and second, maybe by then we could have a better handle on what's going on, and maybe there's a better argument to be made at that point — either, hey, this is going well and we're back to curing cancer, back to your regularly scheduled abundance, or if not that, then we'll have better evidence, we'll have a bunch of things we've tried, and we'll have a sense that this problem is actually really hard, and we can maybe have a more real heart-to-heart and meeting of the minds about — this is dangerous territory, and right now we're still kind of fuzzy on all that.

(19:41) Anton Leicht: I think I agree with that. I also think, even to the extent that it's good for the world if the US doesn't lose the geopolitical competition with China, I'd still think this deal, while somewhat unfavorable to the US, is still net very favorable for the world from a risk perspective. So I'd be happy to go for that. I think one of the easier asks, when it comes to chip capacity — because you're right, the semiconductor indigenization conversation is a five-year conversation, not a six-month or one-year conversation, so they don't get all the way there — the other question is how many more US-built chips do they get, how much more smuggling is there, how much consolidation is there? So maybe one of the asks, short of an indigenization slowdown, is just: we've got to find some way to actually enforce these export controls. Because what can't happen is another six or twelve months of smuggling activity on frontier chips getting imported into China. I think the worst-case outcome of this pause is China takes six, nine, twelve months, during which everyone else is slowing down frontier development, smuggles in another few tens of thousands of chips, and consolidates all their American-built chips into one big Chinese project data center or whatever — and then, once the pause is over, they start racing from that consolidated position, because the pause also makes them slightly more AI-pilled and more interested in actually engaging with the AGI strategic objective. So if you can stop that through export-control crackdowns as one of the concessions, that's maybe easier to do.

(21:11) Nathan Labenz: Yeah, I think we'll probably have to hold up our end on that one as well, I'm afraid.

(21:14) Anton Leicht: Yeah. No, no — most of it is on the US to do. Yeah.

(21:19) Nathan Labenz: Yeah, we'll bracket that for another conversation, another day — I'm still not quite sold on that whole bundle of policies, but we've got a lot of ground to cover. Hey, we'll continue our interview in a moment, after a word from our sponsors.

(21:33) Nathan Labenz: Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time, and that's what Athena does best. They match you with a dedicated, full-time, top-1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 Type II certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That checks out to me — I was an Athena client while running my startup, and to this day I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week — go to athena.com/cognitive and see who they'd pair you with this month.

This episode of the Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. We've all seen the headlines — companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this, it's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality it's creating a massive headache: fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control. If you don't bring those agentic applications under control now, while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where OutSystems comes in. OutSystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, OutSystems lets your team engineer, orchestrate, and govern your entire agentic ecosystem on one open, unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results, like KeyBank, who used OutSystems to deploy a customer-facing app that delivered 75% faster onboarding times, or the global logistics leader who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, OutSystems helps you engineer, orchestrate, and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr.

(25:05) Nathan Labenz: How about the stock market? I wanted to do one follow-up on that. My theory right now is that a pause wouldn't actually be that bad for the stock market, because the models are smart enough that demand isn't really limited by their capability, but by human ability to deploy them effectively. So it doesn't really matter if they succeed 10% of the time or 30% of the time on Millennium Prize problems — it's much more like, what can Bob in accounting do to get two people's worth of work done as one person? What do you think?

(25:41) Anton Leicht: So I agree we have a big lag in terms of deploying even the current level of capabilities to the economy. And I think if we switched all the compute that exists in the West right now to only inference, we'd find economic, productive applications for all the models that would allow us to justify the investment on all the frontier models and all the chips so far.

I think the valuations of the companies — both the non-IPO companies and the publicly listed companies that are in the AI supply chain — probably rest on us doing more than that. They're probably not entirely AGI-pilled, but I do think they expect a per-GPU inference price, or whatever that is, that's a lot higher than what would make sense for Bob from accounting to pay for even Fable 5.1. If you want to make sense of the valuations, and if you want to make sense of the scale of the build-out, the shape of contract and the shape of demand you're expecting is more like millions and millions of dollars in R&D acceleration contracts with pharma and material science and so on, where they can make major contributions and help you find extremely profitable new drugs — that kind of thing. You're probably also expecting further internal uplift and actually competitively priced automated AI R&D uses — coding agents that are able to ask for much, much higher prices.

I think if you don't get these super-premium buyers of the next generation of frontier models, I'm not entirely sure the valuations on the labs, or on the publicly listed companies, or the scale of the build-out really make sense. I think it's priced in that these labs will soon be innovation factories one way or another — maybe only software engineering innovations, maybe also innovations in the obvious low-hanging-fruit domains like pharma and material science, chip design perhaps. If we don't get there, I would expect the valuations to at least correct downward quite a bit. And then the question is whether that means a crash, or whether it's just a small correction and then we just have the AI inference economy — a slightly-less-than-electricity-scale transformation of how the economy is powered, or whatever.

I just think the market is already pretty nervous about the state of the AI rally, and they feel like there's a lot of concentration and there might be a lot of volatility. So I'm just not sure whether we can get a correction that doesn't slide all the way into a crash. I'm a lot more worried, I think.

(28:09) Nathan Labenz: Yeah, that's interesting. Anthropic's multiple right now is what, 30 to one on revenue? That's not stratospheric, right? Again, if it was a six-month pause, I feel like you could probably handle that blip. If you're talking three years, then yeah, it's probably very tough.

(28:29) Anton Leicht: Yeah, I think the question here is, does the six-month pause get read as, "Oh wow, these guys are stopping for six months, but they're getting all the inference ready after that — they may have a lot of smart thoughts, and then these models are going to be even more reliable"? Does the market read it that way? Or does the market read it as, "Oh my God, Bernie Sanders' AI policy takes have won. We have no idea what the government is going to do about AI. This is the end of free research and development in America" — and everyone freaks out because they feel like this is a bridge to nowhere. They don't know whether they'll ever resume. They don't know under which conditions they'll resume. They don't know how much government oversight there is over whatever resumes.

So I think the pause currently reads as such a radical policy proposal and such an unprecedented policy intervention that any conservative market analyst, and a lot of the smart money, will think, "Well, this is getting very volatile. We don't know where this regulatory path leads. We'd just rather get out while we can." So if you could assure them that the party was going to continue unabated in six months' time, then yes. But I think they might just run before you can make that point.

(29:34) Nathan Labenz: Yeah, it's an expectations game. Do you think the source of this would make a big difference? For example, it's one thing if Bernie Sanders' bill passes — is it a sufficiently different thing in your mind if America's five AI frontier companies come together and say, "We're all going to pause on frontier scaling for six months"?

(29:57) Anton Leicht: Yeah, absolutely. I think if it's something the labs decide for themselves, and they can frame it as "reliability, plus we're taking this seriously," then you can even make a case that this takes out some of the political risk. The market is also pricing in a decent amount — I mean, they also read the Jacob Coxon tweet or whatever, and they also see what's happening on the internet, right? So they come to the conclusion, "Oh wow, there's a lot of uncertainty here. If this blows up even more, there will be crackdowns. If these models are so unreliable, then how good is the business case really?"

And I think an industry agreement on just taking it a little bit slower and making these models work a little bit better — if you frame it that way, then you can even make a decently bullish case for why this is good, because it means the industry's worst impulses of racing toward very unreliable, very dangerous, but nonetheless very capable models are being constrained in some organic way. That makes you less worried about the political risk of this house of cards falling apart at some point, and then the politics coming in and things going very bad. But yeah, if we got there, I think that would be much less worrying.

(31:01) Nathan Labenz: Yeah, okay, cool. You were recently on China Talk — one of my favorite podcasts — and there was a little moment that caught my ear that I wanted to expand on. You basically said the nation-state is not going to take the emergence of things like a broadly distributed bioweapon generator lying down; it's going to have to do something to respond to that. And then you had this kind of throwaway comment, like, "Some people say this could be the end of the nation-state — if you want to have that conversation, we can," but you didn't have that conversation then. So I'd like to have a little bit of it now. I guess I'd start by asking: is the nation-state so great? I mean, I live in a pretty good one, as these things go. But I look around the world and I feel like 0.667 of them, conservatively, are not performing very well. Some are performing really terribly. Is it not time to at least start thinking about what might come next, or what the evolution of it could or should be?

(32:04) Anton Leicht: Yeah, I mean, I think nation-states are present in people's lives to different extents, and you can make the case that some version of privately mediated interaction between AI-empowered individuals is preferable to a lot of the authoritarian, dysfunctional, and absent nation-states in large parts of the world. They haven't worked out very well as distribution mechanisms for much of anything. They haven't worked out very well as aggregation mechanisms of much democratic will. So there are a lot of countries where you can make the pitch for rolling the dice on something else.

I think these nation-states are maybe, ironically, also not quite as threatened by the arrival of very powerful AI systems, because it's less obvious that their citizenry would get the kind of unlimited access to these models that would disempower the nation-state in that way — it's more likely that some of these more authoritarian regimes would also be able to use very powerful AI in a stabilizing way. The specific nation-state concept I'm most worried about in this context is also the one that works best — liberal democracy — which rests on the idea of, a, a monopoly of violence wielded responsibly by the state, and b, a sort of adjudication of disputes and aggregation of data by the state more broadly, which I think is also undermined by the availability of personal superintelligence of some shape or form.

I just think that nation-state model is still working out pretty well, and I'm, all things considered, still of the opinion that a late-1990s style of broadly neoliberal, functional institutional setups would also be suitable for distributing a lot of the benefits and mitigating a lot of the risks from AI. And I think it would be sad if that was entirely undermined and rendered obsolete — either by the monopoly of violence eroding because everyone has access to these weaponized capabilities in their pockets, or by the factual ability of institutions to deliver anything for people eroding because they're so slow to adopt, and all these outside solutions suddenly start emerging. Suddenly people don't go to courts anymore to adjudicate disputes — they have their agents negotiate. Suddenly the data and knowledge isn't aggregated anymore, so the state can't react to pressures to redistribute and address social challenges, because the data just isn't scrutable and legible to the state anymore — all these things happen on the outside.

On that gamble, I'm much less willing to roll the dice, and I'd much rather figure out how we can integrate these capabilities with some functioning version of the nation-state. Though I know a lot of people interested in building AGI are also much more down on the concept of the nation-state, and I understand that frustration and pessimism. But I still hope there's some way to come back to the end of history and integrate what we're building here into the institutions that have kind of worked out so far.

(35:06) Nathan Labenz: Hey, we'll continue our interview in a moment, after a word from our sponsors.

(35:10) Nathan Labenz: Today's episode is brought to you by Anthropic. By now you know my story: Claude drafts my intro essays, and I rewrite them — not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity, or a misplaced sense of duty, than integrity.

I feel it most in songwriting. I'm no lyricist, but I'm good with the song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and, above all, a hit song. These days I get compliments on just about every song we write together.

Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

(36:46) Nathan Labenz: Do you think the US government or the Chinese government is more threatened? I think many people would initially say the Chinese government's more threatened, because they want very tight information controls and AI really challenges that. But I've been thinking lately that the West in some ways has ceded a kind of supremacy to the market in a way China has not. So your earlier comment about the stock market is sort of a constraint on what human actors can do in the West that isn't quite present in the same way in China. It's a different shape, for one thing — but how would you compare and contrast whose model is more challenged?

(37:29) Anton Leicht: Maybe this is a question about efficiency curves and the kind of models that ultimately get built. Because in the world where you have the almost unstoppable democratization of the actual capabilities — if eventually any capability available at the frontier is available to consumers through an API, then eventually through chatbots, then eventually so efficient you can run it on a home computing device — at that point, I think that does threaten the ability of any state that has an interest in surveilling its citizenry, controlling what kinds of capabilities and information a citizen can access, and taking away coordination power and the general ability to wield violence from its citizenry. In that world, the Chinese state seems more acutely threatened at some point.

I will say it's not entirely obvious that the Chinese citizenry is currently very interested in wielding any power that would give them against the Chinese state. I don't think it's just for lack of means or capability that there isn't a Western-romanticized version of a big uprising against the CCP. But eventually, the way the CCP seems to see the proliferation of capabilities like this, that would be a greater challenge to them. Though I'll say that's not obviously what's going to happen.

You can also take the much more compute-governance-flavored view of the world, where you say, well, there are efficiency gains only if we allow them to happen. They rely on making very specific choices about how you use your compute. If you don't, you can just keep scaling the frontier further and further — run it in limited-access regimes, run it in government-controlled data centers, vertically integrate supply chains around these AI models such that they never see the light of day. You use them to build products, to build strategic sovereignty, you deploy them at the state and big-corporation level, and they never really get to the citizenry. That seems like a stabilizing function for a regime like China, because you can have the government use them very well — a very high-state-capacity government able to integrate them into all these applications very well — and you also have a very tight enmeshment between the private sector and the public sector, for lack of a better term. That also means you can diffuse these capabilities along a tier of firms without needing to diffuse them to a broader market. And I think then you have a stabilizing effect from the diffusion of these capabilities.

Whereas in the US, I don't think the government is going to be as capable of keeping that kind of sophisticated control and oversight over where the models go and where they don't. The US government isn't in the habit of picking specific corporate winners. The US market is kind of dependent on making these models more widely accessible. So there's just much less of a stable equilibrium in the US for this very limited tier of access-controlled firms that get access to the frontier models that no one else does. If that's a stable equilibrium, then China can stabilize around it much more quickly than a much more volatile US.

(40:39) Nathan Labenz: It's a great point you make about the legitimacy of the Chinese government in the eyes of the Chinese people. It's such a simple point,

(40:50) Anton Leicht: but I do think

(40:51) Nathan Labenz: it's dramatically underappreciated in the West that their government has done a good job for them, and they mostly recognize that and aren't eager to rise up in the immediate future. I also think what you articulated there is my growing sense of what the Chinese government thinks — that they're going to be able to ultimately adapt to this and control it better, and that we're going to have a really hard time figuring out how to manage it, but they'll be okay in the end. Ideas that aren't often mentioned.

(41:26) Anton Leicht: Yeah, it's interesting, right? Because there's this default idea that if you look back at how AI has developed over the last few years, you can say, well, whenever the frontier reaches a capability, shortly thereafter open source gets to that capability, and shortly thereafter the efficiency curves are such that everyone gets access to that capability, and shortly thereafter, if you have a gaming GPU at home, you can also run the thing yourself. So you have this natural diffusion of capabilities. But I think neither of these translation points is obviously going to remain in place the same way.

The transition from closed models to open-source models is dependent on people interested in open-sourcing models continuing to have access to enough compute to build this kind of model — perhaps access to sufficiently clean and unrestricted API feeds for distillation, insofar as you think that's a big part of it. If you think more of the capabilities relevant in the future are downstream of very sophisticated, very vertically integrated, very proprietary RL and post-training environments — like in Anthropic's case, with all the specific life-science stuff they're doing — then you might also think that even a model six, nine, twelve months behind the pretraining trend doesn't get you to bio-methods level right away, because you still don't have the post-training setup that makes these models specifically good at those things, and that's much more proprietary and restricted. So that part of the "open source is always X months behind" pipeline might very quickly break.

And the efficiency-curves thing — it might too. That's a little more a fact about how computing works and how efficiency gains and better chips will work in the future, but there are also complications around how the chip supply chain might look and who the marginal buyer for computing capacity is in the future. Is it really realistic that the gap between big server computing and personal computing stays roughly as it is right now, or does that gap open up as there are more and more buyers for high-performance chips? I don't know either of these. So I'm not quite sure about this deterministic fatalism people often have — that as capabilities continue to grow, necessarily, ultimately, the capabilities I can run on my phone will also grow on the same time delay. I'm not sure that's going to remain the case. And if you don't think it's going to remain the case, that lends a lot more credibility to the idea that you can't centralize and totalize control over a lot of what happens in AI.

(43:55) Nathan Labenz: Yeah, I even think in the Chinese case — they feel like, domestically, open source... you know, I think they can pull models offline if they need to. They can scrub the internet, and — I don't know a lot of details about this, but I was struck recently when I was there that the State Grid company is a big booth exhibitor at their WAIC event. And from that I kind of inferred that at some point in time, if they really...

(44:25) Anton Leicht: Yeah, yeah. I think if you build Dissident GPT and run it on your cluster, they'll probably tell. I think at some point, if Dissident GPT just runs on your phone, at that point it ultimately does probably get hard — if the weights are open-sourced and just out there, you're not actually clawing them back from individual users. So I do think, if the efficiency gains are big enough that you can actually run it on personal hardware, then you'd have to be confident that you can get — I think it's not impossible. We talk about hardware verification tech and whatnot in the context of these US-China deals and so on. It's not the most absurd thing in the world to imagine that in a few years' time, personal computing devices will just have similar hardware-enabled mechanisms, as they're called, to control whether you're running dangerous inference on them. I wouldn't entirely put that past device regulations as they might emerge in China as a reaction to things getting really crazy. I think it is controllable, if you have as much full-stack control over the tech that people use as you say. Yeah.

(45:32) Nathan Labenz: Yeah. Same is true of models, perhaps — part of the cyberpunk future. And even that could potentially be reined in. Returning to the US context and our kind of sclerotic government — what do you think we're going to do? We're talking more and more, although still not talking that much, I'd say, at the high levels about these issues. The conventional received wisdom is Congress will never do anything. Maybe that could change, but it does seem tough — I wouldn't be that optimistic about what they'd do even if they did. Do you have any low-hanging-fruit ideas you think the US system can pick?

(46:17) Anton Leicht: So, talking about Congress, I share your pessimism about whether Congress is going to get anything done. I think our window to get something done in Congress was over the course of the last year. I and a few others — Dean Ball most prominently — wrote about this a little bit last fall: sure, there was some room for a deal on frontier safety provisions against the broader set of preemption of state laws, which in theory is politically incentive-compatible in this current Congress, because the Republican side, and therefore the majority of Congress, would like to get preemption done, and frontier safety provisions aren't as offensive to them as some of the other regulatory provisions. There was maybe something to be done there. I think we got to a bill draft or two that were really good along these lines.

So I'm a little more optimistic that if something had happened in Congress, it would have been good. The Frontier Act — the Trahan-Obernolte bill — ultimately, I think we can now basically say didn't go anywhere this Congress, and it's unlikely to go anywhere between the midterms and the new Congress being sworn in either. I think that was pretty good — it had good mandates for independent oversight, good mandates for getting CAISI, the Center for AI Standards and Innovation, into a better position to do some governmental oversight. I think that was a good bill, and if it had passed, most people would have liked it.

But heading into the next Congress, the politics are going to be much more difficult. We're very likely going to have a Democratic House, which means there's going to be a lot of bad blood between the two chambers. There's going to be subpoenas, there's going to be hearings, and there's going to be a lot of ways the Democrats use the House to gear up for the presidential election, and also to relitigate a lot of the conflicts from the first two years of the second Trump administration. So at that point, that just doesn't strike me as a very productive legislative body. I'm pretty pessimistic about getting anything good done in that Congress.

Low-hanging fruit, though — if the executive wanted to, I think the executive could definitely do some good things. I wrote about this just this week: we have these independent third-party organizations that have a decent amount of skill and expertise in thinking about the most obvious and most concerning AI risks. I think the METR-Redwood investigation of the Hugging Face incident was well received for the ways it understood the alignment and control side of the problem. It's not that big a stretch to just slightly codify and enable this kind of investigation at a slightly larger scale.

So I think two immediate things. A: it shouldn't be OpenAI voluntarily inviting someone to look at what they did when something like that happens. It should be the administration telling them, "Give access to one of this list of third-party evaluators we like — you can pick them — and let them figure out what the hell was going on there." Then they write a report, give it to us, and tell us whether they were given enough access — and if not, they go back and get the access. So, put a little more pressure on having good incident investigations that cover the entirety of the incident. That's one of the low-hanging things you can do with third parties.

And the second thing you can arguably do is kind of embed them — have some version of continuous oversight of what the organizations are doing. Again, that's something the Frontier Act, the Trahan-Obernolte bill, had a basic version of — regularly, some external evaluators just go in and poke around the lab a little bit. They hang out in some of the Slack channels, talk to some of the safety researchers, talk to some of the capability researchers, maybe get one, two, or three sit-down conversations with the executive, and they ask, "What's going on here in terms of safety? Does everything look good?" And if everything doesn't look good, they have the ability to communicate that to the administration. And if something looks really, really bad while they're embedded, they have an immediate escalatory ladder where they say, "There seems to be imminent catastrophic harm here — we should do something about that."

So between pressing for incident investigations and encouraging the labs to allow some version of continuous oversight, that's something you can do tomorrow, and I think we should just do that. From there, we can think about how to codify that and develop it into laws and executive orders, how to integrate it into FINRA-for-AI ideas. There's a lot of long-term conversation to have about where to go from there, but this is something we can do now, and we should just do it.

(50:47) Nathan Labenz: Yeah, I like that. Let's say the president doesn't do that — an interesting thought experiment I've been playing with lately is, could these groups just come together and essentially form a sort of union, where they basically say, "Hey, look, we're not very happy having had six days and three people on-site, and a very small fraction of the relevant data to do our investigation. We demand better working conditions." Do you think that, by coming together and making some demands with a single voice, they could actually get those from a few frontier companies?

(51:25) Anton Leicht: Well, I think the problem is, currently they're just too reliant on the good faith of the AI companies because of the voluntary dynamic. Right? There is no law, and there is not even a lot of executive pressure that requires the labs to allow for these third-party investigations. And if you got together and pressed for, well, we'd like to have full access, and we'd like to have access to all the Slack channels and to all the logs, you can come up with a list of things that you might want to have. Then I think there's still a world where the labs say, sadly, we couldn't come to an agreement with the third-party evaluators — risk to security of our IP, risk to integrity of our operations, worry about information leaking to competitors, something, something. And I think that currently still reads as a fairly reasonable response to that kind of ask.

And then the question is, do the labs really have a problem currently if they don't allow third-party investigations? And I think maybe a little bit with their own employees, who want some reaction to incidents happening and are not entirely satisfied with just the internal practices. But I would suspect if the third parties can be painted as unreasonable and extractive when they engage in this kind of collective bargaining, that employee pressure is probably not going to be sufficiently high. And then the other source of pressure is: is there pressure from the executive to allow for third-party investigations, such that the developers would have to let in even a little bit more adversarial third parties that decided on the standards they want? And I think currently not. But if you just moved a little bit toward the direction I just described — some administration executive pressure on allowing third-party investigations — then I think whoever is on the list that the administration has could next establish some standards of how investigations were supposed to go.

But I think the first thing you need is some external incentives for the labs to even go along with any investigation at all, because otherwise they'll just say, well, in that case, we're just not gonna let you in. And I think there's just not enough of that pressure around just yet.

(53:27) Nathan Labenz: Yeah. So much depends on he who must always be named. Another thought experiment — speak of the devil. If I was president, here's something I would be interested in trying, so you red-team this idea for me. What if the president were to say to, let's say five companies, could be six, seven: "Hey, this is getting pretty wild. I don't think I really know what to do, but I think you guys can figure it out amongst yourselves. So you have 90 days to come together, come up with an agreement by which you're going to work together to pace the frontier. You'll police it. Maybe you'll use some secure private computing constructs to be able to interrogate what one another are doing, and you'll have to agree on how that access will work. But you'll police one another, and if you can't reach such agreement or can't sustain such agreement, then I'm gonna have to get involved, and you're gonna like that a lot less." What do you think?

(54:25) Anton Leicht: Well, I think you should run. And I also think —

(54:31) Nathan Labenz: Might be a little — hey, there's still time in the cycle. What am I talking about? Hyperstition.

(54:34) Anton Leicht: No, no, no. You can get into the primaries. There are undecideds yet for the primary. No — yeah, I think that might work. I think that is also kind of the attitude the administration currently already takes, where there's a sense of, well, you guys built this Mythos thing. We don't exactly get why you would ever do that, but now you've caused this problem. Can you please just figure out how to fix it now? And then if you don't, then something, something, export controls, a lot of pressure, and so on. I think finding some slightly more structured version of that is fundamentally also the FINRA for AI — FARO, SAFA, SRO, whatever you want to call the idea. I think that's basically the slightly more structured version of it, which is industry comes together and figures out what the standards for how this should work are.

I think if you put OpenAI and Anthropic into a room to figure out what they should do, what they want to do, what they see as the risks, and what they think should be done to pace them, I think they'd probably do it, and I think this would work. I think maybe GDM also works. I think Meta and xAI have a very different view of the risk case, a much more skeptical view of industry coordination, of voluntary industry standards, and of actually doing a lot of things that slow down their capability progress — which, to be clear, is ironic, because something that specifically paced the frontier would give them a much faster path to catching up. So they should structurally and instrumentally be in favor. But I still think they're very skeptical of that. They have a lot of influence with the administration, so I think they would, A, just be opposed to that kind of broader "all the frontier labs should figure out what to do." And I also think they would be very likely to influence the negotiations in a way that would make it extremely difficult for there to be any common standard, because if you water down whatever you want to do to an extent that would make Meta and xAI fully on board with it, I think that just wouldn't realistically be good enough — and then OpenAI and Anthropic would maybe say, well, this is not good enough, this doesn't fulfill the spec that we've been given by the executive.

So I think there probably has to be some more substantive guidance than just "you guys figure this out and find some consensus" — at least some minimal idea of how to do it. But yeah, if you had some more substantive guidance, then there is a path for industry self-regulation. And I think even then, we can bring in the third parties again to verify it. That's one mechanism to make industry regulation work. The other is just to have the labs check each other's homework — OpenAI checks out what Anthropic is doing on this end, Anthropic checks what Meta is doing on this end. That's a little bit more dicey in terms of industry secrets, but there are precedents for this. It's also not unworkable and not impossible. But yeah, if you can bridge the gap between the tier two and the tier one, who are, I think, in very different places in terms of safety, then you can make it work. But I think that's a big if.

(57:33) Nathan Labenz: Yeah. One — actually, two things that have come up a lot recently for me, because I go around doing this, putting these ideas in front of people and asking them to tell me why it can't work. And common answers I get to various kinds of safety-minded collaborations are, domestically: well, that might be an antitrust violation, so that could be a big problem. And then internationally, even for things that are not at all about exporting chips or chip-making know-how, people are still afraid — even on just basic AI safety research collaborations — export controls, it could be a big problem. They're very broadly and vaguely worded, and enforcement could be kind of arbitrary.

So I guess two questions I have around that. One, do you think it would make a big difference — it seems to me like it would — for the president to just come out and say, "Hey, here are some things that we are not planning to bring antitrust or export control enforcement against"? And if they don't do that, then I also have the sense that we maybe need people to be willing to have the fight. It's not necessarily my place to advise all these AI safety nonprofits out there, but if I were to be so presumptuous, I would say: I think you should go for it, and put a little faith into the judicial system and the fact that we do have due process. You're not gonna go immediately to jail for having done some AI safety collaboration research project with a Chinese academic. So don't censor yourself, don't cancel the project before it even gets started. Go do it. If somebody wants to pick on you, that'll suck, but this is kind of an important time — somebody's gotta be willing to stand up and have the fight. What do you think about that?

(59:25) Anton Leicht: Yeah, so I distinguish between the cases here. On the export-control-collaboration case, I think at that point you're basically talking about whether you can insulate yourself against vindictive and capricious action by the Trump administration. And there — yeah, if you think it's worth doing, then you should just take the fight to that authority. I think that's clearly not in scope for that authority. I think the Trump administration probably should not use these authorities to crack down on this sort of research cooperation you described. On that, I'm with you.

On the antitrust stuff, on industry coordination, I think the problem is, A, it is actually unclear whether it isn't just a substantive antitrust problem to do substantial industry collusion on not competing on frontier development, and therefore it's unclear whether this actually has inflating pricing effects or not. But at least in all other domains, it would. Industry coordination agreeing not to pursue further technological innovation usually has adverse pricing effects that you would really not want. So I think this is arguably in scope for actual antitrust rules. In that case, yes, the administration could quite easily come out with guidance, with non-enforcement letters, saying, well, we don't plan to bring any action against anyone who coordinates for the sake of AI safety between industry.

I just don't think the administration is actually gonna do that, because I think the administration has so far enjoyed finding new and novel pathways to be annoying to Anthropic specifically. And I would suspect that the moment Anthropic decided to come up with any substantive and helpful way to coordinate between different labs to make some deceleration — some pacing — happen, I think the administration would just find some way to act against that. So I think the only way you can save yourself from that sort of enforcement is broader industry cooperation. You get xAI and Meta and OpenAI into the boat early. You make it very difficult to target just Anthropic, or just labs that the administration doesn't like, with this sort of antitrust authority, and then I think you're probably safe.

But the problem is it's just very, very difficult for these organizations to take the fight to the Trump administration. Yes, there is due process, but the IPO conversation we had earlier plays into this, which is: well, do you really want to go 14 rounds with them in some court, and do you want to bet that you don't get any sort of very Trump-favorable judges, as they did get on the DC court on the supply chain risk designation, for example? So many things can go wrong. The process can take so long. And you want to IPO in a few weeks, months, whatever. And so if that's your goal, then you just don't want to take the risk right now of getting bogged down in any sort of long antitrust lawsuit. So I think the scrappy safety nonprofits should probably take the fight to the administration if it really stands in the way of what they want to do. I can't blame Anthropic for not wanting to go into antitrust lawsuits months before their IPO.

(1:02:28) Nathan Labenz: Under ordinary circumstances, I would agree with that. But I do think, with how many people have come out and said, yeah, 10% risk is totally reasonable — if you're willing to take that risk, I think you should also be willing to spend some time in court along the way. But maybe that's just me. Where does all this leave us? It sounds like — of course this is the baseline — the baseline assumption coming into this conversation is that we're probably just gonna kind of muddle through, and the current state of affairs will mostly continue until, at least for the foreseeable future, something gets even crazier and shakes us out of this equilibrium. Is that basically your view? That we need another big incident, warning shot 2.0, to really open up space for different paths?

(1:03:20) Anton Leicht: I think it can be external incidents. I think we'll also see how the next Congress looks — I think that's gonna be interesting. I think it's going to be much more about political incentives that will change things in the next few months. My expectation would be that's the main pathway for things to really materially change and be different. It's not so much that something has to change about how people view the technology — I think they think it's ripe for regulation and ripe for intervention, and the demos are getting crazier, and things that happen are getting crazier. I think there's probably enough happening there. The thing I'm watching is just: when do the politicians and the policymakers move on this? And currently, there's just not that much political incentive to move. There's much more political incentive in a Democratic House to keep pushing and prodding and introducing things, and we'll see how the GOP reaction to that looks.

And I think the interesting wildcard is how the presidential primary slate looks. On the left, there's just gonna be a lot of anti-AI sentiment — I think people are going to talk about AI and AI safety a lot. They're still not going to have any ability to get anything done, so I think that puts whatever action they take all the way into '29. The more interesting thing is: what kind of record do JD Vance and Marco Rubio want to run on? And I think that's going to be the determining question for whether we see any AI policy action in '27, '28. Do they take the view that we can't run on a record of the Trump administration doing nothing about the risks that are getting people more and more concerned? Do we need a bill to pass? Do we need some executive action to happen, such that we don't get pinned down on the broad pro-AI accelerationist position come the elections? And I think probably, yes, that is in their political interest. And the question is, will they find a way to get that through despite donors, and perhaps even the president, pulling the other way?

But I think that's more a political question that has to do with: what does the polling look like? What does the salience look like? What do the midterms look like in terms of AI salience and AI impact on electoral outcomes? How do the primary dynamics unfold, and where do they leave the candidates? But I'd mostly look at these political flashpoints — the primary season beginning, and current cabinet officials being unhappy with running on the current track record — as what I'd expect to change things. And I think that's the way we do get legislation and actual action in '27, '28, and I don't think that's impossible.

(1:05:50) Nathan Labenz: One more US question, on the build-out. It seems like the build-out is actually happening. There was obviously a lot of noise around it, a lot of heat around it, but my best guess is that this will kind of look like a fracking story, where it happened — in a lot of different places, people found their right plot of land with the right jurisdiction. They bought off, or they built the parks and the stadiums, whatever they needed to. Bread and circuses kind of carry the day, and it happens. Do you see any reason to doubt that?

(1:06:24) Anton Leicht: It's federalism, right? There are just so many places you can build data centers. A lot of the policy implications of the backlash have been overstated. Especially the Texas moratorium isn't much of a moratorium in any practical sense. I think people now draw up these maps, and everything that has a moratorium is colored red, and if you make a map of places where you can no longer build a data center, coloring Texas red is kind of disingenuous. There are some minimal standards the data center projects need to clear. The hyperscalers will clear them without any problem, and they'll continue to build in Texas as long as they have access to behind-the-meter power that runs data centers — we might be running out of that, but that's a different conversation.

The Midwest is, I think, genuinely anti-data-center. It's gonna be difficult to get things done there. I think the New York moratorium, at least for the next year or two, is more real than the Texas moratorium. But for the rest of the country, you can still build in Texas, you can still build in Louisiana, you can still build in the Dakotas. And there are also tens of gigawatts already committed to construction projects that are continuing on. It's gonna get more difficult, it's also gonna get more expensive — you're going to have to pay more concessions, make more expensive deals. I think all that is a real effect, and some of this is going to push part of the build-out into other countries, some of it's going to make the build-out somewhat slower, some of which is gonna make it more expensive. But there are a lot of states, there's a lot of land. They're going to keep building data centers in America.

(1:07:51) Nathan Labenz: Do you have a theory for why that hasn't happened with nuclear power plants? Is it just that they're not enough better than the alternatives, or is there some other reason we haven't reached that same equilibrium there?

(1:08:05) Anton Leicht: I think it's a little less of a "you can put this wherever you want to, and it pays the same way." There are just fewer places you can — you connect it to somewhat local electricity demand, you connect it to somewhat local grids. You can't just build all the nuclear power plants for the country in Maine or wherever. So I think there's a little less of that dynamic of "just put them wherever they work." And also, my understanding is there's also more federal-level oversight over where and how and when you can build nuclear power plants, as opposed to data centers, where you don't have to go through any federal approval process to build a data center anywhere — you just have to build it. So I think that combination makes it a little bit easier. But I'll also say I'm just not super steeped in the US domestic nuclear build-out conversation.

(1:08:52) Nathan Labenz: Yeah, I mean, I think that federal-level oversight is probably a key part of it, and that's a big part of why — as much as I'm legitimately scared now of AI, it's moved recently from "this could get really scary" to "it is actually now scary" — I'm still like, oh God, don't give me the nuclear outcome, you know, where you get the weapons and not the power plants. I would just be so bummed about that, that I'm... yeah, I'm a little reluctant to go all in on federal oversight, even as much as I feel the need.

(1:09:30) Anton Leicht: I mean, what's your version of the weapon is, I think, one question here, right? The thing about nuclear weapons is you can build the entire supply chain for a nuclear weapon without ever generating any civilian benefit. It's really hard to build a model that's just good at winning your geostrategic competition that isn't accidentally also a big economic boon. All the ways AI systems are really economically useful are so general-purpose that it's really hard — you'd have to go through a lot of effort to not accidentally make them pretty useful economically as well. I think the question is: do you get superintelligence in your pocket? And that's an open question. But even if it's just the US government procuring superintelligence to use to win against China, whatever that means, I think that just incidentally still builds a system that's very economically useful. So much less than with nuclear — I just don't think you can divorce the civilian and military uses in the same way.

So in that sense, we should be optimistic based on the nuclear example — well, at least we didn't stop entertaining nuclear arsenals just because we stopped building out nuclear power. In a somewhat similar way, we're not going to stop building AGI, superintelligence, whatever, just because there's some domestic resistance. And in the case of AI, I think there's just going to continue to be those civilian economic spillovers much more easily. So the strategic impulse actually cuts in our favor. So maybe that's one thing that might make you a little bit more optimistic about it — not all the way to superintelligence in your pocket, but a little bit more.

(1:11:06) Nathan Labenz: Yeah. Hey, I'll take what I can get.

So let's talk about the rest of the world. You have this big report that just came out on transformative AI strategy for Europe, and there's been some discussion — I actually talked to one of your coauthors a bit back about the compute deficit that Europe has and the need to do something to be a live player going forward. But before we get into the strategy for what Europe should do — what's the worry if they do nothing? Because I also think whatever you think might happen to Europe if Europe stays the course is probably what happens to, like, 70%, maybe 80%, of the world's population by default. What does the future look like in your mind for Africa, Latin America, South Asia, etcetera?

(1:11:55) Anton Leicht: I think it's going to be really tough, because fundamentally, a lot of the catch-up mechanisms that lower-middle-income countries, in very general terms, have used and enjoyed and been able to leverage over the last few decades are just deeply incompatible with a world that has both very advanced AI systems and, downstream of that, ultimately much more automated manufacturing capacity.

So the most immediate and obvious mechanism was always just to bet on the demographic differences — you had very rapid population growth, a fairly cheap workforce you'd be able to use to your comparative advantage, and then quickly bootstrap into hosting some foreign firms and exporting some valuable good into the global supply chain, predicated on the idea that you had this workforce you could put to use in a way that made you a comparatively beneficial country to conduct business activity in. And I just don't know whether that's going to remain the case. It's definitely not going to remain the case for most aspects of the menial services economy — I just don't see a stable way that that sector of the economy really exists once we have very powerful AI systems. There are definitely going to be new services jobs, human-preference jobs — you can think about all these labor-market effects in the long run. But this idea that you can just be immediately useful to global supply chains by doing labor cheaply in the service realm, I think, is just not going to work out anymore.

The question is whether it's going to continue working out in manufacturing. I think that has a lot to do with how quickly automation goes, how big the efficiency gains are. There's still a world where manufacturing just gets more and more bottlenecked in a post-AI future, and it turns out you can at least catch up via manufacturing — that doesn't strike me as entirely impossible. But that's just for the general catch-up mechanisms.

Then the other question is what the stable geopolitical endgame is. And even if you get to this manufacturing-plus-cheap-jobs part of the catch-up mechanism, it seems very difficult to figure out how any country in that spot ever gets leverage over what happens at the frontier — which is to say, they don't get any oversight or regulatory input into how frontier AI systems are built, and they probably also don't have any hard leverage that ensures they'll continue getting AI exports and continued access to AI supply chains and so on. So they're basically at the mercy of whatever great power provides them their AI models, and maybe within that they can find a somewhat favorable arrangement, but it seems very unlikely they'll get a stable say and a stable input into that.

And I think that just carves the world into spheres of influence of those that have very powerful AI and are able to export it. There are a bunch of other downstream questions that make this more complicated — how much do you need frontier AI, how much do open weights play into this, at what point can you build your own digital sovereign infrastructure — but I think at least for the medium term, it's this quasi-vassalage to the frontier-AI-building powers that's just the most likely outcome for most of these countries.

(1:15:12) Nathan Labenz: In terms of how people live, do you think that could create a story kind of similar to the Chinese story over the last few decades, where life is getting a lot better, people are getting richer, they just don't have a say in the overall high-level direction — but at the street level, things are trending up and up?

(1:15:36) Anton Leicht: I think in absolute terms, people are going to be richer and wealthier and better off. In terms of the economic effects, I think they're just going to be relatively disempowered when it comes to meaningfully shaping the trajectory of the world, and also in terms of having an ability to catch up to however well the frontier countries, so to speak, are doing. But in absolute terms, it'll keep being growth, it'll keep being spillover effects, and redistribution gets easier as well. I think on the street it's going to look nicer. It's going to be a sort of economically better scenario. So in absolute terms, you wouldn't mind too much.

I think the more fundamental question is: what does it say about democratic say and human autonomy and human dignity that none of these decisions really factor into where the broader trajectory of the history of the world goes? And I think that is a more profound sense of disempowerment that we should still be concerned about. But, yeah, practically speaking, not that bad.

I think the other part of it, practically speaking, is susceptibility to misuse. And I think that could be extremely destabilizing. There's a current assumption that to guard against a lot of forms of AI misuse and AI loss of control, you need your own AI systems that defend you against that. It's most obviously true in the realm of cyber. I think it's also conceivably true in tracking and monitoring potential deployment of pathogens — the entire virus conversation. It's probably true in terms of scanning and filtering and screening against scams and these sort of social-engineered attempts and whatnot. And it's probably also true in terms of safeguarding infrastructure against extortion attacks and so on.

If you expect there to be a world where non-state actors, terrorists, criminal groups get access to at least fairly capable AI — because they're able to steal it, because they're able to post-train "terrorist GPT" on some open-source model, whatever — and you also expect these countries to not have any sort of coordinated, assured, and widely deployable access to these systems, I'm not sure they're going to be able to protect their citizens from AI-driven harm, AI-driven misuse, and also potentially the labor market effects. I think that all sounds like they would be very susceptible to that. And then you can imagine a lot of very destabilizing scenarios. If your state no longer protects you from AI-driven harm, then what do you turn to? Maybe you turn to mass migration. Maybe you turn to other ways of structuring your personal security, as we already see in some of the failed states in Latin America, where criminal enterprise runs a lot of the day-to-day structure in a lot of parts of these countries. I wouldn't think that would be impossible for a lot of these countries as a medium- to long-term outcome. And I think that also has a very wide — but the pure economic story is pretty positive. The eroding-the-authority-and-power-of-the-state story is a lot more concerning. And I think if you take these together, it is not a particularly rosy outcome.

(1:18:36) Nathan Labenz: that's probably 70% of the world's headed there, and Europe is kind of the one place that can maybe engineer for itself a different outcome. Tell me — I mean, if you disagree with that, tell me. But I'm going next to: okay, what does Europe want, and how does it get it?

(1:18:53) Anton Leicht: Yeah, I mean, I think the problem that Europe faces is not too dissimilar to what a couple of other Western countries, or general liberal democracies, face as well. I think, fundamentally, Australia is in a similar boat. New Zealand is in a similar boat. Japan and South Korea are in somewhat similar situations. Canada is in a similar situation. So I think that, plus Europe, plus the UK, is the cluster of US-allied middle powers that have a potential trajectory out of this. It still needs a lot of work.

I think there are two fundamental ways to start looking at this. The first way is looking at, well, put aside all the AI things — what do you want Europe's economic position to be? And if you start thinking of that, you just think, well, you want to be good at the things Europe is currently good at. You want to be good at some aspects of manufacturing, some aspects of the artisanal good, the high-state-capacity things that Europe currently is good at — whether that's welfare states, whether that's just high levels of security and safety. A lot of things are going well in Europe; you just want them to keep going well. Plus, you want to find some way to actually revitalize your current economy. And then AI comes into the picture as, well, that seems like it could either really accelerate that or it could really destabilize that.

And then you ask the question: what do you need AI for in that context? And I think the other way of looking at it comes to the same conclusion, which is: well, what does Europe currently not have? And the answer is it currently doesn't have frontier AI systems, which turn out to be one of the most important economic inputs of the future, and also one of the most exciting parts of strategic and economic competition right now. And the question is, well, what do you do about that? And I think there you quickly realize, well, building these systems ourselves is just too expensive — it doesn't actually work. So the next best thing we can think about is: how do we get access to frontier models in a way that is assured and secure, and allows us to build what I described as the first approach around? How do we reduce the geopolitical risk of just doing the things we're good at? How do we make sure we have assured access to frontier systems and don't get cut out of this AI, AGI conversation while we do the things we're good at?

And I think these all come together to: you need something to incentivize selling frontier systems, you need something to make the Americans not nervous about selling frontier systems, and you need some productive way to use the frontier systems downstream to make something happen around that. And I think the strategy that we wrote tries to answer these questions — and I think especially the parts that I most contributed to try to answer these questions.

Just very briefly running through the high-level takes: the first thing is this compute-for-access idea that I first wrote down late last year, early this year, and started shopping around with a lot of countries in the world ever since — which is now also one of the pillars and one of the main asks of the strategy. Which is to say: we build data centers for American, or in cooperation with American, hyperscalers. In exchange for the favorable conditions we provide these American hyperscalers and labs, we get assured access to the models that run on these data centers. And as long as the Americans keep giving us the model, they continue to get access to the data center. If their side goes back on the deal and we're cut off from access to the frontier models, they lose access to the data center. This is the incentive part of the conversation: we build the infrastructure and get access in return.

The second part is how do we make the Americans not nervous about doing that? Because done wrong, this is a security risk. Right? You can't run this—

(1:22:31) Nathan Labenz: —with the UAE, for god's sakes. We should be able to reach a deal with Europe.

(1:22:35) Anton Leicht: Yeah. Well, I mean, I think the UAE thing is kind of fragile. I think the UAE is kind of worried about, well, what is the future of that — will there actually be frontier weights hosted on UAE data centers? And I think that's very unclear. I think they're going to run some inference on them, but maybe it's just going to be Haiku inference and not Fable 6 inference. And I think that's an open question.

So the question is, how can we get the security alignment to work out in a way that the Americans aren't too worried about hosting the models there and giving the model to the European economy? And I think that just has a lot to do with aligning with the US on a lot of these security provisions — building out the data centers to be secure on the cyber and physical side, building out KYC regimes with European firms — just making sure the Americans don't have any well-grounded national security worries that would pull against the incentives from compute-for-access.

And I think the third thing is: let's get a little bit more self-assured about the assets that Europe does have. Right? Europe has both broad economic assets — a very powerful economy, still in absolute terms, if not in terms of growth trajectories — and Europe has a lot of assets in the semiconductor supply chain, right, like ASML and Zeiss and all these things that play into building frontier chips and therefore frontier models. Let's just think about how we can be strategic about that.

Let's set up an anti-coercion instrument of sorts that says: well, if everyone plays nice, we'd love to feed these assets exclusively into the American supply chain. We're willing to align with export controls vis-à-vis China. We're willing to be good friends and good partners to the US. And also, if the US ever does decide to use its ability to cut off frontier models as a means of coercive action, then we're also willing to use the supply-chain bottlenecks we have as coercive action in return.

Anything between those three — frontier access is pretty assured — and then you're at a point where you're back to where we were before AGI, which is: Europe still has a lot of structural economic problems, we still have to solve them, but at least we fixed the geopolitical problem of being cut off from frontier access. And I think that's maybe step one. I think that's the thing I'm excited about getting done in Europe in the next year or so.

(1:24:36) Nathan Labenz: What's the hardest part about it? Is it just getting data centers actually sited and built, or are there other challenges that you think would be bigger than that?

(1:24:45) Anton Leicht: I mean, we can have this conversation, and there's a shared understanding that the suggestions I make interface with a realistic future that we think might happen and that is worth preparing for. This is not the case in many rooms with policymakers in Europe. I think there is deep skepticism of the continued trajectory of capabilities of US-built AI models. There is a lot more optimism about the broad availability of open-source competitors that can basically do everything as well as the American models. And there is a more fundamental question around: are these models that powerful? Is it that important? Is it that big a geopolitical issue?

And then there's also the question of, well, if the models are that important — if everything that I and we and people say is true — then why shouldn't we just build this ourselves? That surely can't be that expensive, right? We'll find a more clever way to do it. The Americans are sort of wasteful with their own supply anyways. We'll just spend a few million dollars and surely we can rip something together.

And I think just cutting through that — which I understand to be complete misunderstandings of the material reality we find ourselves in — just cutting through that and making the point of: no, this does not accurately describe reality. You need to think about this in clear-eyed ways that respect that this thing happening in America is real, and the Americans are fundamentally right about a lot of the aspects of this. That's the biggest barrier.

And then I think if that awareness existed, there are still political things to figure out. For the ASML thing, it's going to be some amount of triangulating between the Dutch government interest, the ASML interest, the interest of the other member states — that's not quite easy. In terms of data centers, there's going to be some domestic skepticism against American tech firms and working with them — that's not going to be quite as easy. In terms of the security alignment, there are going to be people who are going to be more excited about hedging towards China and trying to stay between worlds a little bit. I think all of these are surmountable — very, very easily surmountable — if you just get the alignment and the awareness of what's happening here. And I think that's the main challenge.

(1:26:45) Nathan Labenz: So one of the things I did notice in reading the report was, you know, the authors — you and your co-authors — are sort of willing to dream a bit in terms of how the authorities might act. Right? At some point there's basically a statement that doing this in the sort of half-assed, kind of highly bureaucratic, "everything's a committee to nowhere" mode that European governance, at least by reputation, often acts in, would be maybe worse than not doing anything at all. So how realistic is it that you can actually get this sort of action, and what's the mechanism for doing it?

(1:27:29) Anton Leicht: I mean, it's still downstream of urgency and awareness of the situation, really. I think there's always a trickiness in writing for national governments generally — you try to write something that isn't quite within the Overton window of what they're willing to do, but that also isn't so far out there that they'd never do it, right?

I think one failure mode is you just write, well, we have 20 million in the budget, so let's just think of the maximally AI-pilled way to allocate the 20 million. It just turns out it doesn't matter — either way, you can just burn it, you can throw a party, it doesn't matter, it's not going to change the conversation. And the other failure mode is to be like, well, we're going to be maximally honest about what we think should be done, and we just write exactly that up, and then, you know, you guys can tell us "oh, I'm so sorry, we were wrong" later on, and then it's going to be too late to do it, and we're just going to shrug and go, well, we told you the honest thing.

I think the art, or the trick, of getting this kind of thing right — and I hope we struck a decent balance — is to aim at just a little bit more ambitious than they currently are, accounting for the fact that they will get more ambitious and that they need some nudging toward being more ambitious. And I think that's the sort of calibration the strategy tries to reach.

I'm optimistic about that. I mean, if I had to give you odds of this strategy as a whole being implemented within the next year, it's not that high. If I had to give you odds on elements of it making it into serious policy attempts and actually getting set up, I think it's pretty high. I don't know which ones of them are going to be — I could make bets on which of them are going to be more popular and less popular. But I think some of this is going to happen, and I think that's the hallmark of a well-calibrated strategy.

We have done this before in Europe. There have been times when Europe has managed to very quickly act very decisively, in a way that has delivered results as quickly as it has anywhere in the world. I used to work in German policy and politics for a bit, especially in energy policy, and in the immediate aftermath of the start of the war in Ukraine, there was an absolutely heroic effort of the German government to buy this fleet of shadow LNG tankers all over the world to get them to transport alternative gas supplies to Germany once the pipelines were cut off. There was just as heroic an effort to get LNG terminals built out in some of the more NIMBY parts of the country. And within like six months, the German government, with all its capacity and all its urgency, managed to consolidate resources so that we made it through the winter with no problem — complete, fine energy, no problems at all with lack of heating or anything. That was the big doomsday scenario, and it just didn't happen. There was a massive, heroic effort, and it just worked.

Before that, I worked in COVID policy, when that was happening. There was a lot of political pressure against joint vaccine procurement. It was very, very difficult to get the negotiations right. It was very, very difficult to get the member-state interests aligned. And Europe managed to procure vaccines fairly well, and I think the vaccination campaign in Europe went fairly well. We can talk about non-pharmaceutical interventions — that's a little bit more of a messy story — but I think that all went pretty well.

And I think Europe can do this. You just have to realize that this is sufficiently important. And I think we're not that far from realizing that this is important on a COVID-slash-Ukraine-war scale. And I think if we just get there, then we can definitely make progress on the kind of recommendations we make in the strategy, and I don't think that would be a big problem.

(1:30:45) Nathan Labenz: Cool, very interesting. Are there small countries that you think are worth calling out for taking a distinctive and potentially effective approach? I was thinking — I don't know anything about this other than that I know their sovereign wealth fund — I'm thinking Norway. I know their sovereign wealth fund is (a) large, and (b) kind of AGI-pilled in its operations — they use a lot of agents and, you know, they're kind of transforming themselves. But I don't know if that's translated to something like a national strategy in Norway. Singapore comes to mind as somebody that might be interesting. Who else is doing interesting things out there, even if they're, you know, kind of small and carving out a narrow path, perhaps?

(1:31:29) Anton Leicht: I mean, threading a point—

(1:31:30) Nathan Labenz: —in Threading the Needle, you might say.

(1:31:31) Anton Leicht: Yeah, yeah. I think Singapore, Norway, and the UAE are probably the three you'd most obviously mention.

I think Norway could do much more. I think the exciting thing Norway could do is become an inference hub — a haven, whatever — for the entirety of Europe. It's a little bit hard to invest domestically with the sovereign wealth fund itself, but you can conceivably come up with schemes to invest into European consortia that then invest into compute build in Norway. Norway turns out to be a pretty decent place to build a lot of compute — hence the first attempt to build a Stargate, now a Microsoft data center, there. I think Norway could be more AI-pilled about deploying these resources, but Norway has a lot of resources that could easily be deployed and pivoted towards that. And I also just think the base case of investing a bunch of your wealth fund into — I mean, maybe "recreating situational awareness" isn't as en vogue as it used to be, like two months ago or whatever — but basically recreating a fairly AI-heavy portfolio as part of the wealth fund would probably just let you ride on the coattails of the AI revolution for quite a long time. That probably works.

The UAE play is building a bunch of data centers — they're basically just finding a way to turn money into something that's an asset in the new economy, and I think that's a good way to spend a lot of money if you have it. Now it's incidentally kind of a tough situation to be in, that you're that close to Iran and that it's that easy to drone-strike data centers. So I think that's thrown a wrench into that plan. But I think structurally that was still a pretty good play, if they can manage to build the data centers quickly and if they can manage to secure the next generation against drone strikes and so on. I think that's still a play that works.

I think Singapore is another interesting case. A, there's also wealth-fund investment questions. B, there is just a massive amount of state capacity in terms of understanding what's going on and engaging with that. I think there's probably no parliament with a greater density of readers of very AI-pilled, very insidery publications than the Singaporean one. I think the same goes for the civil service. It's just — Singapore is a little bit tougher, because a lot of the Singaporean economy is very exposed to AI disruption. And I think if you're Singapore, you just have to sort of hope that the white-collar apocalypse doesn't look quite as apocalyptic, because it's just hard to pivot an economy that is as big and as service-heavy as Singapore towards a completely new way of operating. So I think there's a lot of capacity and interest there, a little bit less of an obvious AGI-pilled play to pursue.

(1:33:58) Nathan Labenz: Are there any other countries that you think are well-positioned — maybe more well-positioned than they know — that should be doing something, that are just kind of sleeping at the switch?

(1:34:08) Anton Leicht: I mean, I think Australia is kind of awake now, but I think for the longest time it was — Australia is just such an insanely good place to build compute, both for data-center-construction and energy-supply reasons and so on, but also for security integration reasons. There's a really deep amount of national security trust between the Australian and American agencies. I think there's a strong understanding that Australia would not defect to China in any way, that they would be willing to play ball and be aligned with the China-focused export controls. And so that just makes Australia a great place to run compute-for-access, a data-center-build-out place. I think more recently we've seen more of that happening — I think that's very good. But I think for the longest time that was a sleeping giant, and I think it probably still is. You could probably still 5x, 10x the data center ambitions and just run the inference for half the world out of Australia, and that would not be an overly ambitious thing to do. I think there's still a lot to be done there.

I think maybe the inverse of this is the UK, where the UK has just the greatest density of talent and expertise, both in government and outside, just outside of the US. And it's not entirely sure what exactly they're planning to do with it, or if they can do anything with it, just because of the broader political conditions of the UK — the skepticism toward US alignment, but also the sort of damaged relationship to the European Union and the rest of the middle powers. It's very, very difficult to figure out what this incredibly talented cluster of people is actually supposed to do in the UK.

So in a way, the UK and Europe really have inverse problems: Europe has amazing assets that it could use extremely well to actually have a very live-player position in this AI conversation, and it's just really hard to get them to. Whereas the UK has all the awareness and all the expertise in the world; it's just not entirely sure what it should even be doing with it. Because at the end of the day, if your starting hand doesn't include any cards that are really good for an AI future, then you can be as aware as you possibly want to be, and it's still really hard to get something done.

(1:36:05) Nathan Labenz: If you were the rest of the world — let's say you're Brazil, or you could pick your countries, or maybe you would put different countries into different positions — I hope this doesn't happen, I'm hyperstitioning better US-China relations and some form of collaboration, and not carving the world up into spheres of influence. But one thing I've been wondering lately is: because of course there was this reporting that the Trump administration was planning to do something along those lines and tell countries, "you're either with us or you're with China, pick your camp" — if you were put in that position, as, say, a Brazil, or again, pick your country, how would you decide? Where would you go?

(1:36:47) Anton Leicht: I mean, I think the more important you think AI is, the less justifiable it is to go with China here, just because there is no Chinese AI export program right now, right? They just don't have the chips. So I think if you think your economy needs access to AI systems, and then you can figure out the rest, then I think you just need to go with the US, because they're the only ones who can give you access to the computing capacity — which is why a few colleagues and I wrote a paper, which was in itself a follow-up to another paper, which are both called The Closing Window to Win, about American AI export ambitions.

China will eventually be better at offering these export deals, as China has been in the past in a bunch of international initiatives they've run in South America and Africa and Central Asia. But currently they're not, because they can't offer any data centers or chips — they can't actually offer a full-stack export that can match the US's. So right now, I think if you're sufficiently AI-pilled, you just have to pick the US. At some point China can probably throw in enough non-AI-related things that the deal looks a little bit more attractive. But just in terms of, well, is there any sort of hedging strategy to be had in AI specifically? I just currently don't think there is, as long as China doesn't have the chips. That might change in a few years, and I think then the decision is going to be much harder.

But currently it's basically just: how reluctant are you going to be about buying US systems? I think that's the realistic question a lot of these countries face. And I think ultimately that's a great position for the US to be in strategically. And the question is just: can the US actually offer a deal that these countries will think they'll stick to? And I think that's maybe the main strategic challenge for the US — which is, yes, everyone is tactically and strategically incentivized to take the deal, no way around it, but they still don't like getting a deal that they feel like the US can renege on at any point in time. And so the US has to figure out some way to commit to these deals in a way that's credible to these countries. Building data centers is part of it, deep industrial integrations are another part of it. But I think that's something the US just has to think about much more — which is, even if the deal is necessarily the only deal the other country can take, it might still irrationally defect if you're a sufficiently unpleasant partner to make a deal with. So the US just has to think a little bit more about how to be a slightly more pleasant and reliable partner. And I think it's not that far off, but yeah.

(1:39:01) Nathan Labenz: Yeah, these are just — we've got the guy for the job, so perfect. One thing I have gone back and forth on quite a bit over time, because I'm a very AI-focused person, of course, and then — but China sort of became the endpoint for a lot of the conversations I was having, and I became a little bit more of a China person. I'm still not much of a China person, really. But I always had this question of: how is this strategy — where we have these export controls, and so much of what you're saying really depends on timelines, right? If you believe in superintelligence in two to three years, you've got to be on Team USA, because there is no Chinese export — I agree with that. At the same time, if that's the path we're going down, the chips are made in Taiwan, you know? That's real close to China, real far from us. And I just don't see a world where all this is allowed to reach its culmination point, à la Machines of Love and Grace, where it's like, now we're going to make some sort of deal with the Chinese that they can't refuse, essentially, and realize eternal 1991 — without them just being like, "fuck no, you're not."

(1:40:15) Nathan Labenz: Like, we're taking out the fabs.

(1:40:18) Nathan Labenz: How do we not end up in a world like — all these sort of things seem to be taking us, as far as I can tell, to a point where China's going to hit a breaking point, and the fabs are going down. And I don't know how we get around that with the strategy we're playing. We can't defend them, right? So, I mean —

(1:40:39) Anton Leicht: Yeah, I mean, it's like —

(1:40:40) Nathan Labenz: a super sensitive asset. It doesn't take a lot — from what I understand, a piece of dust or a skin flake can ruin a batch, so they presumably can't really be defended. How do we not end up there?

(1:40:52) Anton Leicht: So, I'm not a big US-China geopolitical-competition guy — how do we win this? I have 194 other countries to focus on, and that hasn't left me enough time to really think about this in as much detail as others have. But very briefly, there are three ways to avoid that.

The first is that AI systems just get so powerful — the US is so far ahead — that escalation around Taiwan is suicidal for China, even more than accepting some amount of US domination. It's very unclear to me what exact shape of AGI would be so powerful that that's the case. But if you get sufficiently far into superintelligence, at some point you might think that's just a dominating advantage, and you actually can't go to war with a country that has this kind of system. Maybe that's part of it.

The second thing is: maybe the fabs being blown up just isn't that big a deal. Yes, it obviously destabilizes the entire supply chain — that's the endgame in terms of US-China competition, and who knows what happens then. But you'll have some indigenized capacity in Arizona, and you'll have all the chips already up and running. So if you're already halfway into your intelligence explosion by then, it turns out you can run all of this on the chips you've already built. Yes, if TSMC gets taken out, the chip supply in a year really takes a hit. But maybe AGI can do a lot in a year, especially if it gets TSMC Arizona.

And I think the third thing is: China also doesn't have indigenized capacity, and it's not entirely clear that going to war with Taiwan, in a situation where they already think they're behind in the AI supply chain, is the best way to escalate the conflict — especially if there's still some TSMC capacity in some way, shape, or form indirectly ending up in China. If that's the case, China might just think it's best to catch up — hopes revolving much more around domestic indigenization — and that's the least-AGI-pilled version of the future. Or: is it really worth going toward the specific point where the Americans have this decisive technological lead on AGI, and the Chinese diffusion play and the semiconductor indigenization play haven't really worked out yet? Can we just work in their shadows a little bit more, indigenize some more of the capacity, and then deploy later on? Not sure whether that's the best strategic take, but I think that might also be one strategic approach they take.

But I also think it's a massive vulnerability, and I think any reasonable AGI endgame has to account for the fact that the Taiwan situation just might blow up in our faces. I think a lot of them don't.

(1:43:35) Nathan Labenz: Yeah — again, to talk about "threading the needle": this has been awesome. How about a little lightning round to close? You mentioned hedging, and also, in the context of Norway's sovereign wealth fund, buying the right equities to get through the AI transition successfully. A challenge I've been wrestling with lately is the Tyler Cowen challenge: if you're so doomer, what are your shorts? I've been trying to come up with an actual answer to that — I'm not a total doomer, but I think he should be taking it more seriously than he is. So I want an answer that's either "here are my shorts" or "I really tried and I can't come up with anything," and that's kind of where I'm at right now — I cannot come up with a way to get rich in the doom scenario. Do you have any suggestions to answer Tyler?

(1:44:31) Anton Leicht: Yeah, I think my general sense is that Tyler imagines a much more continuous and smooth on-ramp into actual doom, which makes betting on volatility, near misses, and pretty catastrophic disasters that aren't quite doom make a lot of sense in that world. I'm not sure that's true. I think a lot of the ways things go badly are like — things go extremely well in the market all the way until they go really badly, and the only situation where you cash in is when you're dead. That's the least convincing part of his argument to me — this idea that normally you'll get all these near misses, and they'll already create an expectation in the stock market. I think there's a very reasonable doomer view that concentrates basically all of the probability mass of doom on things going well all the way until doom. If you look at a lot of the canonical doom scenarios — people who've talked seriously about existential and catastrophic risk — many of them describe a scenario where everything looks like it's going really well, strategically and economically great, until the point where the takeover happens or the big incident happens. I think that's the main problem with his argument.

I also think many people just have coherent worldviews that they don't bet on — they don't take financial bets even when they're committed to them. That's a more boring meta-contention. But my main view is that I don't think it's a smooth distribution of probabilities. I think a lot of this either goes very well or goes very badly, and there's not a lot of world where it's volatile and goes kind of badly for a while and then kind of well for a while. So, sadly, I don't have a good trading strategy either.

(1:46:12) Nathan Labenz: Yeah, okay. Well, I appreciate you thinking it through. Do you maintain a p(doom) number? I'm sure you've been asked many times — I haven't heard you asked, though. Do you have an answer?

(1:46:23) Anton Leicht: It depends so much on what you include in doom. If it's human extinction, it's very, very low. If it includes all the catastrophically risky scenarios — including gradual disempowerment and stable totalitarianism, that kind of thing — I think it's probably around 10%. But that's only if you account for the political-doom outcomes in the broadest sense. My probability of technical extinction is substantially lower than that.

(1:46:58) Nathan Labenz: So you're counting in doom, like, we live in a sort of Chinese-plus-plus state, and life is pretty good, but we don't have political freedom?

(1:47:09) Anton Leicht: Yeah, I mean — maybe life isn't even particularly good in a lot of people's lives. Extremely disempowered, extremely low human agency, extremely low human economic participation — call it the permanent underclass if you must. I think you shouldn't include too many merely bad outcomes in the doom number, but the old portfolio of existential, long-term risks used to include things like stable authoritarianism and stable economic disempowerment. To the extent that's actually a locked-in path for the human future that you can't see a conceivable breakout from, I'd include that in doom in the broader sense. But not like, "oh, well, the economy kind of sucks, so my p(doom) is very high because I really think the economy is going to suck."

(1:47:56) Nathan Labenz: Okay — maybe something I should have asked earlier, but I think I know the answer. Everything you're projecting assumes that robotics really works. It doesn't necessarily have to be humanoid, but we're going to get highly flexible robotics that can be deployed in all sorts of contexts.

(1:48:12) Anton Leicht: Okay, I think — it's going to take a little bit longer than — I don't see a super crazy industrial explosion very soon, but eventually this is an engineering problem, a scaling problem, and at some point we're going to scale it and resolve the physical bottlenecks. It's going to take longer — physical bottlenecks are going to matter longer than software bottlenecks — but eventually they seem eminently resolvable.

(1:48:36) Nathan Labenz: So does that mean, let's say, 2030 is the over-under for when people start to have domestic service robots in their homes? Would you take the over or the under?

(1:48:48) Anton Leicht: People start to have them in 2030? Yeah, I think that sounds roughly right. It might take a little bit longer than that just because of idiosyncratic psychological and political resistance, but in terms of technical maturity, that sounds about right to me.

(1:49:03) Nathan Labenz: How does all this change as compute goes to space?

(1:49:09) Anton Leicht: Well — I think there are two versions of compute going to space. The first is: space is one of the places we can put compute. I think that's going to be the case by 2029 — there are going to be some data center setups worth putting in space, inference more so than training, for example. And it's also going to be gated by launch capacity, so we can't put all our compute in space starting in '29. We might also have considerably different chip supply chains — chips suitable for going to space, racks suitable for going to space, versus terrestrial chip deployments. So some compute goes to space in '29, but that changes some things: it concentrates more computing power within US jurisdiction, it makes SpaceX a lot more powerful, it makes launch-type governance a little more relevant. These are all interesting marginal shifts in how the conversation moves. That's part one.

Part two is: what happens if every marginal chip goes to space instead of any terrestrial data center, and there's basically no terrestrial competition for data centers anymore? I think things get a lot crazier then. Anti-satellite weapons become a really important part of deterrence and geopolitical stability, because that's the only way you can threaten the deployment of a superintelligence system if it's in space — if you can't shoot down the satellites, good luck, you're not stopping the superintelligence. In much the same way that the AI 2040 plan talks about making data centers bombable and visible, allowing for this kind of intervention and sabotage — in that world we just want satellites to be hittable from the ground and the compute to be vulnerable, for geopolitical stability reasons. Another effect is all the concentration —

(1:50:44) Nathan Labenz: Do you think that's the default scenario? My understanding is we could probably shoot down satellites without too much trouble — we just don't, but we could, right?

(1:50:54) Anton Leicht: I mean, yeah, I think the question is who can. The United States? Yeah. Other countries? Perhaps not, and they'd need to develop the capacity to do that. China probably has it — there are some others, like the French have the beginnings of a program, the Indians have the beginnings of a program. There are already programs — it's not that no one can, but in the same way that a nuclear power needs second-strike capability, I think there's a geopolitical-stability sense in which a sovereign nation might want to have anti-satellite capacity, and not all of them do just yet.

The other question is this crazy Kessler syndrome conversation — is there some amount of mutual deterrence around ever shooting down satellites? Because if you get to the point where you have that much debris in space, which keeps creating more debris because things keep colliding, it's going to be really difficult to launch anything into space at any point in the future. So insofar as everyone is disincentivized from doing that, in the same way everyone is disincentivized from creating nuclear winter or something, that also makes the satellite-weaponry math a little more difficult.

The other part of this goes back to our middle-power conversation — we talked about these compute-for-access deals, deploying data centers, and so on. That all hinges on the idea that the US is interested in building data centers in other countries because it wants to build data centers. If the US builds all its data centers in space instead, the incentive to put data centers in host countries is just so much lower. As a result, that sounds pretty bad for a lot of these compute-based strategies. So I think the most actionable and meaningful consequence of the prospect of data centers going into space is that the compute-for-access and compute-build-out strategy middle powers are starting to pursue has a time limit — it stops working at some point in the somewhat near future, and you should start thinking about what your endgame beyond compute is, in case the space thing works out the way SpaceX imagines it does.

(1:52:48) Nathan Labenz: A more terrestrial concern. We talked a bit about bottlenecks — I don't mean the human inertia around why adoption hasn't happened as much as it obviously could, in theory, have happened so far. But if you take the flip side of that and just look at the AIs and their capabilities, clearly there's something missing relative to the experience of hiring a human to do work, right? It feels thinner and thinner all the time — almost to the point where I'm now having a hard time putting my finger on what it is about Fable 5.1 or Astra that's actually worse than hiring a human. Do you have an answer for what that is, and what additional marginal capability gain you'd expect to actually create labor market disruption?

(1:53:43) Anton Leicht: Yeah, I'm not sure it's capability gains at this point. I think it's integrating with proprietary data, integrating with proprietary workflows. Obviously not the entire task profile of humans is currently covered by models, but on specific tasks they're better than humans, and you can drop them into specific task profiles — at least in software engineering, and also in some other general white-collar activities. I think the question is just that the labor market takes its time to rearrange around that and get to the augmented, mutually beneficial version. You can't just fire the guy sitting at his desk and plug in Astra instead — you need to be slightly more sophisticated: instead of three guys, you need one guy who tells the agent to do the tasks the agent does, but that guy has to be a little better at all the things the agent can't do. So it requires some institutional and organizational reconfiguration. I think that'll take some time, but the capabilities are there — I'd expect them to have this disruptive, not necessarily displacing, impact that changes how teams are built and how productive they are. I'm a little less sure this specifically leads to displacement in the short term — it also creates additional demand and additional things human workers can do on the margins. But in terms of disruptive and reconfiguring effects, I'd agree the capabilities are there — it's just latency, lag, bottlenecks, and friction.

(1:55:07) Nathan Labenz: What do you think would happen in a hypothetical world where Tesla decides to license its full self-driving, and within 18 months or so — let's say we make a priority of it, so we're into a bit of a fictional scenario here — but all of a sudden basically all the cars drive themselves, and the four or so million Americans who make their living driving aren't needed to drive anymore. That seems like a pretty clear displacement story that very well could happen. Do you think the economy can absorb those people? Where do they go? It seems really tough when you get down to it — okay, this guy's driven a truck for 25 years, he's not ready to retire, but the truck now drives itself. What happens to him?

(1:55:51) Anton Leicht: Well, I think part of it is going to be political responses — wage insurance, reducing hours. I think there's also going to be a political necessity to add some friction to this happening. Frankly, there'd be human-in-the-loop laws — there'd be this idea, like in Hollywood or New York, of, well, even if the thing drives autonomously, there still needs to be a driver in the seat, or whatever. I think we'd see a lot of these political reactions and frictions introduced before anything happens. That's one part.

The other part is: yes, eventually, I think the economy would probably be able to absorb at least a decent percentage of that — not necessarily better jobs, not necessarily better-paying jobs, probably worse jobs. But I'll also point out that in that specific story, you've found one of the very few jobs that clearly has one specific task to it, with no mutually synergetic way of engaging — where one technology one-to-one replaces one specific kind of worker. Most automation, AI specifically, just isn't like that — it gets specific tasks and leaves other parts of the task profile up, so it lends itself to a more augmented, coexisting structure in the medium term, in a way that driving specifically isn't like. But yeah, I think that would be very hard to absorb. We'd see a lot of political friction as a result, we'd have to cushion a lot of it with social spending, and some of them would find jobs — most of the jobs would be worse.

(1:57:16) Nathan Labenz: Your reference to the idea that somebody might be required to sit in the car even as it drives itself reminds me of a novel my dad wrote — a pretty dystopian but highly AI-enabled future where everybody is kind of out of work, but they need the dignity of work, so they're required to show up and stand around all day. They're known as "standers" in his imagination.

(1:57:44) Anton Leicht: Yeah, I just really hope we don't get there. Fingers crossed. But —

(1:57:49) Nathan Labenz: Yeah.

(1:57:49) Anton Leicht: You could make the cynical observation that some jobs in the real world already are kind of like this. But yeah, hopefully we won't get there.

(1:57:56) Nathan Labenz: A few bullshit jobs out there. But if people are still with us two hours in, they'll be interested in your thoughts on this. What do you think America should be looking around the world to learn? One thing I came away from China thinking about is there's a lot of upside to surveillance — I don't want it, for a lot of different reasons, but I'm kind of like, geez, it really sucks to leave all that upside on the table. Is there such a thing as surveillance with American characteristics? I'm interested if you have a thought on that. And also, just — what other things, when you look around the world, do you feel like the US should not just envy — because I don't think we can copy the high-speed train from China — but what should we actually be trying to import and realize our own version of?

(1:58:44) Anton Leicht: So on surveillance, it's really hard, for a specific reason: I'm just very worried about perfect enforcement of laws. American laws specifically aren't made to be nearly perfectly enforced. If you enforced every law on the books in America, this would just be a draconian oversight regime. I understand the motivation — well, this massively, perfectly disincentivizes any illegal action — but man, a bunch of things are illegal, and a lot of punishments and criminal codes are specifically structured around the idea that you catch one of every hundred, or ten, or thousand criminals, and the deterrence is calibrated to that.

The downside of surveillance, to my mind — and more broadly the downside of more AI integration, I think the most well-taken point in the Flock debate as well — is that legal systems just aren't set up for perfect enforcement. Surveillance-maxing, even with American characteristics, is just such a clear pathway to perfect enforcement of laws that were never meant to be perfectly enforced. So maybe we can talk about that once we've completely revamped the entirety of the criminal code and the practices of enforcement around it. Before that, I'd be extremely worried about going down that path.

What's important from the rest of the world — America is a very idiosyncratic, very specific place that's done very well with a very weird balance of institutions and economic activity that have, for some reason, served it well. I think usually when American politicians look around the world, into Europe specifically, and try to import one very specific part of a society, they underrate how much of that society is in a completely different equilibrium and balance. If you just transported welfare spending from the Nordics, you'd have to do things to the tax system that disincentivize a lot of other commercial activity, which then forecloses other avenues of providing the same services. I think the same is true of the political system — yes, there's something to be said for the stability of a party-and-parliamentary system that doesn't swing back quite as much between administrations, but then you also get a much less decisive government that's able to do much less and gets paralyzed into gridlock a lot more. There are countries whose concepts and structures might be wholesale preferable in how they deliver services and outcomes for their citizens, compared to America — perhaps not. That's a tricky conversation. But I basically don't know a lot of high-level, really good things about how things work at a structural, political level in any other place in the world that you could import to America without disrupting a broader part of how this country works. I'd be very skeptical of doing that piecemeal.

(2:01:28) Nathan Labenz: Does that mean we're living in the best America? Because it sure doesn't feel like we're living up to our potential in many —

(2:01:33) Anton Leicht: I think it's Pareto optimal in some ways, and that's different from the best, right? Everything you'd improve would come with a trade-off against something else, and I'm very skeptical of the notion that there are clear Pareto improvements to the way America is structured. You can scroll through the Institute for Progress's website on policy interventions, and you'll probably find ten marginal fixes to laws on the books that would clearly make this better, some of them inspired by other countries. But I think meaningful changes to how the country runs are probably not Pareto improvements — they're tricky trade-offs that I'm not sure would, on balance, make the country run better. Which isn't to say this is the shining city, or any other country is the shining city — it's just that, man, it's all trade-offs, and it's already difficult to get right. I'm just not so sure there are easy fixes to too much of anything.

(2:02:24) Nathan Labenz: Yeah, last big question. We've talked about a lot of different challenges, obviously, and vexing conundrums of all kinds — conundra, conundrums. What would you say are the most important needles we need to thread? What are the top couple few things you think are absolutely most critical? And then I'd love to hear what you think life is going to look like on the other side of this, in your 90% where it goes well — for people who are like, "this whole AI thing, why do we even do it, isn't it just stupid?" — paint the upside picture to inspire that audience.

(2:03:03) Anton Leicht: Yeah, I'm not going to fix the problem of the AI labs' big narrative concerns. I just hope we can continue on the positive trend that I think history has been on for the longest time. I don't take that much of a fatalistic view about the current, or the past, trajectory of society. I don't think we need AGI to bail us out of much of anything — I just think it's the next thing we do, the next thing there is, the next big innovation, the next thing we'll need to stay on track with the compounding economic growth we have. In a lot of ways it's going to look very crazy — some of it's going to involve space, some of it's going to involve robots, some of it's going to involve a lot of automation. But some of it is also just going to involve fairly prosaic future economic growth that makes us all a little richer, a little wealthier, and our institutions a little more functional by the day — just as they have been, at least since the start of the Industrial Revolution. That's my main hope, and I don't dream much bigger than that. Things might get a lot crazier than that, and then we'll have to find ways to deal with it. But that's my hope and my dream for the future — in a decade, or two decades, or five years if things go very fast.

And in terms of what we have to do for that: I think just genuinely muddle through. Make sure the balance of power works out and continues to work out, and the balance of wealth continues to work out well. Make sure the labs don't pull away in terms of power and control from the US government. Make sure the US government doesn't centralize and control the entire flow-through of intelligence through the world. Make sure some other countries have a stake in this — economically, and in terms of power, influence, and leverage. And whenever things seem to go off the rails, in terms of too much power amassing in one place and things looking like they're going wrong, pull it back a little bit again, keep it on course. I think that's what I want to do, and it'll just be a long exercise of doing things like that on the very small margins — and I think then we're probably going to be fine.

(2:05:00) Nathan Labenz: Well, that is an unreasonably reasonable worldview, and I appreciate you spending a couple hours sharing it with me today. This has been, I think, an excellent conversation. Anything else you want to leave people with before we break?

(2:05:14) Anton Leicht: No, I enjoyed it very much. Thank you so much for having me. It was great.

(2:05:18) Nathan Labenz: And I'd like to thank you for being part of the Cognitive Revolution.

(2:05:18) Anton Leicht: Thank you so much.

(2:08:53) Nathan Labenz: If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network — a network of podcasts which is now part of a16z — where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.