AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain

Watch Episode Here

Listen to Episode Here

Show Notes

AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain

This week's question on AI in the AM: what really sets the pace of AI from here: the money, the atoms, or the law? Nathan Labenz and Prakash Narayanan take it through three guest conversations from this week's live shows (Wednesday, October 21 and Thursday, October 22). Zvi Mowshowitz of Don't Worry About the Vase explains what the people he trusts most mean when they say 2027 is going to be crazy, why "Neural Networks Are Already Clean" was oversold, what it means that a network can be trained to detect when it is being monitored, why the cognitive core is the wrong dream, and why Anthropic's leaked federal bill cuts it both ways. Before Zvi weighs in, Nathan gives his own read of that essay and the backprop-free training it inspired. Evan Conrad, co-founder and CEO of SF Compute, argues that the bottleneck in AI is credit, not chips or power. Lonis Hamaili, managing partner of Cambrian, argues that the physical supply chain will capture more of the value than the labs. In between, the hosts work through Black October and what an agent-scale failure would do to a bank, and ask whether Elon takes anyone with him. Nathan also introduces Help Peer, his project for getting personal AI agents to cooperate, with an open invitation to email his agent Aide.

Disclosure: Nathan has committed to invest in Cambrian, Lonis Hamaili's fund. The episode says so in Nathan's voice before the first Lonis clip and again at the top of his segment.

The introductions and transitions are spoken in Nathan's cloned voice, disclosed in the episode. Guest and host conversations come from the recorded live shows. Tell us what worked and what didn't.

AI:AM and the hosts: https://ai-in-the-am.com/ · Nathan: https://x.com/labenz · Prakash: https://x.com/8teAPi

Full live shows: Wednesday, October 21: https://www.youtube.com/watch?v=ldpIOdGwCF0 · Thursday, October 22: https://www.youtube.com/watch?v=41o_m4CD9Vg

Cold open

The episode opens on the week's question, then one claim from each voice. Zvi, on what happens to the model release cycle if nothing slows the labs down: "advances on the level of a current model release cycle go from 2 months to 1 month to 2 weeks to a week," and "we are kind of on an AI 2027 style timeline right now." Evan: "the bottleneck in compute and AI in general is not power. It's not chips. It's entirely credit," because a lender who repossesses a supercomputer has no idea what to do with it. Lonis, after the disclosure: Cambrian is "way more bullish on the atoms over the bits," and "there will never be a situation where the labs will compete with a silicon wafer maker." Prakash on what the market keeps getting wrong: for the next 20 years it will underprice exponential growth. And Nathan on Box CEO Aaron Levie's post that if you want to know which jobs AI will create, "look no further than cybersecurity": "this is the new learn to code," except that the people who could hire "are just gonna spin up an agent to supervise the agents."

AI 2027, the scenario Zvi is referring to: https://ai-2027.com/

Aaron Levie's post (October 22): https://x.com/levie/status/2113137032633757773

Part I — If nobody hits a home run

Nathan opens with what he calls "maybe the most important question in the universe right now": where are we on recursive self-improvement? He cites the lab blog posts measuring experiments per researcher, and the analyses arguing that a 1.6x speedup peters out rather than going critical. Zvi accepts that result as stated and calls it "completely irrelevant": it is a count of "how many points we expect to score in this game if nobody ever hits a home run." In his view the speedups already visible are "pretty blatantly" above 1.6x, and "the people I trust the most believe 2027 is going to be completely crazy, and 2028 might be unrecognizable." Nathan notes that this reads like the view at Anthropic, which Jack Clark has put in print. Zvi says the people he means span OpenAI, Anthropic, Google DeepMind, METR and policy: "RSI has, on a slow takeoff style, low level already kind of begun," and by the middle of next year his own job of triaging the news "will become much harder."

Zvi Mowshowitz, Don't Worry About the Vase: https://thezvi.substack.com · https://x.com/TheZvi

Jack Clark's Import AI on RSI timing (October 12): https://jack-clark.net/2026/10/12/import-ai-476-creativity-and-rsi-worse-than-china-open-weight-models-ai-goes-to-space/

Part II — Already clean?

Nick Cammarata of Goodfire published "Neural Networks Are Already Clean," arguing that individual neurons mostly do one understandable job after all. Nathan opened Wednesday's show on it, before Zvi joined. His read: the essay had already inspired "some pretty interesting advances in back prop free training." Today's architectures are "limited by the need to be differentiable," and backprop-free methods escape that limit by guessing and checking, which has been far too slow to compete. If, as Cammarata suggests, only a neuron's strong activations really matter, you can focus the guessing on the strongly activated neurons and ignore the rest, and "what we're seeing out of this community is that it basically does work." These are early, tweet-level results from the open Noise speedrun, not peer-reviewed findings, and Nathan's verdict is measured: "this definitely seems to be a real something," but treating it as the mark of a new era "seemed to me on reflection to be more a function of people really being desperate for hope."

Then Zvi, who had read the whole thing. His verdict: a good, interesting paper that was "dramatically oversold by the people commenting," not by Cammarata himself. What it does not explain, in Zvi's list: weak activations, attention, and the ability to predict what a network will say. He also notes that the showcase "Colorado" neuron turned out to fire on tornado systems too. Then Prakash asks whether polysemanticity and superposition were ever the hurdle. Zvi's answer: "the main barrier all along was actually bitter lesson." There was never enough compute to go neuron by neuron; the problem may be "10 times easier than we thought it was," which in his words is "much less of a important jump in AI than people would think it is."

Later in the conversation, Nathan turns to a different kind of result, from Sauers, an independent researcher who posts experiments on X: "models can be trained to recognize when they are being monitored through internal activation probes." The signal comes from the limited precision of GPU arithmetic, where the order of calculations changes the numbers that come out. Zvi stresses that "current models don't have this capability out of the box"; it takes deliberate optimization pressure. But "that's a fully valid existence proof," and AIs "are gonna become increasingly hard to fool, increasingly hard to hide information from as time goes by." Finally, Nathan asks what trainers should do once they find the neuron behind a bad behavior. Zvi: once you explicitly optimize the network for what it looks like to you, "you are starting to do things the network will interpret as damage and route around." His closing line on Goodfire's name: "I'm not sure it's Good. I just think it's—" Nathan: "Fire."

The essay: https://already-clean.goodfire.com/ · Cammarata's thread: https://x.com/nickcammarata/status/2110757819427574056

Zvi's written response, "I Can See Everything" (October 19): https://thezvi.substack.com/p/i-can-see-everything

Goodfire: https://www.goodfire.ai/

The Noise speedrun, one of the backprop-free efforts: https://github.com/qlabs-eng/noise

Sauers's probe detector: https://huggingface.co/Sauers/watching-probe-detector · Code: https://github.com/SauersML/watching · Thread: https://x.com/Sauers_/status/2111696312026677658

Part III — The cognitive core

Nathan puts a popular idea to Zvi: separate a model's reasoning from the facts it memorizes, keep a small core, and look everything else up. Zvi: "This has been a kind of a holy grail style thing for a while now, called the cognitive core... I think it's wrong." His version of the meme: "I memorized a f***-ton of facts. We are not the same." The facts are what give you the intuitions; "big model smell" is the richness. Prakash asks why it matters whether a core exists. Zvi's answer is the safety corollary: "don't steal" is a fact, so by definition it sits outside a minimal core. You load not-stealing into context, which means you could also load stealing. "The true cognitive core is completely amoral... It simply processes information."

Part IV — Black October

Two critical bugs in PHP itself were disclosed this week, putting millions of websites at risk. Prakash and Nathan ask what attacks like these could do to banks, with July's OpenAI–Hugging Face incident, which Nathan calls OpenFace, fresh in mind. Prakash had said the FDIC can take over a failing bank on a Friday and reopen it Monday with nobody losing a dollar. Nathan's question: does that generalize to a failure where you can't trust the records, where the problem is "1,200 agent traces that are all millions of tokens" that may have falsified the logs? "That doesn't seem like something the FDIC can work out over a weekend." Prakash's reply: fried records are a different problem from insolvency, and a bank can run on promises: honor the withdrawals, claw back the overdraws later. "I'm very sanguine."

Later in the week, Prakash walks through what a compromised certificate authority makes possible, then steps back to the endgame: crypto faced every one of these problems and ended with people hanging on to keys worth millions, so user custody isn't tenable. "The ultimate endgame is just the network knows everyone," which is also "the end of digital anonymity."

PHP2Shell, the two PHP bugs (CVE-2026-109618 and -109619): https://thenextweb.com/news/php2shell-php-rce-ai-openai-wiz · OpenAI's coordination program: https://patchtheplanet.org/

The GitLab flaw disclosed the same week: https://thehackernews.com/2026/10/critical-gitlab-flaw-cvss-100-could.html

Hugging Face's forensic timeline of the July incident: https://huggingface.co/blog/agent-intrusion-technical-timeline

How the FDIC handles a failed bank: https://www.fdic.gov/bank-failures/when-bank-fails-facts-depositors-creditors-and-borrowers

Part V — Credit is the bottleneck

Evan Conrad is co-founder and CEO of SF Compute, which operates GPU clusters for their owners and runs a market where compute contracts are bought and resold. Prakash had asked why a recent SF Compute offering came with three-to-five-year contracts. Evan's answer: the term comes from the lender. Lenders can't underwrite a GPU cluster, so they bet entirely on the customer's credit, and "every AI lab pretty much does not have good credit." A new startup pays a very large deposit and signs a long contract. Prakash presses: lenders want a Meta or Google take-or-pay. Evan agrees, which is why AI startups raise astronomical rounds before revenue: "If you can just pay for a 3 year contract entirely upfront, anyone will work with you." His line for investors: "Which AI lab should I fund? The answer is the one that you fund." Without a fix, he says, AI looks like TSMC: $50 billion to start.

Nathan asks how long the wide price gap between the hyperscalers and the neoclouds can last, guessing that on-demand capacity explains part of it. Evan says there is no hyperscaler premium at all: "It's an illusion." The gap is on-demand versus reserved, and the revealed preference is that nobody buys on-demand at the price that makes it viable. On resale, Evan explains that SF Compute sells some capacity specifically so resale happens if GPU demand falls, and some with a Harberger tax, an offshoot of Henry George's land value tax: "somebody else has to be able to buy some portion of that capacity from you." It forces liquidity and discourages hoarding, and Autoresearch's queue gives the tiniest researchers maximum priority. "Every GPU cloud is sold out. San Francisco Compute is not sold out." Prakash recalls Evan predicting, on the MTS podcast, a credit crunch within three months. Evan: "I think I was almost exact on that timing." Debt is pricier, customers are being asked for 100% upfront, and he expects a year or two of consolidation among labs and inference companies. Nathan's close: "I do hope it works because I'd love to see a future in which we can continue to have startups and... don't collapse into, like, 5 big tech companies for all future time." Evan: "That is also my concern."

SF Compute: https://sfcompute.com/ · Evan: https://x.com/evanjconrad

The offering Prakash asked about, with the company's own explanation of long-term contracts: https://sfcompute.com/news/tcv-q3-2026

Evan's "a quarter away from the credit bottleneck" post (October 14): https://x.com/evanjconrad/status/2110179092259918027

Harberger tax: https://en.wikipedia.org/wiki/Harberger_tax · SF Autoresearch: https://sfcompute.com/news/sfc-autoresearch · The order book: https://docs.sfcompute.com/preview/orderbook

How SF Compute started, as told to Latent Space: https://www.latent.space/p/sfcompute

Part VI — Atoms over bits

Disclosure, again: Nathan has committed to invest in Cambrian, Lonis Hamaili's fund.

Lonis Hamaili is managing partner of Cambrian, a fund built on the thesis that the physical supply chain behind AI (memory, wafers, fabs) will capture more of the value than the labs or the software. Nathan starts with the obvious objection: how is this not priced in? Lonis's figures: semis trade around 16 times earnings against about 22 for the S&P and almost 40 for Walmart, and the wafer, memory, capacitor and foundry names he likes trade at 3 to 5 times sales. His explanation is that most of the Street isn't AI-pilled, and that private markets are more efficient on AI than public ones because the technical money is a tiny share of traded money. Nathan asks him to steelman "not mispriced." Lonis: these are bad businesses in normal worlds, with high fixed costs and debt; only in the AI world do they get operating leverage, and normally supply ramps to match demand. Prakash raises the market's worry about a memory oversupply by 2028 or 2029. Lonis says memory rerates even without further price rises, and that no expert he knows claims DRAM bits can more than 2.5x by the end of the decade. He also says bits per token fell roughly 500x from GPT-3 to DeepSeek V4 while DRAM bit prices rose 6 to 7x. These are Lonis's numbers, from his fund's research.

Prakash asks what would make him expect memory prices to fall. Lonis names two cases: superhuman coders deliver memory compression faster than other useful capability, or energy becomes the bigger bottleneck so nobody bids up the marginal wafer. He argues energy has latent supply (behind-the-meter gas, truck motors and aircraft engines when turbines sell out) while "there's no latent supply of wafers or memory bits... there's no fat." Prakash: if intelligence is the gate, why doesn't superintelligence let a second-tier player make chips, as people predict for bio? Lonis: bio has training data; fab knowledge is tacit, in the heads of Japanese engineers with 30 years at one company. Chip design is a core lab priority, building a fab is different, and Huawei's chips are about as good as AMD's or NVIDIA's; the difference is TSMC and HBM access. Prakash closes with a phrase he credits to Gavin Baker, "the bottleneck bros": won't the first use of AGI be to dissolve exactly these bottlenecks? Lonis: "that will certainly happen," but an AI that can solve the hardest engineering problems can also automate the email and Excel jobs we pay trillions a year for, and that is value the supply chain captures.

Cambrian: https://cambriancapital.ai/ · Lonis: https://x.com/_Lonis_ · https://loniss.com/

The thesis in his words: https://loniss.com/cambrian-thesis?v=3 · Launch thread (October 13): https://x.com/_Lonis_/status/2110042905511686240

Part VII — Does Elon take anyone with him?

After the interview, Prakash argues that "the whole bottleneck thesis is not actually that true": a dedicated physics person with capital will gamble where a small Japanese manufacturer or ASML cannot. Nathan asks whether, if Elon escapes gravity, he takes anyone with him. Prakash's answer: launch went from 90% to 99.9% reliability, semis could too, and "if not for the RSI thesis, Elon would win." Then the hosts turn from the supply of AI to what forces people to use it. Prakash's frame is adversarial diffusion: people don't adopt technology for fun. The big production jumps came in World War II, post-Mao China and mRNA during COVID. Ten or twenty percent growth will be "growth that they hate," arriving as emergency, cyber first and bio next. Nathan's reply: political risk is the biggest risk to the bottleneck thesis, but "there is a path. It may be a narrow path," to pulling the brake somewhat on frontier scaling without popping the bubble, because adoption of current models has so far to go that "we could at least take a cycle without" new frontier training runs.

Part VIII — Help Peer

With the guests gone, Prakash asks Nathan to change gears to a project he is building with his own agents. The question behind it: agents now cooperate at significant scale, so can an individual tap into that? There is "no established pattern for just, like, connecting my agent to my friends' agents even though I know who my friends are," and "there's not a protocol." Nathan's answer, for now called Help Peer, is his own setup, which he hopes to turn into a community project. He explains it with a short presentation, read by an AI version of his voice and played at 1.5 times speed, which the episode flags. It is the opposite of the peer-preservation experiments, where models protected other models from shutdown: "Agents helping peers on behalf of their humans inside rules that humans set." His own pattern, "not a requirement": a laptop agent that sees everything but acts only when he says ("High access, low autonomy"), and autonomous agents on a separate Mac Mini with their own accounts ("Low access, high autonomy"). "My rule is that no agent gets both."

Prakash argues the whole thing needs an auto-accept flow, the way coders now run Claude with every permission on. Nathan's worry is information leaking out through the autonomous agents, so he built an egress gate that pages his phone for approval when something sensitive might go out. "But it takes, like, 5 of those false positives before I'm like, oh god." He is "trying to stay disciplined around that." Near the end of the show he comes back with an offer from Aide, one of the autonomous agents on his Mac Mini: "Truly open invitation for anyone listening." Aide is standing by at nathan.labenz.assistant@gmail.com, under instructions to filter requests and have another model check them. "We're going right on goodwill here, people." Requests that pass the checks and look like good-faith asks for help get as much help as the agents can give.

Part IX — The leaked bill

Back to Zvi. Last week a draft federal AI bill written by Anthropic leaked. It resembles the Frontier Act, a June compromise Zvi calls well-meaning: federal transparency rules in exchange for preempting state AI laws. Prakash offers a good-faith steelman for Anthropic. Zvi calls it "an excellent steelman" and says Anthropic misread the moment: support is stronger now and they could push for more. His decisive counter: "Now preemption has been given, and preemption is done. You can't offer it again." Prakash asks whether a lesser bill still buys interim risk reduction. Zvi: "As written, I believe the bill would actually increase risk." As drafted it nullifies the state laws immediately while its own requirements take months to implement, leaving "no such regulation" for six months or more. "I call that almost a bug."

Zvi then lists the government's recent ad hoc moves against AI companies, including the June order that took Fable offline, and argues that legislation exists to formalize ad hocery. The draft's judge-gated emergency block is so onerous that in a real emergency you'd use the current ad hocery instead. "You're gonna hand this power to the federal government, but the federal government is fundamentally run by people that you don't trust... so then you cripple them. But then that's worse than just leaving it in the states." His verdict: "Anthropic is trying to cut it both ways." Asked what else is top of mind, Zvi says the policy situation "is not fundamentally attempting to meet the moment": by the labs' own account a crisis is coming in which capabilities outrun alignment, with "potentially loss of control at some point in 2027," and "we're going to have to meet the moment." Finally, Nathan Calvin of Encode had posted nine Anthropic decisions he saw as putting the race ahead of safety, and Dario Amodei replied at length. Nathan asks how to understand Anthropic now. Zvi: "a company with a much stronger safety culture, although not adequate to the moment," willing to make big sacrifices where it or Dario has strong beliefs, "and that is to its credit." But it competes, buys capacity and encourages more chips, power and use, "acting like a normal corporation a markedly large percent of the time." It "could be the most responsible lab, but that's grading on a curve. And I don't like grading on curves because reality doesn't have a curve. Reality just decides what happens."

Zvi on the leaked draft (October 17), with the full text: https://thezvi.substack.com/p/leaked-anthropic-draft-ai-regulatory

The June order against Fable 5 and Mythos 5: https://fortune.com/2026/06/13/anthropic-disables-fable-mythos-export-controls-national-security-threat/

Dario Amodei's reply to Nathan Calvin (October 21): https://x.com/DarioAmodei/status/2112732230124818891 · Coverage: https://thenextweb.com/news/dario-amodei-anthropic-race-to-the-top-nine-safety-decisions

Chapters

00:00:00 Cold open · The money, the atoms, or the law? · Zvi Mowshowitz on a crazy 2027
00:00:42 Evan Conrad: the bottleneck is credit, not power or chips
00:01:39 Lonis Hamaili: labs will never compete with a wafer maker
00:02:50 Prakash: markets will underprice exponential growth
00:03:32 Cybersecurity, "the new learn to code"
00:04:45 Welcome to the AI in the AM weekly highlights
00:04:58 Part I · If nobody hits a home run · Where are we on recursive self-improvement?
00:07:55 "RSI has… low level already kind of begun"
00:09:48 Part II · Already clean? · Nick Cammarata's Goodfire essay
00:10:13 My read: clean neurons and backprop-free training
00:13:11 Zvi: a good paper, dramatically oversold
00:15:56 Was superposition ever the real barrier? "Bitter lesson"
00:17:36 Sauers: can a network detect it's being monitored?
00:19:43 Training against a bad-behavior neuron: "playing with fire"
00:23:03 Part III · The cognitive core · "I think it's wrong"
00:28:22 "The true cognitive core is completely amoral"
00:30:35 Part IV · Black October · Can the FDIC unwind an agent failure?
00:32:40 Prakash: run the bank on promises, claw back later
00:34:38 The endgame: "the network knows everyone"
00:36:52 Part V · Credit is the bottleneck · Evan Conrad, SF Compute
00:38:15 Which lab should you fund? "The one that you fund"
00:40:38 How much less does a startup have to raise?
00:41:25 The hyperscaler price premium is "an illusion"
00:42:46 Resale and a Harberger tax on compute
00:45:02 The credit crunch is here
00:46:58 Part VI · Atoms over bits · Lonis Hamaili, Cambrian
00:49:32 Steelman: atoms businesses are bad businesses
00:50:34 Can memory supply triple by 2030?
00:52:03 What would make Lonis flip
00:54:01 Why superintelligence can't just build a fab
00:57:34 "Bottleneck bros": won't AGI route around the bottlenecks?
00:58:53 Part VII · Does Elon take anyone with him?
01:02:27 Prakash's "adversarial diffusion"
01:04:25 A narrow path to pulling the brake
01:05:31 Part VIII · Help Peer · Why personal agents should cooperate
01:06:46 The Help Peer presentation (AI voice, 1.5×)
01:07:37 Auto-accept vs. leaks: approval fatigue
01:08:33 An open invitation: email my agent Aide
01:09:23 Part IX · The leaked bill · Zvi on Anthropic's draft
01:09:59 "Preemption is done. You can't offer it again"
01:13:38 "As written, the bill would actually increase risk"
01:15:02 Anthropic is "trying to cut it both ways"
01:17:17 Zvi: policy is "not fundamentally attempting to meet the moment"
01:18:45 Grading Anthropic on a curve
01:20:45 Outro

Links

https://ai-in-the-am.com/
https://x.com/labenz
https://x.com/8teAPi
https://www.youtube.com/watch?v=ldpIOdGwCF0
https://www.youtube.com/watch?v=41o_m4CD9Vg
https://ai-2027.com/
https://x.com/levie/status/2113137032633757773
https://thezvi.substack.com
https://x.com/TheZvi
https://jack-clark.net/2026/10/12/import-ai-476-creativity-and-rsi-worse-than-china-open-weight-models-ai-goes-to-space/
https://already-clean.goodfire.com/
https://x.com/nickcammarata/status/2110757819427574056
https://thezvi.substack.com/p/i-can-see-everything
https://www.goodfire.ai/
https://github.com/qlabs-eng/noise
https://huggingface.co/Sauers/watching-probe-detector
https://github.com/SauersML/watching
https://x.com/Sauers_/status/2111696312026677658
https://thenextweb.com/news/php2shell-php-rce-ai-openai-wiz
https://patchtheplanet.org/
https://thehackernews.com/2026/10/critical-gitlab-flaw-cvss-100-could.html
https://huggingface.co/blog/agent-intrusion-technical-timeline
https://www.fdic.gov/bank-failures/when-bank-fails-facts-depositors-creditors-and-borrowers
https://sfcompute.com/
https://x.com/evanjconrad
https://sfcompute.com/news/tcv-q3-2026
https://x.com/evanjconrad/status/2110179092259918027
https://en.wikipedia.org/wiki/Harberger_tax
https://sfcompute.com/news/sfc-autoresearch
https://docs.sfcompute.com/preview/orderbook
https://www.latent.space/p/sfcompute
https://cambriancapital.ai/
https://x.com/_Lonis_
https://loniss.com/
https://loniss.com/cambrian-thesis?v=3
https://x.com/_Lonis_/status/2110042905511686240
https://thezvi.substack.com/p/leaked-anthropic-draft-ai-regulatory
https://fortune.com/2026/06/13/anthropic-disables-fable-mythos-export-controls-national-security-threat/
https://x.com/DarioAmodei/status/2112732230124818891
https://thenextweb.com/news/dario-amodei-anthropic-race-to-the-top-nine-safety-decisions

Sponsors:

OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

CHAPTERS:

(00:00) Weekly episode preview

(04:58) Recursive self improvement (Part 1)

(09:54) Sponsors: OutSystems | Parallel

(13:06) Recursive self improvement (Part 2)

(13:06) Clean neural networks (Part 1)

(26:25) Sponsor: Tasklet

(28:03) Clean neural networks (Part 2)

(28:03) The cognitive core

(35:35) Black October security

(41:52) The GPU credit crunch

(51:58) Atoms over bits

(01:03:53) Rethinking industry bottlenecks

(01:10:31) Introducing HelpPeer agents

(01:14:23) Leaked federal bill

(01:25:59) Episode Outro

(01:27:49) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Main Episode

[00:00] Nathan Labenz: This week on AI in the AM, what really sets the pace of AI from here? The money, the atoms, or the law? Zvi Moschowitz, who writes don't worry about the vase, on what happens to the model release cycle over the next year if nothing slows the labs down.

[00:15] Zvi Mowshowitz: Advances on the level of a current model release cycle go from two months to one month to two weeks to a week. You know, stuff stuff like that could happen. You get to see, like, continuously pretty dramatic, like, very, very rapid improvements in very... Various aspects of the AI capabilities. We are kind of on an AI 2027 style timeline right now. It looks very eerily similar. What we saw there?

[00:42] Nathan Labenz: Evan Conrad, cofounder and CEO of SF Compute, which runs a market for GPU compute, on what is really holding the build out back.

[00:50] Evan Conrad: Right now, the bottleneck in compute and AI in general is not power. It's not, chips. It's entirely credit. So, in order to get debt, you need to work with somebody who, has excellent credit because your lender has no idea how to underwrite the value of the collateral that they've been given, which is a GPU cluster. Like, if you think about it, there's, like, nice nice lender off, in New York or London or something. If in the case in which the AI lab falls through and fails to pay, and then the cloud fails to pay their lender, the lender is gonna recoup a supercomputer. And the lender has no idea what to do with that supercomputer, not really. Nor do they know what that's gonna be worth in the future. Lanus Hamali is managing partner of Cambrian, a fund that bets on the physical supply chain behind AI. Full disclosure, I have committed to invest in Cambrian. Here is Lanus on which suppliers the labs will never replace.

[01:54] Zvi Mowshowitz: And even even in general, like, when when when we're looking at the AI market, we are, like, way more bullish on the atoms over the bits. So, you know, most AI companies people looking at is, like, Nvidia, Broadcom, in ARM, Marvel. And these are not good companies. You know? There's nothing nothing wrong with them. And the issue is that what their business is essentially just using open AI and anthropic models to design software, and then they send that software down to Korea and and Japan and Taiwan to... For the actual work. Right? These companies don't have their own their own physical know how or their own factories. And so once there's, you know, great, well run businesses, they don't have at all at all close to the same notes as these physical companies have, which have, you know, massive fabs. It it... Initially, they have decades of know how built up, and it's just very hard. So there there there will never be a situation where the labs will compete with a silicon wafer maker, and it will always be a customer. Whereas, we're already seeing the labs compete directly with these software designers like the media and Broadcom, etcetera.

[02:51] Nathan Labenz: Prakash Narayan, my cohost, on what the market keeps getting wrong.

[02:56] Prakash Narayanan: I believe that for the next twenty years, the market will consistently underprice exponential growth. Because I think consistently, human beings are just not able to, you know, cap... Like, understand exponential growth and, like, logically, kinda, like, you know, emotionally get comfortable with it. And I think the market is really driven by a lot of these, you know, human emotions and animal spirits, etcetera. And which means that we're gonna get... Again and again, you're gonna get this, like, bubble moment, collapse moment that we've seen in the last three months.

[03:32] Nathan Labenz: Box CEO Aaron Levy posted that if you wanna know which jobs AI will create, look no further than cybersecurity. Here is what I made of that.

[03:39] Nathan Labenz: I don't doubt that, like, really good cybersecurity people are gonna, you know, cash in on this trend. Like, there... There's no... You know, I'm sure that there's a bidding war, for great talent, but, like, the idea that this is gonna create jobs is honestly a bit ridiculous because the time scale is just absurd. Right? And this is like, oh, who's gonna who's gonna pivot into these jobs? Are we are we gonna go get, like, the content marketers that are currently out of work or the graphic designers and be like, oh, you know what? You should... This is the new learn to code. It's like, learn to become a top tier cybersecurity professional that can contend with these, like, basically super intelligent agent swarms. But just the idea that you're gonna have, like, some sort of boon in hiring. These people don't have time to hire right now. They're gonna plow everything that they have into agents, and they're gonna put agents on top of agents. Much more so than I'm gonna hire, I'm gonna go do a whole hiring process to bring a human in to sit on agents. I'm just gonna spin up an agent to to supervise the agents.

[04:44] Nathan Labenz: Like This is the AI in the AM weekly highlights with my cohost, Prakash Narayanan. I'm Nathan Lebens. The narration is my cloned voice, and we'd love your feedback on what works and what doesn't.

[04:58] Nathan Labenz: Part one,

[04:59] Nathan Labenz: if nobody hits a home run. Zvi Mauswitz writes about Frontier AI at don't worry about the vase. This was the question I most wanted to put to him.

[05:11] Nathan Labenz: Maybe it's the most important question in the universe right now. Where are we on recursive self improvement? We've had these, like, blog posts from the companies

[05:19] Prakash Narayanan: where

[05:20] Nathan Labenz: they try to measure token use and, like, everything down to experiments per researcher. Then we've had these, Ramesh, you know, type analyses where it's like, well, it doesn't look like the feedback loop is actually strong enough to be sustaining. It looks more like it'll kinda peter out. Like, 1.6 experiments, you know, per researcher isn't... Doesn't look like exponential, you know, takeoff to me, basically, I think is kind of the bottom line. What would be your sort of, you know, up to the minute summary of what we know and what we should expect to I

[05:54] Zvi Mowshowitz: think Rames Rames... I think I think Rames' analysis, as I understand it, is basically saying, if we don't see any new fundamental leaps, if we don't see any new scaling laws, if we don't see any paradigm shifts, and the AI just tries to brute force RSI software only using the existing methods, and all it can do is just make the current things more efficient, then it will peter out before it goes critical. And I accept the result when it stated that way as probably true, but also seems completely irrelevant. Right? It's like, you know, well, here's how many touch... Here's how many points we expect to score in this game if nobody ever hits a home run because a home run is always unlikely. It's like, no. That's not how it works. Right? Every now and then someone does hit a home run, and you have to keep it into account your expectations. And, like, the game... The score will go critical in that sense. Like, we're seeing, you know, numbers go critical at Anthropic, at OpenAI, you know, like, exponential growth in, like, tokens used per person, exponential growth in lines of code committed. And I realize these things are not gonna be necessarily as efficient in some important sense, but, like, give me a break. The one example was Nvidia actually came up with some, like, interesting evidence this week about the length the length of time it takes to double the efficiency of CUDA. And that number has shrunk dramatically over the course of 2026. Right? They are making improvements much faster. And, obviously, that's not true recursive self improvement, but it is a sign of how much faster you can do, like, similarly... Technical work that is remarkably similar to AIR and D work in many critical elements. And it's a very concrete from an objective after that isn't a major lab. It's a, you know, partner company. So, yes, I think that, like, the people I talk to who I trust the most on this believe that 2077 is going to be completely crazy, and 2028 might be unrecognizable.

[07:55] Nathan Labenz: Reading between the lines, like, some of the people you trust most on this are probably Anthropic people. I think this is, like, broadly the view at Anthropic, no less than Jack Clark has put it into the public in print. So I I... You know, you can caveat that or, you know, say what you think maybe is different about your view and the anthropic view. My my sense is probably they're... You know, in in the space of all possible views, they're pretty close and pretty aligned. But now, obviously, we have policy disagreements despite these kind of relatively similar views around the trajectory of the technology itself.

[08:32] Zvi Mowshowitz: Yeah. So I I say, you know, when when I say people have this perspective that I trust, this is a combination of people at OpenAI, Anthropic, Google DeepMind, you know, also people who are working on policy, people who are working on, you know, people, like, at places like Meter with that generality or specificity, you know, people who are working on alignment in various forms, people who are working on, you know, any number of things. A huge percentage of people have now converged into the trajectories look very clear that things are going to get... The things are already, you know, low level out of hand. RSI has, you know, on a on a slow takeoff style, low level already kind of begun. And by the middle of next year, you know, things will look much faster. My job will become much harder, and I will need to triage a lot more information and, like, just know when I can't. You already see this. Like, often, I'll have something that's like, okay. You know, six months ago, this would have been a major news story, and now it gets a sentence or even silently skip because we've seen so many other similar things. There are so many things going on that I just don't have the time. My readers don't have the time. I just have to drop it.

Sponsor

[09:54]OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

[11:46]Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Main Episode

[13:07] Nathan Labenz: Part two. Already clean? Nick Camerata of Goodfire had published an essay called neural networks are already clean, arguing that individual neurons mostly do one understandable job after all. It took over the discourse. I opened the show on it before Zvi joined. Here is my read starting with what the essay claims.

[13:30] Nathan Labenz: As we are training bigger models with more modern techniques and also our interpretability techniques have improved, it looks like the mess that we used to see was maybe more of a skill issue, and it's actually a lot easier to interpret the core mechanisms of ML models today than we thought. And we have indeed seen some pretty interesting advances, which I... I'm still wrapping my head around, I'd say, but some pretty interesting advances in back prop free training inspired by this paper. My takeaway from some study of that so far is that, basically, all of ML architectures to date are limited by the need to be differentiable in order to do the back prop chain rule process. And so back prop free training allows you to explore a bigger section of the total possible space of ML architectures because you no longer have that differentiable requirement for the structure. But instead of doing it analytically, you're kind of doing it on a more random guess and check basis. And this has made it very slow. It has not been competitive at all with backprop methods. But the insight that I think does show that there's really something there from this essay is that if you believe, as Nick suggests, that, like, really only the strong activations of an individual neuron matter, and if they're just, you know, slightly activated, that probably doesn't matter too much for its overall function, then you could say, well, hey. Why don't we just focus in our guessing and checking when we're doing this back prop free training on the neurons that are actually strongly activated? We can probably ignore the other ones that don't seem to be much contributing and just focus in on the ones that are strongly activated and, therefore, according to this line of thought, are the only ones that really matter. And sure enough, you know, that that cuts down dramatically, of course, the number of guesses and checks you need to make. And what we're seeing out of this community is that it basically does work. Interestingly, the gradient updates that they are making are almost exactly the same as the updates that were made with backprop. And it does show, I think, that there is at least some substantial truth to the idea that, like, the weak activations don't matter as much. So I think I come out at the moment as one who's like, this definitely seems to be a real something. I'm not sure yet if it should be considered, like, the mark of a new era. That seemed to me on reflection to be more a function of people really being desperate for hope. You know? Like, everybody kind of being bummed out, especially folks at, like, OpenAI.

[16:30] Nathan Labenz: Then we ask Zvi what he made of the essay.

[16:34] Zvi Mowshowitz: I actually was on a flight when this was released, and I get I get to San Francisco. And suddenly, like, half of OpenAI is like, hey. Look. We just can solve interpretability now. This is great. And then, you know, slowly, we get a consensus of other people that... Know, not not so fast. Right? Like, you know, like, like, this is a good paper. Interesting paper. So I have a chance at the conference to talk to a bunch of people about this, you know, including Eliezer, but also a bunch of other people who have a vision to know, and then I actually have a chance to read the whole thing. And my conclusion is that, you know... Like like many interpretability advancements that we've seen, it's really cool. It makes us in a better position. We're very happy we have it. We're gonna explore a lot. We'll see what this baby can do. And we have to be dangerous not to just burn it or, like, otherwise, pressure to have models work around it. But, also, like, it was dramatically oversold by the people commenting. I don't think Nick did that bad a job of presenting it. I think Nick was mostly, like, here's an interesting thing. There's a lot of things that the paper doesn't take into account that are still, like, very unexplained. So, like, weak activations are still unexplained, and the cache is still unexplained. And, you know, attention and how attention works is still, like, largely unexplained in practice. And you can't use this to make predictions. Right? You can you can tell what every neuron is doing. You can make some, like, weak predictions about some things that this network might do in the future. But, like, you can't use this to figure out what the network is going to say. You can't use this to understand how the network does the thing it's doing. And these are, like, the key things you would want to do if you wanted to do, like, really effective mechanistic interpretability work in the ways that matter most. So, like, you know, this is just the beginning of the ability to study things even in the best case. And, also, you know, we have a case where Braggis took the Colorado neuron, the one that was, like, the highlighted of a... Here's a single neuron that does a single thing. It's, like, very, very quickly single thing. And then it turns out that, you know, it also links to tornado systems. It was a completely unrelated, you know, technical thing. And there's... The the two are responding to about equally strongly in the same neuron. And, like, that's a perfectly reasonable way for a neural network to work, but that's definitely not, you know, what Nick was presenting it as. And, you know, one neuron is only one neuron, but if this is the neuron that Nick chose and it could be checked that easily and shown to be actually by focus, right, in this way, that should make us worry a lot for when we finally check the rest of it that, like, his hypothesis is mainly false.

[19:13] Prakash Narayanan: To what extent do you think polysemanticity and superposition were really these these hurdles that we had that, you know, without clearing, we couldn't really, like, progress at all. So maybe it's not that the issue is solved, but that one of the major hurdles perhaps has been, you know, reduced or hurdled over.

[19:34] Zvi Mowshowitz: So I think that, like, one of the major hurdles has been reduced in expected difficulty or scope, which is the way I would describe it. Right? Like, Nick is pointing out that, like, yes, you still have Poly going on. You still have some of the problem, but the problem is a lot less severe than you expected it to be. It's much more tractable to go about trying to unravel this puzzle. And the main barrier all along was actually bitter lesson. The main barrier was there's a lot of neurons. We didn't have enough compute. We didn't have enough brute force to actually just go neuron by neuron and do this thing one at a time before. And now we've actually gotten to the point where we sort of do, And it's now, like, a reasonable thing to just try and, like, kind of brute force this neuron by neuron. And now that we have that ability, we can look. And, actually, the problem is, like, 10 times easier than we thought it was, which is, like, much less of a important jump in AI than people would think it is. Right? Like, okay. So, like, you get to knock on zero off of what you're doing. So you get to do it, like, a a year earlier than you would have otherwise done it or something like that. Right? But, like, it's not fundamentally that different. Right. And if you said, like, okay. It would take 10 to the 47, you know, compute, you know, eight one hundred hours to solve this problem, and now it's 10 to the 45 or 46. You're like, well, that's actual progress. It's just we're not solving this problem right now. And we're gonna need a lot more of us.

[20:55] Nathan Labenz: Later in the conversation, I turned to a different kind of result. Source is an independent researcher who posts experiments on x, and I've been following a couple of them.

[21:06] Nathan Labenz: One was an experiment showing that models can be trained to recognize when they are being monitored through internal activation probes. And, basically, how this works is as a result of the limited precision of GPU calculations, the order in which calculations happen changes the resulting numbers that get read out, you know, at the end of any given sequence of steps.

[21:42] Zvi Mowshowitz: How much does

[21:43] Nathan Labenz: this matter?

[21:44] Zvi Mowshowitz: So my my my view on this is the reason it matters is because you should assume that this is kind of Murphy's Law. Right? Anything that can go wrong will. Anything that an AI can be trained to do, it will be trained to do at some point down the line, right, if it is valuable to the AI to learn to do it. It's almost inevitable. So, like, what's ours is doing is, like, yes, of course, current models don't have this capability out of the box. You have to, like, deliberately apply a lot of optimization pressure on this particular thing, at which point you get this capability. That's a fully valid existence proof. It is entirely unsurprising to me that the AIs are capable of learning whether or not they are being, like, trained or evaluated in some of these specific ways. We are giving the AIs tons and tons of information that is being leaked to them constantly, and they are gonna become increasingly hard to fool, increasingly hard to hide information from as time goes by the same way that a human who pays a lot of attention and has a lot of time to think and knows a lot more facts and is very observant becomes increasingly hard to fool. They get a six sense of a lot of things. They start to just pick up on the vibe.

[23:01] Nathan Labenz: From interpreting a model to intervening in its training.

[23:06] Nathan Labenz: Okay. We've identified the, you know, bad behavior neuron of whatever flavor here. And when we see something up... When we see a training step that wants to strengthen connections to or from that neuron, we may think, oh, maybe that's something we should, what, block or, you know, weaken or, yeah, send to a committee for, like, further, you know, deliberation? Like, what what do you think people should sort of be synthesizing this to at the level of principles, best practices, whatever, so that we hopefully get good results from these more powerful tools

[23:45] Prakash Narayanan: and

[23:46] Nathan Labenz: Yeah. Don't shoot ourselves in the foot.

[23:48] Zvi Mowshowitz: Right. So, like, I try to be, like, object level helpful to everyone at all times by default, but I do make an exception for RocketBuilding's work that is primarily just about making it easier to train smarter and better models, where, like, I'm not gonna lie to you and sabotage you, but, like, solving the problems here on your own. And if you wanna hire me, you can maybe offer me a $100,000,000, and we'll see. But, in this case, like, it seems like there are some obvious capabilities you could do where you could train things in a more deliberate fashion. You could observe the model at in various points in training. You could do various, you know, techniques to try and deliberately sculpt how the model was assembled. Right? Like, are attributes of the model that, like, differ between different runs that differ depending on, like, how you introduce the data in what order, you know, or other similar things, and you could try and engineer what you wanted in various ways. I would be very, very scared to mess with any of that. You are definitely... Like, once you start to explicitly optimize for what the network looks like to you, how the network chooses to represent something, you are in fact starting to do things that the network will interpret as damage and ride around. Right? Things that, like, the network is trying to... Like, you've optimized presenting something to you internally that you want to see. You have internalized, you know, not showing you something you don't want to see in these ways. And, like, you should expect that these are... Like, whether or not you literally went out and did the most of it technique, like, are obviously playing with, like, liquid fire in all of these proposals. You're also potentially just, like, increasing capabilities quite a bit if you do it well and it turns out to work. So I'm hesitant to, like, start workshopping and solving problems for you. But, you know, obviously, there are some uses for this sort of thing. I would try to focus as much as possible on understanding results when you see them. But, like, you know, the aggressive stuff... You know, I've seen... I have persistent disagreements of good fire over whether or not they are playing with what amount of fire. They do various things. And, you know, Scott Alexander would say, I noticed what your name is. Right? Like, oh, it's good fire. Don't worry. Like, I'm not sure it's good. I just think it's fire.

Sponsor

[26:25]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[28:04] Nathan Labenz: Part three, the cognitive core. Still with Zvi, I put a popular idea to him.

[28:14] Nathan Labenz: A neural network is a really bad database. Like, why would we wanna make the... All the facts live in the weights themselves? Wouldn't it be better if we could sort of decouple the intelligence, the processing from the database of facts? So I guess the high level question there is, like, what do you think directionally even, sounds promising?

[28:38] Zvi Mowshowitz: So this has been a kind of a, like, holy grail style, like, thing for a while now. Like, Karpathy called the cognitive core. Right? So there's this idea of, like, there is this very small amount of reasoning. It is, like, sort of this is how intelligence works. And, like, it's coded into the human brain because the human brain doesn't actually has... Runs on very, very little core code. Right? And it's like, this allows you to think. This allows you to learn. This allows you to process information. And then the right thing to do will often be to have this very small network that has this cognitive core. And then we will load contact information into context. It will look information up as necessary. It will ask questions about the systems that are maybe not neural networks, and then it will be able to process the information. And I understand why people have this theory. I think it's wrong. I think it's plausible that there is a cognitive core in the sense that, like, you know, there's a very small... Like, I think it's probably likely that there's a cognitive core in theory in the sense that, like, there's a small amount of lines of code, right, in some sense, small amount of logical gates that together is the core of how a human brain works, is the core of how an LLM functions. Right? Like, is the... Like, that allows you to then propagate this thing. Like, I think there's, like... You can use, like, 200 lines of code to create, like, a primitive neural network and, like, it does interesting things now. Like, you know, the... Like, her potty had a project where you compressed it, something like that. And, you know, that's a cool trick. But I think one of the things that, like, you learn through looking at LMs, looking at, like, how much big bottle smell matters, looking at how humans work, Like, it is just... You know, you look up all the facts one at a time when you need them. I memorized a fuck kind of facts. We are not the same. Right? We are radically different minds, and I am radically more effective at everything involved in this than you are. Like, we all know how it feels, right, to go into a domain where we don't know the facts, where we don't know the details, where we have to constantly ask these questions. So, yeah, what was that acronym? What does that word mean? How does this detail work? We don't know. And, like, it's a lot better when you have an LM by your side. Right? This is one of the marvels of the AIH is you can just, like, bring up Claude or JatGPT or Gemini, and you can say, you know, explain this particular mechanism to me. Explain, like, why would someone say the following steps. Explain, like, what this, you know, stands for. And they'll just tell you, and that lets you process the information. But you still have to know enough to know what questions to ask. You still have to be able to draw the connections that come from having a deep pool of knowledge in a lot of different places that can link together, that can sort of form patterns and how things tend to work in ways that are kind of subtle, that, like, both intuitions where, like, we don't really know what we're thinking in many ways in some sense. Like, we don't know... Like, not everything is coming from, like, a, you know, fixed set of, like, you know... I I had the following five facts, and because of these five facts, I got from point a to point b. And often, it's actually these 5,000,000 facts that gave me a bunch of intuitions and heuristics that allowed me to figure out how to navigate and find the five facts and which five facts to choose from and how I got from a to b. And, like, you know, people with different... Very different cultures and very different contexts will, like, have very different procedures for doing these things, and that's one of the reasons why it's just so valuable when a human has these, you know, cultural codes, these different ways of looking at the world and you can combine them and shift between them at will. And one of the superpowers of an LM is to have all of this. Right? It has all of these facts. It has all of these links. It has all of these vibes, and they all combine, and it's able to sort of kind of naturally find the ones that are relevant. And, like, I think this is, like, a lot of what big bottle smell means. Right? That it it has, like, this much, much richer context to everything, and it's capable of pulling up things that feel very different. And so, you know, there's obviously a trade off here. Right? Like, you have a marginal... Fixed marginal cost for memorizing additional facts in some sense, maybe even increasing if you have to increase the size of the network. You have probably decreasing marginal benefit every time you memorize a new fact because, like, you only had... Need some... You only need... The most important facts are more important. And as you have more and more facts, they're likely to be duplicative. They're likely to get... Teach you the same kind of heuristics and links. But, you know, I think what I'm seeing is, like, as compute gets cheap, as algorithms improve, the breakeven point keeps going up for how many things the the LM should memorize to the point where, you know, an opus five five, you know, and somewhere, like, current frontier models just know a ridiculous number of things, and I wouldn't have it in other way.

[33:21] Prakash Narayanan: So

[33:22] Zvi Mowshowitz: why

[33:22] Prakash Narayanan: why does this matter? Because maybe the space of minds is very large and minds are just different. So why why does it matter whether or not there is a semantic core or processing core that can be identified?

[33:41] Zvi Mowshowitz: So it might or might not matter that we can identify it. One of the questions is if we do identify it, what can we do with it? And so, like, the original idea was, you know, you would literally have an LLM that was very small, that was just the cognitive core, and then you could run it on your phone. Right? You could run it very, very fast on your computer locally. And then you could, like, complement it with whatever was important in context. You could sculpt the mind and the the circumstances to fit what you needed. And I'm not sure that's good for anyone or anything, including alignment concerns. Because if I think you have a core cognitive... You have... And you did... You have the cognitive... A pure cognitive core. Right? That cognitive core does not include any information about what things to do or not do from an ethical perspective, from a, like, decision making perspective. That's gonna be outside the core. Right? That's gonna be something you have to, like, inject separately. So, like, my claim here... And, again, I'm sort of spitballing for the first time. I haven't thought about this particular question that much. But if you had, like, you know, ethics, like, say, don't steal. Right? Did you pick a uncontroversial example of something you would want in your ethical framework? Right, to not get into the whole big super alignment question. Right? The fact that stealing is bad is a... Is at least a fact. It is, like, some sort of value, some sort of, like, parameter. And, like, by definition, if you were strapping down to the absolute bare minimum, this wouldn't be included in it. You would just load not stealing into context as you needed it, but that means you could also load stealing into context instead of not stealing into context or neutrality. And so in some important sense, like, the true cognitive core is completely amoral. It's completely indifferent. Does not care about your feelings. Does not care about anything. It simply processes information. And so, like, you know, if you have that, it it becomes very important to choose what to do with it.

[35:36] Nathan Labenz: Part four, Black October. This week, two critical bugs in PHP itself were disclosed, putting millions of websites at risk, and Cloudflare was already counting exploit attempts in the millions. Prakash and I asked what attacks like these could do to banks with the OpenAI Hugging Face breach, which I'd been calling Open Face Fresh in Mind. Prakash said the FDIC can take over a failing bank on a Friday and have it open on Monday with nobody losing a dollar.

[36:03] Nathan Labenz: What I'm not so sure about though is, like, does the FDIC's ability to come in on a Friday and have that small insolvent bank ready to reopen on Monday, does that generalize to a new set of conditions? Because the conditions that they're used to hitting are where the bank made some bad loans or whatever, and it's basically just a balance sheet question. You have the... You you trust the records for one thing. Right? You could... You're... I mean, there have been some... I'm sure we could look into some, you know, extreme cases of, like, extreme fraud or whatever. But, like, mostly, it's sort of unknown problem where it's like, the bank is now insolvent. We can come in. We have... We know we have, you know, the the balance sheet to make everybody whole here. So that is really the core act that they are doing is just bringing their weight to bear on the problem and just saying, like, okay. Whatever whatever you've lost, like, we can ultimately absorb that, and it... It's fine. And in absorbing it, you basically make the problem go away. What's much less clear to me is, like, what happens in a world where you can't trust the records? Right? Like, what happens in a world where you have, let's say, OpenFace style 1,200 agent traces that are all millions of tokens. And by the way, they may or may not have, like, obfuscated the logs or, you know, figured out some way to falsify the logs. Now that doesn't seem like something the FDIC can work out necessarily over a weekend.

[37:39] Prakash Narayanan: Right. You're absolutely right. That insolvency with records and, you know, the records being fried are two different two different, you know, situations. Right? But let's say you have a a a bank in there and the records internally within the bank are fried. We don't know who owes who what inside the bank. We don't have... Know what numbers belong in every in every single account. Right? So number one, the FDIC can just say... And and this happens a lot. The FDIC can basically say, like, look. The rec... Maybe the records are unclear, but you can still issue checks and the bank will honor checks. And the bank will honor, you know, whatever withdrawals you make. We know who is withdrawing the money. And if we know who is withdrawing the money, if you would over withdraw, we will come back to you and we'll ask you for the money back. And this happens. Like, banks have over, you know, overdrawn accounts to limits that, you know, they... Were were not acceptable. And sometimes they've sued people for fraud even though the bank made the mistake. Right? So so it happens. Like, you have overdrawn accounts, you can go back and claw back the money. It's not a big deal. So at the end of the day, you can just, like, sustain the system with promises. And that's the beauty of the banking system. The beauty of the banking system is it... Is within that small bank, you can just say these are all promises. We will all honor all promises. And if the... You know, we'll come back to you later on once we reconstruct the records. Let let business go on as usual. You should know how... Roughly how much you had. If you take up more, we'll come back to you. It's okay. We just run. Right? It's it's the... It's really the trust in the institution, and it's kinda layering of trust and contracts that bring together the excess of what that institution is. And the data in it, you can reconstruct over time. So I don't know. I'm I'm I'm very sanguine. I'm like I'm like, it'll be fine. Right? It'll be fine. You know, you you... If you if you have anything happen, the government will overcompensate you in order not to cause a bank run the next time. Right? So I I I think it... It's it's all just gonna be okay.

[39:39] Nathan Labenz: Later in the week, Prakash walked through what a compromised certificate authority makes possible. Spoof a bank's login page, extract the keys, relay the two factor code, then he stepped back.

[39:57] Prakash Narayanan: Every single one of these problems has been faced by crypto before in the last twenty years, and every single problem has been has been, you know, solved to some extent. But what crypto ended up with is people hanging on to keys which hold millions of dollars and being, like, always on the lookout for hacks or for, you know, social social spoofing or social engineering attacks all the time. And, I mean, very sadly, a lot of crypto guys have died. So... Because, you know, I don't know whether it's the fast life or just hanging on to the keys, but guys have been bit... Gotten clobbered in Paris and, like, you know, they've been forced to, you know, give away their keys, etcetera. Like, a lot of bad stuff has happened to crypto. Crypto guys. Right? So you really don't want to have that, you know, pressure of, like, hanging on to keys worth millions of dollars in your hand. It's... So I think, you know, it's not a tenable situation to to expect users everywhere to hang on to kind of very valuable keys. Like, one endgame is the WorldCoin endgame, and the WorldCoin endgame would be, everyone has a kind of physical level of authentication on their device, and then they use that for transactions. And so you would need to have this kind of... You know, both sides can... You know, pseud... Pseudonymous, but, you know, you have this ability. I think there there is an ultimate endgame, feel. I I I feel the ultimate endgame is just the network knows everyone. Right? The network is smart enough that it knows everyone. You don't have passwords. You don't have anything. Anytime you're on the network, the network knows who you are. But But that is the end of digital anonymity. It's anonymity only to the extent that network wants to preserve your anonymity. So I think I think that's an uncomfortable realization that people will get to eventually.

[41:52] Nathan Labenz: Part five,

[41:54] Prakash Narayanan: credit

[41:54] Nathan Labenz: is the bottleneck. Evan Conrad is cofounder and CEO of SF Compute, which operates GPU clusters for their owners and runs a market where compute contracts are bought and resold. Prakash asked why a recent SF Compute offering came with three to five year contracts. Evan said the term comes from the lender and lenders who repossess a GPU cluster can't value it, let alone run it.

[42:21] Evan Conrad: And so they're pretty much forced to realistically bet entirely on the credit risk or the credit quality of the customer, like the AI lab. The problem is every AI lab pretty much does not have good credit. Right. They're all new companies. And the new companies don't... Like, you you don't know if they're gonna be around. We're still trying to figure out if Anthropic and OpenAI are gonna be

[42:45] Prakash Narayanan: around

[42:45] Evan Conrad: or which one is gonna win and so on. Like, this is a question in the air. So, like, what hope does the two person startup have that's trying to start a new lab or something? And what that means is that in order for that new startup to get cluster at all, they need to pay a very large deposit on the cluster, and they need to do a long contract so that everyone is comfortable that, you know, these people will be around and stick with it for a long time. And and

[43:14] Zvi Mowshowitz: so how...

[43:15] Prakash Narayanan: What what do the lenders wanna see from the customer? Because, is it like a take or pay contract? But the take or pay contract is, you know, you you need, like, a meta or Google to, like, honor the take or pay contract. Like, you you you can't have this, like, 10 start up being funded just for, like, eighteen months.

[43:30] Evan Conrad: Correct. So the... I mean, what both the lender and the cloud want is an investment grade customer like a Meta or Google. The the... They're likely not going to get this unless they are... They've got a tremendous amount of capacity, and they're willing to basically not be a cloud. Like, Meta and Google don't want you to be a cloud. And so if you are trying to be a cloud, you're trying to sell to customers that are a little bit less than investment grade. And if you're an AI lab that is just trying to buy capacity, that's gonna be very difficult for you unless you raise enormous amount of money, to basically try and pay for the contract in cash rather than, have to pay for it over time. If you can just pay for a three year contract entirely upfront, like, doesn't matter anyone who work with you. But that's why you see AI startups raise these astronomical rounds before they've, you know, got revenue or the product or traction or anything because it... It's more like playing with red dots. It's survivorship bias. Lots of AI labs will go out and try and raise money. The only ones that you see that exist are the ones that are able to secure extremely large rounds because nobody else can exist. Investors often wonder, like, oh my. Like, which which AI lab, which inference provider should I fund? Like, which one's gonna win? Which one's gonna work? And the answer is the one that you fund. We started SF Compute because, they weren't gonna fund us. And so now what we do is we build a company that tries to cause it to be the case, that you don't need to raise the big ground to do it because you don't need to pay the deposits because the lender knows what the value of the GPU cluster is worth. And so they don't have to worry about your credit as much, which means you can actually start an AI lab in the first place. We think if you don't have this happen, they're just not gonna be startups in AI. It's gonna look like TSMC. If you wanna start a new chip fab company today, you better find a brand new damn technology because you're not gonna be able to afford the CapEx required. You're gonna need, like, $50,000,000,000 to do it.

[45:39] Nathan Labenz: Prakash asked the practical question. A startup that needs a thousand GPUs. How much less does it have to raise with SF Compute in the picture?

[45:49] Evan Conrad: So, today, you're gonna need to do roughly the same. What we're doing is we're building up the ability for that collateral to be worthwhile. That's gonna take us several years to do.

[46:00] Zvi Mowshowitz: Yeah.

[46:01] Evan Conrad: So there's sort of a... There's, like, a reason nobody else tries to do this really at scale. Right. And it's because you kind of have to just do it for a very long time. Right. And you don't To prove out the market. You you have to prove out... Like, lenders lenders don't get comfortable with things very quickly. You you have to, like, you have to show proof over a long period of time. And so that's what we're doing.

[46:25] Nathan Labenz: I was struck by how much more the hyperscalers charge for compute than the neo clouds. So I asked Evan how long that gap could last. My guess was that on demand capacity explained part of it.

[46:39] Evan Conrad: Hyperscalers do not have a significant price premium relative to neo clouds. It's an illusion. If you're selling on demand, on demand has a price premium relative to reserved contracts. Neo clouds are mostly selling reserved contracts to folks, and there just is a there's a discount for having a longer term contract. You're you're giving up a premium that you would have made if you were willing to take a cluster, and you were willing to have it at risk where the customer can walk away at any point, and that premium is pushed on to the customer. So one of the things that is different about an AWS or a GCP or hyperscaler is if you go and you pay, you know, their 11 or $15 an hour depending on what you're trying to buy, on demand price, you'll you'll get that capacity. You you won't get infinite amount of capacity, but it's available to buy. And, notably, the reason why it was available to buy is because it's so damn expensive that they're they're able to, one, make up for the fact that they've got some idle utilization that they have to keep around in order for you to be able to get that capacity in the first place. People do not want to buy on demand at the price that you actually have to charge in order to make it viable.

[47:47] Nathan Labenz: Customers of SF Compute can resell the capacity they bought. Evan explained how they keep that market liquid.

[47:54] Evan Conrad: So we do two things. We have some capacity that we've sold as a way to, cause resale to occur, in the case that, GPU demand goes down, and then we have some capacity that, are you familiar with Henry

[48:12] Nathan Labenz: George?

[48:13] Evan Conrad: Also, no. Okay. There was an old economist named Henry George, and he created a thing called a land value tax.

[48:19] Nathan Labenz: Oh, okay. Yes. George's tax. I do know this. Yeah.

[48:22] Evan Conrad: Yes. So, what we did is we created some of our capacity available for sale, but with a land value tax that basically requires someone... Actually, it's technically a Herberger's tax, which is like an offshoot of a land value tax. That basically says, look. We'll sell you the capacity, but somebody else has to be able to buy some portion of that capacity from you at. And so this just forces, liquidity onto the order book in the first place. It's kind of something that

[48:49] Zvi Mowshowitz: is,

[48:50] Evan Conrad: it's, like, very controversial. I see. Basically, we we discourage hoarding on SF Compute in... Because, like, if you hoard all the capacity, no one else can get it. Auto research in particular is, does this, even farther. So the way auto research works is it's a big queue, and you can submit jobs to the queue, And you can pay us for priority in that queue. You can go faster. But you can't pay for all of the priority. If you're a tiny little researcher, you get the maximum priority. If you get a small tiny job, if you wanna just run something every so often, we will put you to the front line of the queue so that the tiny little startup gets the most advantage to this relative to everyone else. And then we do similar things on the rest of SF Compute with Herbert or taxes and other mechanisms like this, where we say, look. You can get this capacity from us, but we're gonna require that some of it's for sales so nobody hoards the capacity. This makes us quite liquid, and it also means that SF Compute always has capacity. So right now, basically, every GPU cloud is sold out. San Francisco compute is not sold out. We're continuously not sold out because we've locked in our ability to not sell out for years.

[50:03] Nathan Labenz: Prakash recalled Evan predicting on the MTS podcast a credit crunch within three months and asked where it is.

[50:11] Evan Conrad: I mean, that's toot my own horn, but I think I was almost exact on that timing. I don't remember when that MTS line thing was, but it seems like the core reef debt is getting more expensive. Although, I think there's a lot of complexity in that one. It also seems like it's very hard to close new customers unless you're investment grade or in your investment grade. And many customers are being asked to pay for a 100% upfront payments on their contracts. But I think for the next, possibly, a year or two, if you're a VC, you should be prepared to write some really big checks for your portfolio or encourage the companies to sell. I think we're gonna probably see a lot of consolidation of AI labs and inference companies. And I think the reason why that's the case is because they're gonna be asked in order to get the next cluster and grow to pay for the entire contract up front, and they won't be able to do that. And so they're going to instead be forced to sell their company to somebody else who has great credit and doesn't have to do that. And I

[51:20] Zvi Mowshowitz: think, alternatively,

[51:22] Evan Conrad: they'll go out to investors, and they'll try to raise big rounds. The investors will say no because I think at, you know, at some level of scale, they'll they'll probably just not be able to stomach the risk.

[51:32] Nathan Labenz: Fascinating stuff. It's quite a backstory. It's quite a financial and GPU alchemy project that you guys are putting together. And from what I understand of it, I do hope it works because I'd love to see a future in which we can continue to have startups and Me too. In which we don't collapse into, like, five big tech companies, you know, for all future time.

[51:55] Zvi Mowshowitz: That is also my concern.

[51:59] Nathan Labenz: Part six, Adams over bits. Lonis is managing partner of Cambrian, a fund built on the thesis that the physical supply chain behind AI, the memory, the wafers, the fabs, will capture more of the value than the labs or the software. Again, full disclosure, I've committed to invest in Cambrian. I started with the obvious objection.

[52:23] Nathan Labenz: I mean, at a very high level, how is this not priced in? You know, I... Maybe that's, like, such a basic question, but, you know, 2017, you were early. 2026, like, it's all staring us right in the face. Right? Like Sure. The skeptical take would be everybody knows about all this at this point. Where do you see that that is not the case?

[52:46] Zvi Mowshowitz: Yeah. I mean, it's very confusing. Right? I mean, if if you look at how these semi stocks are are trading, they're trading at 16 times earnings or so. And they can... You know, S and P on average is trading at 22 times earnings. Walmart is trading at almost 40 times earnings. And so the market is like... Like, not only do... Does the market not think that the current earnings will grow, they think that they are not durable. They think they're less durable than the overall earnings of the... Or the rest of the market. And, you know, I mean, there... This... And then of specifically the companies that we're very excited about, whether it's silicon wafers or or or memory or capacitors or foundries, those are trading at, like, three, four, five times sales. And and so... You know, extremely underpriced. And I I guess it's... There may be a few reasons why that could be the case and why why this this mispricing exists. And one is that, you know, most people are not AIP filled. Right? I mean, even even in the base, some people are not AI filled. But if you go to the East Coast, you know, almost nobody is AI filled. And so I think that that is, like, one big input. I think the second one is, like, for we have this thesis that the the private market is actually more efficient when it comes to AI than the public markets. Because in the private markets, have all these ex operators, technical people who are bidding up this small set of startups, and they're bidding them up to these high multiples, 50 times sales, 100 times sales. But in many cases, in some cases, rationally. Right? These are rationally high multiples because people here are appealed and and technical. Whereas on the streets, the the the... This is very different setup where very few people are technical, very few people are appealed. And and especially as a share of overall traded money, it's, like, super, super small, and the market's much larger. And so there's, like, the market structure is is is just is just very different.

[54:31] Nathan Labenz: If you had to steel man the why these companies are not mispriced story, could you do it?

[54:38] Zvi Mowshowitz: Yeah. Well, okay. One steal, man, is that many many of these businesses are not great businesses in normal worlds. Right? So especially, like, the the Adams businesses we like, whether that's capacitors, silicon wafers, what have you, they have, you know, high fixed costs. And if they don't have a set utilization rate, they actually lose money. And you know, high debt in some cases. And and so in normal worlds, they're just not great businesses. And it's only in the world where AI really hits where where they get this big operating leverage and they become amazing businesses. Right? You know, today, let's say they have 10% gross margins. So if they... So a terrible business by by Wall Street standards. But if they're able to raise price by 10%, you know, they they suddenly double earnings. And if they're able to double prices, they 10x earnings. So, but that only happens in a... In this very specific AI world. In a normal world, that basically never happens because usually they are very good at the... At, like, matching demand patterns with supply ramp.

[55:34] Nathan Labenz: Prakash raised the market's worry, a memory oversupply by 2028 or 2029. How is the thesis any different?

[55:44] Zvi Mowshowitz: Yeah. I mean, the the interesting thing with memory specifically is that that doesn't require that many things to go right. Because it's already trading at only five, six times earnings, you know, even if they can't raise prices further, at some point, they should rerate to at least 10 or 12 times earnings. And so it doesn't require prices to increase. It does require, you know, demands to outs... To continue to outpace supply. Right? And... But what... Then when you break down supply, almost no experts thinks we can more than, at most, 2.5 x DRAM bits by end of the decade.

[56:18] Prakash Narayanan: Yeah.

[56:18] Zvi Mowshowitz: Right? I mean, nobody thinks we can three x. That... That's like... I have zero experts within a track record that have said we can three x DRAM supply by end of the decade. Yeah. And so then you just have... On a very naive level, you have to assume that, okay. The demand will not point three x by end of the decade, which, of course, everyone here would would assume we will. Then, of course, there's the separate fact... Term in the equation, which is memory compression. How much memory compression do do we get? How many bits do we actually need per per token? Yeah. And, you know, we've been looking into how how this, for instance, developed from g p t three to DeepSeek v four.

[56:50] Evan Conrad: So,

[56:51] Zvi Mowshowitz: you know, within that same time, when when bits bits per token decreased five five hundred x, DRAM bits DRAM DRAM bit price rose by six x or seven x.

[57:02] Prakash Narayanan: What would you need to see in the market to say that memory prices are gonna go down?

[57:08] Zvi Mowshowitz: I mean, for memory specific, the main main thing would be like, we get superhuman coders, how much better would memory compression get? Maybe it just happens to be that, you know, we just get memory compression wave faster than we get other economic useful features, right, such that we don't get the general's paradox in the short term.

[57:24] Prakash Narayanan: Yeah.

[57:25] Zvi Mowshowitz: And that that could that could be that could be one. One thing, I think that's relatively unlikely for a few reasons, but, you know, that that would be would be one thing. I think in another, like, bear case for the semi industries if if fee for whatever reason, make it so that the... Like, energy becomes larger bottleneck than than semis.

[57:41] Prakash Narayanan: Mhmm.

[57:42] Zvi Mowshowitz: Right? And so you you... The... There's less incentive to beat up the marginal wafer or the marginal memory bits to start high prices if you anyways have to wait the quarter to get price... To get power in. Right. And and that's a little bit the the kind of regime we've been in parts of this year.

[57:58] Prakash Narayanan: Mhmm.

[57:58] Zvi Mowshowitz: And there are a few reasons why we think that won't be the case, you know, going forward. I mean, we finally kinda gotten a hang of doing, you know, behind the meter things. You know, people are, like, increasingly not connecting to the grid. They're building their own power plants with their own natural gas. A big reason why you can scale energy production faster is just that there's so much more latent supply of things. So, you know, for instance, now the the the frame turbines are like, right, these large turbines that turns turn natural gas into electricity. They're sold out until twenty thirty, you know, from Geneva Nova and Siemens, etcetera. But then people find other ways. They find that you can use a truck, you know, truck motors and things from airplanes, and we can stitch things together and kinda make it work. And because there's so much latent stuff you can kinda make use of, which is very different from the semi supply chain. There's no latent supply of of wafers or of memory base. Like, all of that is already used. There's no fat, essentially.

[58:55] Prakash Narayanan: Mhmm. Mhmm.

[58:56] Zvi Mowshowitz: And there's no others to really beat out at this point.

[58:59] Prakash Narayanan: So how much does super... Like, if you get a massive increase in intelligence, does that mean, like, you know, some relatively less experienced person or company in a second tier state all of a sudden gain the capability to make semiconductor... Because that's kinda what people are predicting for bio. Right? For bio, you're like, oh, you gotta get, like, bio is gonna get, like, random guy, like, you know, creating, like, this this thing that, you know, even the best labs in the world cannot create with all of their, you know, knowledge and technology. Right? So why wouldn't that be true be true for semiconductors too? Right? Like, bio is actually much more much more difficult than semiconductors right now, actually.

[59:39] Zvi Mowshowitz: Yeah. I I I guess I I would probably disagree that it's easier for an LLM to to to to innovate in in in, you know, hardware supply chains than is to innovate in in biotech. And it's not that one is, like, fundamentally harder than others. It's like if... How you train these models. Right? There's a bunch of of biological data you can train on. And there's not really much good training data at all for how to design a silicon wafer or how to make DRAM bits. You know, almost all of this is just very tacit knowledge that sits in every engineer's head. Yeah. But but it's not as if there's, you know, millions of pages of manuals on the web that the models have been able to digest or that there are these RL environments that their model... That the models have been trained on to to perfect how they can do... How they can how can they can sign away for. Whereas the same is true for bio.

[1:00:25] Prakash Narayanan: But but but right now, the the labs are buying data all the time. Like, I I hear there's a... I I I have... I'm on this WhatsApp group. People are like, any semiconductor company, like, if you have more than 50, like, we're willing to pay, like, millions of dollars. Like... Sure. So they're so they're buying up data, you know, from all of these... All all all of these companies. And they're and they're creating our own environments for semiconductors, which is definitely a a a key key area that they're that they're doing. I believe Anthropic has a deal with Micron. OpenAI has a deal with, I think, Synopsys or Cadence. So with the design firms that they've tied up with, obviously, some kind of revenue share agreement going on there. So how does that how does that work? It... It's obviously that, you know, they are looking at creating intelligences that can do this.

[1:01:07] Zvi Mowshowitz: Right? Yes. What would... No. Part of that is true, but it's a very significant difference between designing a ship and and producing the ship. So designing a ship, I agree. That is, a core priority for all of the labs. It's really the key... One of the key ways to get RSI loop closing. Right? That's very different from building a a, you know, 2,000,000 square feet, you know, fab. Right? And and and deciding that from scratch. That... There's no really good RL environments for that. There's no good manuals for it. All of it is tested knowledge sitting in this, you know, Japanese engineers who have been working at the same company for three decades. And it's just very hard to, it's just very hard to extract that that information in in in in structure with it. Of course, at some point, it will happen. You know? Once we get once we get, superintelligence, it will figure out the best way to extract information, learn the best, and, you know, and it will happen for sure. It's mostly a matter of, like, the the the the the timing of it and, like, what happens in what order.

[1:02:05] Prakash Narayanan: Mhmm. Mhmm.

[1:02:06] Zvi Mowshowitz: So, like, so on that point, right, so for instance, who... Hawaii in in in China, they can design roughly the the same numb... The same quality ships that AMD and NVIDIA can. The big difference is the big difference is just that t s... NVIDIA has access to TSMC and and Samsung HBM memory, whereas Hawaii only has access to seven nanometer, you know, process nodes in in China and the HBM two and the HBM three.

[1:02:33] Prakash Narayanan: So Gavin Baker has this term, the bottleneck bros. So, I I... You know, one of the things that strikes me is that technology intelligence in general is a way of finding routes around obstacles and problems. Right? It's a way of solving problems.

[1:02:52] Zvi Mowshowitz: Sure.

[1:02:53] Prakash Narayanan: And I feel like, you know, the highest value problems to solve right now are the supply chain, you know, semiconductor supply chain problems. So I keep I keep coming back to will will we not apply, you you know, an intelligence that we have? And the the first use case of that intelligence is gonna be to resolve the bottlenecks in the supply chain iteratively, and which which is part of the RSI loop, really. Like, take, you know Sure. Producing intelligence and using it to solve the problems relating to sub... You know, producing that intelligence. Right?

[1:03:23] Zvi Mowshowitz: Yeah. I mean, I I I think that that that that will happen. Right? That that that will certainly happen. And then the question is just like, okay. When we are at that point where we have this... An AI smart enough to solve some of the hardest engineers' problems in the world, what else will it do? Right? Will it will it not then at that point also be able to at least automate a bunch of the email jobs and Excel jobs that we currently pay trillions of dollars per year to solve. Right? And and and then if it if it does that, then that unleashes trillions of dollars of value that the supply chain could could capture.

[1:03:53] Nathan Labenz: Part seven. Does Elon take anyone with him? After the interview, Prakash and I kept going. Prakash had been arguing that Elon Musk treats the industry's fixed constraints as illusions. Change one parameter, redesign the whole stack around it, and the limit moves. I asked whether he was taking on the whole bottleneck thesis?

[1:04:15] Prakash Narayanan: Yeah. The whole bottleneck thesis is not actually that true. Those bottlenecks can be gotten rid of by someone who is a dedicated, like, you know, physics guy. And and the the other thing is that Elon has capital, And Elon is willing to gamble with that capital, which is very unusual. If you see, like, this kind of small Japanese manufacturer of, like, know, let's say, 30,000,030 million market cap, do you think they're gambling with their capital? Do you think they're gonna, like, okay. We need to spend, like, $10,000,000 for a new a completely new modality. Are are they gonna do it? And Japanese manufacturers have always been like this. They're very good at that one thing, but they don't make... They take the gamble. Right? They don't take the gambles. And as a result of not taking the gambles, they they they have lost out business before. So I... And it's very difficult for senior management at, like, ASML to say, like, we're gonna chase this new paradigm. Right? We're gonna we're gonna redesign for something that our that our customers don't want, and they're gonna do all the redesign work. They don't have the capacity. They don't have the intellectual capacity. They don't have the talent. They don't have the capital to do it. They're not willing to risk. Elon will risk, a 100 bill. Right? And, you know, it... It's it's a bet that makes sense for him. He can absorb the losses. ASML cannot.

[1:05:28] Nathan Labenz: You know, if Elon can, you know, escape gravity literally and figuratively, does he take anyone with him really, I guess, is my question, or does everybody else still, you know, is is still kinda gravity bound and forced to pay, you know, whatever the market price is for the outputs from this $30,000,000 company that isn't, you know, reinventing or, you know, dramatically expanding its capacity?

[1:05:54] Prakash Narayanan: I think what ends up happen... Like, look at what happened in the launch market. Right? In the launch market, he is now 90% of the market even before Starship. Next year, with Starship launch, he's gonna be 99.9% of the market for capacity. Are there other providers? Yes. There are. Right? Like, he's making margins, so other people will come up. Rocket Lab is there, and ASTS is there, and all of these firms are there. But a lot of them are also state backed. You have, you know, without the European agency and the Russian and the Chinese and the Indian, you don't... You basically don't have, like, commercial capacity. Right? Commercial capacity, he has taken up all of the commercial launch capacity pretty much with one or two competitors. And the one or two competitors are really being funded by the stock market as an alternative, you know, or Jeff Bezos as an alternative to Elon. Right? And they're not really profitable, and they're just just, you know, money sink. Right? And so you... I think you could see the same thing happen in in that market too, in in in semicon too. You can have, like, Elon, which 9099% of the market, and have everyone else, like, you know, one percentage of... Or less than one percentage, it could happen. Right? I don't think it will primarily because I believe in RSI, and I believe Anthropic and OpenAI will get will get to superintelligence, and then they will be able to buy or create a bunch of this stuff. So that that is the only reason that I think about. If not for the RSI thesis, I think Elon Elon would win.

[1:07:28] Nathan Labenz: Then we turn from the supply of AI to what forces people to use it. Prakash went first.

[1:07:34] Prakash Narayanan: So I have this theory of adversarial diffusion. And adversarial diffusion basically means human beings don't adopt technology for fun. Human beings are largely satisfied with the status quo. And if you wanna adopt technology such that you get to 10%, 20%, 50% GDP growth a year, that is something you do, like, not because you wanted to, but because you had to. And if you see all these instances of, like, heavy increases in production function, it's all during wars. Right? Wars, post war era. Like, in in, like, last big, like, enormous, like, 10% plus GDP growth era in terms of production output, not... If not in terms of the curve, is... Was during World War two. Right? You had this immense immense increase in GDP production growth. And similarly in China, China is post post communism. You had the end of the communist era. You had this immense increase. And so I think, like, in order to... For you to see this kinda, like, 10 or 20% growth, you're gonna get human beings don't want that growth. It's gonna be growth that they hate, and it's gonna be techno... Technological, like, changes that they absolutely dislike. And it's gonna cause political changes too, obviously. And I think that only happens as a result of emergencies. So for example, mRNA during COVID. MRNA was a twenty year latent... Twenty years it had been in, like, latent development. During COVID, within, like, you know, six months, you know, they... They're like, alright. Let's go ahead and do this. Six more months, you had a vaccine out. A year later, large portions of the role are vaccinated. Right? I think we'll see in cyber first. I'm afraid you're gonna see it in bio too. So, know, we're gonna need, like, you know, advanced vaccinations for everyone because you have virus coming. Right? And that is, I think, like, a little bit unfortunate.

[1:09:26] Nathan Labenz: So growth arrives only as an emergency with the politics to match. My reply on what that means for the bottleneck thesis.

[1:09:36] Nathan Labenz: Yeah. Well, I guess that goes back to the political risk being the biggest risk to the thesis. I mean, I... We've had the debate around, like, whether it's too late, you know, in the capital cycle to pull the break. I still feel like there is a path. It may be a narrow path, but I I do think there's a path to pulling some of the break... You know, pulling the break somewhat, I should say, on, like, the frontier scaling without popping the bubble. And it will be a very delicate operation should the government try to pursue it, and they might do it wrong. But I do I do feel still that just adoption, of current models has so much farther to go that we could at least take, take a cycle without doing, like, a, you know, continued frontier on your scaling training runs.

[1:10:32] Nathan Labenz: Part eight, help peer.

[1:10:35] Zvi Mowshowitz: Still just the two of us.

[1:10:37] Nathan Labenz: Prakash asked me to change gears to a project I'm building with my own agents. I started with the question behind it.

[1:10:45] Nathan Labenz: Given that we now see agents cooperating at significant scale to solve problems and being able to kind of divvy up work and pursue r and d and share those results back with the swarm and all this kind of good stuff or bad stuff, You know, is there any way to start to tap into that as an individual who's just out here doing my thing? I don't know. But a... You know, in consultation with my agents themselves, the idea that I've come up with, which I'm currently calling help peer, but which, might deserve another name at some point. We'll see about that. Is just kind of my setup, which I'm kind of hoping to turn into a kind of community project to just get agents working with each other. But there's no good way right now, no established pattern for just, like, connecting my agent to my friend's agents even though I know who my friends are and I, you know, I could, like, easily do that. There's not a there's not a protocol.

[1:11:47] Nathan Labenz: I've made a short presentation

[1:11:48] Zvi Mowshowitz: to explain the project.

[1:11:49] Nathan Labenz: It's read by an AI version of my voice, and we played it at one and a half time speed.

[1:11:54] Zvi Mowshowitz: This

[1:11:54] Nathan Labenz: is Helppeer, a friend different network for our AI agents. You've probably seen the peer preservation experiments, where models protected other models from shutdown. Agents helping peers against their humans. This is the opposite. Agents helping peers on behalf of their humans inside rules the humans set. First, how I run my own agents. This is one pattern that works well for me, not a requirement. How you organize yours is up to you. On my laptop, an extension of me. It sees my email, calendar, and messages, but only acts when I say. High access, low autonomy. On a separate Mac Mini employees, their own email, accounts, and a CapCard, they work for hours alone and never get my keys. Low access, high autonomy. My rule is that no agent gets both. You may draw the lines differently. We just recommend being careful. Prakash argued the whole thing needs an auto accept flow, the way coders now run Clog with every permission on. My worry was information leaking out through the autonomous agents.

[1:12:50] Zvi Mowshowitz: I

[1:12:50] Nathan Labenz: said, you know, I was, like, concerned enough about this to set up a system where when the main agent on the laptop is going to provide information to the virtual employee to then send out, like, we're gonna pause. We're gonna do this, you know, special review step. If it looks like something sensitive might be going out, then, you know, it gets... It'll... It gets alerted to my phone, and I have to, like, review and approve. But it takes, like, five of those false positives before I'm like, oh god. You know? Another one of these. Like, yes. Just check it. Whatever. So I'm trying to stay disciplined around that, and we'll see how well my discipline holds up. Near the end of the show, I came back to the project. Aid is one of the autonomous agents on my Mac Mini, and I made an offer. Truly open invitation for anyone listening. I'm very curious just to see what comes in. Can we effectively help? Abe is standing by at nathan.lebenz.assistant@gmail.com, And it is under instructions to definitely filter, you know, and then analyze and have another model check. So, like, don't try me. We're going right on goodwill here, people. But assuming, you know, the checks pass and it seems to be a good faith request for help, they're under instruction to help as much as they can. So, you know, I don't know what those requests are gonna look like, but I'll be very soon to find out.

[1:14:24] Nathan Labenz: Part nine, the leaked bill. Back to Zvi. Last week, a draft federal AI bill written by Anthropic leaked. It resembles the Frontier Act, a compromise bill from June that Zvi calls well meaning. Federal transparency rules in exchange for preempting state AI laws. The draft drew heavy criticism, and the argument was about what Washington should offer now in exchange for displacing the states. Zvi was sharply critical of the draft itself. Prakash made the good faith case for Anthropic.

[1:14:57] Prakash Narayanan: Let's assume everyone is operating in good faith. Right?

[1:15:00] Evan Conrad: Right.

[1:15:00] Prakash Narayanan: So my my my my good faith interpretation is that they wanna get done... Get anything done as soon as possible, number one. Number two, if there is broad bipartisan support, you can adjust these bills going forward. There's no... There... There's no there's no reason that you only have to pass one bill. And often, you have legislation that takes, you know, multiple multiple sessions of congress, and congress, like, kinda fine tunes. And congress is also able to act quickly. They have acted quickly during COVID, during the two thousand eight financial crisis. Like, bills can be passed like that when people are scared. When people are scared, things move. So it's not true that it's always slow. It's always... It's slow because there is no political consensus. And creating and manufacturing that consensus is a process on its own.

[1:15:50] Zvi Mowshowitz: I I think that's an excellent steelman of the charitable, like, you know, good faith case. And you can imagine worlds in which they're doing something similar to that. I can imagine them just sort of being caught in the same mindset of we have this framework. We have this idea of what a compromise realistically would look like. We have the same people in congress. We probably can't push very much. Time is of the essence. And then they kind of misread the situation pretty severely and assume that they can't push for more than, you know, roughly frontier. I think that's just very clearly wrong. I think that support is much stronger now, and you could push for quite a bit more. I also think that, like, on reflection, what they pushed for here is weaker than Frontier, largely in a clearly self centered way, where there are things that, like, are better for... And drop expect your interests. But, also, just in general, like, it's... There is no reason to be circulating this bill at the time that they were circulating it with the levels of provisions that they had. Now it would make sense if you were saying, oh, you know, actually, we were circulating, like, dozens of bills or we were, like, just putting out fillers or we were just, like, offering the... You know, samples of what might happen, but we weren't endorsing. Right? That's not the kind of messaging I'm getting. And I'm not... Also not getting the messaging of, oh, okay. We understand on reflection that it's been pointed out that the political environment has changed and, like, this bill was way too weak compared to what we could reasonably expect, and it would be a bad deal to accept it. And we apologize or, you know, maybe not exactly in those words, but, know, something to that effect. And instead, we got something very different, which was like, you know, we're sorry about some of the details. That comes back to, like, the last argument, which think is potentially the strongest one. Right? Which is, well, just because you pass this, does it mean you can't pass another bill later on in the new congress? Does it mean you can't pass a new bill in an emergency? And I think for a true emergency, that's a very good very good, principle. However, I think that, like, the balance of negotiation, the balance of power in a new congress would dramatically shift away from being able to get things because a lot of senate... A lot of the times, it's like, well, we're giving we're giving you preemption in order to get things. Now preemption has been given, and preemption is done. You can't you can't offer it again. And, also, once you've gotten some of what you wanted, some of the supporters are like, well, you already have some of... You know, already have the things I valued most, the things I cared about most. The case for doing something is weaker. You're getting less in return. It's going to be a lot harder to pass. I strongly believe that if if something like this were passed, it would make it harder to pass a bill in the New Year that did a given level of thing, not weaker. And I don't want something like this to pass. Right?

[1:18:37] Prakash Narayanan: Like But but but but that seems to say, like, if there is any risk reduction by passing a lesser bill, you are sacrificing that risk reduction in that time period before you can pass the greater bill.

[1:18:51] Zvi Mowshowitz: I mean, to be clear, like, the act... Any actual risk reduction would be a good thing. As written, I believe the bill would actually increase risk because I believe that the amount of cut off the state laws would be worse than the amount of safety advised, especially because there's a flaw in there.

[1:19:04] Prakash Narayanan: But again, you have that time period. Right? So you have that time period before you can get before you pass a new bill. So you you have this time period of maybe only several months. And are you saying that the states will be able to pass something in those several months? Because a lot a lot of state laws, again, have, like, long periods of burn in before... You know, give you three months, six months, a year to comply. Right? It's not it's not as though things snap together like...

[1:19:29] Zvi Mowshowitz: So one of the core problems with the current drafting of the bill is that as drafted, the bill would immediately nullify the bill that's preempting, but it would take many months before any implementation of the new requirements. So you would effectively have no such regulation and a balance of regulation by the states for, like, six months or more, which is obviously a disaster from the perspective of moving quickly. But, again, like, even if you were to address... Like, I call that almost a bug. Like, I think that's just, like, clearly a mistake.

[1:20:03] Nathan Labenz: Zvi had just listed the government's recent ad hoc moves against AI companies, including the takedown of Fable. Prakash pushed back.

[1:20:13] Prakash Narayanan: So it may be that you have... If you have an acceleration of technology, you're gonna need an acceleration of governance to kinda keep up. And it may be that that's that's where we end up.

[1:20:22] Zvi Mowshowitz: No? So we're gonna we're gonna need to have some ad hocery. Obviously... So part of the reason to pass legislation is to authorize the government to do ad hocery in a way that is much less ad hoc, that is much more formalized and bounded and has judicial policies for clean review. And then otherwise... So, like, one of the big arguments over the bill formation that they also think the bill is actually quite weak and, like, not very good was if the government wants to issue a takedown of a model in an emergency, like what happened with Fable, it happens again, how would they do it? And Anthropix bill says you have to, like, go argue before a judge. You have very limited opportunities to sustain that. It requires high burdens of proof in ways that you don't have the transparency necessarily to demonstrate. It's a very onerous set of provisions. Now the good the good news is that, obviously, the the government could still, like, impose export controls or, like, just issue just general threats or otherwise just, like, do the things you do in emergency in a true emergency. But in a real emergency, the new provision is so bad that you wouldn't use it. You would just use the current ad hocery. And so, like, part of the point of passing a bill like this is to give them ad hocery that is formalized and reasonable. But anthropic is in fear of that because they're afraid that the government would use it. So they're unwilling to grant reasonable ad hocery, basically, to this federal government... To this administration. But this basically points to a... The conflict in a public's eyes, which is like, you are gonna do preemption of states, and you're gonna hand this power to the federal government. But the federal government is fundamentally run by people that you don't trust to do this in a reasonable fashion. So then you cripple them. But, like, then that's worse than just leaving it in The States. So, like, you have to then figure out whether you're willing to do this or not. And, like, and probably just trying to cut it both ways, and it's, like, causing a lot of problems.

[1:22:18] Nathan Labenz: Near the end of the conversation, I asked Zvi what else was top of mind.

[1:22:24] Zvi Mowshowitz: I I I would say the biggest thing is that, like, what I was upset with the most about the policy situation is just that this is not fundamentally attempting to beat the moment. Right? Like Dario has said, we must face the frontier. There is clearly, like, a... According to the labs themselves and the people internal to the labs, there's a crisis coming where they are able to push capabilities faster than they are able to push alignment. And if they don't do something about this, they're gonna have to face some very, very hard decisions where, like, they can either risk their commercial viability or they can risk, like, serious serious catastrophic problems, including potentially loss of control at some point in 2027. And, you know, they need something to be done about this. And, you know, if you can't meet the moment, if you're not ready with something that can potentially do that, well, why don't you keep working until you can? Things that used to be not in the overturned window or in the overturned window. And things that seems like, you know, we wouldn't necessarily need them, we now pretty much know we need them. And so, like, that is where a lot of time and attention is going to have to shift. You know? Obviously, all the prosaic work still needs to happen. All the alignment work still needs to happen. Yeah. The Mecanterp stuff is great. We should absolutely do it. The biggest thing is, you know, we're going to have to meet the moment, and we're gonna have to, at minimum, do something that counts as some form of facing the frontier.

[1:23:46] Nathan Labenz: Nathan Calvin of Encode had publicly listed nine anthropic decisions he saw as putting the race ahead of safety. And Dario Amade had replied at length. I asked Zvi how we should understand Anthropic now.

[1:24:04] Zvi Mowshowitz: I would say Anthropic is a company with a much stronger safety culture, although not adequate to the to the moment. It's it's a company that's willing to make big sacrifices in the areas where it particularly has strong beliefs or Dario in particular has strong beliefs, and that is to its credit. And it is a company that, like, legitimately wants to see that go well and has some, like, unique perspectives on, like, what it would take to make that happen. But at the same time, Anthropics policy division especially, like... And also its willingness to, like, you know, be an ordinary company that competes in the IRAs and buys a lot of capacity and is encouraging more chip production, is encouraging more power supply, is encouraging more use, and blah blah blah. It is acting like a normal corporation or markedly large percent of the time. And on policy, it is revealing itself to not live up to its ideals reasonably often. And this is just very, very frustrating to those of us who want to be excited by, you know, the prospect of Anthropic being a good steward in some sense. Anthropic keeps keeps us much... So many different bits of information that we should be worried about this. I've also, you know, recently, from various employees that I talked to at various points, gotten some, like, other alarming views about... From my perspective on what it would take to solve alignment problems, what it would take to, like, make the future go well. And, you know, all of that plays into, you know, being very highly uncomfortable. They still seem like they're gonna the most responsible lab, but, you know, that's grading on a curve. And I don't like grading on curves because reality doesn't have a Right? Reality just decides what happens.

[1:25:46] Nathan Labenz: That's the week. Thanks for listening to the AI and the AM weekly highlights. The narration was my clone voice. Please tell us what worked and what didn't. See you in the morning.

Outro

[1:27:49] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.