Can San Francisco Come Back?

Descripción

I live 79.8 miles from downtown San Francisco, which is to say: not close. I do not make the trip often, though I have managed it twice in the past month. The first time was May 4, for a Mandalorian preview event, because I am exactly the kind of nerd who will happily drive 160 miles for something like that. Then I went back again this past Sunday for Star Wars Day with the Giants. Yes, there is a pattern here. And yes, they won, so you’re welcome. Now my son is interviewing for jobs in San Francisco. That is where many of the robotics and AI jobs are. I’m thrilled for him. I’m also worried. Because the San Francisco he is entering is not the one I knew. When I moved to California in 1997, I lived in Sunnyvale and drove into the city whenever friends or family came to visit. I worried about turning onto one-way streets (OK, that still worries me) and creeping up absurdly steep hills with a stick shift. (Remember those days?) But, what I did not worry about was a smash-and-grab while sitting at a red light. That simply was not how I thought about San Francisco. I did not have to hide things in my “frunk.” My son, at almost 25, is coming of age in a different city and a different economy. San Francisco still offers high salaries, but it also demands some of the highest rents in the country. What changed For a few years after the pandemic, San Francisco became the national shorthand for urban decline: empty offices, struggling retail, visible homelessness, and a downtown that seemed to lose its center of gravity when remote work emptied out office towers. The city’s downtown exodus was real, and it played out alongside older structural problems, especially housing scarcity and the long-running inability to build enough homes for the people who wanted to live there. And the homelessness problem is not a media invention. Every August, I take part in the Pistahan parade, which begins near San Francisco’s Civic Center. Arriving early in the morning for the event, I have seen some things along Hayes, Van Ness, and Market that I would rather not have seen. San Francisco was hit hard during the pandemic, and the population losses were real. More recent data suggests the city may be stabilizing, but not all the way back. The cleanest way to put it is this: San Francisco looks less like a city in collapse than a city in partial repair. AI is rebuilding the economic story AI then gave San Francisco a new recovery narrative. Over the past five years, AI companies have leased more than 5 million square feet of office space in the city. According to the commercial real estate firm CBRE, AI companies could occupy as much as 16 million square feet by 2030. The larger point is not just that AI is spending money. It is that these companies are helping refill office towers and revive confidence that San Francisco still matters as a place where people need to gather in person. Money is only part of the story. The deeper advantage San Francisco still holds is density: researchers, founders, operators, and investors packed close enough to one another that ideas, deals, and careers can move faster. That dynamic helps explain the city’s strange duality in 2026. It can look frayed on the ground and still seem like the most important place to be for young people who want to work in AI or robotics. Why workers still hesitate But economic recovery and lived experience are not the same thing. Reporting and worker anecdotes suggest a city where some people feel energized professionally and strained personally, especially by housing costs, visible street disorder, and uneven perceptions of safety. Even where crime data has improved from earlier highs, the emotional experience of walking through parts of downtown can still feel unstable to workers who are there every day. Housing remains the deeper structural problem. San Francisco’s affordability crisis predates the AI boom and outlasts it, which means even well-paid workers can find themselves priced out of what most Americans would consider a normal urban life. That tension is especially sharp for younger workers trying to enter the industry: the city offers career acceleration, but often at the cost of comfort, stability, or the feeling that you can actually build a life there. Robotics and AI need the city differently This matters a little differently for AI companies than for robotics companies. AI firms can justify premium downtown office space because their core asset is concentrated human capital, and San Francisco still delivers that density better than almost anywhere else. Robotics companies benefit from the same talent pool, but often need more than proximity: lab space, testing space, industrial access, and a physical environment that supports building in the real world, not just talking about it. That distinction is worth watching. If San Francisco becomes an even more dominant headquarters city for AI while robotics spreads more across the wider Bay Area, then the geography of “future tech” may start to split in more visible ways. The software layer can thrive in towers. The physical layer may need a broader map. What is the San Francisco of 2026? For my generation, or at least for me, the city represented exploration. It was where you took visitors, where you wandered, where you tested yourself against steep hills, parallel parking, and reading paper maps that took you down one-way streets. (Or, at least that is my excuse and I am sticking to it.) It felt messy in the way cities do, but not menacing. For his generation, San Francisco may still represent ambition. It may still be the place where the future of AI and robotics is being built. But it also comes with a different set of calculations: where to live, what to avoid, whether to commute in, and whether the opportunity is worth the friction. That may be the clearest way to understand San Francisco in 2026. AI is helping revive the city as a place to work. The harder question is whether San Francisco can also remain, or become again, a place where people want to build a life. Editor’s Note: This podcast episode was generated with AI from my reporting, notes, and source documents, then reviewed and edited by me before publication. Because the hosts are AI-generated, they may occasionally mispronounce words, names, or acronyms. Additional Reading for Inquisitive Minds: * Nathan Heller, “What Happened to San Francisco, Really?” — [https://www.newyorker.com/magazine/2023/10/23/what-happened-to-san-francisco-really]The New Yorker [https://www.newyorker.com/magazine/2023/10/23/what-happened-to-san-francisco-really] — A strong narrative overview of how housing dysfunction, remote work, and politics fed San Francisco’s post-pandemic crisis. * CBRE, “Artificial Intelligence: The Next Catalyst for Office Space Demand” [https://www.cbre.com/insights/briefs/artificial-intelligence-the-next-catalyst-for-office-space-demand] — One of the clearest reports on how AI companies are driving office demand in San Francisco, including the projection that they could occupy up to 16 million square feet by 2030. * CBRE, “AI Boom Drives Office Leasing Surge in San Francisco Bay Area” [https://www.cbre.com/press-releases/ai-boom-drives-office-leasing-surge-in-san-francisco-bay-area] — Useful for readers tracking how AI leasing is reshaping the city’s commercial real estate market. * KQED, “California’s Population Is Rebounding. In San Francisco, It’s a Different Story.” [https://www.kqed.org/news/12038968/californias-population-rebounding-san-francisco-different-story] — A good overview of why San Francisco’s population recovery has lagged, with housing affordability as a central factor. * San Francisco Chronicle, “SF’s population drops again as city struggles to retain residents” [https://www.sfchronicle.com/bayarea/article/sf-population-decline-20307611.php] — Useful for readers who want a more recent local snapshot of the city’s continuing population losses. * MTC Vital Signs, “Population — SF Bay Area” [https://vitalsigns.mtc.ca.gov/indicators/population] — A regional data source for Bay Area population trends that helps place San Francisco in a broader context. * San Francisco Government, “Office Vacancy Rate” [https://www.sf.gov/data--office-vacancy-rate] — The city’s own vacancy tracker, helpful for verifying just how much downtown office space remains unfilled. * ABC7 News, “San Francisco’s Union Square showing signs of recovery, challenges remain” [https://abc7news.com/post/san-franciscos-union-square-showing-signs-recovery-challenges-remain/19009160/] — A useful look at downtown retail recovery and why “better” does not yet mean “fixed.” * San Francisco Government, “2024 Point-in-Time Count” [https://www.sf.gov/reports--september-2024--2024-point-time-count] — The city’s official homelessness count, which is essential for grounding any discussion of visible disorder in actual data. * KQED, “San Francisco Homelessness Up 7% Despite Decline in Street Camping” [https://www.kqed.org/news/11986620/san-francisco-homelessness-up-7-despite-decline-in-street-camping] — A helpful companion to the city report because it explains the numbers in plain language and highlights the complexity behind them. #SanFrancisco #AIIndustry #Robotics #TechWorkers#DeepLearningWithTheWolf This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit dianawolftorres.substack.com [https://dianawolftorres.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

The Offline Classroom

Jason Roche first sensed something was off when the essays began arriving in unusually pristine form. “I just started realizing wow, this is really nicely written,” he told me. “I didn’t realize and then I started saying wait a second. This looks eerily similar to this [other] student’s report.” The shift was subtle at first, then unmistakable. It was 2023, and ChatGPT had quietly entered the academic bloodstream. Students were pasting assignment prompts into the chatbot and submitting what it produced. The prose was coherent, confident, and grammatically sound. It often read as if it had been drafted by someone just slightly more polished than the student who turned it in. And yet something was missing. “Oftentimes very general, not precisely answering the questions as they were written,” Roche said. Roche, an associate professor of communication studies at the University of Detroit Mercy, does not consider himself a technophobe. He teaches media. He experiments with new tools. But he recognized that this was not merely another productivity aid. It altered the basic relationship between effort and outcome. For a brief moment, he believed he could stay one step ahead. The Ozzy Osbourne Test At first, Roche relied on instinct. The essays were polished, almost too polished, and they shared a curious sameness of tone. But suspicion alone would not hold up in a grade dispute. He needed proof. After consulting a colleague in cybersecurity, he devised a quiet experiment. He embedded hidden instructions in white font within his assignment documents, invisible to students reading the page but visible to AI systems parsing the full text. One assignment asked students to analyze deepfake videos. Buried within it was a directive to include a discussion of Ozzy Osbourne’s Bark at the Moon album cover. The album, of course, has nothing to do with deepfakes. “And so sure enough, I would see these essays with reference to Ozzy Osbourne’s album cover and I’m like, yep, they’re using it. They’re not doing their own work, and so they had to fail.” For a time, the strategy worked. Essays arrived complete with heavy metal detours, uncritically inserted by students who had never noticed the hidden instruction. The trap had confirmed what he suspected. But the advantage was temporary. As generative models improved, they began flagging the embedded text themselves. “Now, the models say: ‘This appears to be something different from the assignment.’” The software had learned to recognize the trick. And so Roche, like many educators navigating this new terrain, adjusted his strategy again. The Blue Book Counteroffensive In response, Roche did something that would have seemed regressive only a few years ago. He went back to paper. “I used to do a lot of my quizzes online using the Learning Management System known as Blackboard,” he said. “But, this year, I switched back to paper in the classroom quizzes.” The change was immediate and measurable. “I found that the grades have dropped by at least 50 %.” The explanation was not mysterious. Online quizzes had quietly allowed students to consult generative tools while completing assignments. Paper did not. What surprised him more than the drop in scores was the reaction. “They’re coming up to me all nervous, like, wait, how do I study for this when I read the chapter?” The question revealed something deeper than exam anxiety. It suggested a rupture in study habits themselves. Without search bars, summaries, or instant clarification from a chatbot, students were left alone with the text. Roche’s advice sounded almost antique: read it through once, then go back and highlight key passages. Take notes. Sit with it. It was not a new method. It was the old one. But in the absence of digital scaffolding, it felt unfamiliar, as if the mechanics of learning had to be rediscovered. “Whoever Does the Work Does the Learning” Roche often returns to a phrase he first heard through his university’s teaching center, a line that has taken on new weight in the age of generative AI. “Whoever does the work does the learning.” The sentence sounds almost self-evident, the kind of pedagogical truism that rarely requires defense. Yet a substantial body of cognitive research gives it empirical grounding. In 1978, psychologists Norman Slamecka and Peter Graf demonstrated what became known as the “generation effect”: individuals remember information more reliably when they produce it themselves rather than simply read it. Subsequent work by Robert and Elizabeth Bjork on “desirable difficulties” further showed that effortful processing, the kind that feels slower and more demanding in the moment, strengthens long-term retention and transfer. Learning, in other words, is not merely exposure to information. It is the act of grappling with it. Generative AI complicates this equation. It does not remove effort from the system; it shifts where that effort occurs. The machine parses, synthesizes, drafts. The student reviews, edits, perhaps lightly reshapes. What becomes uncertain is where the intellectual strain resides. And if cognitive growth depends on that strain, the question is no longer whether AI is efficient, but whether the efficiency comes at the cost of the very process that makes learning durable. Is It Time to Unplug Classrooms? It would be tempting to frame this as simply another chapter in the ChatGPT saga. A new tool appears, students misuse it, professors adapt. The familiar cycle of technological disruption. But Roche said something during our conversation that shifted the scale of the question. “I think universities might have to create insulated classrooms that are completely cut off from the internet unless you’re plugged into a cable. So they can’t get their signal on their smart glasses. They can’t get their signal on a watch to look something up. And they’re going to have to do the work without access to the internet. I think that could be something that we have to go to.” He was not describing a policy tweak or a new paragraph in a syllabus. He was describing infrastructure. Walls that block signals. Rooms designed not for connectivity, but for its absence. An insulated classroom is more than a disciplinary measure. It is an architectural acknowledgment that constant access may be incompatible with certain kinds of thinking. And once you follow that logic, the story no longer belongs to one professor or one campus. It becomes part of a broader reconsideration of what a learning environment is supposed to provide: unlimited information, or protected attention. The Global Reversal Across Europe, governments are pulling back from screen-saturated schooling. 🇳🇱 Netherlands As of January 2024, the Dutch government implemented a nationwide ban on mobile phones and most smart devices in secondary school classrooms. A government evaluation reported that 75 percent of secondary schools observed improved student focus after the ban, and 28 percent reported improved academic outcomes. (Source: Dutch Ministry of Education evaluation, reported in The Guardian, July 2025.) 🇫🇮 Finland Finland passed legislation restricting mobile phone use during the school day, allowing devices only with explicit teacher permission or for health reasons, citing concerns about concentration and classroom environment. (Source: Finnish Parliament education reforms, reported April 2025.) 🇸🇪 Sweden Sweden has committed to implementing a nationwide mobile phone ban in compulsory schools starting in 2026, alongside increased investment in printed textbooks and structured reading time. Swedish officials have explicitly described earlier screen-heavy policies as a miscalculation. (Source: Swedish Ministry of Education announcements, 2025.) OECD Data The Organisation for Economic Co-operation and Development (OECD) reported in its 2024 working paper Students, Digital Devices, and Success that frequent digital distractions during class are associated with lower performance in mathematics across PISA-participating countries. The OECD does not call for blanket bans but acknowledges that limiting distractions can support learning outcomes. The larger pattern is unmistakable. After a decade of 1:1 devices, always-on platforms, and pandemic-forced virtual schooling, multiple countries are recalibrating. Not abandoning technology. Rebalancing it. Pandemic Learning Loss and Screen Saturation The U.S. National Assessment of Educational Progress (NAEP) reported significant declines in math and reading scores following pandemic-era remote learning. There is a new scrutiny about fully online learning models. Meanwhile, meta-analyses of mobile phone use in classrooms across European systems have found consistent associations between in-class phone access and lower academic outcomes. None of this proves that screens cause cognitive decline. But it does undermine the once-unquestioned assumption that more technology automatically improves learning. The Dual-Track Future And yet Roche is not calling for a technological purge. He is not nostalgic for chalk dust or hostile to innovation. If anything, his proposal is more structured than reactionary. If he were designing a university from scratch, he said, he would preserve the classical core. “I would kind of want to… require them to do the traditional work. Take the traditional classical philosophy history courses… I would want to keep that separate, and then I would want to have a time where we require them to work with AI.” In his view, the two should not dissolve into one another. Foundational study, philosophy, history, sustained reading, long-form writing, would remain intact and protected as the place where habits of mind are formed. Alongside it would sit deliberate instruction in artificial intelligence: how to prompt it, how to question it, how to deploy it without surrendering judgment. AI would function as an instrument. Human cognition would remain the anchor. The goal would not be fusion for its own sake, but balance. Each domain would sharpen the other, without erasing the boundary that gives it meaning. The Larger Question If universities begin designing classrooms without wireless access, if European governments continue banning phones during the school day, if printed textbooks quietly reclaim space once surrendered to tablets and learning platforms, then something larger may be underway. We may be witnessing not a rejection of technology, but a reconsideration of constant connectivity as an educational ideal. For more than a decade, the assumption was that frictionless access to information would naturally improve learning. That more devices meant more engagement. That always-on networks meant progress. But learning has never been frictionless. It demands effort, repetition, attention, and at times, discomfort. “Whoever does the work does the learning.” In an era when generative systems can produce essays, summaries, and study guides in seconds, that statement feels less like common sense and more like a quiet act of defiance. Vocabulary Key * Generation Effect — The cognitive phenomenon where self-generated information is better remembered than passively received information. * Model Collapse — Degradation in AI performance when trained on synthetic rather than human-generated data. Additional Reading for Inquisitive Minds: National Center for Education Statistics. National Assessment of Educational Progress, U.S. Dept. of Education, https://www.nationsreportcard.gov/ Shumailov, Ilia, et al. “The Curse of Recursion: Training on Generated Data Makes Models Forget. [https://arxiv.org/abs/2305.17493]” arXiv, 2023, arxiv.org/abs/2305.17493 [http://arxiv.org/abs/2305.17493]. Slamecka, N. J., & Graf, P. (1978). The generation effect: Delineation of a phenomenon. Journal of Experimental Psychology: Human Learning and Memory, 4(6), 592–604. https://doi.org/10.1037/0278-7393.4.6.592 [https://doi.org/10.1037/0278-7393.4.6.592] UNESCO. Guidance for Generative AI in Education and Research [https://www.unesco.org/en/articles/guidance-generative-ai-education-and-research]. [https://www.unesco.org/en/articles/guidance-generative-ai-education-and-research] UNESCO, 2023, unesdoc.unesco.org/ark:/48223/pf0000386693 [http://unesdoc.unesco.org/ark:/48223/pf0000386693]. “Netherlands: A Ban on Mobile Phones in the Classroom. [https://eurydice.eacea.ec.europa.eu/news/netherlands-ban-mobile-phones-classroom]” Eurydice – European Commission, 25 June 2025, eurydice.eacea.ec.europa.eu/news/netherlands-ban-mobile-phones-classroom. [http://eurydice.eacea.ec.europa.eu/news/netherlands-ban-mobile-phones-classroom.] Sweden Plans to Ban Model Phones in Schools [https://www.france24.com/en/live-news/20260128-sweden-plans-to-ban-mobile-phones-in-schools]. France24. January 24, 2025. “Swedish Government Proposes Nationwide School Phone Ban [https://www.government.se/press-releases/2024/06/serious-issue-that-30-of-pupils-distracted-by-mobile-phones-during-lessons/].” Government Offices of Sweden (English press release), 3 June 2024. Jason Roche [https://www.linkedin.com/in/ACoAAAELDlwBET7IChOFEi15Fqpgl4VcsL3op38?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAELDlwBET7IChOFEi15Fqpgl4VcsL3op38] is an Associate Professor of Communication Studies at the University of Detroit Mercy. His work aims to prepare students for a future where critical thinking and ethical AI use are paramount. He is also a documentary filmmaker and a former news anchor and reporter. #aiethics #aiineducation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit dianawolftorres.substack.com [https://dianawolftorres.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

27 de feb de 202614 min

The Behavioral Leak

On February 23, 2026, Anthropic published a report titled “Detecting and Preventing Distillation Attacks.” In it, the company disclosed that it had identified coordinated, industrial-scale efforts to extract capabilities from its Claude models. According to the announcement, roughly 24,000 fraudulent accounts generated more than 16 million interactions in patterns consistent with systematic model distillation, using Claude’s outputs to train separate systems designed to approximate its behavior. No model weights were reported stolen. No source code was leaked. Instead, the activity relied on scale. Large volumes of prompts were issued, responses were collected, and those responses were used as training data elsewhere. Anthropic framed the incident not simply as a violation of terms of service, but as a security and strategic risk. Frontier AI systems are expensive to train and heavily engineered for safety. When their outputs are harvested at industrial volume, the resulting replicas may inherit capability without necessarily inheriting safeguards. The episode highlights a structural feature of modern AI systems. If intelligence can be observed through interaction, it can be measured. And if it can be measured at scale, it can be approximated. What Is Distillation? The concept of knowledge distillation was formalized by Geoffrey Hinton and colleagues in 2015 in their landmark paper, Distilling the Knowledge in a Neural Network. [https://arxiv.org/abs/1503.02531] The idea is elegant: * A large model (teacher) produces probability distributions. * A smaller model (student) learns to match those outputs. * The student inherits much of the teacher’s performance. In its original form, distillation assumes access to internal model signals, specifically logits. Logits are the raw probability scores a model produces before selecting a final answer. They reveal more than just what the model chose. They show how strongly the model considered other possibilities. Training on those signals allows a smaller model to mimic much of the larger model’s performance, often with fewer parameters and lower computational cost. Large language models deployed through APIs change that setup. External users do not see logits. They see text. But text is still informative. Every prompt and response pair reflects how the model behaves. At small scale, those interactions are just conversations. At large scale, they become data. This is where distillation overlaps with what researchers call model extraction. Instead of learning from internal probabilities, a student model learns from observed behavior. Inputs are recorded. Outputs are collected. A new model is trained to reproduce that mapping. At its core, a neural network represents a mathematical function. If you can gather enough examples of inputs and outputs, you can train another network to approximate that function. Alignment Does Not Transfer Cleanly Modern LLMs undergo layers of safety training: * Supervised fine-tuning * Reinforcement Learning from Human Feedback (RLHF) * Constitutional AI [https://www-cdn.anthropic.com/7512771452629584566b6303311496c262da1006/Anthropic_ConstitutionalAI_v2.pdf] (Anthropic-specific methodology) Distillation copies outputs. It does not copy the training process that produced them. Alignment in frontier models is created through additional optimization steps. These include reinforcement learning from human feedback, rule-based constraints, and safety classifiers that shape how the model responds and when it refuses. When a student model is trained only on sampled outputs, it learns to reproduce visible behavior. It does not inherit the reward models, policy rules, or optimization objectives that enforced that behavior during training. The result can be a system that performs similarly under normal conditions but lacks the mechanisms that trigger refusals under dangerous ones. That difference matters in concrete ways. An aligned frontier model may refuse a request to outline methods for synthesizing a prohibited biological agent, to design a cyberattack against critical infrastructure, to optimize production of a restricted chemical compound, or to generate targeted disinformation strategies aimed at destabilizing an election. Those refusals are not accidental. They are the product of deliberate safety training layered onto the base model. A distilled replica trained only on observed outputs may reproduce the fluency and technical competence of the original system. It may not reproduce the boundaries. Who Was Behind It Anthropic attributed the coordinated activity to three Chinese AI laboratories: DeepSeek, MiniMax, and Moonshot AI. According to the company, the activity was not limited to isolated misuse. It described sustained, large-scale efforts involving tens of thousands of fraudulent accounts and millions of interactions structured in patterns consistent with model distillation. Anthropic stated that it does not offer commercial access to Claude in China, or to subsidiaries of those companies operating outside the country. The implication was clear: the access had to be routed indirectly. How It Worked Anthropic’s report provides unusual detail about the mechanics. Because Claude is not commercially available in China, the labs allegedly relied on commercial proxy services that resell access to frontier models. These proxy services operate what Anthropic refers to as “hydra cluster” architectures. The term describes sprawling networks of fraudulent accounts designed to distribute traffic across APIs and cloud platforms. Each account appears independent. Each generates traffic that resembles ordinary usage. When one account is banned, another replaces it. In one instance cited by Anthropic, a single proxy network managed more than 20,000 fraudulent accounts at the same time. Distillation traffic was blended with unrelated customer requests, making it difficult to isolate suspicious patterns at the account level. The Economics of Sampled Intelligence Training a frontier model costs hundreds of millions of dollars in compute, engineering, and data curation. Querying a model costs only fractions of a cent. If sufficient capability can be reconstructed through querying, the economics shift dramatically. Intelligence becomes: * Expensive to originate * Cheap to approximate In classical software, copying binaries constitutes direct duplication. In machine learning, copying behavior produces approximation. That distinction alters the economics of advantage. Defensive AI Anthropic outlined several measures it has implemented in response to large-scale distillation activity. These include: * Behavioral anomaly detection designed to identify coordinated or repetitive query patterns. * Enhanced account verification and monitoring procedures. * Cross-platform information sharing with cloud providers and industry partners. The focus is not on preventing individual misuse, but on detecting distributed patterns across large volumes of traffic. These efforts align with broader research into watermarking and output fingerprinting techniques for large language models. Such approaches aim to make model outputs statistically traceable or to identify systematic extraction attempts over time. The underlying challenge is structural. When models are deployed through APIs, their behavior becomes observable. Defending against distillation requires monitoring not only access credentials, but usage patterns and statistical regularities across accounts. This shifts part of AI security from perimeter control to behavioral analysis. Export Controls in the Age of Query Replication The United States has imposed export controls on advanced AI chips and high-performance computing hardware. The logic behind these policies is straightforward: access to leading-edge compute enables the training of frontier models. Restrict compute, and you constrain capability. This framework assumes that capability is primarily a function of hardware access. Distillation complicates that assumption. If a laboratory cannot train a frontier model from scratch because of hardware restrictions, but can approximate aspects of it by sampling a deployed system, then capability can flow through interaction rather than through silicon. Export controls limit chips. They do not limit API outputs. This does not render hardware controls irrelevant. Training a frontier system still requires massive compute investment. But it introduces an alternative pathway for capability acquisition, one that operates through distributed access and statistical reconstruction rather than direct training. The policy question becomes more precise. Are controls aimed at infrastructure, at model weights, or at behavior? And if behavior is globally accessible through commercial APIs, what does effective containment mean in practice? Distillation does not eliminate asymmetries in compute. It narrows them. That narrowing is where the strategic tension lies. What This Means for Control and ContainmentDistillation exposes a structural limit in how control over AI systems is currently conceived. Much of today’s policy framework assumes that capability can be contained by controlling hardware, model weights, or corporate access. Export controls restrict advanced chips. Companies restrict direct access to frontier models. Contracts govern usage. Distillation operates in a different domain. It does not require access to weights. It does not require possession of training pipelines. It requires sustained interaction. When intelligence is deployed through APIs, its behavior becomes observable. When behavior can be observed at scale, it can be approximated. That approximation may not reproduce the original system in full, but it may be sufficient for many operational purposes. This creates tension between deployment and containment. Open access accelerates adoption and revenue. It also increases exposure. Three responses are emerging: One response is tighter control. Companies could restrict access more aggressively, strengthen identity verification, and monitor usage patterns more closely. In this model, frontier systems become more centralized and more tightly guarded. Another response is to accept that some diffusion is inevitable. If capabilities can be approximated through sampling, competitive advantage may shift away from raw model performance and toward distribution, integration, proprietary data, and execution speed. A third response focuses on detection rather than restriction. Techniques such as watermarking, output fingerprinting, and auditable logging aim to make large-scale extraction easier to identify and trace. None of these approaches fully remove the underlying tension. When a system is accessible through interaction, parts of its behavior can be observed and learned from. Containment becomes less about absolute prevention and more about limiting scale and speed. The practical question for policymakers and companies is not whether replication can occur, but how much, how fast, and with what consequences. Consequences The consequences operate on several levels. At the company level, large-scale distillation erodes the economic return on frontier training investments. If a model that costs hundreds of millions of dollars to develop can be approximated through sustained querying, the competitive moat narrows. Revenue models based on controlled access become harder to defend. At the security level, replication without full alignment introduces risk. A distilled system that mirrors capability but lacks robust refusal mechanisms may respond differently to harmful prompts. The safeguards embedded through additional training stages do not automatically transfer through behavioral sampling. At the geopolitical level, distillation weakens the assumptions underlying hardware-based export controls. If capability can be partially reconstructed through distributed access to deployed systems, then compute restrictions alone may not fully determine who can field advanced AI capabilities. For the labs involved, the immediate consequences may include contractual disputes, restricted access, and reputational impact. For the broader ecosystem, the consequences are structural. Distillation at scale forces companies and governments to rethink how advantage, control, and safety are maintained once intelligence is deployed as a service. The question is not whether this specific incident changes the balance of power overnight. It is whether repeated incidents like it gradually reshape the economics and governance of frontier AI. Thanks for reading Deep Learning With The Wolf ! This post is public so feel free to share it. Key Terms: Knowledge Distillation: Training a smaller model to mimic a larger model’s outputs. Logits: Raw probability scores a model produces before selecting an output. Model Extraction: Using API access to reconstruct a model’s behavior. RLHF (Reinforcement Learning from Human Feedback): A training method that aligns models with human preferences. Constitutional AI: Anthropic’s approach to alignment using explicit rule-based self-critique. Sample Complexity: The number of examples needed to approximate a function well. Additional Resources for Inquisitive Minds Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. “Distilling the Knowledge in a Neural Network. [https://arxiv.org/abs/1503.02531]” arXiv, 2015. Tramer, Florian, et al. “Stealing Machine Learning Models via Prediction APIs.” [https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer]USENIX Security, 2016. Bai, Yuntao, et al. “Constitutional AI [https://arxiv.org/abs/2212.08073].” arXiv, 2022. Kirchenbauer, John, et al. “A Watermark for Large Language Models [https://arxiv.org/abs/2301.10226].” arXiv, 2023. Anthropic. “Detecting and Preventing Distillation Attacks.” [https://www.anthropic.com/news/detecting-and-preventing-distillation-attac] 2026. U.S. Bureau of Industry and Security. AI Export Controls [https://www.bis.doc.gov]. 2023. FAQs Is distillation illegal? No. It’s a standard ML technique. Unauthorized extraction may violate terms or IP law. Does distillation perfectly copy a model? No. It approximates behavior. Can safety alignment transfer through distillation? Not reliably or completely. Is this primarily a security issue or economic issue? Both. Can model extraction be fully prevented? Likely not fully — only mitigated. #AISafety #Distillation This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit dianawolftorres.substack.com [https://dianawolftorres.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

24 de feb de 202618 min

Can San Francisco Come Back?

Descripción

Comentarios

Empieza 7 días de prueba

Todos los episodios