“Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta

“Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta

Subtitle: Yes, but less than had they been schemers. OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted1. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term. Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences. Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback. Background The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of [...] --- Outline: (01:23) Background (03:46) Implications (03:58) These AIs can't be trusted in an intelligence explosion (05:06) This misalignment poses direct takeover risk (07:35) What the incident tells us about takeover risk generally (08:51) The naive fixes likely make misalignment worse The original text contained 5 footnotes which were omitted from this narration. --- First published: July 23rd, 2026 Source: https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by [https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration] --- Narrated by TYPE III AUDIO [https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=redwood_research&utm_campaign=ai_narration].

“AI Futurism Reading List” by Alexa Pan

We recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used. The selection reflects my opinionated views of the field, focuses particularly on topics we happen to focus on at Redwood, and doesn’t aim to be comprehensive. I selected readings that I thought described conceptual frames and hypotheses in AI futurism that are regularly used by me and my coworkers. I think it is a good exercise to consider whether you agree with their theses and ways in which their predictions have fared well or badly in light of recent evidence. If you have suggestions for this reading list, please let me know. How to use this reading list This reading list has a core and extended section. * Core readings are organized into 4 weeks. Each week covers <8 hours of foundational context on a topic. * Topics are chosen for (1) general importance for AI risk threat modeling and/or [...] --- Outline: (01:00) How to use this reading list (01:49) Core readings (01:53) Week 1: Timelines / takeoff modeling (05:29) Week 2: Misaligned AI takeover threat modeling (10:20) Week 3: Control (13:25) Week 4: Governance / strategy (17:51) Extended readings (17:58) Trading with AIs (18:44) Power concentration/coup prevention (19:15) Acausal stuff (19:54) Moral patienthood (21:13) AI biorisk / other AI x-risk (22:23) Model spec (23:41) Better futures / Post AGI governance (24:06) Space governance The original text contained 2 footnotes which were omitted from this narration. --- First published: July 2nd, 2026 Source: https://blog.redwoodresearch.org/p/ai-futurism-reading-list [https://blog.redwoodresearch.org/p/ai-futurism-reading-list?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration] --- Narrated by TYPE III AUDIO [https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=redwood_research&utm_campaign=ai_narration].

“The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t” by Alek Westover, Alexa Pan, Sebastian Prasanna, Arun Jose

Subtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model. Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: 1. Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. 2. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment. In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment. Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...] --- Outline: (02:07) Why distillation might transfer capabilities but not misalignment (03:10) Ideas for implementing distillation for capabilities (04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed (05:38) Why distillation for capabilities and incrimination might both fail (05:44) Reason 1: Capabilities and misalignment might be too tightly linked (08:29) Reason 2: Context-dependent misalignment (09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason? (10:34) How we can empirically study distillation techniques (11:35) Appendix: details on ideas for implementing distillation for capabilities (12:02) Basic ideas (14:17) Idea 1: Iteratively filter and distill (15:14) Idea 2: Inoculate against misalignment (16:08) Idea 3: Reduce cognitive slack (16:47) Idea 4: Distill and audit The original text contained 2 footnotes which were omitted from this narration. --- First published: June 18th, 2026 Source: https://blog.redwoodresearch.org/p/the-distillation-double-bind-distilling [https://blog.redwoodresearch.org/p/the-distillation-double-bind-distilling?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration] --- Narrated by TYPE III AUDIO [https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=redwood_research&utm_campaign=ai_narration]. --- Images from the article: Diagram showing AI alignment: misaligned teacher creates students, one benign, one misaligned. [https://substackcdn.com/image/fetch/$s_!KuFJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa61b3c36-164f-492a-aa58-4505930867ad_1520x1320.webp]https://substackcdn.com/image/fetch/$s_!KuFJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa61b3c36-164f-492a-aa58-4505930867ad_1520x1320.webp ---------------------------------------- Graph showing capabilities versus misalignment, dividing AI models into four quadrants. [https://substackcdn.com/image/fetch/$s_!TFwp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b563f42-9ac2-4fd7-ab32-ab5fdd6b7d92_1742x1144.webp]https://substackcdn.com/image/fetch/$s_!TFwp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b563f42-9ac2-4fd7-ab32-ab5fdd6b7d92_1742x1144.webp Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts [https://pocketcasts.com/], or another podcast app.

“Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models” by Anders Cairns Woodruff

Subtitle: Models' no-CoT time horizon has doubled roughly every year. by Francis Rhys Ward, Dewi Gould, Anders Cairns Woodruff et al. (see full author list at the end) PAPER LINK About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)? If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dangerous planning. Models that reason substantially without a CoT might also drift further from human patterns of thought, since their reasoning is no longer constrained by text in the pretraining prior. As a result, they would be harder to understand and might be more likely to scheme. Extending Ryan Greenblatt's research, we investigate this by measuring models’ ability to complete tasks without any CoT on a suite of 43 benchmarks spanning different domains. We compare AI reasoning ability to humans using the estimated 50% time horizon (TH)---the typical time taken for a human to perform a task that the LLM performs with [...] --- Outline: (02:28) Methods (04:59) Results (06:34) FAQ (08:09) Conclusion --- First published: June 10th, 2026 Source: https://blog.redwoodresearch.org/p/estimating-no-cot-task-completion [https://blog.redwoodresearch.org/p/estimating-no-cot-task-completion?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration] --- Narrated by TYPE III AUDIO [https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=redwood_research&utm_campaign=ai_narration]. --- Images from the article: Figure 1: Our no-CoT THs (green) compared to METR’s with-CoT THs (purple). Until the release of GPT-4, with- and without-CoT THs increased at a similar rate. Since GPT-4 with-CoT THs have grown at roughly twice the rate of no-CoT THs. [https://substackcdn.com/image/fetch/$s_!66Jp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb53eb978-46f8-47fc-9b19-944e0cd47f57_2048x1253.png]https://substackcdn.com/image/fetch/$s_!66Jp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb53eb978-46f8-47fc-9b19-944e0cd47f57_2048x1253.png ---------------------------------------- Figure 2: For each model, we fit a logistic curve to predict the probability it successfully completes tasks as a function of human time (left) or o3-mini reasoning tokens (right) needed to complete the task. [https://substackcdn.com/image/fetch/$s_!Vcdy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21f01c15-7c8a-4edc-845f-a7894afbdc3d_2048x881.png]https://substackcdn.com/image/fetch/$s_!Vcdy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21f01c15-7c8a-4edc-845f-a7894afbdc3d_2048x881.png ---------------------------------------- Figure 3: No-CoT TH (left) and token horizon (right) over time. Each point is a model plotted by release date; lines are exponential fits, with shading showing 95% CIs. [https://substackcdn.com/image/fetch/$s_!Youn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3467f6ac-26d1-472c-8fb3-1be3e3e7ab34_2048x593.png]https://substackcdn.com/image/fetch/$s_!Youn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3467f6ac-26d1-472c-8fb3-1be3e3e7ab34_2048x593.png Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts [https://pocketcasts.com/], or another podcast app.

“Efficient tradeoffs and the safety-usefulness tradeoff model” by Buck Shlegeris

Subtitle: When is "increasing safety budget" a useful concept? I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold. The model suggests two ways a safety-concerned person can increase safety: * Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously. * Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...] --- Outline: (04:17) Rushed reasonable developers (06:08) Limited political will (08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness (12:59) Overall thoughts (14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model --- First published: June 8th, 2026 Source: https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety [https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Source+URL+in+episode+description&utm_campaign=ai_narration] --- Narrated by TYPE III AUDIO [https://type3.audio/?utm_source=TYPE_III_AUDIO&utm_medium=Podcast&utm_content=Narrated+by+TYPE+III+AUDIO&utm_term=redwood_research&utm_campaign=ai_narration]. --- Images from the article: Graph showing relationship between Safety and Usefulness with safety budget and tech improvements. [https://substackcdn.com/image/fetch/$s_!K2La!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0689739d-d976-4dfc-8a86-103807de72b3_682x481.png]https://substackcdn.com/image/fetch/$s_!K2La!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0689739d-d976-4dfc-8a86-103807de72b3_682x481.png Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts [https://pocketcasts.com/], or another podcast app.