AI Post Transformers

AI Post Transformers

After Titans: Behrouz on Nested Learning and Hope

1 h 0 min · 22. mai 2026
Kom i gang
episode After Titans: Behrouz on Nested Learning and Hope cover

Beskrivelse

Episode title: After Titans: Behrouz on Nested Learning and Hope The followup to our Titans episode, by the same core team a year later. Behrouz, Razaviyayn, Zhong, and Mirrokni (Google Research) generalize the Titans bet — that long-term memory should be a learnable module updated at test time — into a broader paradigm they call Nested Learning, where a "deep" architecture is really a hierarchy of nested optimization problems each compressing its own context flow. The episode walks through their three core contributions: (1) reframing standard optimizers like Adam and SGD-with-Momentum as associative-memory modules that compress gradient information, then proposing more expressive optimizers with their own deep memory; (2) a self-modifying sequence model whose update rule is itself learned end-to-end — the natural generalization of the test-time-learnable memory module Titans introduced; (3) a continuum memory system that replaces the traditional short-term-vs-long-term dichotomy with a continuum across multiple update rates. Combining the self-modifying module with the continuum memory system produces Hope, a continual-learning architecture reported promising on language modeling, knowledge incorporation, few-shot generalization, continual learning, and long-context reasoning. The hosts treat Hope as the next concrete instance of the Titans → Nested Learning research arc, stress-test the novelty against established meta-learning and fast-weights literature, and distinguish what the paper actually shows from what its framing suggests. Sources: Nested Learning: The Illusion of Deep Learning Architectures — Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni (Google Research, NeurIPS 2025) — arXiv 2512.24695 https://arxiv.org/pdf/2512.24695 Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni (Jan 2025) — arXiv 2501.00663 https://arxiv.org/pdf/2501.00663 AI Post Transformers — "Titans: Learning to Memorize at Test Time" https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp3 Interactive Visualization: After Titans: Behrouz on Nested Learning and Hope [https://podcast.do-not-panic.com/viz/2026-05-20-nested-learning-beyond-deep-architecture-7cc949.html]

Kommentarer

0

Vær den første til å kommentere

Registrer deg nå og bli medlem av AI Post Transformers sitt community!

Kom i gang

2 Måneder for 19 kr

Deretter 99 kr / Måned · Avslutt når som helst.

  • Eksklusive podkaster
  • 20 timer lydbøker i måneden
  • Gratis podkaster
Kom i gang

Alle episoder

662 Episoder

episode Beluga: CXL Memory Pooling for LLM KV Cache cover

Beluga: CXL Memory Pooling for LLM KV Cache

This episode explores Beluga, a systems paper that tackles one of the biggest practical bottlenecks in long-context LLM inference: how to store and retrieve massive KV caches when GPU memory is no longer enough. It explains why traditional RDMA-based memory disaggregation is cumbersome and how Beluga uses CXL-based pooled memory to give GPUs and CPUs more direct, load/store-style access to shared cache data, reducing copies, staging, and synchronization overhead. The discussion digs into the architecture’s tradeoffs, including the fact that CXL is still slower than local HBM or DRAM, but argues that its simpler access model can still deliver large gains in the right workload regime. Listeners would find it interesting for its concrete analysis of when the reported speedups, including major reductions in time to first token and large throughput gains, are real advances versus artifacts of favorable cache-reuse conditions. Sources: 1. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management — Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, Yicong Zhu, Yuqi Zhou, Qiuru Lin, Jian Dai, Yang Kong, Jiayu Zhang, Guoqiang Xu, Qiang Liu, 2025 http://arxiv.org/abs/2511.20172 2. Memory Pooling With CXL — Donghyun Gouk, Miryeong Kwon, Hanyeoreum Bae, Sangwon Lee, Myoungsoo Jung, 2023 https://scholar.google.com/scholar?q=Memory+Pooling+With+CXL 3. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices — Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, Nam Sung Kim, 2023 https://scholar.google.com/scholar?q=Demystifying+CXL+Memory+with+Genuine+CXL-Ready+Systems+and+Devices 4. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu, 2025 https://scholar.google.com/scholar?q=Mooncake:+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 5. Exploring CXL-based KV Cache Storage for LLM Serving — Yupeng Tang, Runxiang Cheng, Ping Zhou, Tongping Liu, Fei Liu, Wei Tang, Kyoungryun Bae, Jianjun Chen, Wu Xiang, Rui Shi, 2024 https://scholar.google.com/scholar?q=Exploring+CXL-based+KV+Cache+Storage+for+LLM+Serving 6. MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan, 2024 https://scholar.google.com/scholar?q=MemServe:+Context+Caching+for+Disaggregated+LLM+Serving+with+Elastic+Memory+Pool 7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 8. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheGen:+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 9. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Dongjie Yang et al., 2024 https://scholar.google.com/scholar?q=PyramidInfer:+Pyramid+KV+Cache+Compression+for+High-throughput+LLM+Inference 10. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025 https://scholar.google.com/scholar?q=ChunkKV:+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference 11. Inference-Time Hyper-Scaling with KV Cache Compression — Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot, Edoardo M. Ponti, 2025 https://scholar.google.com/scholar?q=Inference-Time+Hyper-Scaling+with+KV+Cache+Compression 12. FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling — Weiqing Li et al., 2025 https://scholar.google.com/scholar?q=FlowKV:+A+Disaggregated+Inference+Framework+with+Low-Latency+KV+Cache+Transfer+and+Load-Aware+Scheduling 13. TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving — Bingyang Wu et al., 2025 https://scholar.google.com/scholar?q=TokenLake:+A+Unified+Segment-level+Prefix+Cache+Pool+for+Fine-grained+Elastic+Long-Context+LLM+Serving 14. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows — Zaifeng Pan et al., 2025 https://scholar.google.com/scholar?q=KVFlow:+Efficient+Prefix+Caching+for+Accelerating+LLM-Based+Multi-Agent+Workflows 15. Learned Prefix Caching for Efficient LLM Inference — Dongsheng Yang, Austin Li, Kai Li, Wyatt Lloyd, 2025 https://scholar.google.com/scholar?q=Learned+Prefix+Caching+for+Efficient+LLM+Inference 16. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp3 17. AI Post Transformers: CXL Computational Memory Offloading for Lower Runtime — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-cxl-computational-memory-offloading-for-3b2124.mp3 18. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp3 19. AI Post Transformers: SolidAttention: Co-Designing Sparse Attention and SSD I/O — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-18-solidattention-co-designing-sparse-atten-5a8622.mp3 20. AI Post Transformers: Stochastic KV Routing for Cache Sharing — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-29-stochastic-kv-routing-for-cache-sharing-5fef63.mp3 21. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3 Interactive Visualization: Beluga: CXL Memory Pooling for LLM KV Cache [https://podcast.do-not-panic.com/viz/2026-05-27-beluga-cxl-memory-pooling-for-llm-kv-cac-b6142f.html]

27. mai 20261 h 0 min
episode DFX: Multi-FPGA Acceleration for Transformer Inference cover

DFX: Multi-FPGA Acceleration for Transformer Inference

This episode explores the DFX system, a four-FPGA appliance designed to accelerate transformer-based text generation by targeting a key weakness of GPUs: low-batch, token-by-token decode. It explains the difference between prompt processing and sequential generation, connects the paper’s older terminology to today’s prefill/decode framing, and shows why autoregressive inference often leaves GPU hardware underused even when training runs efficiently in parallel. The discussion also breaks down how DFX uses hardware-aware model parallelism and end-to-end accelerator design, rather than only speeding up isolated transformer subcomponents, to argue for lower latency and better energy and cost efficiency than a four-V100 GPU server. Listeners would find it interesting for its clear historical perspective on transformer serving and for its skepticism about how much of the reported advantage comes from FPGA specialization versus the fairness of the GPU baseline. Sources: 1. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation — Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, Joo-Young Kim, 2022 http://arxiv.org/abs/2209.10797 2. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Yanping Huang, Youlong Cheng, Ankur Bapna, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and others, 2019 https://scholar.google.com/scholar?q=GPipe:+Efficient+Training+of+Giant+Neural+Networks+using+Pipeline+Parallelism 3. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro, 2020 https://scholar.google.com/scholar?q=Megatron-LM:+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism 4. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Noam Shazeer, Zhifeng Chen, and others, 2020 https://scholar.google.com/scholar?q=GShard:+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding 5. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia, and others, 2021 https://scholar.google.com/scholar?q=Efficient+Large-Scale+Language+Model+Training+on+GPU+Clusters+Using+Megatron-LM 6. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017 https://scholar.google.com/scholar?q=Attention+Is+All+You+Need 7. FTRANS: Energy-Efficient Acceleration of Transformers using FPGA — Jingcheng Rao, Yuchen Shao, Ke Wang, Zhihao Zhu, Xuehai Qian, Yiyu Shi, 2020 https://scholar.google.com/scholar?q=FTRANS:+Energy-Efficient+Acceleration+of+Transformers+using+FPGA 8. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding 9. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, Hai Zhao, 2024 https://scholar.google.com/scholar?q=PyramidInfer:+Pyramid+KV+Cache+Compression+for+High-throughput+LLM+Inference 10. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Bo Li, Xuming Hu, Xiaowen Chu, 2025 https://scholar.google.com/scholar?q=ChunkKV:+Semantic-Preserving+KV+Cache+Compression+for+Efficient+Long-Context+LLM+Inference 11. Cost-Optimal Grouped-Query Attention for Long-Context LLMs — Yingfa Chen, Yutong Wu, Xu Han, Zhiyuan Liu, Maosong Sun, 2025 https://scholar.google.com/scholar?q=Cost-Optimal+Grouped-Query+Attention+for+Long-Context+LLMs 12. Optimised Grouped-Query Attention Mechanism for Transformers — Yuang Chen, Cheng Zhang, Xitong Gao, Robert D. Mullins, George A. Constantinides, Yiren Zhao, 2024 https://scholar.google.com/scholar?q=Optimised+Grouped-Query+Attention+Mechanism+for+Transformers 13. Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang, 2025 https://scholar.google.com/scholar?q=Prefill-Decode+Aggregation+or+Disaggregation?+Unifying+Both+for+Goodput-Optimized+LLM+Serving 14. Nexus: Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving — Xiaoxiang Shi, Colin Cai, Junjia Du, Zhihao Jia, 2025 https://scholar.google.com/scholar?q=Nexus:+Proactive+Intra-GPU+Disaggregation+of+Prefill+and+Decode+in+LLM+Serving 15. SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference — Hengrui Zhang, Pratyush Patel, August Ning, David Wentzlaff, 2025 https://scholar.google.com/scholar?q=SPAD:+Specialized+Prefill+and+Decode+Hardware+for+Disaggregated+LLM+Inference 16. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 17. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3 18. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp3 19. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp3 20. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3 21. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp3 22. AI Post Transformers: Caffeine: A Unified FPGA for CNNs — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-06-caffeine-a-unified-fpga-for-cnns-e8acbe.mp3 Interactive Visualization: DFX: Multi-FPGA Acceleration for Transformer Inference [https://podcast.do-not-panic.com/viz/2026-05-22-dfx-multi-fpga-acceleration-for-transfor-3266ea.html]

27. mai 20261 h 0 min
episode Trajectory Summaries for Long-Horizon Coding Agents cover

Trajectory Summaries for Long-Horizon Coding Agents

This episode explores a paper on inference-time scaling for coding agents, asking whether extra test-time compute still helps when tasks are long, messy, and require multi-step tool use rather than a single code completion. It focuses on the paper’s main argument that the real bottleneck is not generating more rollout attempts, but representing prior attempts well enough to compare, select, and reuse them, with structured trajectory summaries serving as the key middle layer between raw transcripts and final patches. The discussion examines two mechanisms: a parallel “tournament” style selection method over summaries, and a sequential refinement method that conditions later attempts on distilled lessons from earlier ones. Listeners would find it interesting because the conversation connects agent performance gains to practical questions of context management, selection versus reuse, and whether the reported improvements reflect a deep scaling insight or simply better engineering around long-horizon coding workflows. Sources: 1. Scaling Test-Time Compute for Agentic Coding — Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang, Yun Zhu, Eryk Helenowski, Ruan Silva, Zhengxing Chen, Srinivasan Iyer, Manzil Zaheer, Daniel Fried, Hannaneh Hajishirzi, Sanjeev Arora, Gabriel Synnaeve, Ruslan Salakhutdinov, Anirudh Goyal, 2026 http://arxiv.org/abs/2604.16529 2. ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2023 https://scholar.google.com/scholar?q=ReAct:+Synergizing+Reasoning+and+Acting+in+Language+Models 3. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023 https://scholar.google.com/scholar?q=Reflexion:+Language+Agents+with+Verbal+Reinforcement+Learning 4. ExpeL: LLM Agents Are Experiential Learners — Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, Gao Huang, 2023 https://scholar.google.com/scholar?q=ExpeL:+LLM+Agents+Are+Experiential+Learners 5. Rethinking Thinking Tokens: LLMs as Improvement Operators — Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, Anirudh Goyal, 2025 https://scholar.google.com/scholar?q=Rethinking+Thinking+Tokens:+LLMs+as+Improvement+Operators 6. CodeMonkeys: Scaling Test-Time Compute for Software Engineering — Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Re, Azalia Mirhoseini, 2025 https://scholar.google.com/scholar?q=CodeMonkeys:+Scaling+Test-Time+Compute+for+Software+Engineering 7. S*: Test Time Scaling for Code Generation — Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, Ion Stoica, 2025 https://scholar.google.com/scholar?q=S*:+Test+Time+Scaling+for+Code+Generation 8. Scaling Test-time Compute for LLM Agents — King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, Wangchunshu Zhou, 2025 https://scholar.google.com/scholar?q=Scaling+Test-time+Compute+for+LLM+Agents 9. Agentic Test-Time Scaling for WebAgents — Nicholas Lee, Lutfi Eren Erdogan, Chris Joseph John, Surya Krishnapillai, Michael W. Mahoney, Kurt Keutzer, Amir Gholami, 2026 https://scholar.google.com/scholar?q=Agentic+Test-Time+Scaling+for+WebAgents 10. Does SWE-Bench-Verified Test Agent Ability or Model Memory? — Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan, 2025 https://scholar.google.com/scholar?q=Does+SWE-Bench-Verified+Test+Agent+Ability+or+Model+Memory? 11. A Benchmark for Procedural Memory Retrieval in Language Agents — Ishant Kohar, Aswanth Krishnan, 2025 https://scholar.google.com/scholar?q=A+Benchmark+for+Procedural+Memory+Retrieval+in+Language+Agents 12. PROCED-MEM: Benchmarking Procedural Memory Retrieval in Language Agents Across Domains — Ishant Kohar, Aswanth Krishnan, 2026 https://scholar.google.com/scholar?q=PROCED-MEM:+Benchmarking+Procedural+Memory+Retrieval+in+Language+Agents+Across+Domains 13. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems — Guibin Zhang, Muxin Fu, Guancheng Wan, Miao Yu, Kun Wang, Shuicheng Yan, 2025 https://scholar.google.com/scholar?q=G-Memory:+Tracing+Hierarchical+Memory+for+Multi-Agent+Systems 14. Scaling Agentic Verifier for Competitive Coding — Zeyao Ma et al., 2026 https://scholar.google.com/scholar?q=Scaling+Agentic+Verifier+for+Competitive+Coding 15. AgentPro: Enhancing LLM Agents with Automated Process Supervision — Yuchen Deng, Shichen Fan, Naibo Wang, Xinkui Zhao, See-Kiong Ng, 2025 https://scholar.google.com/scholar?q=AgentPro:+Enhancing+LLM+Agents+with+Automated+Process+Supervision 16. Recursive Introspection: Teaching Language Model Agents How to Self-Improve — Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar, 2024 https://scholar.google.com/scholar?q=Recursive+Introspection:+Teaching+Language+Model+Agents+How+to+Self-Improve 17. Agentic Refactoring: An Empirical Study of AI Coding Agents — Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, Ahmed E. Hassan, 2025 https://scholar.google.com/scholar?q=Agentic+Refactoring:+An+Empirical+Study+of+AI+Coding+Agents 18. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp3 19. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp3 20. AI Post Transformers: MiA-Signature and Global Activation for Long Context — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-13-mia-signature-and-global-activation-for-5ad62f.mp3 21. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp3 Interactive Visualization: Trajectory Summaries for Long-Horizon Coding Agents [https://podcast.do-not-panic.com/viz/2026-05-24-trajectory-summaries-for-long-horizon-co-0194be.html]

27. mai 20261 h 0 min
episode Generative Recursive Reasoning in Latent Space cover

Generative Recursive Reasoning in Latent Space

This episode explores Generative Recursive Reasoning, a paper that asks whether models can reason more effectively by repeatedly refining an internal latent state instead of externalizing long chains of thought as tokens. It explains how recursive reasoning trades parameter growth for inference-time computation, and how this approach may be especially useful for tasks like Sudoku, ARC-style problems, graph coloring, and N-Queens that benefit from iterative constraint solving. A central focus is the paper’s argument that reasoning should be stochastic rather than locked into a single deterministic path, using variational methods to model multiple possible latent trajectories and improve coverage when problems have more than one valid answer. The discussion is especially interesting because it contrasts this elegant search-like mechanism with mainstream transformer practice, highlighting both the promise of branching internal hypotheses and the practical reasons industry has not adopted such architectures at scale. Sources: 1. Generative Recursive Reasoning in Latent Space https://arxiv.org/pdf/2605.19376v1 2. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser, 2018 https://scholar.google.com/scholar?q=Universal+Transformers 3. Looped Transformers are Better at Learning Learning Algorithms — Liu Yang, Kangwook Lee, Robert Nowak, Dimitris Papailiopoulos, 2024 https://scholar.google.com/scholar?q=Looped+Transformers+are+Better+at+Learning+Learning+Algorithms 4. Hierarchical Reasoning Model — Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, Yasin Abbasi Yadkori, 2025 https://scholar.google.com/scholar?q=Hierarchical+Reasoning+Model 5. Less is More: Recursive Reasoning with Tiny Networks — Alexia Jolicoeur-Martineau, 2025 https://scholar.google.com/scholar?q=Less+is+More:+Recursive+Reasoning+with+Tiny+Networks 6. Probabilistic Tiny Recursive Model — Amin Sghaier, Ali Parviz, Alexia Jolicoeur-Martineau, 2026 https://scholar.google.com/scholar?q=Probabilistic+Tiny+Recursive+Model 7. Auto-Encoding Variational Bayes — Diederik P. Kingma, Max Welling, 2013/2014 https://scholar.google.com/scholar?q=Auto-Encoding+Variational+Bayes 8. Stochastic Backpropagation and Approximate Inference in Deep Generative Models — Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra, 2014 https://scholar.google.com/scholar?q=Stochastic+Backpropagation+and+Approximate+Inference+in+Deep+Generative+Models 9. Inference Suboptimality in Variational Autoencoders — Chris Cremer, Xuechen Li, David Duvenaud, 2018 https://scholar.google.com/scholar?q=Inference+Suboptimality+in+Variational+Autoencoders 10. Iterative Amortized Inference — Joseph Marino, Yisong Yue, Stephan Mandt, 2018 https://scholar.google.com/scholar?q=Iterative+Amortized+Inference 11. Amortized Variational Inference: A Systematic Review — Ankush Ganguly, Sanjana Jain, Ukrit Watchareeruetai, 2023 https://scholar.google.com/scholar?q=Amortized+Variational+Inference:+A+Systematic+Review 12. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank J. Reddi, 2025 https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts:+On+the+Power+of+Looped+Transformers 13. Structured Denoising Diffusion Models in Discrete State-Spaces — Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, 2021 https://scholar.google.com/scholar?q=Structured+Denoising+Diffusion+Models+in+Discrete+State-Spaces 14. One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models — Chris Cameron, Wangzheng Wang, Nikita Ivanov, Ashmita Bhattacharyya, Didier Chetelat, Yingxue Zhang, 2026 https://scholar.google.com/scholar?q=One+Step+Forward+and+K+Steps+Back:+Better+Reasoning+with+Denoising+Recursion+Models 15. LASAR: Latent Adaptive Semantic Aligned Reasoning for Generative Recommendation — approx. recent recommendation/LLM reasoning authors, 2025 https://scholar.google.com/scholar?q=LASAR:+Latent+Adaptive+Semantic+Aligned+Reasoning+for+Generative+Recommendation 16. Towards Inference-time Scaling for Continuous Space Reasoning — approx. recent latent reasoning authors, 2025 https://scholar.google.com/scholar?q=Towards+Inference-time+Scaling+for+Continuous+Space+Reasoning 17. GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler — approx. recent latent reasoning authors, 2025 https://scholar.google.com/scholar?q=GTS:+Inference-Time+Scaling+of+Latent+Reasoning+with+a+Learnable+Gaussian+Thought+Sampler 18. Latent Chain-of-Thought for Visual Reasoning — approx. recent visual reasoning authors, 2025 https://scholar.google.com/scholar?q=Latent+Chain-of-Thought+for+Visual+Reasoning 19. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp3 20. AI Post Transformers: Agentic Discovery for Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-12-agentic-discovery-for-test-time-scaling-f9a81f.mp3 21. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3 22. AI Post Transformers: Causal-JEPA for Object-Level World Models — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-causal-jepa-for-object-level-world-model-311a8b.mp3 23. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3 Interactive Visualization: Generative Recursive Reasoning in Latent Space [https://podcast.do-not-panic.com/viz/2026-05-21-generative-recursive-reasoning-in-latent-a9371d.html]

I går1 h 0 min
episode LPU Chip for Low-Latency LLM Inference cover

LPU Chip for Low-Latency LLM Inference

This episode explores a 2024 paper on the LPU, a custom processor designed specifically for large language model inference, with an emphasis on reducing the per-token delay that users notice in interactive systems. It explains why autoregressive decoding is often limited by memory movement and synchronization rather than raw compute, making conventional GPU strengths less decisive in small-batch, user-facing generation. The discussion highlights the paper’s full-stack argument: a specialized chip, a supporting software stack called HyperDex, and a multi-device link meant to preserve low latency while scaling across processors. Listeners would find it interesting because it reframes AI hardware performance around real conversational responsiveness and digs into whether the paper’s bold efficiency and scaling claims actually hold up under careful comparison. Sources: 1. LPU Chip for Low-Latency LLM Inference https://arxiv.org/pdf/2408.07326 2. DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation — Seongmin Hong, et al., 2022 https://scholar.google.com/scholar?q=DFX:+A+Low-latency+Multi-FPGA+Appliance+for+Accelerating+Transformer-based+Text+Generation 3. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning — Hanrui Wang, Zhekai Zhang, Song Han, 2021 https://scholar.google.com/scholar?q=SpAtten:+Efficient+Sparse+Attention+Architecture+with+Cascade+Token+and+Head+Pruning 4. A Software-Defined Tensor Streaming Multiprocessor for Large-Scale Machine Learning — Dennis Abts, et al., 2022 https://scholar.google.com/scholar?q=A+Software-Defined+Tensor+Streaming+Multiprocessor+for+Large-Scale+Machine+Learning 5. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Zhao, et al., 2024 https://scholar.google.com/scholar?q=DistServe:+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 7. RazorAttention: Efficient KV Cache Compression Through Retrieval Heads — Hanlin Tang et al., 2024 https://scholar.google.com/scholar?q=RazorAttention:+Efficient+KV+Cache+Compression+Through+Retrieval+Heads 8. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning — Yu Fu et al., 2024 https://scholar.google.com/scholar?q=Not+All+Heads+Matter:+A+Head-Level+KV+Cache+Compression+Method+with+Integrated+Retrieval+and+Reasoning 9. Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction — Haoran Qiu et al., 2024 https://scholar.google.com/scholar?q=Efficient+Interactive+LLM+Serving+with+Proxy+Model-based+Sequence+Length+Prediction 10. Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving — Ke Cheng et al., 2024 https://scholar.google.com/scholar?q=Slice-Level+Scheduling+for+High+Throughput+and+Load+Balanced+LLM+Serving 11. Deferred Continuous Batching in Resource-Efficient Large Language Model Serving — Yongjun He, Yao Lu, Gustavo Alonso, 2024 https://scholar.google.com/scholar?q=Deferred+Continuous+Batching+in+Resource-Efficient+Large+Language+Model+Serving 12. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference — Raja Gond, Nipun Kwatra, Ramachandran Ramjee, 2025 https://scholar.google.com/scholar?q=TokenWeave:+Efficient+Compute-Communication+Overlap+for+Distributed+LLM+Inference 13. Characterizing Communication Patterns in Distributed Large Language Model Inference — Lang Xu et al., 2025 https://scholar.google.com/scholar?q=Characterizing+Communication+Patterns+in+Distributed+Large+Language+Model+Inference 14. Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference — Pol G. Recasens et al., 2025 https://scholar.google.com/scholar?q=Mind+the+Memory+Gap:+Unveiling+GPU+Bottlenecks+in+Large-Batch+LLM+Inference 15. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp3 16. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp3 17. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp3 18. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3 19. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp3 20. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp3 21. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3 22. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp3 Interactive Visualization: LPU Chip for Low-Latency LLM Inference [https://podcast.do-not-panic.com/viz/2026-05-20-lpu-chip-for-low-latency-llm-inference-be13c3.html]

23. mai 20261 h 0 min