Beyond Language Modeling: An Exploration of Multimodal Pretraining

Beyond Language Modeling: An Exploration of Multimodal Pretraining

In this episode, we discuss Beyond Language Modeling: An Exploration of Multimodal Pretraining [https://arxiv.org/pdf/2603.03276v1] by Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, Mike Lewis, Nicolas Ballas, Amir Bar, Michael Rabbat, Jakob Verbeek, Luke Zettlemoyer, Koustuv Sinha, Yann LeCun, Saining Xie. The paper investigates native multimodal foundation models by training from scratch on diverse visual and language data using the Transfusion framework. Key findings include the effectiveness of Representation Autoencoder for unified visual representation, synergy between vision and language data, emergence of world modeling from unified pretraining, and the role of Mixture-of-Experts in efficient multimodal scaling. The study also reveals a scaling asymmetry with vision requiring more data than language, which MoE architectures can balance to enable truly unified multimodal models.

Mode Seeking meets Mean Seeking for Fast Long Video Generation

In this episode, we discuss Mode Seeking meets Mean Seeking for Fast Long Video Generation [https://arxiv.org/pdf/2602.24289v1] by Shengqu Cai, Weili Nie, Chao Liu, Julius Berner, Lvmin Zhang, Nanye Ma, Hansheng Chen, Maneesh Agrawala, Leonidas Guibas, Gordon Wetzstein, Arash Vahdat. The paper presents a novel training paradigm combining mode seeking and mean seeking to decouple local video fidelity from long-term coherence using a Decoupled Diffusion Transformer. It employs a global Flow Matching head trained on limited long videos for narrative structure and a local Distribution Matching head aligned with a frozen short-video teacher to ensure local realism. This approach enables fast synthesis of minute-scale videos that maintain both high-quality local details and coherent long-range motion, significantly improving the fidelity–horizon trade-off.

Recursive Language Models

In this episode, we discuss Recursive Language Models [https://arxiv.org/pdf/2512.24601v2] by Alex L. Zhang, Tim Kraska, Omar Khattab. The paper introduces Recursive Language Models (RLMs), a novel inference approach that enables large language models to handle extremely long prompts by recursively processing prompt snippets. RLMs significantly extend effective context length by up to 100 times and outperform standard LLMs and existing long-context methods on multiple tasks without increasing computational cost. Additionally, the authors develop RLM-Qwen3-8B, a recursive model that notably improves performance over its base model and rivals GPT-5 on several long-context benchmarks.

PaperBanana: Automating Academic Illustration for AI Scientists

In this episode, we discuss PaperBanana: Automating Academic Illustration for AI Scientists [https://arxiv.org/pdf/2601.23265v1] by Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, Jinsung Yoon. The paper presents PaperBanana, an autonomous framework that generates publication-ready academic illustrations using advanced vision-language and image generation models. It coordinates specialized agents to retrieve references, plan, render, and refine images through self-critique. Evaluated on a new benchmark from NeurIPS 2025 diagrams, PaperBanana outperforms existing methods in faithfulness, clarity, and aesthetics, and also effectively creates high-quality statistical plots.

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

In this episode, we discuss World-Gymnast: Training Robots with Reinforcement Learning in a World Model [https://arxiv.org/pdf/2602.02454v1] by Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, Sherry Yang. The paper introduces World-Gymnast, a method that fine-tunes robot policies using reinforcement learning within a video-based world model conditioned on vision and language. This approach significantly outperforms traditional supervised finetuning and simulator-based RL in real-robot tasks, achieving up to 18x and 2x improvements, respectively. World-Gymnast also enables training on diverse instructions and novel scenes, offering a promising path for scalable robot learning outside controlled environments.