DeepSeek-R1: Reasoning LLMs via Reinforcement Learning

Introduction to Reinforcement Learning

In this episode we explore Reinforcement Learning, an AI framework used in systems such as ChatGPT. Reinforcement Learning, a subfield of Artificial Intelligence, is a method for machines to learn optimal decision-making through trial and error by receiving rewards or penalties for their actions. This beginner-friendly introduction covers fundamental aspects, such as basic terminology like agents, environments, and rewards, alongside core concepts like the Markov Decision Process. The text further explains the workflow of reinforcement learning, outlines its key characteristics including sequential decision-making and delayed feedback, and categorizes common algorithms and types like positive and negative reinforcement. Finally, it showcases practical applications of this technology across diverse fields, including robotics, autonomous vehicles, and game playing.

DeepSeek-R1: Reasoning LLMs via Reinforcement Learning

We talk about DeepSeek-R1, a novel language model with enhanced reasoning capabilities achieved through reinforcement learning (RL). The researchers explored training methodologies, including DeepSeek-R1-Zero which uniquely utilizes large-scale RL without initial supervised fine-tuning (SFT), demonstrating emergent reasoning behaviors. To improve readability and further boost performance, DeepSeek-R1 incorporates a multi-stage training process with cold-start data before RL and achieves results comparable to OpenAI's o1-1217 on reasoning tasks. Furthermore, the paper discusses the distillation of DeepSeek-R1's reasoning abilities into smaller, more efficient models, showcasing their strong performance on various benchmarks.

Biology of a Large Language Model

In this first episode we dive into this paper from AnthropicAI called Biology of a Large Langage Model where the autors present a detailed investigation into the inner workings of the large language model Claude 3.5 Haiku, employing a methodology centered around attribution graphs to understand how it processes information and generates responses. Through various case studies, the authors explore phenomena such as multi-step reasoning, planning in poetry generation, and multilingual understanding, uncovering specific circuit components and their functions. The research also examines the model's ability to handle harmful requests, its tendencies toward hallucination, and the faithfulness of its chain-of-thought reasoning. Ultimately, this work aims to reverse engineer the mechanisms within advanced language models to improve our understanding and assess their capabilities, while also acknowledging the limitations of current interpretability methods. Here is the full paper: https://transformer-circuits.pub/2025/attribution-graphs/biology.html [https://transformer-circuits.pub/2025/attribution-graphs/biology.html]