包括最近一篇关于当代LLMs控制推理水平机制的文章。 理性模式仍然一如既往。 事实上，今天几乎所有的当代LLMs都是推理模型或提供推理模式。
对于那些对推理模型如何工作感兴趣的人，包括推论缩放，强化学习，以及提炼，我写了最近我所写的Build Reasoning Model(从scratch)，这个模型从PyTorch的地面上执行这些，根据我的经验，这仍然是了解和学习事情如何真正运作的最好方法。
在过去几个月里，我还坐下来录制了我想与你们分享的讲座。 因此，下面，你会找到8个推理模型-从scratch编码视频(内容11小时).
我们从预先训练的LLM开始，在Python和PyTorch执行评价和改进的方法。 所以，如果你喜欢我的编码LLMs 从去年的地面课程，这本质上是续集(但你也可以不以先前的课程为先决条件观看).
将很快回来，同时，我希望你喜欢编码！
LLMs，推理模型，和agents的关系和搭配方式
我经常被问及推理模式与今日的AI agents有何关系
一个推理模型本质上是从常规的LLM升级，即通过强化学习增加可核查的奖励(和可选的推论缩放技术)后训练，使其更有能力解决更具挑战性的问题。 经典地，推理模型是为数学和代码开发的，但能力也可以概括到其他问题领域。
一个推理模型然后作为agent拉带内部的引擎发挥作用，其中agent拉带是一个围绕LLM/推理模型的软件或工具框架，以解决更具有挑战性的问题(bigger编码任务，旅行预订，以及所有其他类型的个人助理任务). 牵引装置的任务是复杂的上下文管理，并让LLM获得工具。
图1：常规LLMs，推理模型和agent牵引装置的引擎类比。 图片来自编码Agent文章的我的组件。
在上图所描绘的这种类比中，模型是引擎，agent将新增控制在让我们用于各种事物的机械上(作为简单的聊天器接口的升级). agent既可以使用常规的LLM，也可以使用推理模型，但推理模型是比较典型的选择，因为它们的能力更高。 (如上所述，大多数流源LLMs是具有可变推理努力设置的推理模型。 )
因此，底线是推理模型作为"引擎"，仍然是理解现代LLM agents的重要组成部分。
上述专题的配套资源如下：
- 建立大语言模型(从剪辑)书和视频
- 《构建一个理性模型》(从剪辑)书(和下面的视频)
- Agent编码文章在Substack上的组件
下面的视频专注于开发能够为这种系统提供动力的推理模型。
范围和先决条件
推理-从scratch视频聚焦于一个小型的预训Qwen3模型，我们自己在Python和PyTorch中执行推理方法。 从一个预先训练的模型开始，让我们集中关注有趣的部分：评价，推论时间的缩放，强化学习，和蒸馏。
关于先决条件，我建议对Python和PyTorch感到舒适。 我将推理材料设计为独立内容，所以之前的Build A Ligh Language Model(从Scratch)内容被推荐，但并不严格要求。
计算资源时，可以在CPU上通过早期的文本生成和评价实例工作。 对于后来的实验，特别是培训，一个GPU使工作更加实用。 第一部视频涵盖了设置和一些更多的建议；存储器还包括硬件和环境建议。
图2：涵盖的专题概述。
伴书
这些视频与我的书，“构建一个理性模型”(来自Scratch)。 如果你想更详细地按自己的速度阅读解释，通过练习来工作，等等，我在此推荐这本书为伴。
对于最深入的学习经验，我的建议是读一章，通过它的代码和相应的视频一起工作，然后尝试练习。
该书从曼宁和Amazon获得，印刷版以颜色发行！
图3:Build a Reasoning Model(来自Scratch)印刷本的随机摘录。
感谢所有收到副本或推荐给朋友的人。 你的支持帮助我继续创作这些材料。
视频
接下来的影片，对于更简短、更注重概念的概述，我还建议我关于链接的AI理性模型课程在学习中(通过该链接访问时应免费24小时)
该课程大约90分钟，并解释了推理模型是如何工作的，涵盖了推论-时间缩放，强化学习，和蒸馏的主要思想。 如果你喜欢主要的想法
为了了解这篇文章的主要部分，这里有8个主要讲座视频。
1. 动机和代码设置
第一段视频首先解释了常规LLMs，推理模型，和agents的关系。 接下来，视频的主要话题是设置编码环境。 这包括安装Python和PyTorch的依赖(与uv)等。 我也讨论了硬件的选择和常见的设置问题。
所涉主题
- 常规LLMs、推理模型和agent牵引装置之间的关系
- 通过执行守则学习的益处
- 这个视频系列的范围
- 安装 Python 依赖关系
- 选择硬件并检查CUDA或Apple硅支持
- 使用带有VS代码的远程GPU，排除环境故障
观看视频 1 —— 设置指令
2. 装入基本模型、文本生成和 KV 缓存
在即将到来的视频中加入推理方法之前，我们需要一个工作基础模型。 与其从零开始训练它，因为那是另一本全书(见Build A Lige Language Model (From Scratch))，不如从预先训练过的模型开始。
在这里，我们加载Qwen3 0.6B型机的前训练重量。 然后，我们通过流式文本生成器实施文本生成过程，并加入KV缓存和模型编译，以提高文本生成效率。
这些执行为我们提供了基线和基础组成部分，我们将在以下视频中评价和改进。
所涉主题
- 将文本编码为符号并将其解码为文本
- 下载Qwen3重量并在选定的计算设备上加载模型
- 检查变压器形状、记录器和下一个预测
- 实施贪婪的解码和流转文本生成
- 处理序列终结符和测量生成速度
- 添加 KV 缓存以重用注意键和值
- 将 KV 缓存与火炬组合。 编译和比较硬件性能
观看视频 2 章代码 2
3. 建立评价和强化学习的核查机构
在这个讲座中，我们一步一步地构建一个数学答案验证器，从答案提取到正常化和检查数学等效。 然后我们用它来评估MATH-500上的模型。
我们用这个验证器来评价基础模型在MATH-500评价数据集上的性能。
然而，核查员在系列的后期也有第二个工作：为强化学习提供可核查的奖励信号(RLVR).
所涉主题
- 核查人介绍及其如何融入LLM评价方法(其他方法载于本书附录)
- 将回答核查作为RLVR的奖励信号(详见第6章)。
- 核查者流动，包括提取最后答案、使答案正常化和检查数学等同性
- 为MATH-500建造一个回答分级器和评价循环
- 比较MATH-500的基础和推理模型
观看视频 3 章代码
4. 推断时间尺度：取样和自我一致性
现在我们探索改进答案的方法，同时保持模型的权重固定。 但在实施具体的推论-缩放技术之前，我们首先执行方法，从同一个即时生成不同的答案(这是专有的LLM聊天器的标准做法，如ChatGPT).
具体来说，我们实施温度缩放、多鼻取样和顶p过滤，以产生不同的反应。 然后，我们通过自我一致，利用多数人对其最后答案的表决，将多种反应结合起来。
我们还审查了答案准确性和产生更多候选人所需的额外计算之间的权衡。
所涉主题
- 比较在培训和推论期间如何额外计算效益模型的业绩
- 一个温柔的引言通过思维链的提示推论
- 实施温度缩放和多鼻取样，以产生不同的答案
- 添加上P过滤器，以控制不同答案的质量
- 通过自我一致性综合候选人的答案，这是最流行的推论缩放技术之一
- 根据额外推论计算费用评估MATH-500准确性
观看视频 4 — 第4章代码
5. 推断时间尺度：评分和自我改进
在之前的视频的基础上，我们执行评分功能，以比较同一提示产生的多个答案。 我们还可以使用这些评分方法，在自我一致性中断绝关系(在之前的讲座中涵盖).
这里所覆盖的评分方法包括计算符和序列日志概率(这在后来的强化学习章节中也将相关). 然后，我们实施一个自我完善的循环，其中模型批评一个答案并试图加以修订。
评估让我们检查这些额外步骤对我们的模式是否有益。
所涉主题
- 利用评分函数比较多个答案和断开链接
- 执行简单的高压计分器
- 计算符号概率和序列可能性(也与以下讲座有关)
- 利用日志概率提高数值稳定性
- 为提出批评和订正答案建立自我完善的循环
- 评价MATH-500上的自我改进
观看视频 5 章代码
6. 强化学习：对RLVR实施GRPO
在这个讲座中，我们着手实施一种培训技术，称为“强化学习与可核实奖励”。 我们首先实施简化版的Group相对政策优化(GRPO)，生成各组响应，与我们的核查员进行评分，并在训练期间使用他们的相对奖励来更新模型重量。
所涉主题
- 生成推理痕迹并理解答案长度与解决问题能力之间的关系
- 讨论和比较两种重要的LLM培训后方法：强化学习与人类反馈(RLHF)和强化学习与可核实奖励(RLVR)
- 将核查分数转化为奖励和相对优势
- 计算序列日志概率和简化的GRPO损失
- 将培训循环与伐木和检查站储蓄结合起来
- 评价检查站，检查培训稳定性和记忆需要
观看视频 6 章代码
7. 强化学习：跟踪、剪辑
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
I have written a lot about reasoning models here over the last two years on Ahead of AI, including a recent article on mechanisms to control reasoning effort levels in contemporary LLMs. Reasoning models are still as relevant as ever. In fact, almost all current-generation LLMs today are reasoning models or offer a reasoning mode.
For those who are interested in how reasoning models work, including inference scaling, reinforcement learning, and distillation, I wrote my recent Build Reasoning Model (From Scratch), which implements these from the ground up in PyTorch, which, in my experience, is still the best way to understand and learn how things really work.
Now, in the last few months, I also sat down and recorded accompanying lectures I wanted to share with you. So, below, you will find eight reasoning model-from-scratch coding videos (11 hours of content).
We start with a pre-trained LLM and implement methods to evaluate and improve it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up course last year, this is essentially the sequel (but you can also watch it without the prior course as a prerequisite).
I’m currently taking a short break and will be back soon with more exciting LLM research roundups next month. In the meantime, I hope you enjoy coding along!
How LLMs, reasoning models, and agents relate and fit together
As a small side note, and since I often get asked how reasoning models relate to today’s AI agents, here’s a short section on this before getting started with the videos.
A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to other problem domains as well.
A reasoning model then functions as the engine inside an agent harness, where the agent harness is a software or tooling framework around the LLM/reasoning model to tackle even more challenging problems (bigger coding tasks, travel booking, and all other sorts of personal assistant tasks). The task of the harness is sophisticated context management and giving the LLM access to tools.
Figure 1: An engine analogy for conventional LLMs, reasoning models, and agent harnesses. Figure from my Components of a Coding Agent article.
In this analogy depicted in the figure above, the model is the engine, and the agent harness adds the machinery that lets us use it for various things (as an upgrade from a simple chatbot interface). An agent can use either a conventional LLM or a reasoning model, but reasoning models are the more typical choice as they are more capable. (As mentioned above, most current-gen LLMs are reasoning models with variable reasoning effort settings.)
So, the bottom line is that reasoning models, as the “engine,” remain an important part of understanding modern LLM agents.
The resources accompanying these topics shown above are:
- The Build a Large Language Model (From Scratch) book and videos
- The Build A Reasoning Model (From Scratch) book (and videos below)
- The Components of a Coding Agent article on Substack
The videos below focus on developing the reasoning model that can power such a system.
Scope and prerequisites
The reasoning-from-scratch videos focus on a small pre-trained Qwen3 model, where we implement the reasoning methods ourselves in Python and PyTorch. Starting with a pre-trained model lets us focus on the interesting part: evaluation, inference-time scaling, reinforcement learning, and distillation.
Regarding the prerequisites, I recommend being comfortable with Python and PyTorch. I designed the reasoning materials as standalone content, so the previous Build A Large Language Model (From Scratch) content is recommended but not strictly required.
Compute-resource-wise, you can work through the early text-generation and evaluation examples on a CPU. For the later experiments, especially training, a GPU makes the work much more practical. The first video covers the setup and some more recommendations; the repository also includes hardware and environment recommendations.
Figure 2: An overview of the topics being covered.
The companion book
These videos accompany my book, Build a Reasoning Model (From Scratch). If you like to read through the explanation in more detail and at your own pace, work through exercises, and so on, I recommend the book as a companion here.
For the most in-depth learning experience, my suggestion is to read a chapter, work through its code alongside the corresponding video, and then try the exercises.
The book is available from Manning and Amazon, and the print edition is in color!
Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) print book.
Thank you to everyone who has picked up a copy or recommended it to a friend. Your support helps me keep creating these materials.
The videos
As a last, quick side note before getting to the main videos below: For a shorter, more conceptual overview, I also recommend my AI Reasoning Models course on LinkedIn Learning (it should be free for 24h when accessing it via the link there, which shouldn’t then require a LinkedIn Premium subscription).
The course is about 90 minutes and explains how reasoning models work, covering the main ideas behind inference-time scaling, reinforcement learning, and distillation. It’s a good optional preview if you’d like the main ideas without working through the full implementations below.
To get to the main part of this article, here are the eight main lecture videos.
1. Motivation and code setup
The first video starts with an explanation of the relationship between conventional LLMs, reasoning models, and agents. Next, the main topic of the video is setting up the coding environment. This includes installing the Python and PyTorch dependencies (with uv), among others. I also discuss hardware choices and common setup issues.
Topics covered
- The relationship between conventional LLMs, reasoning models, and agent harnesses
- On the benefits of learning through code implementations
- The scope of this video series
- Installing Python dependencies
- Choosing hardware and checking CUDA or Apple Silicon support
- Using a remote GPU with VS Code and troubleshooting the environment
Watch video 1 · Setup instructions
2. Loading a base model, text generation, and KV caching
Before adding reasoning methods in the upcoming videos, we need a working base model. Instead of training it from scratch, since that would be a whole other book (see Build A Large Language Model (From Scratch)), we start with a pre-trained model.
Here, we load the pre-trained weights of a Qwen3 0.6B model. Then, we implement the text-generation process via a streaming text generator and add KV caching and model compilation to make text generation more efficient.
These implementations give us the baseline and base components we will evaluate and improve in the following videos.
Topics covered
- Encoding text as tokens and decoding it back into text
- Downloading Qwen3 weights and loading the model on the selected compute device
- Going over tensor shapes, logits, and next-token predictions
- Implementing greedy decoding and streaming text generation
- Handling end-of-sequence tokens and measuring generation speed
- Adding a KV cache to reuse attention keys and values
- Combining KV caching with torch.compile and comparing hardware performance
Watch video 2 · Chapter 2 code
3. Building a verifier for evaluation and reinforcement learning
In this lecture, we build a math-answer verifier step by step, from answer extraction to normalization and checking mathematical equivalence. We then use it to evaluate the model on MATH-500.
We use this verifier to evaluate the base model’s performance on the MATH-500 evaluation dataset.
However, the verifier also has a second job later in the series: providing the reward signal for reinforcement learning with verifiable rewards (RLVR).
Topics covered
- Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)
- Reusing answer verification as a reward signal for RLVR (more on that in chapter 6)
- The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence
- Building an answer grader and an evaluation loop for MATH-500
- Comparing base and reasoning models on MATH-500
Watch video 3 · Chapter 3 code
4. Inference-time scaling: sampling and self-consistency
Now we explore ways to improve answers while keeping the model weights fixed. But before implementing specific inference-scaling techniques, we first implement methods to generate diverse answers from the same prompt (this is standard practice in proprietary LLM chatbots, such as ChatGPT).
Concretely, we implement temperature scaling, multinomial sampling, and top-p filtering to generate diverse responses. Then we combine multiple responses through self-consistency, using majority voting over their final answers.
We also examine the tradeoff between answer accuracy and the extra computation needed to generate more candidates.
Topics covered
- Comparing how additional compute benefits model performance during training and during inference
- A gentle intro to inference-scaling through chain-of-thought prompts
- Implementing temperature scaling and multinomial sampling to generate diverse answers
- Adding top-p filtering to control the quality of the diverse answers
- Combining candidate answers through self-consistency, one of the most popular inference scaling techniques
- Evaluating MATH-500 accuracy against the cost of additional inference compute
Watch video 4 · Chapter 4 code
5. Inference-time scaling: scoring and self-refinement
Building on the previous video, we implement scoring functions to compare multiple answers generated from the same prompt. We can also use these scoring methods to break ties in self-consistency (covered in the previous lecture).
The scoring methods covered here include computing token and sequence log probabilities (this will also be relevant in the later reinforcement learning chapters). We then implement a self-refinement loop in which the model critiques an answer and attempts a revision.
The evaluation lets us check how useful those additional steps are for our model.
Topics covered
- Using scoring functions for comparing multiple answers and breaking ties
- Implementing a simple heuristic scorer
- Computing token probabilities and sequence likelihoods (also relevant for the following lectures)
- Using log probabilities for improved numerical stability
- Building a self-refinement loop for generating critiques and revised answers
- Evaluating self-refinement on MATH-500
Watch video 5 · Chapter 5 code
6. Reinforcement learning: implementing GRPO for RLVR
In this lecture, we move on to implementing a training technique called Reinforcement Learning with Verifiable Rewards (RLVR). We first implement a simplified version of Group Relative Policy Optimization (GRPO), generate groups of responses, score them with our verifier, and use their relative rewards for the model weight updates during training.
Topics covered
- Generating reasoning traces and understanding the relationship between answer lengths and problem-solving abilities
- Discussion and comparison of two important LLM post-training methods: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
- Turning verifier scores into rewards and relative advantages
- Computing sequence log probabilities and a simplified GRPO loss
- Assembling the training loop with logging and checkpoint saves
- Evaluating checkpoints and checking training stability and memory needs
Watch video 6 · Chapter 6 code
7. Reinforcement learning: tracking, clippe
Auto-fetched and translated by AI Atlas. All rights belong to the original author.