[Agent Memory / 强化学习] MemPO源码学习笔记 --- (7)--- 混合机制
0x00 概要
现有的基于强化学习的 Memory 管理方法往往缺乏一种有效机制针对 Memory 的更新内容进行引导优化，Memory 的内容难以保证质量。
MemPO(Self-Memory Policy Optimization)使模型对 Memory 进行自管理，并引入了基于有效信息含量的 Memory-level 的优势估计，引导 Memory 保留对解决任务更有效的信息，进而提升记忆有效性。
MemPO的独特切入点：让模型把记忆写在每轮开头()，形式上像"自我对话的草稿纸"，既是记忆又是思考链的一部分。这样，变成可训练的策略变量，用RL信号端到端地教会模型"什么值得记、怎么记”。RL 直接端到端优化这一行为，无需额外的记忆模块。
MemPO 的信息如下：
- 论文标题：MemPO: Self-Memory Policy Optimization for Long-Horizon Agents
- 论文地址：arxiv.org/abs/2603.00…
- 代码地址：github.com/TheNewBeeKi…
- 模型和数据集地址：huggingface.co/collections…
0x01 Mem Reward 机制
1.1 公式
Memory Advantage 公式(分两步)如下：
Step 1 - 计算 mem_reward (A1):mem_reward_k = P_mem_k - P_full_k
其中:
P_full_k = exp (mean (full_logp [k] × mask_k) ) ← 完整上下文→答案概率
P_mem_k = exp ( mean (mem_logp [k] × mask_k) ) ← 仅摘要→答案概率
k = 某条轨迹的某一轮
Step 2 - 归一化 (A2):mem_adv_k = (mem_reward_k - pool_mean) / (pool_std + ε)
其中:
pool = 同一 question 所有 16 条轨迹 × 所有轮次的 mem_reward(约 48 个值: 16 条 × 3 轮 / 条)
pool_mean = mean (所有 48 个值)
pool_std = std (所有 48 个值)
Step 1 含义拆解:
P_mem - P_full(Step 1): "只看笔记能答对的概率－看完整历史能答对的概率"
> 0：笔记比原文更聚焦 → 好笔记 → 正奖励
= 0：笔记无损压缩 → 中性
< 0：笔记丢了信息 → 差笔记 → 负奖励
直觉：如果你把一本书浓缩成一页笔记后，靠笔记答题，比翻全书还准 → 说明你抓住了重点
Step 2 含义拆解:
(mem_reward-pool_mean)/ pool_std (Step 2):
"我这轮的笔记质量比大家的平均水平好多少？"
pool包含同一question下所有轨迹所有轮次的mem_reward
→ 跨轨迹：不同搜索路径写出的笔记互相比较
→ 跨轮次：第2轮笔记和第4轮笔记也互相比较
→ 归一化：产生标准化的正负信号
写入位置：mem_adv[i，start_t:end_t+1]=adv
"只在<mem>...</mem>这几个token的位置赋值"
→ 其余位置 = 0
→ 只有"写笔记"的token受到笔记质量的梯度信号
1.2 P_mem 和 P_full 的完整计算细节
第一步：构建三类输入(tool_agent_loop.py)
第二步：ans_mask的构造(精确标记答案token)
第三步：一次性批量计算log_prob
# 将full_traj和mem_traj 合并成一个大 batch (减少 GPu 调用次数)
traj_input = tokenizer.pad(
{"input_ids": concat_full_traj_ids_list + concat_mem_traj_ids_list},
return_tensors="pt",
padding="longest",
return_attention_mask=True,
padding_side="left",
)
answer_input = tokenizer.pad(
{"input_ids": concat_answer_ids_list},
return_tensors="pt",
padding="longest",
return_attention_mask=True,
padding_side="left",
)
ans_mask_tensor = pad_sequence([torch.tensor(sub) for sub in concat_ans_mask_list], batch_first=True, padding_value=0, padding_side="left")
# responses(要计算 log_prob 的 token) = answer tokens，repeat 2 次对齐 full+mem
traj_batch = DataProto.from_dict(
{
"responses": answer_input["input_ids"].repeat(2, 1), #[full组+mem组】×answer
"input_ids": traj_input["input_ids"],
"attention_mask": traj_input["attention_mask"],
"position_ids": (traj_input["attention_mask"].cumsum(dim=1) - 1).clamp(min=0),
},
)
traj_log_probs = self.worker_group.compute_log_prob(traj_batch)
# 返回每个answer token在对应上下文下的 log P(token|context)
第四步：threshold 过滤(核心工程细节)，仅保留模型置信度>50%的token
第五步：计算概率并得出mem_reward
# P_full：完整上下文下，模型对答案的几何平均概率
P_full = exp(
sum(logp[full_ans_mask]) / (count(full_ans_mask) + 1e-8)
)
# P_mem：仅凭<mem>上下文，模型对答案的几何平均概率
P_mem = exp(
sum(logp[mem_ans_mask]) / (count(mem_ans_mask) + 1e-8)
)
#记忆奖励="记忆版"预测概率-"完整版"预测概率
mem_reward=P_mem -P_full #范围大致[-1，+1]
可视化一个具体例子。
题目：爱因斯坦出生城市的市长是谁？
第1轮<mem>：
"问题1：爱因斯坦出生于乌尔姆。问题2：市长未知，需继续查询。"
全文上下文 → P_full(answer="乌尔姆；某某某")=0.35
仅<mem>上下文 → P_mem(answer="乌尔姆;某某某")=0.12
mem_reward[0]=0.12 - 0.35=-0.23 ← 记忆不完整，扣分
第2轮<mem>：
"乌尔姆市长为张三，两题已解答：乌尔姆；张三。"
全文上下文 → P_full=0.68
仅<mem>上下文 → P_mem=0.71(<mem>比全文更精炼！)
mem_reward[1]=0.71-0.68=+0.03 ← 记忆充分，轻微加分
1.3 为什么不是只用P_mem？
这是一个很关键的设计决策，背后的逻辑是消除"答案本身难易度"的影响。即，P_full是基线(baseline)，把"这道题本来就难／容易"的因素剥离掉，让奖励信号只反映记忆本身的质量，而非题目难易度。这和GRPO中的"组内均值"作为基线是同一种思想一用相对分而非绝对分来评估。
1.3.1 问题：如果只用P_mem
问题A(简单)："法国首都是哪里？
答案Z="Paris"
P_mem("Paris"|<mem>法国首都是Paris</mem>)=0.95 ← 高
问题B(困难)："2018年中美贸易战中美国征收关税的具体税率是多少？
答案Z="25%"
P_mem("25%"|<mem>贸易战关税25%</mem>)=0.65 ← 低
用P_mem打分：
模型A:简单题 0.95 → 奖励高
模型B:难题 0.65 → 奖励低
但这不公平！难题即使写得很好，P_mem 也低 → 模型会学到：回答简单题比回答难题更"合算" → 产生偏好简单问题的偏差。
1.3.2 解决方案：用差值P_mem-P_full作为基线校正
P_full = P(Z|完整上下文) 这是"上界"，最好情况下答案的概率
P_mem - P_full = "仅靠记忆的能力”相对于"看完整上下文的能力”的差距 = 记忆造成的信息损失(负数表示损失，越接近0越好)
如下问题可以看出来差值P_mem-P_full的作用。
问题A(简单)：
P_full=0.95(给完整历史，答Paris很容易)
P_mem=0.93(仅凭记忆，答Paris也不难)
差值 = -0.02 → 记忆几乎没有损失，轻微负奖励
问题B(困难)：
P_full = 0.70(给完整历史，答25%还是有点不确定)
P_mem = 0.65(仅凭记忆，稍微差一点)
差值 = -0.05 → 记忆损失较小，负奖励稍大
问题B(记忆写得很差)：
P_full = 0.70(完整历史还能猜到)
P_mem = 0.20(只看记忆完全猜不到)
差值 = -0.50 → 记忆严重不足，大负奖励
1.3.3 直觉类比
就像考试打分：
单独用P_mem(绝对分)：
简单题答90分 → 奖励高
难题答65分 → 奖励低 → 难题永远得不到高奖励，模型学偏了
用P_mem - P_full(相对分)：
简单题：能力上限95分，你答了 93分 → 差2分(优秀)
难题：能力上限70分，你答了 65分 → 5分(良好)差
难题：能力上限70分，你答了 20分 → 差50分(记忆太差！)
→ 评估的是"相对于你的能力，记忆保留了多少"
1.4 P_mem - P_full 的含义
核心问题：摘要是否「足够好」，好到能替代完整的多轮对话上下文？
由这个核心问题开始，我们来看看P_mem - P_full的直觉含义。
1.4.1 直接含义
轨迹如下
- full_traj = 当前轮生成前的完整上下文(system + question + 之前所有轮次的完整对话)，长度递增
- mem_traj = mem_sys_prompt_ids(system + question) + 当前轮摘要内容的内容，非常短
两者都被要求预测同一个正确答案：
- P_full = 模型基于完整上下文预测出正确答案的概率
- P_mem = 模型仅基于摘要预测出正确答案的概率
三种情况：
因此，直觉含义如下：
- 如果你看了一本书的完整内容 VS 只看了别人写的摘要笔记，谁能更好地回答问题？
- 如果笔记质量很高，可能比翻全书更快更准确找到答案一这就 是P_mem>P_full 的含义。
1.4.2 更精确的数学意义
P_mem - P_full ≈ 衡量"记忆造成的信息损失"
- 如果P_mem ≈ P_fu1l→差值 ≈ 0 → 包含了回答问题所需的全部信息 → 奖励为0(中性，已经够好了)
- 如果P_mem << P_full→差值很负 → 缺失了重要信息→ 惩罚 token，促使模型写更好的记忆
- 如果P_mem > P_full → 差值为正(少见) → 比完整上下文更精准地提示了答案 → 奖励token(记忆甚至做了有用的"压缩摘要")
1.4.3 信用分配
通过把写记忆纳入GRPO的信用分配链路，MemPO驱动模型学习哪些信息值得在中保留(记忆奖励正向鼓励)，哪些无关信息应当省略(负向惩罚)。让模型知道哪些中间信息值得占用上下文预算、哪些噪声应该被主动丢弃。这种机制从根本上缓解了长程多轮交互中上下文爆炸的问题一一一模型被迫学会在有限的空间内压缩本轮关键信息，而非依赖无限增长的完整历史。
然而，MemPO目前优化的是"信息充分性"而非"压缩效率”，且跨轮次的长程信用分配仍然依赖最终结果奖励来间接传导，而非对每轮记忆进行前的多步评估。"
信用分配链路
比如：
标准 RL: 奖励 r → 反传给 /<action> token
MemPO: 奖励 r + mem_reward → 反传给 <mem> token
└─ mem_reward = P_mem - P_full
明确衡量"这段记忆的信息价值"
的生成行为确实进入了信用分配链路，不再是"附带生成的文本”。
有效信息
如何让模型知道哪些信息值得占用上下文预算”？
- 实现层面：MemPO没有显式建模"上下文预算"(token长度限制)。它的记忆奖励P_mem-P_full 只衡量"信息充分性”，不衡量"压缩效率”。
- 实际效果：如果写得很长但信息充分，P_mem高→mem_reward 正→仍然被鼓励。MemPO 鼓励的是**"信息留存”，而非"高效压缩"**。真正的"上下文预算感知"需要额外的长度惩罚项，当前代码中没有这一项。
丢弃噪声
"哪些噪声应该被主动丢弃”一实际上是隐式实现的机制：
轮次t的<mem>写了无关信息：
mem_traj[t]=[系统提示+<mem>无关内容</mem>]
P_mem预测答案的概率很低
mem_reward[t]=P_mem-P_fu1l < 0 ← 负奖励
→ 这段<mem>的token获得负梯度
→ 模型逐渐学会"写有关信息，不写无关噪声"
但这里的"丢弃"是通过负梯度间接引导，而不是模型显式地判断"这个信息不重要”。区别在于：
- MemPO的记忆行为是被动地被奖励/惩罚
- 真正的"主动丢弃决策"需要像MEM1那样有显式的keep/discard动作空间
长程交互难点
解决长程交互难点体现在：
- 上下文长度爆炸：AgentMemory.prepare_prompt()只保留最近一轮工具结果，强制模型靠传递信息
- 信用分配到中间行为(写记忆这一动作)
未完全解决的
一跨轮次的长程信用分配：第1轮写的 对第6轮答案的贡献，无法被当前 P_mem-P_full 精确衡量(只看该轮的即时答案预测能力)-训练中rollout轮次≤8，但推理时外推到更长轮次是否稳定是开放问题
1.4.4 场景3
我们接下来看看场景3 "答错但记忆好”的设计意图
答错 + 好mem → (-outcome_adv)+(+mem_adv)
<mem>区间：-outcome+(+mem_adv) → 惩罚被部分抵消
其他token：-outcome → 正常惩罚
为什么要保护"答错但记忆好"的情况？ token会被保护还是惩罚？为什么这样设计？
- 解耦能力学习：写好和最终答对是两个不同能力。模型可能：
- 很好地压缩了信息(mem好)
- 但在推理阶段犯了逻辑错误(答错)
如果mem写得好却因为别的环节答错而被惩罚，模型会"学废"记忆摘要能力。
- 长期信用分配：是在第2、3、4轮开头写的，而答案可能在第5轮才产生。没有mem_adv 的保护，好的记忆策略会被稀释在漫长的轨迹中无法获得正确归因。
- 防止退化：如果模型发现"反正答错了全被惩罚"，可能干脆写空的以避免任何风险。mem_adv 确保即使答错，好的记忆策略仍有正向激励。
- 部分保护。区间的advantage=(-outcome）+(+mem_adv），惩罚被部分抵消。设计原因：写好和最终答对是两个独立能力，如果因为推理环节答错就惩罚好的记忆策略，模型会"学废"记忆摘要能力。代码中注释掉的条件版本*(advantages.sum()>=0）选择不保护这种情况，作者最终没有采用。
注意：代码中有一个被注释的条件版本：
advantages + mem_advantages * (advantages.sum() >= 0) #含义：只对答对的轨迹施加mem_adv
这个版本选择不保护场景3，作者最终没有使用它一说明实验证明保护"答错但好mem"是更优策略。
1.5 baseline
MemPO 训练中有两个 baseline（对应两种 advantage）：
1.5.1 Outcome Advantage 的 Baseline
baseline = 同 question 16 条轨迹 score 的均值
例：question Q1 的 16 条轨迹：
scores = [1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
baseline = mean = 0.625
adv_i = (score_i - 0.625) /std
→ 答对的：正 advantage
→ 答错的：负 advantage
1.5.2 Memory Advantage 的 Baseline
baseline = 同 question 所有轨迹所有轮次 mem_reward 的均值
例：question Q1, 16 条轨迹，每条 3 轮有<mem>：
pool = [r_11, r_12, r_13, ← 轨迹 1 的 3 轮
r_21, r_22, r_23, ← 轨迹 2 的 3 轮
...
r_16_1, r_16_2, r_16_3] ← 约 48 个值
baseline = mean (48 个值)
adv_k = (mem_reward_k - baseline) /std
→ 比平均好的 mem：正 advantage
→ 比平均差的 mem：负 advantage
对比标准 PPO 的 Baseline
标准 PPO: baseline = V (s_t) ← Critic 网络估计的 value
GRPO/MemPO: baseline = group_mean ← 同组统计量
优劣：
- Critic: 精确到每个 token 位置，但有估计误差，需要额外训练
- group_mean: 只有一个全局值，但零误差，零额外开销
一句话：MemPO 的 baseline 就是 "同组平均水平"— 不用额外模型估计，直接从数据中统计得到。
1.6 等价性
此处的问题是：P_full 与训练时的 baseline 在数学上是否等价？因为P_full中，应该已经有部分压缩的记忆吧？
1.6.1 P_full 与 GRPO baseline 的差异
full_traj 包含如下，可以看到，P_full 中的"完整上下文"本身已经包含了前面几轮的压缩内容。它不是"纯净的原始搜索结果”。
full_traj = 完整轨迹 token 序列：
系统提示
+ 原始问题
+ Round 1 模型输出："<mem>我的记忆1</mem><think>...</think><search>..·</search>
↑ 这里已经包含了模型自己写的<mem>！
+ Round 1 搜索结果
+ Round 2 模型输出："<mem>我的记忆2</mem><think>...</think><search>..·</search>"
↑ 包含了Round2的<mem>
+ Round 2 搜索结果
+ Round T 模型输出(最终答案Z)
1.6.2 P_full 作为 baseline 与GRPO mean的对比
GRPO baseline(组内均值)：
同一道题生成16条轨迹{T1, T2, ..., T16)
baseline = mean(ri, r2,..., r16)
→ 消除"这道题本来就难"的因素 → 不同轨迹的 baseline 相同(都用组均值)
P_full baseline(本条轨迹的上界)：
baseline=P(Z丨本条轨迹的完整上下文)
→ 消除"这条轨迹本来就难"的因素(包括之前<mem>的质量) → 每条轨迹有各自的 baseline！
两者不等价，用一个例子说明：
同一道题，两条轨迹：
轨迹A:前几轮 <mem> 写得很好 → P_full=0.85（完整历史很充分）
轨迹B:前几轮 <mem> 写得很差 → P_full=0.50（完整历史也混乱）
GRPO baseline=mean(reward_A，reward_B，...) ← 两条共用同一个均值
P_full baseline ≠ 相同 ← 两条轨迹各有自己的P_full
这导致了一个微妙的问题
mem_reward=P_mem(当前轮)-P_full(当前轨迹)
P_full包含了"前几轮的质量"作为baseline
场景：前几轮很好，但当前轮写差了
P_full 高(因为完整历史包含好的早期<mem>)
P_mem 低(当前轮的<mem>质量差)
mem_reward=低-高=大负数 → 惩罚当前轮
场景：前几轮很差，当前轮稍微好一点
P_full 低(完整历史本来就乱)
P_mem略高于P_full
mem_reward=略正 → 奖励当前轮
所以：mem_reward实际衡量的是："当前轮的相对于包含前序记忆的完整上下文，是否能独立提供足够的答案信息？"
1.6.3 与GRPO 双层baseline 的关系
最终 advantage = GRPO advantage + mem advantage
GRPO advantage.(outcome baseline):
baseline=同组16条轨迹的平均奖励
目标：整条轨迹比同组平均好吗？
mem advantage(记忆baseline)：
baseline=本条轨迹的P_full(完整上下文)
目标：当前轮<mem>是否保留了足够信息？
两层 baseline正交，各自消除不同的方差来源：
GRPO baseline → 消除"问题难易度"
P_full baseline → 消除"当前轨迹的具体上下文"
1.6.4 一个潜在问题
由于 P_full 包含了前序的 ，存在这样的情况：
如果模型学会"在Round 1的里写全部信息"：
Round 1 <mem>=超级详细的笔记
Round 2~T <mem>=简短甚至空白
那么：
P_full(包含Round1 的好记忆)仍然高
P_mem(只看当轮记忆)很低
mem_reward =大负数→惩罚了这种"前置记忆"策略
MemPO实际上在每一轮都要求：
"这一轮的<mem>必须独立地包含足够回答最终问题的信息"，而不是"历史<mem>的累积"
这是MemPO记忆奖励的一个重要特性：它激励模型在每一轮都写一个独立自洽的"完整记忆摘要”，而不是把信息分散在多轮中逐步积累。这可能也是一个潜在的问题。
1.6.5 被注释的条件版本
被注释的条件版本mem_adv * (outcome_adv.sum()>=0）有什么优缺点？
优点：防止答错轨迹通过写好来抵消outcome的惩罚信号。如果答错了但mem写得好，的advantage 不会是正值，整条轨迹一致地被抑制。
缺点：答错轨迹中的完全不受mem_reward约束。模型无法学到"虽然最终答错了，但这个记忆策略是好的"，可能> 导致记忆能力学习变慢。作者最终选择无条件叠加说明实验表明保护好的mem更重要。
1.7 连续值
Memory Reward为什么设计为连续值(-1，+1）而不是离散的{0，1}？原因如下：
- 更丰富的梯度信号：离散值只能说"好/坏"，连续值能表达"好多少/坏多少”
- 更好的归一化：同组~48个连续值的mean/std统计更稳定
- 自然的定义：P_mem-P_full是概率差，天然是连续的
- 避免人为阈值：如果离散化，需要设定"什么程度算好的mem"的阔值，而连续值让GRPO归一化自动处理
1.8 mem_reward对不同轮数的奖励强度分析
1.8.2 P_full 的效果
Round 1的完全没有mem_reward 信号
原因：没有历史搜索结果，full_traj和 mem_traj 内容差不多，对比无意义
之后，各轮P_full 会自然增长
- Round 2 full_traj = 初始prompt+R1输出+search1结果 → P_full较低（只有1轮信息）
- Round 3 full_traj = 初始prompt+R1+ search1+R2+ search2 → P_full 中等（2轮信息积累）
- Round 4 full_traj = ... +R3+search3 → P_full 较高（3轮信息积累）
这导致 mem_reward= P_mem - P_full 随轮次增大而更负
1.8.2 归一化
我们再来看看归一化后的实际效果（数值模拟）
假设：16条轨迹，每条4轮，各轮raw mem_reward均值：
- Round 2：-0.10
- Round 3：-0.22
- Round 4：-0.33
池化归一化（mean=-0.22，std=0.13）后：
- Round 2: 归一化均值 ≈ +0.90 → 正优势 (鼓励改进)
- Round 3: 归一化均值 ≈ -0.03 → 中性
- Round 4: 归一化均值 ≈ -0.87 → 负优势 (惩罚相对较差)
1.8.3 核心问题
核心问题：信号强度不一致。
从模型学习角度：
- Round 2 : 系统性获得正奖励 → 梯度方向：继续写好
- Round 4 : 系统性获得负奖励 → 梯度方向：改变写法
但这不是模型真正"写得差”—而是因为 P_full 随轮数天然增大！
本质：归一化以"跨轮次的相对表现”来评判，而不是"绝对的记忆质量”
1.8.4 小结
奖励强度的系统性差异如下：
根本原因：P_full 随轮次递增，而 P_mem 相对稳定，导致后期轮次系统性地获得更负的 raw reward，归一化后反而给早期轮次正信号、晚期轮次负信号。这是 MemPO 记忆奖励的一个结构性不均衡问题。
1.9 非对称过滤的微妙之处
不均衡会导致什么学习策略？
梯度信号的方向如下：
policy gradient: ∀θ log π(τ) × advantage
advantage > 0 → 增加这个 token 序列的生成概率
advantage < 0 → 降低这个 token 序列的生成概率
导致实际可能出现的学习策略
Round 2 (系统性正奖励):
梯度方向：把"在 Round 2 写出高 P_mem 的记忆"的概率拉高
→ 模型学会：第2轮必须写详细、全面的记忆摘要
Round 4 <mem>(系统性负奖励)：
梯度方向：降低"在Round 4写当前这种记忆"的概率
但没有明确方向要写什么一一一只知道不要写这个
→ 可能的结果(几种情况)：
情况A：写更短的记忆(减少token，分母变小，P_mem被提高)
情况B：抄写Round 3的记忆(避免改变，维持上一轮水平)
情况C：写更模板化、稳定的格式(降低分布，提高单词概率)
反直觉的均衡策略
理论最优(适应这个不均衡信号)：
Round2<mem>：把已知的所有信息都写进去
Round3<mem>：略微补充新发现
Round4<mem>：直接复制Round 3的内容(不"升级"记忆)
原因：
- Round2升级记忆有正信号
- Round4升级记忆被系统性惩罚(因为绝对P_mem值与Round2同，但P_full更高，差值更负，在pool中排名更低)
- 所以Round4不升级记忆反而"相对表现较好"
与outcome reward 的交互
final_advantage = outcome_adv + mem_adv
即使mem_adv 鼓励Round4写简单记忆， outcome_adv会反向激励："只有写出足够好的记忆，最终答案才能正确，outcome_reward 才能高"
两种信号的博弈：mem_ady(后期负)vs outcome_adv(答对才有正)
实际均衡：模型在第4一5轮写*足够好*但不超出的记忆
"足够好”=足以让答案正确+不会在Poo1中排太低
一个更令人担忧的可能性
如果模型学到："第2轮把所有信息写进<mem>，后面几轮直接用<answer>输出" → 利用 max_assistant_turns 限制，尽量早结束 → Round2写全，Round3直接答题
效果：逃避了后期的 mem_adv负惩罚！
但：需要 Round 2的搜索信息足够
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
[Agent Memory / Enhanced Learning] MemPO Source Learning Notes ---(7) -- Mixed Mechanism
0x00 Summary
The existing Memory management approach, based on enhanced learning, often lacks an effective mechanism to guide and optimize the updating of Memory, whose content is difficult to ensure quality.
MemPO (Self-Memory Policy Optimization) enables the model to self-manage Memory and introduces the Memory-level advantage estimate based on an effective information content, leading Memory to retain more effective information for the task, thereby enhancing memory effectiveness.
MemPO's unique entry point: let the model write the memory at the beginning of each round () in form as "self-dialogue draft paper", both as memory and as part of the chain of thought. In this way, it becomes a training policy variable, using the RL signal end-to-end model "what to remember and how to remember." RL optimizes this behaviour directly to the end without additional memory modules.
MemPO information is as follows:
- Title: MemPO: Self-Memory Policy Optimization for Long-Horizon Ages
- Address: arxiv.org/abs/2603.00...
- Code address: Github.com/TheNewBeeKi...
- Models and data sets address: hugglingface.co/collections...
0x01 Mem Reward Mechanism
1.1 Formula
The Memoory Advantage formula (in two steps):
Step 1 - Calculate mem reward (A1): mem reward k = P mem k - P full k
Of which:
P full k = exp (mean (full logp [k] x mask k)) ← Full context answer probability
P mem k = exp (means (mem logp [k] x mask k))
k = wheel of a track
Step 2 - Normalization (A2): mem adv k = (mem reward k-pol mean)/ (pol std+ε)
Of which:
Pol = all 16 tracks of the same question x mem reward of all rotations (approximately 48 values: 16 x 3 rounds/ articles)
Pol mean = mean (all 48 values)
Pol std = std (all 48 values)
Step 1 meaning dismantling:
P mem - P Full (Step 1): "Only the probability that notes will answer right -- the probability that complete history will answer right."
> 0: Notes are more focused on the original, good notes, positive awards
= 0: Note unlost compression
< 0: The notes are missing.
Intuitive: if you condensed a book into a page of notes, you answer with notes, which is better than a full book, which means you got the point.
Step 2 Dismantling:
(mem_reward-pool_mean)/ pool_std (Step 2):
"How good is my round of notes better than everyone's average?"
Pool contains mem reward of all tracks under the same Question
→ Cross-track: notes from different search paths compared to each other
:: Cross-cycle: round 2 and round 4 are also compared
• Normalization: producing standardized positive and negative signals
Writing location: mem adv[i,start t:end t+1]=adv
"Only these token positions."
The rest of the position = 0
Only token, who writes notes, received a gradient signal of the quality of notes.
1.2 Full computing details for P mem and P full
Step 1: Build three types of input (tool agent loop.py)
Step 2: Structure of ans mask (precision tag answer token)
Step 3: One-time batch calculation log prob
# Merge full traj and mem traj into one big watch (reduced GPu call times)
traj_input = tokenizer.pad(
{"input_ids": concat_full_traj_ids_list + concat_mem_traj_ids_list},
return_tensors="pt",
padding="longest",
return_attention_mask=True,
padding_side="left",
)
answer_input = tokenizer.pad(
{"input_ids": concat_answer_ids_list},
return_tensors="pt",
padding="longest",
return_attention_mask=True,
padding_side="left",
)
ans_mask_tensor = pad_sequence([torch.tensor(sub) for sub in concat_ans_mask_list], batch_first=True, padding_value=0, padding_side="left")
# returns (token for log prob) = answer tokens, repeat 2 full+mem
traj_batch = DataProto.from_dict(
{
"responses": answer input ["input ids"].repat(2,1), #[Full +mem group]
"input_ids": traj_input["input_ids"],
"attention_mask": traj_input["attention_mask"],
"position_ids": (traj_input["attention_mask"].cumsum(dim=1) - 1).clamp(min=0),
},
)
traj_log_probs = self.worker_group.compute_log_prob(traj_batch)
# Returns logP (token|text) for each answerr token under context
Step 4: Threshold filter (core project details), only 50% of model confidence
Step 5: Calculating probability and producing mem reward
# P Full: The geometrical probability of the model to the answer under complete context
P_full = exp(
sum(logp[full_ans_mask]) / (count(full_ans_mask) + 1e-8)
)
# P mem: based on <mem>context only, the geometrical average probability of the answer
P_mem = exp(
sum(logp[mem_ans_mask]) / (count(mem_ans_mask) + 1e-8)
)
# Memory incentive #
Mem reward=P mem-P full#approach[1,+1]
A concrete example can be seen.
Subject: Who is the mayor of the city of Einstein?
Round 1:
"Question 1: Einstein was born in Ulm. Question 2: The mayor is unknown and has to continue his inquiries."
Full context → P full
Only <mem>context →P mem(answer= "Urum; to some extent")= 0.12
Mem reward[0]=0.12 -0.35 =-0.23 ← Incomplete memory, deduction
Round 2:
"The mayor of Ulm is Zhang San. Two questions have been answered: Ulm; Zhang San."
Full text context → P full=0.68
Only <mem>context→P mem=0.71 (<mem> more refined than full text!)
Mem reward[1] = 0.71-0.68 = +0.03 ← full memory, slight points
1.3 Why not just P mem?
This is a critical design decision, and the logic behind it is to eliminate the impact of "the answer itself." In other words, P full is the baseline (baseline) that strips out the element of "the inherent difficulty/easyness" so that the reward signal reflects only the quality of the memory itself, not the difficulty of the subject. This is assessed as the same idea as the GRPO's "group average" as a baseline, with relative points, not absolute points.
1.3.1 Question: If P mem is used only
Question A (simple): "Where is the French capital?
Answer Z is "Paris."
P mem = 0.95 ← high
Question B (difficult): "What is the specific rate of tariffs imposed by the United States in the context of the Central American trade war in 2018?
Answer Z: 25%.
P mem ("25%" <mem> trade war duty 25% </mem>) = 0.65 ← low
Rating by P mem:
Model A: Simple questions 0.95 awards High
Model B: Difficulty 0.65 → Low reward
But that's not fair. Even if the puzzle is written well, the P mem is low, and the model learns that the answer to a simple question is more "calculated" than the answer to the question, and the bias in favour of simple questions.
1.3.2 Solutions: use the margin P mem-P fult as baseline correction
P Full = P (Z| context) This is the probability of an answer in the best case.
P mem - P Full = "The capacity of memory alone" versus "the ability to see complete context" = the loss of information caused by memory (negative numbers mean the loss, the closer to zero)
The effect of the margin P mem-P full can be seen in the following questions.
Question A (simple):
P full = 0.95.
P mem=0.93.
The margin = -0.02 → little loss of memory, slight negative reward
Question B (difficulties):
P Full = 0.70.
P mem = 0.65.
The margin = -0.05 →.
Question B (remembering poor):
P full = 0.70.
P mem = 0.20.
The margin = -0.50.
1.3.3 Intuitive analogy
It's like a test score:
Use P mem alone (absolute):
A simple answer of 90 points. High
The problem is 65 points.
Use P mem-P ful (relative):
Simplicity: 95 points, you answered 93 points.
Problem: A maximum of 70 points. You answered 65 points. Bad
Problem: A maximum of 70 points, you answered 20 points. I'm not sure.
The assessment is, "How much memory is left of your ability?"
1.4 Meaning of P mem - P full
Core issue: Is summary "good enough" to replace full multi-cycle dialogue context?
Starting with this core issue, let's look at what P mem-P Full intuitively means.
1.4.1 Direct meaning
The trajectory is as follows:
- full traj = complete context before current wheel generation (system + full dialogue before Question +), increasing length
-Mem traj = mem sys prompt ids (system + question) + content of current round summary, very short
Both are asked to predict the same answer:
- P full = the probability that the model will predict the correct answer based on a complete context
- P mem = model based only on summary predicting the probability of correct answers
Three scenarios:
Therefore, the intuitive meaning is as follows:
- If you read the whole of a book, the VS only reads summary notes, who better answers the questions?
- If the notes are of high quality, they may be more accurate than a full book.
1.4.2 More precise mathematical meaning
P mem - P Full ≈ Measuring "Memory Loss of Information"
- If P mem ≈ P fu1l→ ≈ 0→ contains all the information needed to answer the question → 奖励 = 0 → (neutral, good enough)
- If P mem < P Full is negative, missing important information, punishment token, prompting model writing to better memory
- If the P mem > P full margin is positive (frequently) → more accurately suggests the answer than complete context → → → 奖励 奖励 奖励 k k k Z
1.4.3 Credit distribution
By integrating writing memory into GRPO ' s credit allocation chain, the MemPO-driven model learns what is worth retaining (the memory reward is positive and encouraging) and what is irrelevant should be omitted (negative punishment). Let the model know which intermediate information is worth taking over the context budget and which noises should be voluntarily discarded. Such a mechanism essentially alleviates the problem of the context explosion in the long-range multi-wheel interaction and forces the model to learn to compress key information in the current cycle in a limited space rather than relying on an entire history of unlimited growth.
However, MemPO is currently optimising "information adequacy" rather than "condensed efficiency" and long-range credit distribution across rotations continues to rely on end-result rewards for indirect transmission rather than multi-step assessments of each memory."
Credit distribution links
For example:
Standard RL: Incentive r →reverse /<action> token
MemPO: r + mem reward
└─ mem_reward = P_mem - P_full
"The value of this memory is clearly measured."
It does enter the credit distribution chain and is no longer an "accompanied text".
Effective information
How can the model know what information is worth taking over the context budget”?
- At the realization level: MemPO does not have a visible model "context budget" (token length limit). Its memory rewards P mem-P full to measure "information adequacy", not "compression efficiency".
- Actual results: P mem reward is still encouraged if it is long and well-written. MemPO encourages the retention of information, not the efficient compression.** The real "context Budget Perception" requires an additional penalty of length, which is not in the current code.
Drop Noise
"What noises should be voluntarily discarded" is actually a mechanism for hidden realization:
The <mem> of t-turn wrote unconnected messages:
Mem traj[t]=[systematic prompt > n/mem>]
P mem predicts that the answer is low.
Mem reward[t]=P mem-P fu1l < 0 ←
♪ This token of 'em gets a negative gradient ♪
The model gradually learns to "write information, not noise."
But the "discard" here is indirectly guided by a negative gradient, not by a model that clearly judges "the message is not important." The difference is:
- MemPO's memory behavior is passive reward/punishment.
- The real "scrambling decision" needs to be as visible as MEM1.
Long-range interaction.
The solution to the long-range intersections is as follows:
- context Long Explosion: AgentMemoory.prepare prompt() preserves only the results of the latest round of tools, forcing the model to pass the message
- Credit distribution to intermediate behavior.
Not fully resolved
Long-Range Credit Allocation: The contribution of Round 1 to Round 6 answers cannot be measured accurately by the current P mem-P Full (reading only the immediate answer predictive capacity of the Round) - Rollout Round 8 in training, but it is open to extrapolation to longer rounds of stability.
1.4.4 Scenario 3
We're going to look at the design of scene three, Wrong but Good Memory.
Wrong answer + good mem → (-outcome adv) + mem adv
<mem> sector:-outcome+ (+mem adv)
Other token:-outcome
Why protect the situation of "mistakes but remembers well"? Will Token be protected or punished? Why is it designed like this?
- Skills learning: two different skills are written and ultimately answered. Models may:
- Very good.
- But there was a logical error.
If mem is good and is punished for something else, the model will "lost" memory of summary.
- Long-term credit allocation: it is written at the beginning of rounds 2, 3 and 4 and the answer may come only in round 5. Without the protection of mem adv, good memory strategies will be diluted in long trajectories without proper attribution.
- Preventing degradation: If the model finds that "the answer is wrong," it might simply be empty to avoid any risk. Mem adv ensures that even if the answer is wrong, the good memory strategy is a positive incentive.
- Partial protection. Inter-sectional advantage = (-outcome)+ (+mem adv) with partial punishment. Reasons for design: Writing and eventually answering correctly are two separate abilities, and if a good memory strategy is punished because the reasoning is wrong, the model "does" memory summary ability. The conditional version * (advantages.sum()>=0), which was quoted in the code, chose not to protect the situation and was ultimately not used by the author.
Note: The code contains an annotated conditional version:
# Meaning: em advantages + mem advantages * (advantages.sum() > = 0) #
This version chooses not to protect scene 3, and the author eventually did not use it to show that the experiment proved that protecting "mim" was a better strategy.
1.5 baseline
MemPO training includes two baselines (for two types of advantage):
1.5.1 Outcome Advantage Baseline
Baseline = average of 16 tracks of the same question
Example: 16 tracks of Question Q1:
scores = [1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
baseline = mean = 0.625
adv_i = (score_i - 0.625) /std
→ Correct: positive advantage
→ Wrong answer: negative advantage
1.5.2 Baseline of Memoory Advantage
Baseline = equal to the question of mem reward for all tracks
Example: Qustion Q1, 16 tracks per 3 wheel with <mem>:
Pool = [r 11, r 12, r 13, x-ray 1 wheel
r 21, r 22, r 23, 3-wheel of ←2
...
r 16 1, r 16 2, r 16 3] About 48 values
Baseline = mean (48 values)
adv_k = (mem_reward_k - baseline) /std
Mem: positive advantage
Mem: negative average
Baseline for comparison PPO
Standard PPO: baseline = V (s t) ← value estimated by the Creetic network
GRPO/MemPO: baseline = group mean ← statistics
Good and bad:
- Critic: Accurate to each token position, but with estimated errors, additional training is required
-group mean: One global value, but zero error, zero extra expenses
One sentence: MemPO's baseline is "the average of the same group" -- without extra model estimates, directly from the data.
1.6 Equivalence
The question here is: is P full and the training baseline mathematically equivalent? Because in P Full, there must have been a partial compression of memories.
1.6.1 Difference between P full and GRPO baseline
Full traj contains the following, and it can be seen that the "full context" in P Full itself already contains compressions from previous rounds. It's not "pure original search results."
Full traj = complete trajectory token sequence:
System Hint
+ Original problem
+ Round 1 Model Output: "My Memory 1 < /mem > <think >... <search > < /search >
It already contains the model itself!
+ Round1 Search Results
+ Round 2 Model Output: "My Memory 2</mem><think>...<search>...</search"
It's got "mem" on Rund2.
+ Round 2 Search Results
+ Round T model output (final answer Z)
1.6.2 P full as baseline against GRPO mean
GRPO baseline (average group):
The same question generates 16 tracks {T1, T2,..., T16)
baseline = mean(ri, r2,..., r16)
→ Elimination of the "almost difficult question" factor
P full baseline (upside of this track):
Baseline =P (full context for Z丨 track)
{\cHFFFFFF}{\cH00FFFF}{\cH00FF00} The removal of the "almost difficult track" factor (including the quality of the previous <mem>)
The price difference between the two is illustrated by an example:
Same question, two tracks:
Track A: The previous rounds <mem> were well written.
Track B: The previous rounds <mem> were poorly written.
GRPO baseline=mean
P full basee, same.
This leads to a delicate question.
Mem reward=P mem (Current Wheel)-P Full (Current Track)
P Full contains "Quality in previous rounds" as baseline
Scene: The first rounds were good, but the current ones are bad.
P Full High
P mem low (less quality of <mem> on current round)
Mem reward = Low-High = Large Negative
Scene: The previous rounds were bad. The current rounds are slightly better.
P full Low
P mem slightly above P full
Mem reward = slightly positive reward for the current wheel
So, the actual measure of mem reward is: "Can the current round provide sufficient answers independently of the complete context that contains the pre-sequent memory?"
1.6.3 Relationship to GRPO double baseline
Final advantage = GRPO advantage + mem advantage
GRPO advantage.(outcome baseline):
Baseline = average incentive for 16 tracks of the same group
Target: Is the entire track better than the same group average?
Mym advantage (memory baseline):
Baseline = P ful (full context) of this track
Target: Does the current round <mem> retain sufficient information?
Two floors straight, each eliminating different sources:
GRPO baseline → Easier to eliminate the problem
P fullbaseline → context to eliminate the current trajectory
1.6.4 A potential problem
Because P Full contains a pre-order, there is a situation:
If the model learns to write "all messages in Round 1":
Round 1 < mem> = super-detailed notes
Round 2 ~ T<mem> = short or even blank
So:
P Full still High
P mem is low.
Mem reward = big negative punishment for this pre-memories strategy.
MemPO actually asks for:
"This round of <mem's > must contain enough information to answer the final question independently, not the accumulation of <mem's"
This is an important feature of MemPO memory awards: It motivates the model to write an independent and communicative "full memory of summary" on each round, rather than spreading information over multiple rounds. This may also be a potential problem.
1.6.5 Annotated version of the conditions
What are the advantages and disadvantages of the annotated version of the condition?mem adv* (outcome adv.sum()>=0?
Advantages: Prevent the wrong trajectory from offsetting the punitive signals of outcome by writing. If it's wrong, but it's written well, the advantage will not be positive, and the entire trajectory will be suppressed in a consistent manner.
Disadvantage: The wrong track is completely free of mem reward. Models can't learn that "though ultimately the answer is wrong, this memory strategy is good," which may slow down memory learning. The author ultimately chose to add an unconditional superscript to show that the experiment shows that the protection of the mem is more important.
1.7 Continuous value
Why is Memory Reward designed to be a continuous value (-1,+1) instead of a discrete {0,’)? The reasons are as follows:
- Richr gradient signal: the discrete value can only say "good/bad." The continuous value can express "good/bad"
- Better integration: more stable statistics of mean/std for the same group ~48 continuous values
- Definition of nature: P mem-P Full is low probability and natural is continuous
- Avoiding artificial thresholds: If dispersed, a broad value of "what good mem" needs to be set, while continuous values allow for the integration of GRPO into automatic processing
1.8 Mem reward Incentive intensity analysis for different rounds
1.8.2 P full Effect
Round 1 has no mem reward signal at all
Reason: No historical search results, full traj and mem traj are similar and meaningless
After that, each round of P ful will grow naturally.
- Round 2 full traj = initial prompt+R1 output+search1 result → P full lower (only 1 round of information)
- Round 3 full traj = initial prompt+R1 + search1 + R2 + search2 → P full medium (2 rounds of information accumulation)
- Round 4 full traj =... +R3+search3 → P full higher (three rounds of information accumulation)
This leads to mem reward=P mem-P full growing negative with the wheel.
1.8.2 Normalization
Let's take a look at the actual effects of integration.
Assumptions: 16 trajectories, 4 rounds each, average for each wheel Raw mem reward:
- Round 2：-0.10
- Round 3：-0.22
- Round 4：-0.33
After imitation (mean=0.22, std=0.13):
-Rund 2: Normalized mean + 0.90 positive advantage (encouraging improvement)
- Round 3: Normalize average values ≈-0.03 → neutral
- Round 4: Normalized average ≈-0.87 negative advantage
1.8.3 Core issues
Core issue: Inconsistent signal intensity.
From a model learning perspective:
- Round 2: Systems get positive rewards Okay.
- Round 4: Systemic negative rewards
But it's not the model that really "writes poorly" -- it's the P full that grows with the number of wheels!
Nature: Normalization is judged by "relative performance over time" rather than "absolute memory quality".
1.8.4 Summary
The systemic differences in incentive intensity are as follows:
The underlying cause: P Full increases with the wheel, while P mem is relatively stable, leading to a more negative raw reward in the post-production round, which, when standardized, gives the early positive and later negative signals. This is a structural imbalance in the MemPO memory reward.
1.9 Asymmetric filter subtleties
What learning strategies would an imbalance lead to?
The direction of the gradient signal is as follows:
policy gradient: ∀θ log π(τ) × advantage
add the probability of generating this token sequence
Decreasing the probability of generating this token sequence
A learning strategy that leads to a real chance.
Round 2 (systemic positive reward):
Gradient: Pulls the probability of writing high P mem memories in Round 2 High
Modelling Institute: Round 2 must write detailed and comprehensive memories of summary
Round 4 <mem> (systematic negative reward):
Gradient: Reduce the probability of writing this memory in Round 4
But I don't know what to write about.
Possible outcomes (in several cases):
Scenario A: Write shorter memories (reduce token, denominator smaller, P mem enhanced)
Scenario B: Copy the memory of Round 3 (avoiding change, maintaining previous round level)
Scenario C: more templateized and stable format (reduced distribution and increase the probability of words)
Counter-intuitive balanced strategy
Theories are the best.
Round2Mem: Write in everything you know.
Round3 <mem>: minor additions to new discoveries
Round4 <mem>: Direct copy of Round 3
Reason:
- Round2 upgrades have positive signals.
- Rund4 upgrades are systematically punished (because the absolute P mem is the same as Rund2, but the P Full is higher, the margin is more negative and ranks lower in the Pool)
- So Rund4's "relatively good" instead of upgrading his memory.
Interactive with outcome reward
final_advantage = outcome_adv + mem_adv
Even if mem adv encourages Rund4 to write simple memories, outcome adv is the reverse incentive: "Only if you write good memories, the answer is correct, outsidecome reward is high."
Game of two signals: mem ady (post-production negative) vs outcome adv
Actual balance: Models written in round 4-5* are good enough * but not beyond memory
"Good enough" = enough to get the answer right + not too low in Poo1
A more worrying possibility.
If the model learns, "Book 2 puts all the information in <mem', and the back rounds go directly to <answer' output" → Use the max assant turns limit, and finish as early as possible →Rund2 complete, Round3 answer
Effect: Mem adv who escaped post-production is punished!
However, the search information for Round 2 is sufficient.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.