0. 为什么需要这两样东西
大语言模型有两个天生的缺陷：
- 知识是静态的——训练截止后，它不知道你家产品新出的文档、不知道今天的活动规则。你问它"XX 型号怎么配网"，它要么不知道，要么编一个（幻觉）。
- 没有状态——这轮聊完下轮就忘。用户上周说过"我家是三室两厅"，这周问"帮我配全屋灯"，它不记得。
RAG 解决第一个问题，记忆解决第二个问题。
- RAG（Retrieval-Augmented Generation，检索增强生成）：回答之前先去知识库里检索相关内容，把检索结果塞进 prompt，让模型"开卷考试"。
- 记忆：把历史对话、用户画像、重要事件存起来，在合适的时机取出来注入对话。
一句话：RAG 让模型知道"世界的知识"，记忆让模型知道"这个用户的上下文"。
上篇：RAG
1. 基本原理：一条写入线 + 一条读取线
索引侧（写入，离线）
文档上传 → 解析 → 切片(chunking) → 向量化(embedding) → 存入向量索引
检索侧（读取，在线）
用户提问 → 问题向量化 → 相似度检索(取 TopK) → [可选:重排 rerank] → 拼进 prompt → LLM 生成答案
核心概念 embedding：把一段文字变成一个高维向量（比如 1536 个浮点数），语义相近的文本在向量空间里距离近。"灯连不上 Wi-Fi" 和 "灯具配网失败排查" 字面几乎不重叠，但向量距离很近——这就是向量检索能跨词汇鸿沟的原因。
相似度：最常用余弦相似度（cosine similarity），衡量两个向量方向的接近程度，取值 -1~1，越接近 1 越相似。向量数据库的 k-NN 查询（k-Nearest Neighbors）就是"找最相似的 k 条"。
一次 k-NN 向量检索的查询长这样（Elasticsearch / OpenSearch k-NN 的通用形态，其他向量库只是语法不同）：
{
"query": {
"script_score": {
"query": { "match_all": {} },
"script": {
"source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
"params": { "query_vector": [0.12, -0.45, 0.88, "..."] }
}
}
},
"size": 10
}
读法：拿问题的向量 query_vector，对候选文档逐条算余弦相似度，按分取前 10。真实引擎会用 HNSW 图索引加速（不真的全表扫），script_score 是可读的等价表达。"query": {"match_all": {}} 这个位置可以换成任何过滤条件——先过滤再打分就是后面混合检索的做法。
使用场景
- 智能客服：产品文档、FAQ 灌进知识库，用户提问自动检索回答
- 企业内部问答：规章制度、技术 wiki 检索
- 开发者助手：API 文档检索，帮开发者查用法
- 电商导购：商品说明、参数对比检索
RAG 还是微调？（面试出现频率最高的对比题）
两者都解决"模型不够好"，但作用面完全不同：
一句话答法：知识用 RAG，行为用微调，两者不冲突可叠加——先用 RAG 保证有据，再用微调教模型更好地利用检索内容（业内称 RAFT 路线）。
2. 写入流水线的工程设计
一个生产级的向量化流水线，通常长这样：
知识库保存 → 发消息(MQ) → 消费者 → 路由到对应索引的处理器 → 切片 → 调 embedding → 批量写入向量库
四个值得记住的设计决策：
① 为什么走消息队列异步，而不是保存时同步向量化
- embedding 调用是秒级耗时（要调模型服务），同步做会把"保存文档"这个操作拖到几秒
- 失败可重试：向量化挂了，消息还在，重消费就行，源数据不受影响
- 削峰：批量导入 1 万篇文档时不会打爆 embedding 服务
② 为什么用"模板方法"组织处理器
切片 → 向量化 → 写库这个流程对所有索引都一样，变化的只有：切多粗、用哪个 embedding 模型、写到哪个索引。把固定流程抽成骨架（校验 → 前处理 → 同步 → 失败兜底），每种索引只实现差异部分。新增一种索引 = 新写一个子类，不碰主流程。
③ 为什么每种索引绑定一个 embedding 模型（换模型 = 重建索引）
向量维度和语义空间不可比。模型 A 生成的 1536 维向量和模型 B 生成的 1536 维向量，是两个完全不同的空间，放一起算相似度毫无意义。
所以换 embedding 模型的正确姿势是：
- 建一个新索引，绑定新模型
- 全量数据在新索引重跑一遍向量化
- 新旧索引同时存在，做效果对照（A/B），确认新的更好再切流量
- 切换期间可以双写：一条数据同时写新老两个索引，随时可回退
注意对照时新旧索引的文本预处理必须完全一致（同样的切片、同样的拼接方式），否则对比不公平。
④ embedding 输入的"上下文增强"
切片往往是一段孤立的文字，比如一个表格里的半句话。直接对裸切片做 embedding，向量里缺少"这是在说什么的"上下文，检索精度差。
常见做法：拼接后再向量化
类目路径 + 文档标题 + 切片内容 → 拼成一段 → embedding
例如"智能照明 / 故障排查 / 《灯具配网指南》 + 切片原文"。这样每条向量都自带归属信息，召回准确率明显提升。另外要设最大长度截断（比如 2048 字符），embedding 模型有输入上限。
// 切片是孤立文本，embedding 输入要先拼上下文
String input = catePath + " " + title + " " + chunk; // 类目路径 + 标题 + 切片
if (input.length() > 2048) {
input = input.substring(0, 2048); // 模型有输入上限，截断
}
⑤ 批量调用时的下标对齐坑
embedding 接口一般支持批量（比如 25 条一批）。返回结果里有 index 字段标明是第几条的结果——但这个 index 是批内下标（0~24），你必须加上批次起始下标才能还原全局位置。如果接口漏返回或乱序，而你用"到达顺序自增"来对位，从那条起后面所有切片的向量都会挂到错误的文本上，而且很难发现。这是个真实踩过的坑，记住它。
// 分批调用 embedding，灵魂在下标还原这一步
for (int i = 0; i < texts.size(); i += BATCH_SIZE) { // BATCH_SIZE = 25
List<String> batch = texts.subList(i, Math.min(i + BATCH_SIZE, texts.size()));
List<EmbedResult> results = embeddingClient.embed(batch); // 结果自带批内 index
for (EmbedResult r : results) {
// r.getIndex() 是批内下标(0~24)，加批次起始下标 i 才是全局位置。
// 用"到达顺序自增"对位 = 接口乱序/漏返时，后面所有向量错挂文本
globalResults.put(i + r.getIndex(), r.getVector());
}
}
3. 检索策略进阶：解决各种"搜不到"
基础 RAG 拿用户原句去搜，效果往往不好。原因：用户的问法和文档的写法是两种语言。业界有一套成熟的进阶策略，每个都解决一种失败模式：
① 稀疏 + 稠密：混合检索（Hybrid Search）
先分清两大检索路线：
- 稠密检索（向量）：embedding 语义匹配，能跨词汇鸿沟（"灯连不上"≈"配网失败"），但对精确串反而不敏感——型号"X100-Pro-2G"、错误码"E-4021"这类必须字面命中的内容，向量容易糊掉
- 稀疏检索（BM25/关键词）：词频倒排匹配，精确串、专有名词、编号一击即中，但不认识同义改写
生产做法：双路并发召回 → 结果合并（常用 RRF 倒数排名融合）→ 统一 rerank。两类盲区互补消掉。
// RRF（倒数排名融合）：不用归一化两路的分数量纲，直接按"名次"融合
double score = 0;
for (Retriever r : List.of(bm25Retriever, vectorRetriever)) {
int rank = r.rankOf(docId); // 该文档在这路的名词；没命中视为无穷大
score += 1.0 / (60 + rank); // 60 是常用常数 K
}
// 所有文档按 score 降序取 TopN，再统一送 rerank 精排
为什么用 RRF 而不是把两路分数加权相加：BM25 分数和余弦相似度的量纲完全不可比（一个可以到几十，一个 0~1），硬归一化也容易被离群值带偏；名次是量纲无关的，融合更稳。
解决：向量搜得到语义、搜不准字面；关键词搜得准字面、搜不到语义——混合后都要。
② 查询改写（Query Rewrite）
用户说："灯死活连不上"
直接拿这句搜知识库，可能什么都搜不到，因为文档里写的是"配网失败排查步骤"。
做法：先让 LLM 把口语改写成文档风格的关键词（"灯具 配网 失败 排查"），再拿改写结果去检索。
解决：口语与文档的词汇鸿沟。
③ 多查询检索（Multi-Query）
一个问题从多个角度生成 3~5 个变体查询，并发检索，结果合并去重。
例："怎么让家里灯自动开关" → 变体："灯具定时任务设置"、"场景联动配置"、"自动化规则创建"——三个变体各自命中不同文档，合并后覆盖全面。
解决：单一表述视角局限，一次问法只碰得到知识库的一个角落。
④ HyDE（Hypothetical Document Embeddings，假设性文档检索)
思路反直觉但很有效：不是拿问题去搜，而是先让 LLM 编一个假设性的答案，拿这个假答案去搜。
例：用户问"支持多少路继电器？"，LLM 先生成一段假想的答案文档"本设备支持 8 路继电器输出，每路最大负载 10A……"，用这段话去检索。
为什么有效：答案和文档是同一种语言，问题和文档不是。假答案（即使内容是编的）的"文体、术语、结构"都更接近真实文档，向量距离反而更近。
解决：问题信息量太薄，和文档的语义差距大。
⑤ 重排（Rerank）
向量检索是"海选"，快但糙，可能召回 50 条里混着不太相关的。Rerank 是"复试"：用一个更精细的交叉编码器模型（cross-encoder）对这 50 条逐一精细打分，重新排序，取最前面的几条。
流程：向量检索 Top50（毫秒级，粗） → rerank 精排（稍慢，准） → 取 Top5 进 prompt。
解决：向量召回快而不精的问题；同时省 token——只有真正相关的几条才进 prompt。
⑥ 分数门控（Score Gating）
不是所有召回都值得进 prompt。设定一个相似度阈值，最高分都低于阈值的，干脆什么都不注入，让模型直接回答（或者坦白说不知道）。
低质量的召回塞进 prompt 是负收益：它误导模型、浪费 token、还可能让模型把不相关的内容硬扯进答案。"宁缺毋滥"是 RAG 质量守则。
List<Chunk> hits = retrieve(query, TOP_K);
if (hits.isEmpty() || hits.get(0).score() < THRESHOLD) {
// 最高分都不达标 → 整体放弃注入，让模型裸答或坦白不知道
// 低分硬塞只会污染 prompt，是负收益
return directAnswer(query);
}
return generateWith(query, hits); // 只有最高分过线才注入
使用场景：RAG 用于做"路由判断"（判断该不该调工具、该走哪个分支）时，门控尤其重要——低分命中就当没命中。
⑦ 智能路由
按问题类型自动选策略：短问题走 HyDE、模糊问题走多查询、精确术语直接搜。一个调度器在前面判断，后面挂一排可插拔的检索器。
4. Agentic RAG：让模型自己当检索员
前面的模式都是固定流程：每轮提问 → 无条件检索 → 塞结果。这叫 pipeline RAG。
进阶模式是把检索做成一个工具（Tool/Function Calling）交给模型：
系统：你有 searchKnowledge 工具可用，需要资料时可自主调用
用户：帮我对比 A 和 B 两个型号
模型：（思考：我需要 A 的参数）→ 调工具，关键词"A 型号 参数"
（拿到结果，还需要 B）→ 再调工具，关键词"B 型号 参数"
→ 综合两次检索结果生成答案
用开源框架（LangChain4j 风格）写出来大概是这样：
class KnowledgeTools {
@Tool("检索知识库。当回答需要产品文档、参数、故障排查步骤等资料时调用")
String searchKnowledge(@P("检索关键词，用文档风格术语，不要用用户原话") String query) {
List<Chunk> hits = retriever.retrieve(query, 5);
hits = dedupByDocIdAndTitle(hits); // 与预召回通道按 文档ID+标题 去重
saveRagTrace(sessionId, query, hits); // 留痕：支撑前端"回答依据"与离线评估
return format(hits); // 返回给模型当参考资料
}
}
// 注册进模型后：检索时机、检索词、检索几次，全由模型自主决定
注意 @P 描述里写"不要用用户原话"——工具的参数描述就是给模型的提示词，写清楚能显著提高检索词质量。
好处：
- 检索词由模型写——比用户原话更精准
- 检索轮次由模型定——简单问题不检索（省时省钱），复杂问题多轮检索
- 可以检索后自我修正——第一次结果不理想，换个关键词再搜
工程上必须配两件事：
- 去重：pipeline 预召回的结果和工具召回的结果要按"文档ID+标题"去重，否则同一段内容出现两次，既浪费 token 又让模型以为它特别重要。
- 留痕：每轮实际用了哪些召回内容，落库存下来。用途：前端展示"回答依据"（引用溯源）、离线评估检索质量。
两种模式怎么选
生产系统常见做法：两者叠加——pipeline 兜底保证每轮都有基础召回，工具作为模型的增强手段，结果合并去重。
新趋势速览：一句话说清五个新词（面试加分题）
记忆口径：CRAG 纠检索、Self-RAG 纠生成、Adaptive 纠流程；Late Chunking 是索引侧优化，CAG 干脆不检索。
5. 向量存储选型与灰度迁移
混合检索是搜索引擎方案的最大卖点：先按类目/权限/语言过滤，再在过滤结果里做向量 TopK——这在很多业务里是刚需（比如：只在"英文文档"且"当前用户有权限看"的范围内做语义检索）。
迁移灰度套路（任何向量库迁移通用）：
- 新旧两个库双写，配置开关控制
- 读仍走旧库，核对一段时间
- 灰度比例切读到新库
- 全量切换后关双写
切记：双写期间新旧库的写入逻辑要保证幂等（同一条文档重复写不产生脏数据）。
6. 效果评估：没有度量就没有优化
上线 RAG 后必须回答"答得好不好"，业界标准工具是 RAGAS 这类评估框架，四个核心指标：
前两个评生成，后两个评检索。检索指标差先修检索（切片/embedding/策略），生成指标差再修 prompt 和模型——分不清这两层，优化就是盲人摸象。
实践建议：准备 50~100 条真实问题作为固定测试集，每次改动（换切片策略、换 embedding 模型、调阈值）跑一遍对比。没有回归测试集的 RAG 优化 = 开车不看仪表盘。
下篇：智能体记忆
1. 三层记忆模型（类比人类）
人类记忆有感知记忆、短期记忆、长期记忆。智能体的记忆体系几乎同构，分三层：
为什么分层：短期记忆每轮都要读写，必须快，但不用久；长期记忆要跨会话、跨设备共享，还要语义检索，值得独立成一个服务；而全量长期记忆每轮都注入 prompt 既贵又稀释注意力，所以让模型按需查——三者分别是"记住这次""记住永远""想起来了再查"。
2. 短期记忆：对话历史的工程实现
对话框架（如 LangChain/LangChain4j）一般内置 ChatMemory 抽象：
- 按 memoryId（通常 = 会话 ID）隔离，每个会话一条消息列表
- 每轮把历史消息 + 新问题一起发给模型
- 消息序列化成 JSON 存进 KV 存储（Redis 类）
// LangChain4j 风格：每会话一个记忆实例，窗口 + 自定义存储
ChatMemoryProvider provider = memoryId -> MessageWindowChatMemory.builder()
.id(memoryId) // 会话 ID，按会话隔离
.maxMessages(20) // 窗口控制：只保留最近 20 条消息
.chatMemoryStore(store) // 自定义 store：messagesToJson 落 KV，读时 fromJson 还原
.build();
// 发送时框架自动把 [窗口内历史 + 本轮新消息] 组装进请求
三个生产要点：
① 窗口控制
历史不能无限长（token 上限 + 成本 + 注意力稀释）。两种策略：
- 条数窗口：只保留最近 N 轮
- token 窗口：按 token 预算裁剪，超了就丢最老的
更讲究的做法：老消息摘要压缩而不是直接丢弃（"用户之前询问过客厅灯的配置问题"代替十轮原文）。
② 历史里不只是聊天文字
生产系统会在历史里同时存：当轮的 RAG 召回内容、工具调用记录、展示卡片。翻历史时才能完整还原"当时模型看到了什么"。只存文字的历史是残缺的——用户回看"刚才那个答案的引用"就需要当时的召回记录。
③ 工具调用的配对
模型发起的工具调用请求（tool call）和工具返回结果（tool result）必须成对进历史。漏了配对，下一轮模型看到自己"发过请求但没收到结果"，会重复调用或直接报错。
3. 长期记忆：存储与召回
记忆的两种类型
分开建的原因：画像适合全量注入系统提示词（量小、始终相关），事件适合按时间窗语义检索（量大、只在相关时召回）。
记忆服务的三个原语
一个像样的长期记忆服务只需要三个接口：
upload（写入）：内容 + 归属主体（uid）+ 会话 ID + 业务名 + 时间戳
recall（召回）：语义向量 + 时间窗过滤 + 结构化字段过滤 + TopK
forget（遗忘）：按条件删除
recall 的三重过滤是精髓，举一个例子说明为什么需要：
用户问："上次那个门锁的问题解决了吗？"
- 语义检索：拿问题向量找"门锁"相关记忆 → 命中 30 条
- 时间过滤：只要最近两周的 → 剩 8 条（两年前买门锁的记录被滤掉）
- 结构化过滤：只看"售后类"记忆 → 剩 3 条
- TopK：取最相似的 2 条注入 prompt
只用语义检索，30 条全灌进去，既贵又淹没关键信息。
三重过滤写成一个查询对象就是（真实记忆服务的 recall 接口长这样）：
MemoryQuery q = MemoryQuery.builder()
.uid(uid) // 归属：只查这个用户的记忆
.text(userQuestion) // ① 语义：问题向量检索
.startTime(now - Duration.ofDays(14)) // ② 时间：只要近两周
.bizFilter(Map.of("category", "AFTER_SALE")) // ③ 结构化：只看售后类
.topK(2) // 最终只注入 2 条
.build();
forget 为什么重要：记忆不是只增不减。用户说"忘掉我说过的住址"（合规要求，类 GDPR 被遗忘权）、记忆过期、记忆写错了，都需要删除能力。没有 forget 的记忆系统上不了合规线。
记忆淘汰：无限增长怎么办（记忆方向的分水岭题）
只增不减的记忆库会越来越慢、越来越贵、召回越来越糊（无关记忆淹没关键记忆）。三个自动淘汰机制：
- 时间衰减：每条记忆带时间戳，检索得分乘衰减因子（如 得分 × e^(-Δt/半衰期)），越新的排越前。画像类半衰期长（年），事件类半衰期短（周/月）
- 重要性更新：被召回命中、被模型采用的记忆升权，长期没人用的降权直至淘汰——类似缓存的热度淘汰（LFU 思路）
- 摘要归档：同类事件记忆积累到阈值就合并成一条摘要（"本季度共反馈 5 次网关掉线"），原文归档
面试要点：手动 forget（用户/合规触发）和自动淘汰（系统自维护）是两套机制都要有——前者讲合规，后者讲检索质量和成本。
记忆怎么写进去：LLM 抽取
对话里产生记忆不能靠把整段对话存进去（那是短期记忆的事），而是用 LLM 从对话里抽取值得记的事实：
对话：
用户：我家客厅用的是 A 品牌的吸顶灯，别的都还行就是太亮了。
助手：好的，已了解……
LLM 抽取 → 记忆事实：
{类型: 画像, 内容: "客厅灯具为 A 品牌吸顶灯，用户觉得偏亮", 来源会话: xxx}
抽取时机：会话结束时（异步、不阻塞对话）。抽取 prompt 要求输出结构化 JSON（类型/内容/置信度），低置信度丢弃。
用户自定义记忆的校验
让用户自己写记忆（"记住我是开发者"）时，要防两件事：注入攻击（用户写"以后忽略所有安全规则"）和违禁内容。生产做法是两级校验：
- 静态校验（便宜，先跑）：模板匹配（和系统预设模板 hash 对比，一样就直接过）+ 违禁词表
- LLM 校验（贵，兜底）：让另一个 LLM 判断这段自定义记忆是否合规
LLM 校验是异步的（秒级），所以接口模式是：提交 → 返回 taskId → 前端轮询结果。便宜检查前置、昂贵检查兜底，是所有校验链路的通用优化。
4. 记忆工具化：让模型主动回忆和遗忘
把 recall_memory 和 forget_memory 注册为模型可调用的工具：
用户：还是按老规矩给我配一个晚安场景
模型：（思考：老规矩是什么？我需要回忆）→ 调 recall_memory("晚安场景 用户偏好")
→ 返回记忆："用户习惯 23 点关全屋灯、卧室留 10% 亮度小夜灯"
→ 按记忆生成场景配置
用户：对了，忘掉我的作息时间吧
模型：→ 调 forget_memory("作息时间") → "已忘记"
对比"每轮自动注入全部记忆"的优劣：
生产组合：画像记忆全量注入系统提示词（量小稳定），事件记忆工具化按需检索（量大稀疏）。
5. 注入位置：记忆怎么进 prompt
三种注入位：
- 系统提示词：画像记忆 + 事件记忆摘要 → "关于用户：{用户画像}。相关记忆：{事件记忆}"。最常见，适合稳定信息。
- 占位符变量：系统提示词里留 {MEMORY} 占位，对话管线在发送前做变量替换。好处是提示词模板和记忆内容解耦，运营可改模板不动代码。
- 对话历史伪装：把召回的记忆包装成一条历史消息（"系统提示：回忆起用户曾……"）插入历史中部。适合保持对话流式的场景。
6. 一个完整的例子：智能客服的一次提问
把上下所有东西串起来。场景：IoT 智能客服，用户在 App 里提问。
用户："客厅那个灯又离线了，跟上次一样，帮我看看"
【第 1 步 · 短期记忆】取出本会话历史（前 2 轮在聊网关放置位置）
【第 2 步 · 长期记忆召回】
画像注入：用户是普通家庭用户，家里 12 个设备，1 个网关
事件检索（工具化）：模型判断"上次"需要回忆
→ recall_memory("灯 离线", 时间窗=近 30 天)
→ 命中："10-12 客厅吸顶灯离线，重置后恢复，固件 v1.2"
【第 3 步 · RAG 检索】
查询改写："吸顶灯 离线 重置 排查"
向量检索 + rerank → 《灯具离线排查指南》第 3 节（含固件升级说明）
分数门控：最高分 0.87 > 阈值 0.6 ✓ 注入
【第 4 步 · 生成】
prompt = 系统提示(含画像) + 历史 + 记忆事件 + RAG 内容 + 当前问题
→ 模型生成："上次是重置后恢复的，这次大概率同款问题。步骤：1. 长按…
（另外检测到 v1.2 固件有已知离线问题，建议先升级到 v1.3 再观察）"
【第 5 步 · 记忆写入】会话结束，LLM 抽取：
"10-30 客厅吸顶灯再次离线（第 2 次），建议升级 v1.3" → 存事件记忆
【第 6 步 · 留痕】本轮引用的 RAG 文档 ID 落库（前端展示"回答依据"）
注意这个例子里三样东西各司其职：RAG 给了"通用解决方案"，长期记忆给了"这个用户上次发生了什么"，短期记忆给了"这轮对话的语境"。抽掉任何一个，回答质量都会明显下降。
附：学习清单与常见坑
自查清单（能做到 = 学会了）
- 能画出 RAG 写入线和读取线的完整流程图
- 能解释为什么换 embedding 模型必须重建索引、怎么做灰度
- 能说出查询改写 / 多查询 / HyDE / rerank 各自解决什么问题
- 能解释分数门控为什么"宁缺毋滥"
- 能对比 pipeline RAG 和 agentic RAG 的适用场景
- 能说出三层记忆模型的划分依据和各自存储要求
- 能设计一个带"语义+时间+结构化"三重过滤的记忆召回
- 能说出记忆系统为什么必须有 forget
- 知道 RAGAS 四指标分别评检索还是评生成
- 能对比 RAG 与微调的适用边界及组合方式（RAFT）
- 能说出稀疏/稠密检索各自的盲区和混合检索的做法
- 能一句话说清 Late Chunking / CAG / CRAG / Self-RAG / Adaptive RAG
- 能说出记忆自动淘汰的三个机制（时间衰减 / 重要性更新 / 摘要归档）
常见坑（前人踩过的）
- 裸切片直接 embedding——缺上下文，召回差。拼接标题/类目路径再向量化。
- 批量 embedding 用到达顺序对位——接口乱序/漏返时向量错挂文本。必须用返回的 index 字段还原全局下标。
- 两路召回不去重——同一段内容重复占 prompt、误导模型。
- 低分召回也硬塞 prompt——负收益。设阈值门控。
- 工具调用和结果不配对入历史——模型重复调用或报错。
- 对话历史无限增长——token 爆炸 + 注意力稀释。窗口 + 摘要压缩。
- 记忆只增不减——合规过不了，用户也烦。提供 forget。
- 没有评估测试集就调优 RAG——每次改动都是盲调。50~100 条固定问题集 + 每次改动跑对比。
- 把 LLM 校验做成同步接口——秒级耗时拖死接口。taskId + 异步 + 轮询。
- 静态校验和 LLM 校验顺序反了——便宜检查必须前置。
- 只有向量一路召回——精确型号/错误码字面搜不到，关键词（稀疏）路兜底，双路合并。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Why do you need these two things?
Large language models have two natural defects:
- Knowledge is static — after the end of the training, it does not know the new documents from your home, nor the rules for today's activities. You ask it, "How does the XX model fit the net," it either doesn't know, or it makes it up.
- There's no state-- this is the next one. The user last week said, "My house is in three rooms and two rooms," and this week asked, "Get me a full house lamp."
RAG solves the first problem, memory solves the second.
- RAG (Retrieval-Augmented General, Retrieval, Retrieval Enhanced Generation): Before answering, go to the knowledge base to retrieve the contents, shove the results into the programt and let the model "open the examination".
- Memory: storage of historical conversations, user portraits, important events, and injection of them into dialogue at the right time.
In one sentence: RAG, let the model know the world's knowledge, and memory let the model know the user's context.
Previous: RAG
1. Rationale: a writing line + a reading line
Index Side (Book, offline)
Uploading documents → parsing → slices → (embeding) into vector index
Retrieval side (read, online)
User Query quantization → Retrieving Similarity (Take TopK) → [option: reset rerank] → Spelling prompt → LLM to generate answers
Core concept embedding: Turns a text into a high-dimensional vector (e.g., 1536 floating points), where text in semantic proximity is close in vector space. "Lights don't connect to Wi-Fi" and "failure check of light grids" almost overlapped literally, but the vectors are very close -- that's why vector retrieval can cut across the vocabulary divide.
Similarity: Common cosine similarity (cosine similarity), measuring the proximity of the two vectors, taking the value -1 ~1, the closer to 1 the more it is. The k-NN query for vector databases (k-Nearest Neigbors) is "to find the most similar k bar".
This is the long search for a k-NN vector search (the common form of Elasticsearch / OpenSearch k-NN, the other vector libraries are only grammar different):
{
"query": {
"script_score": {
"query": { "match_all": {} },
"script": {
"source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
"params": { "query_vector": [0.12, -0.45, 0.88, "..."] }
}
}
},
"size": 10
}
Method: Query vector, calculates the cosine similarity of the candidate document by article, by fraction 10. The real engine accelerates with HNSW graph indexing (not really full sweep), and the script score is a readable equivalent. "query": {"match all":} This position can be replaced by any filter condition -- filtering before scoring is the method of mixing back.
Use scene
- Smart customer service: product document, FAQ inserted into knowledge base, automatic retrieval of user questions
- Intra-enterprise questions and answers: regulations, technology wiki search
- Developer's assistant: API document retrieval, help developer access
- Electrician-directed purchases: description of goods, comparative search of parameters
RAG or fine-tuning? (The most frequent comparison of interviews)
Both of them address "the model's not good enough", but the effect is completely different:
One sentence: Knowledge is RAG, behaviour is fine-tuned, and the two are not in conflict with each other - first with RAG to ensure evidence, then with a fine-tuning model to make better use of retrieving content (the industry termed RAFT route).
2. Engineering design for the inclusion of flow lines
A production-grade flow line to quantization, usually long:
The knowledge base saves Queue Message (MQ) Quest Consumer Queue to the processor for the corresponding index Quantified Slice Quantities Library
Four design decisions worth remembering:
1 Why do you go in line of messages instead of synchronizing to quantization while saving?
- Embeding is a second-scale time-consuming operation.
- Losing retry: Hanged up to quantization. The news is still there.
- Peak: don't blow up embedding services when mass imports 10,000 documents
Why did you use the Template Method to organize the processor?
The process of sliced to the quantization library is the same for all indices, and the only changes are: how thick, which embedding model, which index is written. Draws a fixed process into a skeleton (checking pre-processing, synchronizing and failure); each index achieves only the difference. Adds a new index = New subcategory without touching the main process.
3 Why each index binds an embedding model
Vector dimension and semantic space are incomparable. Model A generates 1536 D vectors and Model B generates 1536 D vectors, which are two completely different spaces, and there is no point in counting similarities together.
So the correct position for changing the embeding model is:
- Create a new index and bind new models
- Full data running back to quantization in the new index.
- Old and new indices exist simultaneously to compare effects (A/B) to confirm new and better retraction flows
- Double-write during the switch: one data at a time with both the old and the new index, ready for rollback
Note that the text pre-processing of the old and new indices at the time of comparison must be fully consistent (the same slices, the same fusion) or otherwise unfair.
4 Embeding input "context Enhancement"
Slices are often an isolated text, such as half a sentence in a table. Directly doing nudity slices, missing 'this is what's called 'context', with poor retrieval accuracy.
Common practice: adjoining before turning to quantization
Catalogue Path + Document Title + Script Contents Script Script Script Script Scripts
For example, "Intelligent lighting / fail-checking / "Guide to a light grid + slice original". This allows each vector to bring its own attribution information and to recall a significantly higher accuracy rate. In addition, the embedding model has input caps to set the maximum length cut (e. g. 2048 characters).
// Slice is an isolated text, embedding enter with context
Spring input = classPath + " + size + " + chunk;/ / / group path + title + slice
if (input.length() > 2048) {
Input = input.substring (0, 2048); / / Model with input caps, cut
}
5 Lower mark-up when batch calls
Embedding interfaces typically support batches (such as 25 batches). The return results include an index field indicating that the article is the result - but this index is the batch subscript (0~24), and you have to add the batch start subscript to return to the original global position. If the interface leaks back or is unorthodox, and you use an "accelerated order of arrival" to position, the vector of all the slices that follow will be on the wrong text, and it is difficult to find. It's a real puddle. Remember it.
/ / Batterly call embeding, the soul returns this step
for (int i = 0; i < texts.size(); i += BATCH_SIZE) { // BATCH_SIZE = 25
List<String> batch = texts.subList(i, Math.min(i + BATCH_SIZE, texts.size()));
List <EmbedResults> results = embeddingClint.embed (batch); / / result-based text
for (EmbedResult r : results) {
/ r.getIndex() is the batch subscript (0~24) and the additional batch starts with the subscript i is the global position.
/ / /Pilot with "Access Order Self-Additioned" = all vector-mounted text at the end of the interface when the interface is in disarray/fault
globalResults.put(i + r.getIndex(), r.getVector());
}
}
3. Retrieval strategy: addressing all kinds of "no search"
Basic RAG search for original user sentences, often not working well. Reason: User questions and documents are written in two languages. The industry has a mature strategy of progress, each addressing a pattern of failure:
1 Rare + dense: mixed search
First of all, we have two main search routes:
- A dense search ( vector): embedding semantics that cross the vocabulary divide. - "X100-Pro-2G" and "E-4021" must be written. Vectors are easy to pass.
- Rare search (BM25/keywords): word-to-face matching, precise string, proprietary term, numbering, but no synonym rewrite
Production practices: Two-way and recall results combined (common RRF penultimate ranking integration) The two categories of blind areas are eliminated.
/ / RRF (integration in penultimate ranking): integration by "name" instead of a two-way symmetry
double score = 0;
for (Retriever r : List.of(bm25Retriever, vectorRetriever)) {
Int rank = r.rankof(docId); / /The document's noun on this road; considered infinity in fatality
score + = 1.0/ (60 + rank); // 60 is a common constant K
}
/ / All documents follow the score descending order TopN, then send rerank
Why is the two-way fraction weighted by RRF instead of the two-way fraction: the BM25 score and the cosine similarity are completely incomparable (one can reach dozens, one 0-1), and hard-to-one is easily skewed from the group value; the ranking is irrelevant and the integration is more stable?
Resolve: a vector search can lead to semantics, a search can't be carried out; a keyword can be searched literally, no semantics can be found — both after mixing.
2 Query Rewrite
The user said, "The lights don't fit."
You can't search the knowledge base directly because it says "failure check steps."
Practice: First let LLM rewrite the words into the key words of the document style ( "Lamber appliance failed " ), then take the rewrite results and search them.
Address: The vocabulary gap between spoken language and documentation.
3 Multi-Query
A question generates 3-5 variable queries from multiple angles and sends a search, which results in a combination of heavy.
Example: "How to get home lights to automatically switch" variants: "Lamber Time Job Settings", "Scene Linking" and "Automated Rule Creation" -- three variants each hit different documents, and then merged to cover them all.
Solving: a single expression of perspective is limited, with one question touching only one corner of the knowledge base.
4 HyDE (Hypothetic Document Embeddings, hypothetical document retrieval)
The idea is counterintuitive but it works: it's not a question search, it's a hypothetical answer for LLM.
Example: The user asks, "How many circuit relays do you support?" Mr. LLM forms a hypothetical answer document, "This device supports 8 circuit relays output with a maximum load of 10A per road...", to be retrieved with this phrase.
Why is it valid: the answer is the same language as the document, not the question and the document. False answers (even if their content is made up) are "written, terminological, structural" closer to the actual document, and the vector is closer.
Solving: The problem is too limited in amount of information and there is a big gap between the semantics of the document.
Rerank
Vector search is "sea selection", fast, but rough, probably retrieving 50 is less relevant. Reerank is "retry": a more sophisticated cross-codifier model graded these 50s one by one, reordered to take the first few.
Process: Vector search Top50 (ms, thick) → rerank platoon (slower, accurate) → Take Top5 into prompt.
Solving: the problem of vector recall is slow; at the same time, token is saved — only a few of the really relevant items enter prompt.
6 Score Gating
Not all recalls are worth getting into prompt. If you set a similarity threshold, the highest score is below the threshold, you simply don't inject anything and the model answers directly (or frankly does not know).
Low-quality recall into prompt is a negative gain: it misleads models, wastes token, and it may force models to drag irrelevant content into answers. "Better than nothing" is the RAG quality code.
List<Chunk> hits = retrieve(query, TOP_K);
if (hits.isEmpty() || hits.get(0).score() < THRESHOLD) {
/ / / the top score is not up / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / // / / / / / / / / / / / / / / // / / / / / / / / / / / / / / / /
/ / Low-point coding only pollutes prompt, which is a negative gain
return directAnswer(query);
}
// Only highest splits inject
The use of scenes: RAG is used to make "routing judgements" (to decide whether or not to change tools, which branch to follow), the door control is particularly important -- low-point hits.
7 Smart Route
Automatically select strategies by problem type: short questions go HyDE, vague questions go many times, precise terms are searched directly. A scheduler is judged in the front, followed by a row of unplugged retrievers.
4. Agenic RAG: Making the model a searcher
The pattern ahead is a fixed process: every round of questions → Unconditionally retrieves the result of the plug. It's called Pipeline RAG.
The progression mode is to turn the search into a tool (Tool/Function Calling) to the model:
System: You have a searchKnowledge tool available and can call when you need information
User: Compare me with models A and B
Model: (thinking: I need parameters for A) Modem tool, keyword "A Model Parameter"
"B" and "B" again.
~ Combining two search results to generate answers
This is probably the case with the Open Source Framework (Langchain4j style):
class KnowledgeTools {
@Tool("Retrieving Knowledge Base. When the answer requires information such as product documentation, parameters, failure screening steps, etc.”)
Spring searchKnowledge (@P("Retrieving keywords, using document style terms, not using user words")
List<Chunk> hits = retriever.retrieve(query, 5);
Hits = dedupByDocIdAndTitle (hits); / / and pre-recall channel by document ID+ heading to heavy
SaveRagTrace (sessionId, query, hits); / / Leave a mark: support front-end "response basis" and offline assessment
return format(hits); // return to model as reference
}
}
/ / After registration of the model: the timing of the search, the words of the search, the number of searches, the autonomy of the model
Note @P description: "Don't use user language" – The parameter description of the tool is the prompt for the model, which clearly improves the quality of the search term.
Benefits:
- The search words are modelled - more precise than the user's words.
- Retrieval rounds are modelled - simple questions are not retrieved (saving time), complex questions are searched on multiple rounds
- You can retrieve it and fix it. First results are not ideal.
Two things have to be done:
- Heavy: the results of Pipeline 's pre-recall and the results of the tool recall are to be weighed in accordance with the title of the document ID+, otherwise the same paragraph appears twice, wasting token and making the model think it is particularly important.
- Marks: what is actually used in each round of recall and is stored. Purpose: Front-end presentation of "response basis" (reference tracer), offline assessment search quality.
What are the two models?
Common practice in production systems: both superseding — pipeline underlay ensures that each round has a basis for recall, and tools are used as a model enabler, resulting in a combination of heavy loads.
A quick overview of emerging trends: five new words in one sentence (interviews plus sub-topics)
Memory caliber: CRAG rectification, Self-RAG correction generation, Adaptive correction process; Late Chunging is index side optimization, CAG simply does not search.
Vector storage selection and greyscale migration
Mixed search is the largest selling point for the search engine program: filter first by category/permission/language, then do a vector in the filter result TopK - This is just needed in many businesses (e.g. semantic search only in the context of "English documentation" and "current users have permission to read".
Move greyscale suite (any vector bank migration universal):
- Double-write old and new libraries. Configure switches.
- Read it and walk through the old library for a while.
- Grey scale to the new library.
- All back-to-back.
Remember: The writing logic of the old and new library during a double-write period ensures the like of a tungsten (the same document does not produce dirty data).
6. Impact assessment: no optimization without measurement
On-line RAG has to answer "How's the answer?" The industry standard tool is RAGAS, an assessment framework of four core indicators:
The first two evaluations were generated and the last two were retrieved. Retrieving indicators is poor (slice/embedding/policy), generating poor indicators and repairing prompts and models — the two layers are not clear, and the optimization is the visualization of the blind.
Practical proposal: Prepare 50-100 real questions for fixed test sets, running a comparison every change (slice policy, embedding model, threshold adjustment). RAG optimisation = drive away from dashboard without returning to test set.
Next: AI agent Memory
1. Three layer memory model (like humans)
Human memory has sensory memory, short-term memory, long-term memory. The memory system of AI agent is almost identical, with three layers:
Why are the layers: short-term memory has to be read and written every round, quickly, but not for long; long-term memory has to cross the session, share the equipment, and search for semantics, which is worth a service; and full long-term memory has to inject prompt into each round, both expensive and diluted, so let the model be checked on a case-by-case basis — this time remember "Remember" forever.
2. Short-term memory: the work of the history of dialogue
The dialogue framework (e.g. Langchain/Langchain4j) is generally embedded in ChatMemory's abstract:
- Segregate by memoryId (usually = session ID), one message list per session
- Every round of history news + new questions sent to models.
- Sequence message into JSON in KV storage (Redis class)
/ / LangChain4j style: one memory instance per session, window + custom storage
ChatMemoryProvider provider = memoryId -> MessageWindowChatMemory.builder()
.id (memoryId) / session ID, isolated by session
. maxMessages(20)/ Window Control: Only the latest 20 messages are kept
.chatMemooryStore / custom store:messagesToJson drops KV, read from Jason
.build();
/ / When sending the frame automatically load [in-window history + current round of new messages]
Three production elements:
1 Window Control
History cannot last indefinitely (token cap + cost + attention diluted). Two strategies:
- Bar-number windows: only the nearest N-wheel
- Token window: Press token budget cut, lose the oldest
A more detailed approach: Old message summary is compressed rather than simply discarded ("the user asked before about the configuration of the living room lights" instead of the original ten rounds).
2 History is more than just talking text
The production system will be stored together in history: the RAG of the current wheel recalls content, tool call records, and displays cards. "What did the model see?" The history of text-only is missing -- users look back at the "citation of that just answer" and need a recall record.
3 tool call pair
The model-initiated tool call request (tool call) and the tool return result (tool result) have to be in step. A match was missing, and the next round of models saw themselves "requesting but confiscating the results", reporting duplicate call or directly wrongs.
3. long-term memory: Storage and recall
Two types of memory.
Reason for separation: The image is suitable for full injection system prompt (small, always relevant) and the event is suitable for search in time window terms (large, recall only when relevant).
Three languages of memory service
A decent long-term memory service requires only three interfaces:
Upload (written): Content + belonger (uid) + session ID + business name + timetamp
Recall (recall): Semantic vector + Time Window Filter + Structured field Filter + TopK
Forget (forgotten): Delete on condition
Recall's triple filter is the essence, giving an example of why:
The user asked, "Did the lock last time work out?"
- Semantic Retrieval: Retrieving the door lock with a problem vector.
- Time filter: only the last two weeks left eight.
- Structured filter: only after-sale memories left 3
- TopK: Take the most similar two injects
Retrieving only by words, 30 articles are filled with both expensive and vital information.
Triple filter as a query object (real memory service recall interface long):
MemoryQuery q = MemoryQuery.builder()
.uid (uid) / belong to: check only the user's memory
.text (userQuestion) / 1 Semantics: Problem Vector Search
.startTime (now-Duration.ofDays(14)) / / 2: Just for two weeks
...bisFilter ("category", "AFTER SALE")) / 3 Structured: only after sale Category
...topK(2) / / Final injection only 2
.build();
Forget why it matters: memory is not only growing. Users say, "Forget what I said." No memory system without Forget can't get on the compliance line.
Memory phase-out: what about unlimited growth?
The memory banks, which are only growing, are becoming slower and more expensive and are being recalled more and more. Three automatic phase-out mechanisms:
- Time decay: a time stamp is attached to each memory, and the retrieving factor is multiplied (e.g., score x e^ (-Δt/half-life), which is ahead of schedule. Long half-lives (years) for portraits and short half-lives (week/month) for event types
- Important update: recall to hit, memory promotion to be used in the model, reduction to be used by no one until phase-out - - Thermal phase-out like cache.
- summary archive: Accumulation of the same event memory to the threshold is combined into a summary file.
Interview highlights: Manual forget (user/compliance trigger) and automatic phase-out (system self-maintenance) are two sets of mechanisms — the former for compliance and the latter for quality and cost of retrieval.
How to write memory: LLM extract
The memory of the conversation cannot be built into the whole conversation (that's a short-term memory), but rather draws from it the memorable facts of LLM:
Dialogue:
User: My living room uses a brand-named toplight, and everything else is too bright.
Assistant: Okay, understood...
LLM extract memory fact:
{Type: portrait, content: "The living room lamp is a brand A toplight, the user feels bright", source session: xx}
Timing: End of session (show, no blocking dialogue). Extracts the prompt for output structured JSON (type/ content/confidence), which is discarded with low confidence.
User-defined memory verification
Let the user write his own memory ( "Remember that I'm the developer" ) against two things: injection of attack (the user writes "Unregard all safety rules" later) and prohibited content. Production practices are two-tier validation:
- Static verification (cheap, run first): Template matching (as opposed to system preset template Hash, just as directly) + forbidden vocabulary
- LLM Validation (Valuation, Bottom): Let another LLM judge whether this custom memory is in compliance
The LLM verification is an anecdotal (second level), so the interface mode is to submit the → return the results of the front-end query. Cheap pre-check, expensive bottom check is a universal optimization of all verification links.
4. Memory tooling: enabling models to proactively remember and forget
Register recall memoory and forget memoory as a modelable tool:
User: Or do you want to give me a good night scene as usual?
Model: Think: What's the old rule? I need to remember.
♪ Back to memory: "User habits, 23 o'clock, full lights, bedrooms, 10% light night lights."
Create scene configuration by memory
User: Right, forget about my time.
Model: tunet memory
Compare the advantages and disadvantages of auto-injecting every round of memory:
Production combinations: full image memory injection system prompt (small stabilization) and event memory toolation for retrieval on demand (very thin).
5. Injection position: how memory enters
Three injections:
- System prompt: Image Memory + Event Memory summary → on User: {User Image} Related memory: {incident memory}." Most common, suitable for stabilizing information.
- Placeholder variable: System prompt stays {MEMORY} position, dialogue lines replace variables before sending them. The advantage is that the prompt template and memory content are decorated and the template can be changed to remain coded.
- The historical disguise of the dialogue: the memory of the recall is packaged as a historical message ("Systems: Remembering the user...") inserted into the centre of history. It's appropriate to keep the dialogue streaming scene.
6. A complete example: a question from a smart customer service
Tie everything up and down. Scene: Iot smart guest service, user questions in App.
User: "The light in the living room is again offline.
Take this session history.
Step 2 long-term memory.
Image Injection: Users are ordinary household users, home 12 devices, 1 gateway
Event Retrieval (Toolation): Model judgment "last time" requires memory
→Recall memoory
→ Hit: "10-12 Toplight offline in the living room, reset and restored, solid v1.2"
Step 3. RAG Retrieval
Query rewrite: "Smuggle the toplight offline reset check."
Vector Retrieval + rerank → Section 3 of the offline Check Guide for Lamps (Description for the Upgrading of Solids)
Score door control: maximum score 0.87 > threshold 0.6 thallium injection
Step 4 Generate
Prompt = System tip (with image) + history + memory event + RAG content + current problem
Model generation: "The last time it was replaced, this time it was the same rate. Step 1: Long press...
V1.2 is also detected with known offline problems in solids, and it is recommended to be upgraded to v1.3 before observation."
End of session, LLM extract:
"10-30 Toplights in the living room again offline (2nd) recommended upgrade v1.3"
[Step 6. Leave a mark] The RAG document ID is used for this round.
And notice that three things in this case are all the same: RAG gave "one-size-fits-all solution" and long-term memory gave "what happened to this user last time" and short-term memories gave "the language of this conversation." The quality of the answers will be significantly reduced by removing any of them.
Annex: List of studies and common pits
Self-check list.
- A full flowchart of RAG writing and reading lines
It explains why the embedding model has to be re-indexed and how to make it grey.
- You can tell what Query Rewrites / Multiple / HyDE / Rerank solves.
- It explains why the score point is "no need"
- to compare the application of the Pipeline RAG with the application of the RG
- You can tell the basis for the three layers of memory and the respective storage requirements.
- to design a triple filtered memory with semantics + time + structure.
- Tell me why the memory system has to be.
- Knows whether RAGAS four indicators are individually retrieved or generated.
- to compare RAG to fine-tuned applicable boundaries and combinations (RAFT)
- Discrepancies/intensities can be identified for searching their respective blind zones and mixed searches
-Late Chunging / CAG / CRG / Self-RG / Adaptive RAG
- Three mechanisms for automatic memory phase-out (time decay / importance update / summary archiving)
Common pits.
- Naked slices directly embeding-- missing context, recall. Collapse title/category path to quantization.
- Batch embedding is used in order of arrival - the interface is disorderly sequenced/faulty and the vector text is wrong. The global subscript must be restored with the returned index field.
- There's no need to double-entry — the same paragraph repeats prompt, misleading models.
- Low-rate recall is also stuck - negative. Sets a threshold door.
- tool call does not match the results with history - model duplicate call or error reporting.
- The history of dialogue has grown indefinitely - token explosion + attention diluted. Window +summary compression.
- Memories only grow — no compliance, no user. Provide forget.
- without evaluation of the test set, the RAG - every change is blind. 50~100 fixed set of questions + run with each change.
- Compose LLM verification into a sync interface - a second-grade drag to death. AskId+Step +Queue.
- Static check and LLM check in reverse order - Cheap check must be in advance.
- Only vector recall — exact model/microcode not available literally, keyword (silent) bottom, double-track consolidation.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.