Embedding 替换怎么评测：把索引成本算进成功任务
一个检索模型在公开榜单上分数更高，接入知识库后却可能让任务更贵。原因常常在检索之后：Agent 拿到片段是否足够作答、是否继续改写查询、是否把整篇文档展开、失败时是否重新调用工具。这些动作会消耗生成模型的 Token 和时间。只看一次向量请求的单价，观察不到整条链路的变化。
本文以 Nemotron 3 Embed 的公开资料为候选入口，交付一套固定检索条件的配对测量方法，以及可以直接复制运行的有限日志核算程序。程序处理人工构造的任务记录，验证账本口径；文中的 A、B 是假设方案标签，不代表任何两个实际模型的胜负。本次没有加载 Embedding 权重、跑 GPU 或取得真实供应商账单。
规格与已确认信息
截至本次核验，NVIDIA 官方的 1B BF16 模型卡列出约 1.14B 参数、2048 维输出和 32768 Token 最大序列长度；8B BF16 模型卡列出约 8B 参数与 4096 维输出。模型卡说明在 34 种语言上评测，许可标为 OpenMDW-1.1。语言覆盖说明哪些公开任务被评估，不能视为每种语言、每个行业都达到同样质量。
这些数字会影响服务成本，但不是业务收益证明。向量维度决定原始向量的存储量，序列上限影响截断策略，参数量影响权重和运行时负担。一个业务文档只需八百字即可回答的问题，不会因为模型支持更长输入就自动变好；把无关附件一起编码，还可能增加计算和召回噪声。
官方 BF16 使用说明区分 query 和 passage 前缀，并要求切片后的向量重新做 L2 归一化。比较时应为每个候选保留符合自身模型卡的输入适配，固定适配规则，再固定业务内容。让所有模型机械使用同一个错误前缀，会把接口失配算成模型质量差异。改变输出维度同样是一项干预，需要另开配置，不能在模型名称相同的情况下悄悄更换。
在本次读取的 1B 卡中，本地 BF16 示例列出的依赖包括 Transformers 5.2.0 及以上、Sentence Transformers 5.4.1 及以上；vLLM 的推荐说明包含 0.25.0。它们是官方示例的版本边界，本机没有执行这些部署步骤。实际运行时应锁定模型 revision、Tokenizer、归一化实现和后端镜像摘要，而不能把一个浮动模型名称当作完整实验版本。
三个候选及硬件边界
以下规格按 2026-10-15 核对的官方发布文章、模型卡归因；这些是公开规格，没有在本机加载模型。
NVIDIA 的 2026-07-16 发布文章标明其图一取自 2026-07-15 的 RTEB 截图，并报告 8B BF16 当时排名第一、平均 NDCG@10 78.5%，1B BF16 为 72.4%。这是厂商发布时的历史结果口径，不是本轮重跑的榜单，也不代表当前排名或业务任务成功率。本文不以这组数字推导实际省钱。NVFP4 的规格与 vLLM 路线另见官方模型卡。
NVIDIA RTX A6000 规格为 Ampere、48GB GDDR6 ECC。只按 8×10^9 参数、每参数 BF16 两字节估算，8B 权重约 16×10^9 字节，即十进制 16GB；这不是完整进程显存。激活、Attention、批次、框架缓存及并发会增加峰值，48GB 也不能自动保证长序列/并发配置可用。A6000 上可把 BF16、FP16 或框架明确支持的量化作为待测路线，不引用 Blackwell 的最高两倍吞吐作为 A6000 预期。本轮没有 GPU 实测。
RTEB 能回答什么
RTEB 是检索基准，可以帮助判断候选是否值得进入业务测试。NVIDIA 的发布与评测文章报告了 Nemotron 3 Embed 的公开检索表现和 Agent 实验。文中排名、分数及吞吐属于厂商给出的特定评测口径；本文不把发布日期的排名改写成永久结论，也不把它与自己的业务任务成功率互换。
公开基准最直接衡量的是排序：与问题相关的材料有没有进入前几个结果，相关程度是否被正确排序。线上知识库还会有权限过滤、过期文档、重复版本、表格解析错误、代码与说明分离等问题。如果答案所在页面根本没有进入语料，换一个更强的向量模型也无法召回它。先检查语料覆盖，再谈向量质量，才能避免把数据缺失归因到模型。
可将业务问题分成事实定位、版本对比、代码查找、多跳组合和无答案五类。事实定位看首个正确片段的位置；版本对比看是否拿到两个有效版本；代码查找还需检查相关函数与调用上下文是否同时出现；多跳问题要核验每一跳的证据；无答案问题要观察是否错误地把近似表述当成答案。平均检索分数不足以替代这些分层判断。
无答案任务尤其影响成本。在没有证据时，Agent 可能不断改写查询直到预算耗尽，也可能很快返回一个错误答案。后者看起来 Token 更少、速度更快，却不满足任务质量条件。因此成本表必须同时带上验收结果和拒答规则。省钱如果来自提前放弃或者更频繁的错误回答，就不属于可接受的替换收益。
为什么更强检索可能降低下游成本
厂商文章中的搜索 Agent 使用 Nemotron 3 Ultra，改变检索用的 Embedding 模型；下游 Token 成本由输入与输出数量按指定定价公式估算。它展示的是特定 Agent、数据集和计价口径下的观察，不是供应商实际账单，也没有为任何自建 RAG 系统证明相同收益。
适合带回项目的是一个可证伪假设：正确证据更早出现，Agent 可能减少重查、少展开文档、降低输入 Token，进而减少任务成本。要核验这个假设，记录的中间环节不能只剩一个总费用数字。至少需要关联任务 ID、查询文本、召回片段 ID、首个有效证据位置、重查次数、输入 Token 和最终验收结果。
例如同一个“新版本接口如何取消任务”的问题，A 首轮召回了旧版指南，Agent 又查了一轮；B 首轮找到新版取消说明，直接给出可验证调用。若二者最后都通过相同验收，B 的少一次查询才是有效行为差异。若 B 返回的是一句未经校验的概括，即使输入少了，也不能把该任务计为同质量的成功。
这条机制也可能被别的组件覆盖。Reranker 已经把正确证据放到第一位时，更换首阶段召回模型未必影响生成；Agent 的终止策略固定要求两次检索时，首轮召回改善也不一定减少调用；需要跨文档组合的问题，第二次检索可能是必要动作。遇到这种结果，应记录“检索改善未传导到下游”，而不是强行用排行榜解释节省。
另一种反例是上下文膨胀。更高召回率让 Agent 收到更多相关但重复的长片段，回答更完整，输入 Token 却增加。此时要检查片段去重、引用筛选和上下文预算是否匹配任务。是否值得接受更高成本，由预先定义的验收条件决定，不能事后只挑对新模型有利的指标。
评测指标必须分三层
第一层：检索质量
Recall 反映所需证据是否进入候选，NDCG 反映相关证据的排序质量，MRR 关注第一个正确结果的位置。多跳任务不能只标一个“正确页面”，应为每一跳保留最小证据集合。无答案任务单独统计误召回后的错误引用；中文、代码、版本和长文档分层报告，以免大量容易问题掩盖少数关键退化。
标注过程要先确定证据是否真的支持回答。含有同一个名词不等于相关；旧版文档可能回答了不同版本的问题；脱离标题的表格行可能缺少限制条件。将相关标签与文档版本、片段位置绑定，再冻结任务集，可以减少评测人员在看见候选结果后临时修改标准的偏差。
第二层：Agent 行为
需要记录重查、查询改写、Rerank 调用、上下文展开和工具重试，并区分重查的原因。第一次找错文档后的纠正，与多跳任务主动寻找第二条证据，都叫第二次搜索，意义却不同。只减少总搜索次数可能奖励没有完成必要取证的 Agent。行为日志应能让人回放到“为何继续搜”这一个决定。
记录失败也应分层。检索空结果、向量服务超时、生成模型限流、工具执行报错、答案验收失败分别保留，不要全部写成一次模型失败。否则换 Embedding 之后恰好减少了生成服务限流，就容易被误解释为检索变好了。配对任务中出现基础设施异常，费用仍应入账；归因时另外标记，不删掉不利样本。
第三层：任务与成本
最终同时报告成功任务数、总尝试数、全量费用、单成功任务费用、端到端延迟以及索引成本。验收必须落到具体条件：代码任务通过约定测试；事实问题引用了有效证据；无答案问题正确拒答。一个自报“完成”的模型文本不等于任务成功。人工复核应使用同一份标准，尽量不暴露候选名称。
任务延迟包含排队、检索、重排、生成与工具，不是把某次模型响应时间贴上任务标签。少量样本可以逐条列出耗时，不能据此声称稳定 P95。并发下的长文档尾延迟、缓存状态、冷启动和服务限流，应在后续负载测试中另测。本文的有限核算程序没有时延输入，因此不输出吞吐或尾延迟结论。
单任务综合成本
费用可以按一次完整任务累计：查询向量、向量搜索、重排、生成输入、生成输出、工具调用、失败处理，再加上索引重建摊销。重试已经产生的各项费用要按尝试累加，不能再次加一个包含相同消费的“重试总价”，否则会重复计费。失败处理只记录额外人工或业务开销；如果这项无法估值，单独列出数量并做敏感性分析，不要默认为零。
单成功任务成本采用本轮所有任务的成本除以通过验收的任务数。失败任务花掉的钱留在分子里，重试后成功的任务在分母只出现一次。这个口径避免把新模型失败更多但单次请求便宜，包装成整体更省钱。全部失败时分母为零，应报告不可定义，并保留总投入；不能显示零元或者删除该配置。
索引重建摊销也会改变结论。设重建成本为十二个假设美元，预计生命周期内处理一百个任务，每个任务分摊零点一二。若实际只用了十个任务，摊销会大幅上升。离线评测中“为了试一个候选建一次索引”的实验支出，与生产持续服务的摊销不是同一口径，应分别列账，再给出预计任务量的高低情景。
比较新旧方案时，还要区分总拥有成本和增量成本。旧索引已经完成，不代表长期重建、存储和维护永远免费；新索引也不应承担已经属于公共数据清洗的全部历史费用。本文算例把 A 的此次额外重建设为零，把 B 的额外重建设为十二，说明一次替换的增量账本，不是长期生产成本模型。
可运行的最小配对核算
下面的代码只需 Python 3 标准库。复制到 paired_embedding_cost.py，在保存文件的目录执行 python3 paired_embedding_cost.py。九条人工记录覆盖四个任务的 A、B 两组尝试：两边均成功三个，失败一个；A 在第二个任务重试过一次。查询次数可以多于尝试数，表示一次 Agent 尝试内部再次搜索。金额使用 Decimal，所有单价均为假设值。
程序先检查任务集合相同、尝试序号连续、只有末次尝试是终态，再累计全量消费。它拒绝把两个成功终态算成同一个任务的两次成功，也拒绝让一个停留在 retry 的任务进入已完成统计。实际日志需要以稳定 task_id 和 variant 连接；本示例的数据直接写在代码里，使读者能够不依赖服务复跑。
from decimal import Decimal as D
from collections import defaultdict
# Synthetic accounting fixture; A/B are labels, not measured model results.
# task, variant, attempt, query calls, input tokens, output tokens, final status
ROWS = [
("t1", "A", 1, 1, 2000, 400, "success"),
("t2", "A", 1, 2, 4000, 500, "retry"),
("t2", "A", 2, 1, 2000, 300, "success"),
("t3", "A", 1, 2, 5000, 600, "failed"),
("t4", "A", 1, 1, 1500, 200, "success"),
("t1", "B", 1, 1, 1800, 350, "success"),
("t2", "B", 1, 1, 1800, 300, "success"),
("t3", "B", 1, 1, 4000, 450, "failed"),
("t4", "B", 1, 1, 1200, 200, "success"),
]
# All prices below are invented USD amounts for arithmetic only.
EMBED = {"A": D("0.001"), "B": D("0.003")}
REBUILD = {"A": D("0"), "B": D("12")}
EXPECTED_TASKS = 100
def summarize(rows):
grouped = defaultdict(list)
for task, variant, attempt, queries, inp, out, status in rows:
if variant not in EMBED or status not in {"retry", "success", "failed"}:
raise ValueError("unknown variant or status")
if any(type(x) is not int or x < 1 for x in (attempt, queries)):
raise ValueError("attempt/queries must be positive integers")
if any(type(x) is not int or x < 0 for x in (inp, out)):
raise ValueError("tokens must be non-negative integers")
grouped[variant, task].append((attempt, queries, inp, out, status))
task_sets = [{t for v, t in grouped if v == variant} for variant in ("A", "B")]
if not task_sets[0] or task_sets[0] != task_sets[1]:
raise ValueError("A/B task sets must match and be non-empty")
summaries = {}
for variant in ("A", "B"):
cost = D("0")
successes = attempts = query_count = 0
for task in sorted(task_sets[0]):
events = sorted(grouped[variant, task])
if [e[0] for e in events] != list(range(1, len(events) + 1)):
raise ValueError("duplicate or missing attempt")
if any(e[-1] != "retry" for e in events[:-1]) or events[-1][-1] == "retry":
raise ValueError("exactly one terminal outcome required")
for _, queries, inp, out, _ in events:
cost += queries * (EMBED[variant] + D("0.0005") + D("0.0008"))
cost += D(inp) / 1000 * D("0.002") + D(out) / 1000 * D("0.004")
cost += D("0.002") # tool cost charged once per attempt
attempts += 1
query_count += queries
if events[-1][-1] == "success":
successes += 1
else:
cost += D("1") # assumed extra failure handling cost, once per task
tasks = len(task_sets[0])
total = cost + REBUILD[variant] * tasks / EXPECTED_TASKS
summaries[variant] = (tasks, successes, attempts, query_count, cost, total)
return summaries
def reject(rows):
try:
summarize(rows)
except ValueError:
return True
return False
result = summarize(ROWS)
assert result["A"] == (4, 3, 5, 7, D("1.0631"), D("1.0631"))
assert result["B"] == (4, 3, 4, 4, D("1.0480"), D("1.5280"))
assert reject(ROWS[:-1]) # unmatched task set
assert reject(ROWS + [ROWS[0]]) # duplicate attempt
assert reject(ROWS[:1] + [("t1", "A", 2, 1, 1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, -1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "retry")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "unknown")] + ROWS[1:])
assert reject([])
for variant, (tasks, successes, attempts, queries, online, total) in result.items():
per_success = f"{total / successes:.6f}" if successes else "undefined"
print(f"{variant}: tasks={tasks} success={successes} attempts={attempts} queries={queries}")
print(f" online_plus_failure_usd={online:.4f} amortized_total_usd={total:.4f}")
print(f" cost_per_success_usd={per_success}")
print("self_checks=9 passed; fixture_only=true; GPU_run=false")
本次实际运行的 stdout 如下，退出码为零，stderr 为空。运行环境为本机 Python 3.13.5；这里只执行核算与九条断言，没有调用模型服务。把价格、任务量或终态改掉后，输出应跟着变化，原来的精确值断言也需要相应更新，不能沿用这个结果证明新的实验。
A: tasks=4 success=3 attempts=5 queries=7
online_plus_failure_usd=1.0631 amortized_total_usd=1.0631
cost_per_success_usd=0.354367
B: tasks=4 success=3 attempts=4 queries=4
online_plus_failure_usd=1.0480 amortized_total_usd=1.5280
cost_per_success_usd=0.509333
self_checks=9 passed; fixture_only=true; GPU_run=false
A 的查询总数为七，B 为四；B 的在线消费加失败处理从 1.0631 降到 1.0480。加入这四个任务分摊的 0.48 重建费用后，B 总额变为 1.5280，按三个成功任务算反而更贵。它演示了“下游节省不足以覆盖变更成本”的可能性，不能证明某个实际 Embedding 模型会产生这些调用次数。
九条断言覆盖两组金额预期、任务集合缺失、重复尝试、成功后继续尝试、负 Token、没有终态、未知状态和空数据。它们帮助检查这个有限输入的账本约束。真实采集还要处理跨服务事件重复、货币换算、迟到账单和缓存读写价格；接入真实日志之前，应扩展输入校验和账单对账，而不是把本例当作完整费用平台。
公平比较 Nemotron、Qwen3 和 BGE
第一步冻结业务任务、语料快照、Chunk 规则、Metadata Filter、Top-K、向量库参数和 Reranker。第二步冻结生成模型、System Prompt、工具权限、终止策略、Token 与时间预算。第三步明确唯一变更：本轮替换 Embedding 及它必需的输入适配。若同时改 Chunk 与 Agent Prompt，应另开实验，把“组合改动的效果”与“模型替换的效果”区分。
Qwen3-Embedding-8B 模型卡提供查询端 Instruct 与 Query 格式，文档端无需附相同指令；BGE-M3 模型卡区分 Dense、Sparse 与 Multi-vector 能力。只比较 BGE 的 Dense 输出，应明确自己测的是这个功能子集；同时开启它的混合检索而另一个候选只有 Dense，就变成检索系统方案比较，结论不再只归因到模型权重。
不同维度产生的索引不能直接混在一起检索。为每个候选独立建立版本化索引，并核验文档数、片段数、过滤条件和构建配置一致。候选的维度、归一化、距离函数必须与其输出契约相匹配；固定同一种错误距离函数并不构成公平比较。第一轮可用同一业务协议、各自合法适配，第二轮再在明确预算内分别调优，并给两轮不同的名字。
配对实验先让每个任务在 A、B 都运行，交错安排顺序，减少时间段负载与缓存差异。随机种子只是可追踪配置，并不保证服务完全确定；生成行为不稳定时，要用预先约定的重复次数保留任务级结果。报告可以列出两边都成功、仅 A 成功、仅 B 成功和都失败四类，分别看 Token 与调用差异，不先筛掉某个候选的失败记录。
可复用的记录表建议包含 experiment_id、task_id、variant、corpus_sha、chunk_config_sha、model_revision、query_adapter_sha、index_build_id、harness_sha、attempt_id、查询与重排次数、输入输出 Token、终态、验收证据和计价版本。缺少项就写未采集，不要补默认值冒充相同条件。尤其 harness_sha 变化后，旧结果仍可保留，但应重新分组比较。
RTX A6000 的现实边界
对一台既有 GPU，选型先看能否在目标长度、批次与并发下完成合法推理，再看吞吐。权重字节的简单估算不包含激活、运行时缓存、向量输出和服务队列，不能充当显存峰值。本文没有在 RTX A6000 上执行加载或检索实验，因此不给出它的吞吐、最大批次和并发容量。
NVIDIA 发布文章将 1B NVFP4 的原生加速放在 Blackwell 部署场景中。不能把那里的吞吐比例移植成 A6000 预期。BF16 与量化配置也需要分别记录后端、设备和精度；在一个后端能够加载，不等于另一个后端具有同等支持。实际测试前先核对官方模型卡与后端文档，失败时保留错误和配置，而不是换一个标签继续填成绩表。
服务容量还会影响下游成本。当大 Embedding 模型让队列更长，Agent 可能触发超时、重试或者回退；这些额外尝试应该进入完整任务账本。单机空闲时的平均向量耗时只能描述那个条件，不足以推断线上任务延迟。把硬件事实、部署可行性、检索质量和任务收益分开记录，才能知道下一步实际需要哪项证据。
常见错误
最容易发生的错误，是用排行榜选模型，然后只拿几个成功回答算 Token。这样同时漏了失败费用、异常重试和索引变更。修正方式是先冻结任务集合，把全部尝试连回任务，再按同一验收统计成功数。小样本允许公开逐条结果，但应避免给出足以代表整个业务的百分比口号。
第二个错误是把检索改善当成回答改善。相似度更高的片段可能是同版本的重复文档，引用看起来完整却没有增加信息；正确片段也可能被 Agent 忽略。需要将召回列表、最终引用和验收证据串起来。只保存向量分数无法检查这一跳，只有最终回答也无法区分检索失败与生成失败。
第三个错误是把估算费用写成付款记录。假设计价适合做方案敏感性分析，供应商返回的 usage 适合核算消费，实际账单用于对账，它们的证据强度不同。厂商定价公式换算的 Token 费用，还不包含你的向量存储、人工复核、缓存优惠与索引生命周期。报告应在金额旁注明计价版本、单位与未包含项。
决策规则
先在看结果前写下质量约束，例如关键版本问题不能错引旧文档、无答案任务必须正确拒答；一般任务成功率底线由业务需求决定，本文不设通用百分比。再核验候选是否满足这些条件。没有通过底线的方案，即使成本低，也不进入“同质量成本更优”的结论。
对合格候选，比较每成功任务总成本、端到端延迟和迁移摊销，同时看任务类型中的退化。若质量相同且在线消费更低，但重建成本高，给出预计任务量下的回收边界；若成本下降依赖更大的缓存命中率，应补冷缓存条件；若收益只在代码任务出现，应限制结论范围，先对那个流量子集试验。
最后保留可回退的旧索引与明确切换条件。Embedding 变更意味着文档和查询编码契约一起改变，回退必须恢复对应索引和适配器，不能只改模型名称。先用有限任务确认质量，再用有代表性的负载确认容量，完成费用对账后才决定是否替换。现在可以立即做的是复跑本文算例，把自己的任务记录映射成相同口径，并列出仍未采集的证据。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Embedding Replace how to evaluate: Count index costs into successful tasks
A search model has a higher score on the public list, but access to the knowledge base may make the task more expensive. Reasons are often found after search: whether Agent has enough footage to answer, whether to rewrite the query, whether to expand the entire document, and whether to re-call the tool if it fails. These actions will consume token and time that generates models. Only the unit price requested in a single vector could not be observed in the entire chain.
This paper is based on Nemotron 3 Embed's open information as a candidate for the delivery of a set of fixed search conditions matching measurements and a limited log accounting program that can be directly replicated. The program handles artificially constructed mission records and validates the calibration of the account book; the A and B in the text are scenario labels and do not represent the success or failure of any of the two actual models. This time no Embeding weight, running GPU or getting a real supplier bill.
Specifications and confirmed information
As of this verification, the NVIDIA official 1B BF16 model card lists approximately 1.14B parameters, 2048 D output and 32768 Token maximum serial length; the 8B BF16 model card lists approximately 8B parameters and 4096 D output. The model card description is evaluated in 34 languages and is marked OpenMDW-1.1. Language coverage indicates which public tasks are assessed and cannot be considered to be of the same quality for each language and industry.
These figures affect the cost of services but are not proof of business gains. Vector dimensions determine the amount of storage of the original vector, the sequence upper limit impact cut-off policy, and parameter count influence weights and operational burdens. A business document requires 800 words to answer questions and does not automatically improve because the model supports longer input; coding unrelated attachments together may add to the calculation and recall noise.
The official BF16 uses instructions to distinguish between query and passage prefixes and to require a later cut vector to be converted to L2. The comparison shall be made by retaining for each candidate input fit for their own model card, fixed matching rules and fixed business content. Allowing all model machines to use the same error prefix will account for interface mismatches as differences in model quality. Changing the output dimension is also an intervention that requires a separate configuration and cannot be changed quietly under the same model name.
In this read 1B card, local BF16 examples of dependence include Transformers 5.2.0 and above, Series Transformers 5.4.1 and above; v LLM contains 0.25.0. They are official examples of the boundaries of the versions, and these deployment steps were not implemented by the aircraft. The actual operation should lock in the model review, Tokenizer, CRS and backend mirror summary, instead of using the name of a floating model as a complete experimental version.
Three candidate and hardware borders
The following specifications are based on official publish articles, model cards, which are checked at 2026-10-15; these are public specifications and are not loaded on this machine.
The NVIDIA 2026-07-16 publish article indicated that its figure I was taken from the RTEB screenshot of 2026-07-15 and reported that 8B BF16 was ranked first and average NDCG@1078.5% and 1B BF16 was 72.4%. This is the historical caliber of the manufacturer publish and is not a runback list of the current round, nor does it represent the current ranking or operational task success rate. This paper does not extrapolate this set of figures to actual savings. The specifications of NVFP4 and the vLM route are also shown in the official model card.
NVIDIA RTX A6000 Specifications are Ampere, 48GB GDDR6 ECC. The 8B weight of approximately 16x10^9 byte, or Decimal 16GB, is estimated at only 8x10^9 parameters per parameter BF16 2Z113Z; this is not a complete process VRAM. Activation, Attention, Batch, Frame cache and SPDC increase peaks, and 48GB does not automatically guarantee long sequences/separation. A6000 can use quantization, which is explicitly supported by BF16, FP16 or the framework, as a route to be determined, without citing Blackwell ' s maximum twice as high as A6000 expected. There are no GPU measurements for this round.
What can RTEB answer?
RTEB is a search baseline that helps to determine whether candidates deserve to be tested for business. NVIDIA ' s publish and evaluation articles reported on the public search performance of Nemotron 3 Embed and the Agent experiment. The rankings, scores and stowaways in the text are the specific calibrations given by the manufacturer; the rankings of publish are not transposed to a permanent conclusion, nor are they interchangeable with their operational tasks success rate.
The most immediate measure of the open baseline is the sorting: whether the issue-related material has entered the previous results and whether the degree of relevance has been correctly sequenced. permission filters, expired documents, duplicate versions, table parsing errors, code separation, etc. If the answer page does not enter the language at all, a stronger vector model cannot be recalled. Checking the textual cover before talking about vector quality avoids the attribution of data missing to models.
Operational questions can be divided into five categories: factual location, comparison of versions, code search, multiple-jumping combinations and no answers. Factual locator looks at the location of the first correct clip; the version contrasts to see if two valid versions are available; the code search also needs to check whether the function in question appears at the same time as the call for context; the multiple jump question needs to verify the evidence for each jump; and the no-answer question needs to see whether the approximation is incorrectly presented as an answer. The average search score is not sufficient to replace these layers of judgement.
The lack of answers affects costs in particular. In the absence of evidence, Agent may rewrite the search until the budget is exhausted, or may soon return to a wrong answer. The latter appears to be less and faster than Token, but does not meet mission quality conditions. The cost schedule must therefore be accompanied by acceptance check results and rejection rules. Saving money from early abandonment or more frequent wrong answers is not an acceptable replacement gain.
Why would a stronger search reduce downstream costs?
The search for Agent in the manufacturer's article uses Nemotron 3 Ultra to change the search's Embeding model; downstream Token costs are estimated according to the specified pricing formula by the number of inputs and outputs. It displays observations under specific Agent, data sets and price calibrations, is not the actual billing of the supplier and does not prove the same return for any self-built RAG system.
A perjury assumption suitable for the return of the project is that the correct evidence appears earlier, and Agent may reduce re-checking, less file development, lower input token, and thus lower mission costs. To verify this assumption, there cannot be only one total cost figure for the intermediate part of the record. At least associated tasks ID, search text, recall clip ID, first valid evidence location, number of rechecks, input token and final acceptance check results are required.
For example, on the question of how to cancel a mission with a new version of the interface, A first round called back an old version of the guide, and Agent checked another round; B first round found a new version of the cancellation statement and gave a verifiable call. If both end up passing the same acceptance check, a small query from B is the valid behavioural difference. If B returns an unverified summary, the task cannot be counted as a success of the same quality even if the input is missing.
This mechanism may also be covered by other components. Reranker has placed the correct evidence first and replacing the first-stage recall model does not necessarily affect generation; when Agent's termination strategy requires two regular searches, the first-round recall improvement does not necessarily reduce calls; and the need for cross-document grouping may be necessary. In the event of such a result, the “retrievation improvement undirected downstream” should be recorded, rather than imposing a ranking to explain savings.
Another example is the expansion of context. A higher recall rate allows Agent to receive more relevant but repeated long clips, more complete answers, and more input token. This is a time to check whether the clips are redacted, the filters cited and the context budget match the task. Whether higher costs are acceptable is determined by the predefined acceptance check condition, which cannot be followed only by indicators that would benefit the new model.
The indicators must be three levels.
First tier: search quality
Recall reflects whether or not the required evidence is candidate, NDCG reflects the sorted quality of the relevant evidence, and MRR focuses on the location of the first correct result. The multiple jump job cannot be marked with only one " correct page ". The minimum collection of evidence should be kept for each jump. Unresolved task to single out the error references after recall; Chinese, code, version and long document stratification reports to avoid a few critical degradations being masked by a large number of easy questions.
The marking process must first determine whether the evidence really supports the answer. Containing the same term is not the same as relevant; old versions of the document may answer different versions; rows of tables separated from the title may lack limitations. By tying the labels to the document version, the location of the clip and freezing the task set, you can reduce the deviations in the criteria that are temporarily modified by the evaluators when they see the results.
Level 2: Agent Behaviour
Record rechecking, query rewrite, Rerank call, context roll-up and tool retry are required and the reason for rechecking is distinguished. The correction following the first search of the wrong document, as opposed to the double-checking of the first attempt to find the second piece of evidence, is called a second search. Only a reduction in the total number of searches may reward Agent for failing to complete the necessary evidence. The behavioural log should be able to put back the decision “Why to continue searching”.
There should also be a hierarchy of record failures. Retrieving empty results, vector service timeout, generating model rate limit, error reporting on tool execution, failure on acceptance check, and failure on acceptance check, do not all fail as a model. Otherwise, Embeding would have just reduced the generation of service rate limit, which could have been misinterpreted as retrieving. Infrastructure anomalies in matching assignments should still be accounted for; additional markings should be made when attributed, without removing adverse sample.
Level three: tasks and costs
Ultimately, the number of successful missions, total attempts, full costs, single successful missions, end-to-end delays and index costs are reported simultaneously. acceptance check had to be placed under specific conditions: the code task had been tested by agreement; valid evidence had been cited for factual questions; no answer had been answered correctly. A self-reported `completed' model text does not amount to mission success. The same criteria should be used for manual review, so as not to expose candidates.
The delay involves queuing, searching, rearranging, generating and using tools, rather than labelling a given model response time. A small number of samples can be listed article by article and cannot be claimed to be stable P95. The long file tail delay, cache state, cold start-up and service rate limit, which is distributed together, should be measured in subsequent load tests. The limited accounting procedure in this paper does not have latency input and therefore does not output throughput or tail delay conclusions.
Combined cost for individual assignments
Costs can be accrued on a full mission basis: search vector, vector search, rearrange, generate input, generate output, tool call, fail processing, plus index reconstruction amortization. The costs already incurred by retry are to be cumulative on an experimental basis and cannot be added again to a “retry total price” that includes the same consumption, otherwise double-counted. Failure processing records only extra labour or business expenses; if this cannot be valued, separate quantities and sensitivity analysis are provided, not default zero.
The single cost of successful missions is divided by the number of missions through acceptance check using the cost of all missions in this round. The money spent on the failed mission remained in the molecules, and the successful mission after retry appeared only once in the denominator. This calibre avoids more failure of the new model, but single requests are cheaper and packaged as a whole is cheaper. When all fail, the denominator is zero, which should be reported as undefined and the total input retained; zeros cannot be shown or the configuration removed.
Amortization of index reconstruction would also change the conclusion. The cost of reconstruction is 12 assumptions in United States dollars, and it is expected that 100 missions will be processed over the life cycle, with a distribution of 1:12 per mission. If only 10 missions were actually used, the amortization would increase significantly. The experimental expenses of the offline assessment, “for a single index to test a candidate”, which are not of the same calibre as the amortization of continuing production services, should be accounted for separately and given the expected high and low workload.
When comparing old and new schemes, a distinction is made between total ownership costs and incremental costs. The completion of the old index does not mean that long-term reconstruction, storage and maintenance will always be free; nor should the new index cover the full historical costs that already belong to public data cleansing. This takes the example of this additional redevelopment of A to zero, of the additional redevelopment of B to twelve, indicating an incremental account that is replaced by one, not a long-term production cost model.
Runable minimum pair accounting
The code below only needs the Python 3 Standard Library. Copy to paired embeding cost.python3 paid embeding cost.py in the directory where the file is saved. Nine manual records cover four missions, A and B, two sets of attempts: three on both sides, one failed; A once on the second mission, retry. You can ask more times than try, meaning that Agent tries to search internally again. The amount is calculated using Decimal, and all unit prices are assumed.
The procedure first examines the same set of tasks, the continuous number of attempts, and only the last attempt is final, before accumulating full consumption. It refuses to count two success stories as two success stories of the same mission, and it refuses to let a mission staying in retry enter the completed statistics. The actual log needs to be connected by stabilizing the tab id and varant; this example of data is written directly in the code to enable readers to run back without relying on service.
from decimal import Decimal as D
from collections import defaultdict
# Synthetic accounting fixture; A/B are labels, not measured model results.
# task, variant, attempt, query calls, input tokens, output tokens, final status
ROWS = [
("t1", "A", 1, 1, 2000, 400, "success"),
("t2", "A", 1, 2, 4000, 500, "retry"),
("t2", "A", 2, 1, 2000, 300, "success"),
("t3", "A", 1, 2, 5000, 600, "failed"),
("t4", "A", 1, 1, 1500, 200, "success"),
("t1", "B", 1, 1, 1800, 350, "success"),
("t2", "B", 1, 1, 1800, 300, "success"),
("t3", "B", 1, 1, 4000, 450, "failed"),
("t4", "B", 1, 1, 1200, 200, "success"),
]
# All prices below are invented USD amounts for arithmetic only.
EMBED = {"A": D("0.001"), "B": D("0.003")}
REBUILD = {"A": D("0"), "B": D("12")}
EXPECTED_TASKS = 100
def summarize(rows):
grouped = defaultdict(list)
for task, variant, attempt, queries, inp, out, status in rows:
if variant not in EMBED or status not in {"retry", "success", "failed"}:
raise ValueError("unknown variant or status")
if any(type(x) is not int or x < 1 for x in (attempt, queries)):
raise ValueError("attempt/queries must be positive integers")
if any(type(x) is not int or x < 0 for x in (inp, out)):
raise ValueError("tokens must be non-negative integers")
grouped[variant, task].append((attempt, queries, inp, out, status))
task_sets = [{t for v, t in grouped if v == variant} for variant in ("A", "B")]
if not task_sets[0] or task_sets[0] != task_sets[1]:
raise ValueError("A/B task sets must match and be non-empty")
summaries = {}
for variant in ("A", "B"):
cost = D("0")
successes = attempts = query_count = 0
for task in sorted(task_sets[0]):
events = sorted(grouped[variant, task])
if [e[0] for e in events] != list(range(1, len(events) + 1)):
raise ValueError("duplicate or missing attempt")
if any(e[-1] != "retry" for e in events[:-1]) or events[-1][-1] == "retry":
raise ValueError("exactly one terminal outcome required")
for _, queries, inp, out, _ in events:
cost += queries * (EMBED[variant] + D("0.0005") + D("0.0008"))
cost += D(inp) / 1000 * D("0.002") + D(out) / 1000 * D("0.004")
cost += D("0.002") # tool cost charged once per attempt
attempts += 1
query_count += queries
if events[-1][-1] == "success":
successes += 1
else:
cost += D("1") # assumed extra failure handling cost, once per task
tasks = len(task_sets[0])
total = cost + REBUILD[variant] * tasks / EXPECTED_TASKS
summaries[variant] = (tasks, successes, attempts, query_count, cost, total)
return summaries
def reject(rows):
try:
summarize(rows)
except ValueError:
return True
return False
result = summarize(ROWS)
assert result["A"] == (4, 3, 5, 7, D("1.0631"), D("1.0631"))
assert result["B"] == (4, 3, 4, 4, D("1.0480"), D("1.5280"))
assert reject(ROWS[:-1]) # unmatched task set
assert reject(ROWS + [ROWS[0]]) # duplicate attempt
assert reject(ROWS[:1] + [("t1", "A", 2, 1, 1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, -1, 1, "success")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "retry")] + ROWS[1:])
assert reject([("t1", "A", 1, 1, 1, 1, "unknown")] + ROWS[1:])
assert reject([])
for variant, (tasks, successes, attempts, queries, online, total) in result.items():
per_success = f"{total / successes:.6f}" if successes else "undefined"
print(f"{variant}: tasks={tasks} success={successes} attempts={attempts} queries={queries}")
print(f" online_plus_failure_usd={online:.4f} amortized_total_usd={total:.4f}")
print(f" cost_per_success_usd={per_success}")
print("self_checks=9 passed; fixture_only=true; GPU_run=false")
This actual run of stdout is as follows, exit code zero, stderr is empty. The operating environment is Python 3.13.5; only accounting and nine assertions are implemented here, and model services are not called. When prices, tasks or finalities are changed, the output should change, and the original precise value assertion needs to be updated accordingly, and the result cannot be used to prove the new experiment.
A: tasks=4 success=3 attempts=5 queries=7
online_plus_failure_usd=1.0631 amortized_total_usd=1.0631
cost_per_success_usd=0.354367
B: tasks=4 success=3 attempts=4 queries=4
online_plus_failure_usd=1.0480 amortized_total_usd=1.5280
cost_per_success_usd=0.509333
self_checks=9 passed; fixture_only=true; GPU_run=false
The total number of queries for A is seven, B is four; B's online consumption plus failure processing decreased from 1.0631 to 1.0480. With the inclusion of 0.48 of the reconstruction costs to be shared among the four missions, the total B becomes 1.52880, which is more expensive on the basis of three successful missions. It demonstrates the possibility that “downstream savings are not sufficient to cover the cost of change” and does not prove that an actual Embeding model produces these calls.
Nine claims cover two sets of expected amounts, missing task pools, repeated attempts, continued attempts after success, negative Token, no terminal, unknown status and empty data. They help check this limited input book constraint. Real collection also addresses cross-service duplication, currency convert, late billing and cache reading and writing prices; before accessing the real log, input checking and bill reconciliation should be expanded rather than using this example as a full-cost platform.
Fair comparison Nemotron, Qwen3 and BGE
The first step is to freeze operational tasks, text snapshots, Chunk rules, Metadata Filter, Top-K, vector library parameters and Reranker. Step two, freeze generation models, Systems Prompt, tool permission, termination strategy, Token and time budget. Step three is the only clear change: this round replaces Embeding and its necessary input. If Chunk and Agent Prompt are changed at the same time, a separate experiment should be conducted to distinguish between the "effects of group changes" and "effects of model replacements".
Qwen3-Embedding-8B model card provides query-end Instrutt and Query formats without the same command for document-end; BGE-M3 model card distinguishes Dense, Sparse and Multi-vector capabilities. By comparing BGE’s Dense output only, it should be clear that it is the functional subset; while opening its mixed search, another candidate, Dense, becomes a search system comparison and the conclusion is no longer attributable to model weights.
Indexes from different dimensions cannot be combined directly. Creates a version index for each candidate and verifies the number of documents, the number of clips, the filter conditions and the configuration of the build. The candidate's dimensions, subsidiarity, distance function must match its output contract; fixing the same wrong distance function does not constitute a fair comparison. The first round could be used for the same operational agreement, each of which was legitimate, and the second round would be streamlined within a clear budget and given two different names.
The pairing experiment starts with each task running in A and B, staggering the order and reducing the difference between the time load and cache. Random seeds are only traceable configurations and do not guarantee full certainty of service; when behaviour is unstable, mission-level results are maintained with the number of pre-agreed duplicates. The report lists four categories of successful, successful A, successful B and unsuccessful, looking at the difference between Token and the call, without sifting a candidate's failure record.
The reusable log sheet proposal consists of options id, task id, variant, corpus sha, chunk config sha, model review, query adapter sha, index build id, harness sha, attempt id, number of queries and reorders, input output Token, terminal, acceptance check evidence and price versions. If an item is missing, the item is not collected and the default value is not applied to the same condition. In particular, after the change, the old results could be retained, but should be re-grouped for comparison.
RTX A6000 Real Border
For an existing GPU, the selection depends on whether the legitimate reasoning can be completed in the target length, batches and simultaneous distribution, and then on the swallowing. A simple estimate of weight byte does not include activation, run-time cache, vector output and service queues and cannot serve as VRAM peaks. This paper does not carry out loading or retrieval experiments on RTX A6000, and therefore does not give it the capacity to swallow, load and co-production.
NVIDIA publish posts the original acceleration of 1B NVFP4 in Blackwell deployment. It's not possible to transpose it to A6000. BF16 and quantization configurations also require separate recording of backends, equipment and accuracy; they can be loaded at one backend, and are not equal to the support of another backend. Check the official model card with the back-end document before the actual test, and keep errors and configurations when it fails, instead of changing a label to continue filling out the report sheet.
The service capacity also affects downstream costs. When the big Embeding model makes the queue longer, Agent may trigger timeout, retry or rollback; these additional attempts should enter the full task book. The average vector time taken while a single machine was idle can only describe that condition and is not sufficient to infer on-line mission delay. A separate record of the hardware facts, the feasibility of deployment, the quality of the search and the benefits of the mission will enable us to know what evidence next step actually needs.
Common Errors
The most likely error is the use of a ranking model, followed by only a few successful answers. This leaves out both failed costs, abnormal retry and index changes. The amendment is to freeze the task set up, re-introduce all attempts, and then count success figures based on the same acceptance check. Small sample allows for the disclosure of article-by-article results, but should avoid giving a percentage slogan that is sufficient to represent the entire business.
The second error is to treat search improvements as better answers. A more similar segment may be a duplicate document of the same version, with references that appear complete without adding information; the correct clip may also be ignored by Agent. Recall lists, final quotations and acceptance check evidence need to be linked. Only the preservation of vector fractions cannot check this jump, and only the final answer does not distinguish between search failure and generation failure.
The third error was to record the estimated costs as payments. The false design prices are suitable for programme sensitivity analysis, the returns of the vendor are suitable for accounting for consumption, and the actual billing is used for reconciliation, with varying evidentiary strength. Token costs of the plant's agreed price formula convert do not include your vector storage, manual review, cache preferences and index life cycle. A price version, unit and item not included in the report should be indicated by the amount.
Decision-making rules
Write quality constraints before looking at results, such as that key version questions cannot lead to old documents wrongly, and that no answer must be answered correctly; the bottom line of the general task success rate is determined by business needs and no generic percentage is set here. Reapproving whether the candidate meets these conditions. There are no bottom-line programmes, and even if they are low-cost, they do not move to the conclusion that “equal quality costs are better”.
For eligible candidates, the total cost per successful mission, end-to-end delay and migration amortization are compared, with degradation in the type of mission. If the quality is the same and online consumption is lower, but reconstruction costs are high, giving the recovery boundary under the expected task; if the cost drop is dependent on a higher cache hit rate, the cache condition should be filled; if the proceeds occur only for code tasks, the conclusion range should be limited and the flow subset tested first.
Finally, the old index that can be rollback is retained and the conditions for the explicit transition are specified. The Embeding change means that the document and the query coding contract are changed together, and rollback must restore the corresponding index and adaptor, not just the model name. Quality confirmation with limited assignments, representative load recognition capacity, cost reconciliation completed before deciding whether to replace. What can be done immediately is to rerun this document, map its mission records to the same calibre and list evidence that remains uncollected.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.