长时间研究的关键承诺,不是任何故障后都从原来的模型步骤继续,而是用户离开时研究不被误停、已发生的效果不被重做、已经保存的事实能够被安全地再次使用。 在允许丢失未提交进度的产品里,执着于恢复整张执行图,可能是在维护比用户真正需要的更昂贵的承诺。
这来自 ETF Agent 对研究恢复机制的一次收缩。我们已经拥有后台研究、数据库状态、浏览器重连、子代理和人工交接,却逐渐发现“恢复”被用来指代几件性质完全不同的事:页面重新打开、进程中断后重建原 Run、用户主动继续未完成的研究。它们共享资料,却不应该共享执行权。
一、先问清哪一种连接断了
设想用户提出一个问题:“比较两只基金的持仓重叠,核对最新披露,并标出覆盖率不足的地方。”Main 已读取部分资料,某个 Child 还在处理 PDF,用户关掉了手机页面。
浏览器关闭不等于服务端 Run 结束。 如果服务进程仍在运行、原 Run 的准入和资源仍有效,后台可以继续做原来的事。重连只读取已有正文、事件和进度,不应为了重新显示页面再调一次模型。
但如果执行所在的服务进程停止,而数据库里只留下若干 ModelTurn、Tool、Child 和等待回执,问题就不一样了。你能读取这些事实,并不等于能重新创建原来那组同时运行的对象。下表把三个经常混用的动作拆开:
- 仅浏览器断开,原服务仍在执行:合理行为:原 Run 在原授权内继续;重连读取原记录;不应该做的事:新建模型调用来“恢复显示”。
- 服务进程中断,普通 Root 没有合法的控制唤醒:合理行为:原 Recovery 负责有限结算、保留已提交内容;不应该做的事:无依据重放旧 Root/Child 执行图。
- 原 Human/Child 控制等待有明确获准的唤醒:合理行为:在原身份、围栏和有限检查点下接续允许的动作;不应该做的事:凭“数据库有旧 Run”推断唤醒已获准。
- 用户明确要求继续刚才的研究:合理行为:创建有独立准入的新 Run,读取仍合法的旧成果;不应该做的事:把旧 Run 的执行权限和额度转移到新 Run。
“Run”是处理一次用户请求的执行尝试;“执行图”则是这次尝试内部的模型请求、工具调用、Child 和等待关系。前者的历史身份可以长期保存,后者不必永久处于可重建状态。
二、旧方案当时为什么合理,现在为何愿意后退?
最早的恢复思路是尽可能把 Run 的内部游标和检查点存下来。发生重启时,重新装配待执行节点,继续原来的模型轮次。这有合理的工程动机:不丢掉昂贵研究工作,尽量避免用户重复提问,也让合法的人工回复能够接回原来的研究。
随着执行系统加入 Child、Peer、人工安全点、模型工具表面版本和并发状态,承诺的范围变大了。恢复一个旧游标,可能需要解释当时的工具目录、专家方法版本、授权、尚未结算的外部调用、并发等待者和历史格式。如果恢复程序只是“成功拼回了对象”,却不知道某次请求是否真的已经执行,重复副作用风险仍然存在。
这次选择不是发现“检查点毫无价值”,而是重新定义产品目标:服务端崩溃后允许丢失尚未提交的进度;保留已经完成的资料和成果;用户可以明确启动新 Run 接着做。 现有合法的 Human/Child 等待与服务器未中断的后台运行仍然保留。
这是比通用持久工作流更弱、也更可解释的承诺。作为对照,Temporal 的 Workflow replay依据持久事件历史重建工作流状态,并区分 replay 与重新执行外部活动。ETF Agent 没有把自身实现称为 Temporal,也不试图在所有历史模型协议上复制同等级别的恢复保证;我们选择按原实际效果和产品可接受的进度损失收缩范围。
三、取消旧图重建,不能取消原效果
中断时至少有三种独立事实。第一种是已经提交的结果:原工具回执、带身份的研究材料、已发布正文或文件。第二种是未确认的效果:请求可能发给了 Provider,响应中断,账目和结果都不完整。第三种是执行位置:原模型本来打算下一步做什么。
第三种可以丢失;前两种不能被顺便改写。
普通 Root 冷重入如果没有被接受的控制唤醒,恢复入口不再直接构造旧模型循环。它沿原 Store/Recovery 的租约与围栏检查身份,只由原终结负责人决定怎样结算。已有合格的正文时可以保留部分结果;没有可交付正文时不能凭几个工具日志冒充完成。已经提交的内容按原 Publication 和来源资格可见,旧执行不因页面重开而再次派发。
图中关系(文字对照):
- 浏览器断开 → 服务端继续运行。
- 服务端继续运行 → 读取原运行。
- 服务端中断 → 原恢复负责人。
- 原恢复负责人 → 封存并结算。
- 封存并结算 → 保留已提交事实。
- 保留已提交事实 → 明确的新运行。
- 明确的新运行 → 读取符合条件的材料。
- 已接纳的控制唤醒 → 范围内检查点恢复。
这张图表示三条不同入口,而不是自动串行流程。浏览器重连只读;普通中断要由原 Owner 安全结算;获准的控制唤醒继续走它已有的受限恢复通道。
为什么要强调围栏(fence)?因为一个进程暂时不在本机内存表里,不代表它已经丧失写入资格。若旧执行尚可能有迟到结果,新的恢复者不能仅凭“看不到它”就夺取终态、清空预算或覆盖已提交正文。原租约、持久执行身份、取消事实和事务约束,才决定哪些操作还能写入。
尤其不能把未收到 ACK 当作未执行:发往外部模型或工具的动作结果未知时,相关耗费应保持未知,而不是推断为零、退款或盲目重试。跨外部 Provider 与本地数据库,这套策略不声称达成端到端 exactly-once。
四、短期继续研究:继承材料,不继承执行权
如果原 Run 被诚实标为中断,用户点击“继续”,新任务究竟能继承什么?
可以继续使用的,是原负责者已保存、当前用户仍获准读取、身份与时点可追溯的材料:旧报告片段、来源观察、合法结果引用、明确的计算输入/输出、尚未解决的研究问题。新 Run 要重新经过自己的准入,继承的是研究背景,不是旧 Run 的未用预算、旧模型请求 ID、租约或某条未执行的工具提议。
用一个合成例子说明:
- 第一个 Run 阅读了基金 A 的公开持仓,保存日期和覆盖范围;计算结果也取得原始结果引用。随后进程中断,原有结论已提交,但关于基金 B 的资料是否足够仍待确认。
- 用户再次打开页面,只能看到原始结算与已保存的部分正文;系统不为了显示历史就重新抓取基金 A。
- 用户明确发起新 Run。系统带入仍有资格的 A 资料、计算结果、方法和“B 的覆盖率未确认”这个待解决问题。新的 Main 可以先核对 B 的最新披露,再决定是否需要补算,而不是重复整套旧 Child 步骤。
这里“复用旧计算”不等于“旧结论仍然适用于今天”。披露日期、计算窗口、覆盖率、来源授权和后续修订都要保留:资料过期就必须说过期,已撤权的内容不能因出现在历史摘要里而重新获得读取资格。模型此前判断“问题已解决”也只是历史评估,不是金融事实。
在现有受控跨进程研究验证中,原 Tool、Python 计算和 ResultRef 可以通过 PostgreSQL 的合法安全点在另一操作系统进程被重新读取,而不重跑已提交的来源和计算。这证明了已保存观察可以跨进程消费这条有限路径,不代表任意崩溃位置、所有历史协议和真实模型研究质量都已验证。
五、保留受限唤醒,不要一刀切关掉恢复
最容易出现的错误,是听完“旧图可以放弃”,就把全部 checkpoint 和 wake 一起删掉。这样会破坏另一类已有的产品能力:Child 的阶段性结果和用户回复,可能正停在一个获准的人工安全点上,等待原身份继续。
因此我们区分两件事。普通 Root 冷重入不默认拥有执行权;已被原控制 Owner 接受的 Human/Child 唤醒,仍可在受限当前格式、原请求封存身份、租约和资源余量内继续。一个旧数据库记录存在,不构成控制唤醒;一个合法的控制唤醒,也不能扩张为对任意历史版本的重放承诺。
同样,完成当前步骤的“状态”并不是继续执行的许可证。外部请求已经提交,结果是否可接受、是否仍有合法后续动作,要由各自持久负责者判断。未知必须保留为未知;当前步骤不可恢复,也不应该抹掉之前已经发布的独立成果。
从工程取舍看,这需要保留一小段当前使用者确实依赖的检查点逻辑,而不是全面删除恢复模块。删掉旧图重建之后,安全点的有效性、当前版本兼容和最终结算反而更值得单独测清楚。
六、把验收从“能重建图”改为“没有丢失该保留的事实”
这次收缩最有意义的验证,不是“模型接着说了几句话”,而是普通冷重入时没有重复外部派发,迟到执行不能改写原终态,原正文/工具结果和未知额度保持,用户新发起的继续能读到适用材料,合法 Human/Child 唤醒又没有被错误阻断。
我们在原生产接口的受控测试中检查过普通 Root 的不再重派、原结果投影以及有界的新 Run 继续;部分跨进程、原唤醒和结果交付路径也各有独立证据。截至 2026 年 10 月 27 日,这些工程进展已出现在主干实现与已接受的部分合同里,但独立中断简化 Spec 的全量正式收尾不能由此推断完成,尤其不能写成真实模型对任意故障都能无损续跑。
这种设计不是每个场景都值得采用。如果合同要求某一步必须完成、不可重复或必须跨任意重启继续,就应使用相称的持久执行技术和外部效果协议,而不是靠一个“继续”按钮满足承诺。反过来,如果目标是长时间研究、允许有限进度损失,那么准确保存结果、显式重启下一轮,可能比维护永久兼容的执行图更符合用户需要。
结语:恢复最重要的是保持事实,不是维持幻觉
“任务从哪里继续”听起来只是游标设计,实际首先是产品承诺和权力分配。
我们保留后台不断线执行,保留合法人工安全点的受限继续,也保留已经结算的资料、计算和正文;放弃的是在不知道完整执行事实时,假装能够重建并安全重跑整张旧图。读者下次设计 Agent 恢复时,先分别写清楚三条保证:客户端重连只读什么、服务端崩溃保留什么、用户新任务可以合法复用什么。只有这些承诺需要同一份执行图时,才值得为它付出通用持久重放的复杂度。
延伸阅读
异步研究:离开页面后怎样按时交付讨论交付时点和用户可见通知;当每个模块都能说“不”讨论局部失败与终结权;已保存的结果为何不等于模型下一轮已经消费讨论研究证据的投影与读取。这些问题相关,但不要求共享一个万能恢复管理器。
ETF Agent 技术文集(欢迎 Star 与讨论)。
原文与交流
阅读原文 · ETF Agent 工程文集(欢迎 Star)。
本文由 AI图鉴自动抓取并翻译,版权归原作者所有;如需查看原文请点下方链接。
The key commitment for long-term research is not to continue from the original model steps after any failure, but rather to study when the user leaves without interruption, the effects that have occurred are not redone and the facts that have been preserved can be used safely again. In products that allow for the loss of unsubmitted progress, the commitment to restore the entire implementation map may be in maintaining more expensive commitments than the users really need.
This is from an ETF Agent contraction of the research recovery mechanism. We already have back-office research, database status, browser reconnection, sub-agents and hand-overs, but we have gradually found that “restoration” is being used to refer to a few things that are completely different in nature: re-opening pages, rebuilding old Run after the process has been interrupted, and continuing research that the users have volunteered to complete. They share information but should not share enforcement powers.
First, let's find out which connection is broken.
The user is supposed to ask a question: “Comparison the holdings of the two funds, reconcile the latest disclosures and mark out where the coverage is inadequate.” Main has read some of the information, and one Child is still working on PDF, and the user has turned off his mobile phone page.
The browser does not close to the end of the service Run. If the service process is still operational and the old Run has effective access and resources, the back office can continue to do the same. Repeats to read only the text, events and progress already in place and should not be re-modified to re-show the page.
But the problem is different if the implementation of the service is stopped and only a few ModelTurn, Tool, Child and waiting for a reply are left in the database. The fact that you can read these facts does not mean that you can recreate the original group of objects that run simultaneously. The following table removes three often mixed actions:
- Only browsers are disconnected and the original service is still in operation: Reasonable behaviour: the original Run continues within the original mandate; re-reading the original records; and what should not be done: the new model is being used to “restore the display”.
- Interrupted service processes, ordinary Root, no legal control awakening: Reasonable behaviour: former Recovery was responsible for limited settlement and retention of the submitted content; should not have done so: unsupported replay of the old Root/Child implementation map.
- Former Human/Cild control waits for a clearly authorized awakening: reasonable conduct: continuation of permitted motion under the original identity, fence and limited checkpoints; non-action: awakening is granted on the basis of the “old Run database” assumption.
- Users have explicitly asked to continue the research that has just been done: Reasonable behaviour: creating a new Run with independent access, reading what is still legal; what should not be done: moving the old Run ' s implementation of permission and quota to the new Run.
“Run” is an implementation attempt to process a user request; the “implementation chart” is an internal model request, tool call, Child and a waiting relationship. While the former's historical identity can be preserved for a long time, the latter need not be permanently re-established.
II. Why was the old scheme justified at the time and why are we now willing to retreat?
The earliest idea for recovery was to save Run's internal trip signs and check points as much as possible. When restart occurs, reassembly to the executive node and continue the original model rotation. This has a reasonable engineering motivation: not to lose expensive research work, to avoid as many questions as possible from users and to allow legal manual responses to take back the original research.
The scope of the commitment has grown with the addition of the implementation system to Child, Peer, manual safety points, surface versions of modelling tools and co-optation. The restoration of an old cursor may require an explanation of the then catalogue of tools, expert methodological versions, authorizations, unsolved external calls, and waiting and historical formats. If the resumption process is merely “successfully spelled back on the object”, it is not known whether a request has actually been executed and the risk of repetition of side effects remains.
This choice was not to find that "checkpoints are worthless ", but to redefine product targets: service-end collapse allows for the loss of pending progress; retain completed information and results; and users can clearly start a new Run and then do it. Existing valid Human/Child waiting to run uninterrupted backstage with the server remains.
This is a weaker and more understandable commitment than the generic and durable workflow. By contrast, Temporal's WorkFlow replay rebuilds the workflow status based on the history of protracted events and distinguishes between replay and re-executing external activities. ETF Agent did not refer to its own realization as Temporal, nor did he attempt to replicate the equivalent level of restoration assurance on all historical model agreements; we chose to contract the scale of the loss of progress in terms of actual effects and product acceptance.
iii. Unreconstructing old maps, not the effects
There are at least three independent facts at the time of the interruption. The first is the results that have been submitted: the original tool reply, the research material with identity, the body of publish or the document. The second is the unrecognized effect: the request may have been sent to Provider, the response interrupted and the accounts and results incomplete. The third is the implementation position: what the original model was going to do with next step.
The third can be lost; the first two cannot be rewrited.
Normal Root cold re-entry is no longer directly constructed of the old model cycle without the awakening of accepted controls. It examined the identity along the original Store/Recovery lease and fence, leaving the settlement to the sole decision of the original terminal officer. Some of the results can be retained when a qualified body is available; several tool logs cannot be fabricated without the text being delivered. The submissions are shown in terms of the original Publicisation and source eligibility, and the old execution is not sent again because the page is reopened.
Relationships in figures (text contrasts):
- The browser is disconnected and the service continues to run.
- The service continues to run → to read the original run.
- The end of the service interrupted the original recovery of the responsible person.
- The former head of the restoration was sealed and settled.
- Seal and settle the reservation has been submitted.
- The reservation has been submitted to the fact that the new operation is clear.
- A clear new run to read eligible materials.
- accepted control wakes up the checkpoint within range.
This figure indicates three different entry points, not an automatic serial process. The browser is read-only; normal interruptions are settled safely by the original Owner; the approved controls awaken the continuation of its already limited recovery route.
Why stress the fence? Since a process is not on the memory list for the time being, it does not mean that it has been disqualified. If the old implementation is likely to be late, the new restorer cannot simply “not see it” by taking the final, clean-up budget or covering the text. The original lease, the permanent implementation of identity and the removal of factual and transactional constraints determine which operations can be written.
In particular, the non-receipt of ACK cannot be considered as non-executed: when the results of the actions sent to an external model or tool are unknown, the associated costs should remain unknown, not extrapolated to zero, refund or blind retry. A cross-externalProvider with a local database that does not claim to reach end-to-end end-to-end.
IV. CONTINUING STUDIES: RESPONSIBILITY MATERIALS, NOT RESPONSIBILITY
If the original Run is marked as interrupted by honesty, the user clicks "continue" and what will the new job inherit?
It continues to be maintained by the original responsible person, still accessible to current users, identity and time traceability: old report clips, source observations, legitimate results references, clear computational input/output, unresolved research issues. To re-enter the new Run, the successor is the background of the study, not the old Run ' s unused budget, the old model request ID, the lease or the unimplemented tool proposal.
In a synthetic example:
- The first Run read the Fund ' s A open hold, date and coverage; the results were quoted in the original results. The process was then interrupted and the original conclusions were submitted, but the adequacy of information on Fund B remains to be confirmed.
- The user opens the page again and can only see the original settlement and part of the saved body; the system retakes Fund A instead of showing history.
- Users explicitly launched the new Run. The system is still eligible for A information, calculation results, methodology and “B coverage is not confirmed” to be resolved. The new Main can first check B's latest disclosure and then decide whether it needs to be replenished, rather than repeating the old Child steps.
Here, “renewal of old calculations” is not the same as “the old conclusions still apply today”. The date of disclosure, the calculation window, the coverage, the source authorization and subsequent revisions are to be retained: the information must be timed out and the content of the right of withdrawal cannot be re-qualified for reading because it appears in historical summary. The model's earlier judgement that “problems have been solved” was also a historical assessment, not a financial fact.
The original Tool, Python and ResultRef can be read again in another operating system process through the legal security point of PostgreSQL without reruning the submitted sources and calculations. This proves that the limited path by which observations can be preserved can be consumed across the process does not represent an arbitrary breakdown, that all historical protocols and true model studies have been validated.
V. Retention restricted awakening. Do not cut off and recover.
The most likely error is to delete all checkpoints and make after listening to the old picture. This would undermine another category of existing product capacity: Child's milestones and user responses, which may be parked at an approved manual safety point pending the continuation of the original identity.
So we distinguish between two things. Normal Root cool re-entry does not have default enforcement rights; the humanitarian/Child awakening, which was accepted by Owner, can continue in the restricted current format, original request sealed identity, lease and resource balance. The existence of an old database record does not constitute a wake-up call for control; a legitimate wake-up call for control cannot be expanded into a renewed commitment to any historical version.
Similarly, the “state” of completion of the current steps is not a continuing permit. External requests have been submitted, and the outcome is acceptable and legal follow-up remains to be determined by those responsible for them on a permanent basis. The unknown must remain unknown; the current steps cannot be restored and should not be erased from the independent results of the previous publish.
In terms of engineering trade-offs, this requires retaining a small section of the inspection point logic that the current user does rely on, rather than completely removing the recovery module. The effectiveness of the security points, the compatibility of the current version and the final settlement are all the more worthy of being measured separately after the old maps have been rebuilt.
For acceptance check read The fact that the reservation was not lost
The most significant validation of this contraction was not “the model goes on to say a few words”, but rather the normal cold re-entry without repeating external distributions, late execution without rewriting the original state, maintenance of original text/tool results and unknown quota, continued user-initiated access to suitable materials, and legal human/child awakening was not blocked by error.
In the controlled tests of the original production interface, we examined the normal Root no-reassignment, the original results projection and the continuation of the new run; there was independent evidence of partial cross-process, awakening and delivery paths. As at 27 October 2026, the progress of these works had occurred in the implementation of some of the accepted contracts by the backbone, but the full closure of the simplified Spec independent interruption could not be inferred from this, especially since it could not be written as a true model that would have been able to run without interruption.
This design is not worth every scene. If the contract requires that a certain step must be completed, not repeated or must be restarted arbitrarily, a proportionate durable implementation technology and an external effect agreement should be used instead of a “continuing” button to satisfy the commitment. Conversely, if the goal is to study for long periods of time and allow for limited progress losses, the accurate preservation of results and a visible resumption of the next round may be more appropriate to the needs of users than the implementation map, which maintains permanent compatibility.
Conclusion: Restoring is about keeping the facts, not the illusions.
“Where does the mission continue” sounds like a cursory design, beginning with product commitment and power allocation.
We keep the backstage line of continuous execution, keep the limits of the legitimate artificial security points, and also keep the information, calculations and text that have been settled; what is abandoned is to pretend to be able to rebuild and safely rerun the entire old map without knowing the facts of full implementation. Readers next time they design Agent, write three separate guarantees: what the customer read back, what the service crash retains, what a user can use legally for a new job. Only when these commitments require the same implementation chart will it be worth paying for the complexity of their universal and sustained re-establishment.
Extended reading
Step study: How to deliver delivery points and user-visible notifications on time after leaving the page; when each module can say "no" to discuss partial failure and termination rights; why does the saved result not equal the projection and reading of evidence already consumed in the next round of model discussions. These issues are relevant but do not require the sharing of a one-size-fits-all recovery manager.
ETF Agent Technical Papers (welcome Star and Discussion).
Original language and communication
Read the original ETF Agent Engineering Collection (welcome Star).
Auto-fetched and translated by AI Atlas. All rights belong to the original author.