在去年，我们利用大型语言模型来捕捉苍蝇上的虫子
现在，我们分享我们所看到的结果和积极影响
- 我们的论文《在Meta进行实时捕获试验》充分详细介绍了我们最初的部署和测试。
每一个代码的改变都会改进我们的平台，但同时也是推出新的复杂性。 传统测试无法跟上速度。 在代码获得之前积极主动地抓住风险，使我们能够更早地识别风险，做出更有效的反应，并大规模提供可靠的经验。
在去年，我们分享了我们应用大型语言模型(LLMs)生成软件测试的研究。 更具体地说，我们与Just-in-Time Tests(JiTests)分享了我们的工作，这是一种全新的测试方法
自我们第一次使用发布之后的五个月里，我们运用了我们的Catching Jit Tests方法，今天，Catching JiT Testes不再是研究原型；它们成为Meta基础设施的一部分，它们帮助我们的工程师捕捉到更多负载较少的虫子，并且假阳性的可能性较小。
捕捉吉特试验的优势
大多数基于AI的防止代码中的bug的方法使用LLMs和/或代理AI来根据以往的经验来审查代码和检测断层。 他们的工作过程与人文审查者的工作过程相同，只会更快。 捕捉JiTTTests是一个根本不同的方法(表1)：它生成可执行测试，通过变化显示具体的行为差异推出。 虽然它需要更多的计算步骤，但它捕捉到基于LLM-或agent的代码审查可能没有的bug类。
我们是如何在Meta 执行抓JIT测试的
JiTTests是自动生成和设计失败的，在生产降落前在代码变化(diffs)中表面显示bug. 当它们被应用到像Meta那样大的编码库时，一个挑战就是区分强渔获量(即发现真虫的真阳性失败)和弱渔获量(即可能是假阳性测试).
归结为一个具体问题：对于每一个拟议中的diff，我们如何能自动生成在引起严重问题之前找到最严重虫的测试，并且以较高的真实阳性率进行测试，同时又不妨碍我们工程师的速度？
为此，我们开发了两个并行工作流程：
- “有意”工作流程，推断代码更改的意图和风险，并针对可能的故障点进行测试生成。
- “Dodgy diff”工作流程，将每个代码更改当作变异/错误处理，并生成测试，以揭露其行为差异，而无需说明意图。 它含蓄地声称，" didgy " (即 " dodgy " 一词)。
为了过滤假阳性，我们采用了一种传统的基于规则的方法，加上我们称之为“LLM-as-Judge”的综观方法，将几个最先进(SOTA)LLMs的判断汇总起来，它们都负责将弱渔获量归类为可能的真或假阳性。
我们还继续进行人文审查，着眼于减少审查负荷。 我们的目标是保持人类标签的一致性，但减少工程师被引进审查弱渔获的可能性。
如何运作
整个系统的工作流程分五个阶段进行：
第一阶段：瞄准和触发
该系统在前一天提交的备用能力和目标diff上通宵运行，使用类似于diff Risk Score(DRS)的机制并接受过以往代码更改方面的培训，被认为“危险”。 这有助于确保测试生成侧重于最有可能抑制严重回归的diffs.
阶段2：两个 Diff- Aware 测试-生成工作流程
dodgy diff工作流程将每个diff视为变异体，并使用变异导引的LLM基于测试生成，以产生区分diff行为与母体的测试。 这里的目标是尽量扩大弱渔获量，并以此找到其中的强渔获量。
同时，意向意识工作流程使用LLM通过代码更改，diff标题，以及摘要推断diff的意图。 LLM随后描述了“风险”——试图实施意图的方式可能产生错误。 然后，它利用这些风险描述来构建变种人——每个变种人代表一个可能是推出的bug，同时执行推断的意图。
生成测试可以杀死每个变种人(即区分变种行为和父母行为).
这些测试然后在实际测试中进行。 任何失败都作为弱渔获物捕获。
第3阶段：LLM-现任法官集成(自动评估)
一旦产生弱的渔获量，系统必须确定每个渔获量是虚假的正渔获量还是实际的强渔获量(真正渔获量)。
LLM-as-Judge 综艺节目包括LLMs集，回顾：
- 渔获物测试码
- 测试-执行结果(失败代码、消息、执行跟踪)。
- 父母与子女之间的代码变化(diff)。
- 推断的diff意图。
第4阶段：基于规则的评估员(Rub Fake)
我们以更传统的、基于规则的方法补充LLM-as-judge，称为基于规则的假死环境(Rub Fake)。
LLM-as-Judge拥有丰富的语义学，但其判决是非决定性的，难以核实。 相比之下，Rub Fake提供了经过核实的，决定性的判断。 Rub Fake和LLM-as-Judge合作，对弱渔获物候选人进行了总体排名，排名足够高的候选人值得进一步审议。
第5阶段：" 人与人 " (工程师接触)
基于LLM-as-Judge和Rub Fake的排名，系统随后决定是否接触人类工程师。
我们通过人与人之间的接触来减少摩擦，人类工程师设定标准并审查结果
在Meta捕捉JiT试验的影响
随着测试系统的建立，我们的目标是让工程师参与测试结果。
至此，Meta的589名工程师(占我们外联的64%)与Catching Jit Tests进行了接触。 这些工程师报告说，Catching JiT Testes 出现了真正可操作的发现，真实阳性率为60%.
2026年上半年，Catching JiTTTests在diff阶段捕捉74个严重(SEV)私密窃听器，在登陆生产前，并在该时间段内将私密窃听器的总体发生率降低20%，起到了关键作用。
为什么它很重要
代理软件开发正在帮助整个行业的工程师和开发者比以往更快地写作，审查，以及船舶代码。 但更快的部署意味着还需要更快的测试。 每一个在达到生产前被捕获的bug都是一个终端用户不需要处理的。 LLMs能够提供额外的安全层，能够捕捉传统测试可能错过的
捕捉JiTTTests是一种在技术上和社会上都可以扩展的解决方案。 将这些测试与人与人与人结合的方法结合起来，有助于确保工程师很少需要读取测试代码，只需要确认所观察到的行为变化是否有意。
这种方法是概括性的：工程师们及早确定成功标准，让AI进行测试，在总体层面监测结果，并在政策层面而不是每个项目进行干预。
阅读纸张
我们的论文“在Meta的时空捕捉试验生成”在FSE 2026上发表，并在Product@scale上发表，详细介绍了我们在Meta的基础设施中与Catching JitTests的初始部署和测试。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
- Over the last year we’ve been sharing our work leveraging large language models to catch bugs on the fly — a novel approach we call Catching Just-in-Time Tests (JiTTests).
- Now, we’re sharing the results and positive impact we’ve seen with Catching JiTTests deployed into production.
- Our paper, “Just-in-Time Catching Test Generation at Meta,” fully details our initial deployment and testing with Catching JiTTests.
Every code change improves our platform, but it also introduces new complexity. With hundreds of millions of lines of code constantly evolving at Meta, traditional testing can’t keep pace. Catching risks proactively, before code lands, enables us to identify risks earlier, respond more effectively, and deliver reliable experiences at scale.
Over the last year, we’ve shared our research into applying large language models (LLMs) to generate software tests. More specifically, we’ve shared our work with Just-in-Time Tests (JiTTests), a novel testing approach where we leverage LLMs to catch bugs on the fly, even ones that traditional testing might not catch.
In the five months since we first unveiled this work, we’ve deployed our Catching JiTTests approach to determine its scalability and applicability to the wider industry. Today, Catching JiTTests are no longer a research prototype; they’re part of Meta’s infrastructure, and they’re helping our engineers catch more bugs with less load and a smaller chance of false positives.
The Advantage of Catching JiTTests
Most AI-based approaches to preventing bugs in code use LLMs and/or agentic AI to review code and detect faults based on prior experience. They work through the same review process that a human reviewer would, only faster. Catching JiTTests is a fundamentally different approach (Table 1): It generates executable tests that demonstrate a concrete behavioral difference introduced by the change. While it requires more computational steps, it catches classes of bugs that LLM- or agent-based code reviews might not.
How We Implemented Catching JiTTests at Meta
JiTTests are automatically generated and designed to fail, surfacing bugs in code changes (diffs) before they land in production. When they’re applied to a codebase as large as Meta’s, one challenge is distinguishing strong catches (i.e., true-positive failures that reveal real bugs) from weak catches (i.e., tests that may be false positives).
It boils down to a specific question: For every proposed diff, how can we automatically generate tests that find the most severe bugs before they cause severe issues, and do so with a high true-positive rate and without impeding the velocity of our engineers?
To address this, we developed two tandem workflows:
- An “intent-aware” workflow that infers the intention and risks of a code change and targets test generation toward likely failure points.
- A “dodgy diff” workflow that treats every code change as if it were a mutant/bug and generates tests to expose its behavioral differences without reasoning about intent. It implicitly asserts that a diff is buggy (hence the term “dodgy”).
To filter false positives, we employed a traditional rule-based approach combined with an ensemble approach we call “LLM-as-Judge,” which aggregates judgements from several state-of-the-art (SOTA) LLMs, all tasked with classifying weak catches as likely true or false positives.
We also maintained human review, with an eye toward reducing review load. Our goal is to maintain the consistency of human labeling but reduce the likelihood of an engineer being brought in to review a weak catch.
How It Works
The entire system workflow happens in five stages:
Stage 1: Targeting and Triggering
The system runs overnight on spare capacity and targets diffs submitted the previous day that are deemed “risky” using a mechanism similar to Diff Risk Score (DRS) and trained on previous code changes. This helps ensure test generation is focused on diffs that are most likely to contain severe regressions.
Stage 2: Two Diff-Aware Test-Generation Workflows
The dodgy diff workflow treats each diff as if it were a mutant and uses mutation-guided LLM-based test generation to produce tests that distinguish the diff’s behavior from its parent. The goal here is to maximize the pool of weak catches and with it the chances of finding a strong catch among them.
At the same time, the intent-aware workflow uses an LLM to infer the diff’s intention using the code changes, diff title, and summary. The LLM then describes the “risks”— the ways an attempt to implement the intent could produce errors. It then uses those risk descriptions to construct mutants — each representing a bug that could be introduced while implementing the inferred intent.
Tests are generated to kill each mutant (i.e., distinguish mutant behavior from parent).
Those tests are then run on the actual diff. Any that fail are harvested as weak catches.
Stage 3: LLM-as-Judge Ensemble (Automated Assessment)
Once weak catches are generated, the system must determine whether each is a false positive or actually a strong catch (true positive).
The LLM-as-Judge ensemble consists of a collection of LLMs that review:
- The weak-catch test-case code.
- The test-execution outcome (failure code, message, execution trace).
- The code changes between parent and child (the diff).
- The inferred diff intent.
Stage 4: Rule-Based Assessor (RubFake)
We complement LLM-as-Judge with a more traditional, rule-based approach called Rule-Based False-Positive Killing Environment (RubFake).
LLM-as-Judge has rich semantics, but its judgements are non-deterministic and challenging to verify. By contrast, RubFake provides verified, deterministic judgments. Working together, RubFake and LLM-as-Judge produce an overall ranking of weak-catch candidates, with those ranked sufficiently highly being worthy of further consideration.
Stage 5: Human-on-the-Loop (Engineer Reach-Out)
Based on the ranking from LLM-as-Judge and RubFake, the system then decides whether to reach out to human engineers.
Rather than a human-in-the-loop approach, where a human engineer reviews every output, we’ve reduced friction through a human-on-the-loop approach, where human engineers set criteria and review outcomes at an aggregate level rather than supervising every output.
The Impact of Catching JiTTests at Meta
With the testing system in place, our goal is to get engineers to engage with its results.
As of this writing, 589 engineers at Meta (64% of our outreach) have engaged with Catching JiTTests. These engineers have reported that Catching JiTTests surfaces genuinely actionable findings, with a 60% rate of true positives.
In the first half of 2026, Catching JiTTests played a key role in catching 74 serious (SEV) privacy bugs at the diff stage, before they landed in production and lowering the overall occurrence of privacy SEVs by 20% during that timeframe.
Why It Matters
Agentic software development is helping engineers and developers across the industry write, review, and ship code faster than ever before. But faster deployment means there’s also a need for faster testing. Every bug caught before it reaches production is one that end users never need to deal with. We’ve shown that LLMs can provide an additional safety layer capable of catching what traditional tests may miss, demonstrating that AI-generated testing can scale enough to reduce organizational risk.
Catching JiTTests is a solution that is scalable, both technologically and socially. Combining these tests with a human-on-the-loop approach helps ensure that engineers rarely need to read test code and only need to confirm whether an observed behavioral change was intended.
The approach is generalizable: Engineers define success criteria early, let AI execute the testing, monitor the outcomes at an aggregate level, and intervene at the policy level rather than per item.
Read the Paper
Our paper, “Just-in-Time Catching Test Generation at Meta,” which was presented at FSE 2026 and featured at Product@Scale, details our initial deployment and testing with Catching JiTTests in Meta’s infrastructure.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.