- Facebook Feature Framework(F3)是Meta的架构，用于编写，编译，服务作为广告排名模型基础的机器学习功能，其中单项请求可以要求超过20,000个功能。 我们首先在 AI @ Scale 2020 上介绍了F3.
- F3通过将所有特性汇编成一个单一的合并图来降低执行计算费用，但该图也作为释放单元，因此一个单一的失败或无效的定义阻碍了整个释放管道。
——此帖推出 F3闪电，是F3上方搭建的基础设施层，通过将特征发布与共享执行脱钩来解决这一挑战，因此每个功能都是编译，验证，部署，并独立回滚。
- 最难解决的是保持F3的效率提高。 早期的每层发布原型增加了CPU的使用，因为独立特性不再共享计算。 我们的目标是通过四个优化来缩小这一差距：稳定的计算识别器，一个共享的执行计划，一个覆盖所有特性的共享内存缓冲器，以及用于直接阵列访问的密集指数映射。
- 随着F3闪电，可比的提取特性现在达到的生产~3.6x比以前的F3快，基础设施计算成本没有增加。 这一较短的释放周期也使得同期能够进行更多的模型迭代。
F3 共享执行架构的首页
生产机器学习系统很少直接消耗原始数据。 在模型能够作出预测之前，它的特性服务系统可能需要读取存储的信号，过滤事件到时间窗口，计数或和值，使其正常化，并结合中间结果。
完成这项工作的最简单的设计将每个特性都视为一个独立的功能。 这很容易理解，但规模上却变得浪费，因为特性在计算上并不独立。 一个7天的点击计数和一个7天的印象计数可能会读取同一个事件源，并应用相同的时间窗口过滤器。 点击率特性可以再利用这两个结果。 单独运行每个功能会重复这个过滤器，以及每一个其它共享步骤，每一个请求。
F3通过将特征提取作为定向环形图(DAG)来进行不同处理。 每个节点执行一个工作单位，如读取输入，应用过滤器，或计算集合。 一个边缘意味着一个计算取决于另一个计算的结果。 图中的输出是模型所消耗的特性值。
工程师们用一种高级语言独立编写每个专题。 F3编译器将每个定义变成一个小的图，然后将服务使用的每个特征的图合并为一个合并图(图1). 在特性共享一个操作的地方，例如上面的七天过滤器，编译器会将复制件崩溃成一个单一的节点。 共享的作品再按请求运行一次，而不是每个需要的特征运行一次。 在运行时，特征提取是这个合并图的转录。
共享的图形使每个发布全部或无
F3的合并图高效共享计算，该图也是发行版，一个不正确的定义可能阻碍合并图中的所有其他特征。
图2说明了这一挑战。 它的显示 7独立作者特性，A至G，其中五个定义有效，但D和G包含不相关的错误。 “试航1号”在D上建造失败，因此这7艘舰都无一出现。 在D被更正后，“尝试2”建筑达到G，再次失败。 只有在两个定义都固定在“尝试3”之后，完整的合并图才使它成功生产。
随着F3的缩放，失败次数更加频繁。 F3最初服务于几百种特性，在这种规模下合并成功，发布基本平滑。 今天这个数字超过2万个，不同的特征类型需要自己的检查，取决于它们是如何计算的，因此新的特征和修改过的特性被保存到不同的测试栏。 一个已经消耗的模型在发布之前需要与模型相关的测试功能。 这些测试在计算上非常昂贵，以至于它们只在后来的发布阶段运行，而且由于合并的图船作为一个单元，其中的所有其他特征也等待着它们。
F3 闪电：使每个特性自有的释放和共享计算
F3 发布之后，我们对合并图进行了几轮改进。 每一个都带回了一段时间的释放速度，但是随着更多特性的落地，增益被侵蚀，因为所有特性仍然在合并图中一起运出。 因此，F3闪电建造是为了在保持F3效率的同时，使每个特性都能够单独放行。
如图3所示，F3闪电推出的两处变化有助于实现这一目标：
- 部署颗粒性：每个特征都是可独立重放的配置。 它经过验证、装运和回滚而不影响任何其他特性。
- 效率均等：F3的效率来自合并的图表，因此，赋予每个特性自己的释放可以消除其来源。 在保持每个地貌发布的同时，要匹配效率，就需要对F3闪电如何执行进行若干关键优化。
部署颗粒度：F3闪电隔热方式发布
孤立开始于编译器中，其中每个功能单独编译为单一配置。 这种特征配置包含着提取该特征所需的一切——它需要的每一个节点，顺序，没有向邻居借的东西。 如果特征C有问题，则特征A still ship，因为两个配置之间没有关系。
由于功能在发布时不再合并，因此每个功能都可以在自己的日程上选择和发布。 释放路径变成了一个命令排队，而不是一个批次，即刻或根本不是船只。 变革按其解除封锁的方式排列优先次序。 一个没有模型消耗的新特性在标准验证上可以迅速外出，而改变一个已经读取的特性则首先得到额外的测试。
效率均等：F3闪电保护F3服务效率
我们的第一个原型机只分离了部署。 特性独立运出，但执行不再共享，因此原型使用的CPU比传统F3. 每一个独立的发布配置都作为自己的图到达了服务过程，并被单独执行，重复F3合并图所消除的每一个读取，过滤和汇总。
修补是将共享执行带回而不带回合并。 在启动时，F3闪电不会重建一个合并的图表。 它采取活动特性配置，整合他们共享的工作，并安排为单一执行命令。 回到对等阶段采取了四个步骤：一个编译器先决条件，之后是三个启动优化。
第1步：稳定识别器使共享工作可以识别
要复制执行，运行时间必须确认两个独立编译的配置包含相同的计算。 节点数字不能做到这一点，因为数字是每个功能图的本地数字。 同样的7天滤波器可能是一个配置中的节点2，另一个配置中的节点7，插入一个单一节点后将所有内容重新编号。 传统F3从未遇到过这个问题，因为一个编译器运行时看到了每个功能，可以在全球分配标识符。
F3 Lightning 反之，编译器从作品的成份，而不是其位置，即操作，其参数，以及输入的标识符中得出每个标识符(见图6). 由于每个输入的标识符已经反映了其上游的一切，标识符捕捉了产生结果的整个工作链。 如果输入顺序不影响结果，则输入按标准顺序排列。 集合还包括定义其行为的属性，如时间窗口和分组键。 因此，在同一个输入上定义同一操作的两个特性都会得到相同的标识符，即使它们被编译，发布相隔数周。 如果计算的含义发生变化，则其标识符随其变化。
第2步：每个特点都有一个执行计划
在服务启动时，F3闪电构建了一个覆盖所有活动特性的单一执行计划，使用一个地形类，在它所依赖的节点之后放置每个节点。 当几个特性包含一个具有相同标识符的节点时，计划会保存一个副本，并连接每个消费者与之，因此无论有多少特性使用，计算都会运行一次。
合并保证每次计算运行一次，但计划仍然是执行者在每个请求上逐个指针行走的图。 因此，同样的启动通道将它平整成一个命令列表，因此，每个请求都运行一个单向前传递列表，而不是翻转一个图表。
共享一个计划融资是一个风险：单一的失败计算会影响每个特性。 为防止出现这种情况，如果一个计算失败或无法输入，F3闪电标记其下游的一切不准备，执行者跳过工作。 失败只影响依赖它的节点，独立分支持续运行。
第3步：共享记忆缓冲，因为复制不够
在步骤1和步骤2完成后，没有进行两次计算，然而CPU的使用仍然高于传统的F3. 剩下的计算成本来自内存布局。 每个配置都携带了自己的请求时间数据容器，因此中间结果分散在不同的内存分配中。 按计划行走意味着在他们之间跳跃，每次跳跃都要花费一个缓存误差。
F3 Lightning用每个请求一个缓冲器取代了这些每层集装箱。 启动时，每个功能的引用会反弹到缓冲器中的插槽。
单一缓冲器可作出三项改进：
本地性：连续计划步骤现在触碰相邻的内存，因此一个步骤读出的值通常已经在缓存中。 这恢复了合并图一贯的顺序访问模式。 它还取消了对每项请求的每个空间分配。
一个准备向量：一个单个向量，除值之外存储，取代每个地貌的跟踪，因此执行者可以检查一个值是否已经准备好，而不读取值本身。 标记必须分开，因为空列表是有效的结果，所以“空”不能也意味着“尚未计算 ”。
没有复制品：大多数特征值是列表和地图，消费者不再获得自己的复制品。 共享的插槽中包含着对生产者结果的参考，因此20个特性使用的计算被计算一次并存储一次。 单值(scalars)仍然被复制，因为复制一个小值比跟随指针便宜。 这是安全的，因为每个缓冲器只持续一个请求，执行令保证每个结果都超过其最后的阅读器。
步骤 4：密集指数映射删除最后的人均使用成本
价值现在共存，但找到价值仍然昂贵。 每一次访问都必须解决一个识别码，这意味着要挥霍一把钥匙、探测桶和跟踪指针。 Meta服务量的查询在计算上效率不高。
最终定律与第2步和第3步一样适用了启动时一次工作的原则。 F3闪电指定每个独特的标识符一个连续的整数槽，并重写每个配置中的每一个引用符以指向这些槽，因此标识符只在初始化时使用。 读取一个结果成为直接数组索引进入共享缓冲，在请求路径上没有散列和查找表。
有了F3闪电，可比的提取特征现在达到的生产~3.6x比传统的F3更快，基础设施的计算成本没有增加。 这种较短的释放周期意味着模型可以在同一期间得到改进和更多的再训练。 F3大楼闪电还教过u
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
- Facebook Feature Framework (F3) is Meta’s framework for authoring, compiling, and serving the machine learning features underlying ads ranking models, where a single request can require more than 20,000 features. We first presented F3 at AI @ Scale 2020.
- F3 reduced execution computational costs by compiling all features into a single merged graph, but that graph also served as the release unit, so a single failure or invalid definition blocked the entire release pipeline.
- This post introduces F3 Lightning, an infrastructure layer built on top of F3 to resolve this challenge by decoupling feature releases from shared execution, so each feature is compiled, validated, deployed, and rolled back independently.
- The hardest part to solve was preserving F3’s efficiency improvements. Early per-feature release prototypes increased CPU usage because independent features no longer shared computation. We aimed to close this gap through four optimizations: stable computation identifiers, a single shared execution plan, one shared memory buffer across all features, and dense index mapping for direct array access.
- With F3 Lightning, comparable extraction features now reach production ~3.6x faster than the prior F3, with no increase in infrastructure compute cost. This shorter release cycle also enables more model iterations in the same period.
A Primer on F3’s Shared Execution Architecture
Production machine learning systems rarely consume raw data directly. Before a model can make a prediction, its feature-serving system may need to read stored signals, filter events to a time window, count or sum values, normalize them, and combine intermediate results.
The simplest design to complete this work treats every feature as an independent function. That is easy to understand, but it becomes wasteful at scale, because features are not computationally independent. A seven-day click count and a seven-day impression count may read the same event source and apply the same time-window filter. A click-through rate feature may then reuse both results. Running each feature separately would repeat that filter, and every other shared step, on every request.
F3 approached this differently, by representing feature extraction as a directed acyclic graph (DAG). Each node performs one unit of work, such as reading an input, applying a filter, or calculating an aggregation. An edge means one computation depends on the result of another. The graph’s outputs are the feature values a model consumes.
Engineers author each feature independently in a high-level language. The F3 compiler turns each definition into a small graph, then merges the graphs for every feature a service uses into one merged graph (Figure 1). Where features share an operation, such as the seven-day filter above, the compiler collapses the duplicates into a single node. That shared work then runs once per request, not once for every feature that needs it. At runtime, feature extraction is a traversal of this merged graph.
The Shared Graph Made Every Release All-or-Nothing
F3’s merged graph shared computation efficiently, but it came with a major catch: The graph was also the release, and one malformed definition could hold back every other feature in the merged graph.
Figure 2 illustrates this challenge. It shows seven independently authored features, A through G, where five definitions are valid, but D and G contain unrelated errors. The “Attempt 1” build fails on D, so none of the seven features ships. After D is corrected, the “Attempt 2” build reaches G and fails again. Only after both definitions are fixed in “Attempt 3” does the complete merged graph make it successfully to production.
Failures became more frequent as F3 scaled. F3 originally served a few hundred features, and at that size merges succeeded and releases were largely smooth. Today that number is more than 20,000, and different feature types need their own checks depending on how they compute, so new features and modified features are held to different testing bars. A feature that a model already consumes needs model-related tests before it can be released. Those tests are computationally expensive enough that they run only in a later release stage, and, because the merged graph ships as one unit, every other feature in it waits for them too.
F3 Lightning: Making Each Feature Its Own Release and Sharing Computation
After F3 launched, we made several rounds of improvements to the merged graph. Each one brought back release speed for a while, but the gains eroded as more features landed, because all features still shipped together in the merged graph. So F3 Lightning was built to make every feature releasable on its own while preserving F3’s efficiency.
As illustrated in Figure 3, F3 Lightning introduced two changes to help achieve this:
- Deployment granularity: Each feature is an independently releasable configuration. It is validated, shipped, and rolled back without affecting any other feature.
- Efficiency parity: F3’s efficiency came from the merged graph, so giving every feature its own release removes the source of it. Matching that efficiency while keeping per-feature releases required several key optimizations in how F3 Lightning executes.
Deployment Granularity: How F3 Lightning Isolates Feature Releases
Isolation starts in the compiler, where each feature is compiled on its own into a single configuration. That feature configuration carries everything needed to extract that feature — every node it needs, in order, with nothing borrowed from a neighbor. If feature C has a problem, feature A still ships, because the two configurations have no relationship to each other.
Because features are no longer merged at release time, each one can be selected and released on its own schedule. The release path becomes an ordered queue instead of a batch that ships all at once or not at all. Changes are prioritized by what they unblock. A new feature that no model consumes yet can go out quickly on standard validation, while a change to a feature a live model already reads gets extra testing first.
Efficiency Parity: How F3 Lightning Preserved F3’s Serving Efficiency
Our first prototypes separated deployment only. Features shipped independently, but execution was no longer shared, so the prototypes used substantially more CPU than traditional F3. Each independently released configuration arrived at the serving process as its own graph and was executed separately, repeating every read, filter, and aggregation that F3’s merged graph had eliminated.
The fix was to bring shared execution back without bringing the merge back. At startup, F3 Lightning does not rebuild a merged graph. It takes the active feature configurations, consolidates the work they share, and arranges it into a single execution order. Getting back to parity took four steps: a compiler prerequisite followed by three startup optimizations.
Step 1: Stable Identifiers Make Shared Work Recognizable
To deduplicate execution, the runtime must recognize when two independently compiled configurations contain the same computation. Node numbers can’t do this, because numbering is local to each feature’s graph. The same seven-day filter might be node 2 in one configuration and node 7 in another, and inserting a single node renumbers everything after it. Traditional F3 never had this problem, because one compiler run saw every feature and could assign identifiers globally.
F3 Lightning instead has the compiler derive each identifier from what the work does, not where it sits: the operation, its parameters, and the identifiers of its inputs (see Figure 6). Because each input’s identifier already reflects everything upstream of it, the identifier captures the whole chain of work that produced the result. Where input order doesn’t affect the result, inputs are put in a standard order. Aggregations also include the properties that define their behavior, such as the time window and grouping keys. As a result, two features that define the same operation over the same inputs receive the same identifier, even if they were compiled and released weeks apart. If a computation’s meaning changes, its identifier changes with it.
Step 2: One Execution Plan Across Every Feature
At service startup, F3 Lightning builds a single execution plan across all active features, using a topological sort to place every node after the nodes it depends on. When several features contain a node with the same identifier, the plan keeps one copy and connects every consumer to it, so that computation runs once no matter how many features use it.
Merging guarantees each computation runs once, but the plan is still a graph that the executor walks pointer by pointer on every request. The same startup pass therefore flattens it into an ordered list, so each request runs a single forward pass down the list instead of traversing a graph.
Sharing one plan raises a risk: A single failing computation could affect every feature. To prevent this, if a computation fails or its input is unavailable, F3 Lightning marks everything downstream of it as not ready, and the executor skips that work. A failure affects only the nodes that depend on it, and independent branches keep running.
Step 3: A Shared Memory Buffer, Because Deduplication Was Not Enough
After steps 1 and 2 were complete, no computation ran twice, yet CPU use was still above traditional F3. The remaining compute cost came from memory layout. Each configuration carried its own request-time data container, so intermediate results were scattered across separate memory allocations. Walking the plan in order meant jumping between them, and each jump cost a cache miss.
F3 Lightning replaced those per-feature containers with one buffer per request. At startup, each feature’s references are rebound to slots in that buffer.
Three improvements follow from the single buffer:
Locality: Consecutive plan steps now touch adjacent memory, so the values a step reads are usually already in cache. This restores the sequential access pattern the merged graph always had. It also removes the per-feature allocation on every request.
One readiness vector: A single vector, stored apart from the values, replaces per-feature tracking, so the executor can check whether a value is ready without reading the value itself. The marker has to be separate because an empty list is a valid result, so “empty” can’t also mean “not computed yet.”
No duplicate copies: Most feature values are lists and maps, and consumers no longer get their own copy. The shared slot holds a reference to the producer’s result, so a computation used by 20 features is computed once and stored once. Single values (scalars) are still copied, because copying a small value is cheaper than following a pointer. This is safe because each buffer lasts only for one request, and the execution order guarantees each result outlives its last reader.
Step 4: Dense Index Mapping Removes the Last Per-Access Cost
Values now lived together, but finding one was still computationally expensive. Every access had to resolve an identifier, which meant hashing a key, probing buckets, and following a pointer. Lookups weren’t computationally efficient at Meta’s serving volume.
The final fix applied the same principle as steps 2 and 3, doing the work once at startup. F3 Lightning assigns every unique identifier a consecutive integer slot and rewrites every reference in every configuration to point at those slots, so identifiers are used only during initialization. Reading a result becomes a direct array index into the shared buffer, with no hashing and no lookup table on the request path.
With F3 Lightning, comparable extraction features now reach production ~3.6x faster than on traditional F3, with no increase in infrastructure computational cost. That shorter release cycle means models can be improved and retrained more often in the same period. Building F3 Lightning also taught u
Auto-fetched and translated by AI Atlas. All rights belong to the original author.