此次对Meta隐私意识基础设施(PAI)的深度潜水
- 当数据通过Meta的多种数据和处理系统移动时，必须坚持隐私要求。 从历史上看，支持一项新的要求意味着在每个系统中分别执行同样的执行逻辑。 PrivacyLib用一个运行在Meta的数据和处理系统的客户端内部的普通库来取代它。
- 一个系统与PrivacyLib合并一次，新的隐私能力插上，作为再利用这种合并的模块。 隐私Lib每秒进行数十亿次检查，每个模块都模拟和推出变化。
- 对于内部的AI agents，其数据访问路径可能无法事先得知，工具调用支持路径将agent上下文带入隐私Lib. 所传播的上下文将每个访问连接到它的会话，因此模块可以观察数据使用，并屏蔽不允许的操作。
在Meta，许多隐私要求最终得出同样的问题：我们能证明一个数据只有允许它去的地方吗？在Meta的规模上，我们不只依靠人工隐私审计，这种审计可以随着系统的变化而脱离同步，而是开发了隐私意识基础设施(PAI)，以便在基础设施本身内回答这个问题。 PAI提供扎根于Meta基础设施的高效可靠的一流隐私构件，使产品能够快速发展。
在前几篇文章中，我们谈到了数据理解、数据线条和政策区(我们的信息流控制技术)如何在PAI中发挥作用。 现在，我们正在潜入PrivacyLib，一个普通的库，运行在客户内部我们的数百个数据和处理系统在数据访问点。
PrivacyLib在更广泛的PAI工作流程的最后阶段发挥重要作用：持续执行和监测数据流动。
在PrivacyLib之前，我们把隐私执法单独整合到每个数据的客户和每个要求的处理系统的客户中。 采用这种办法，跨系统执行新的要求可能需要几个月。 每项新的要求，无论是目的限制还是数据流动发现，都需要相同的一套能力：涵盖广泛的系统；在数周内而不是数月内部署；在不成为一个暂时性问题的情况下运行在每个数据访问上；在不减少车队的情况下推出执行变化。 随着时间推移，随着要求和系统的演变，这种办法更加难以扩大规模。
隐私Lib将系统集成与要求的执行分开。 我们把每个系统整合一次当在数据访问点可以满足一项要求时，我们将其作为一个共享于已经集成的系统的隐私Lib模块加以实施，因此，一个新的要求不再需要在每个系统中单独实施。
模块有两种：
- 执行模块可以阻止数据访问。 例如，政策区强制执行目的限制。
- 观察模块只记录数据访问。 例如，隐私探测器收集用于发现数据流的运行时信号。
随着内部AI agents成为获取数据的新方法，我们还增加了Agent可观察性模块，以观测agent相关数据访问事件，并帮助保护agents能够到达的数据。 已经与PrivacyLib集成的系统在没有新的集成工作的情况下捡到了它，这就是收购的集成后：每个新的隐私能力作为模块而来，而不是作为一个项目。
Meta规模的工程挑战和隐私方法
隐私Lib在 Meta 运行于广泛的数据访问路径：
- 使用100多个图书馆来获取数据服务，从关键/价值商店到AI功能。
- 提供数千个已部署图书馆版本的制作服务。
- 每日有数百万资产上网。
- 每秒有十亿张支票
如此规模的运营造成了三个工程挑战：
走过：保护信件元数据
以说明Meta的规模，让我们走过一个隐私要求
用户通过诸如Messer等支持端到端加密的应用交换消息。 在信使传递信息时，根据适用的法律，其后端系统处理非内容信息数据(即信息元数据)，如信息发送或发送时的信息，以提高产品性能，发现滥用行为，保护用户免受有害行为的影响。
多种这类后端服务使用在服务过程中运行的客户端库Logger将这个元数据写入数据仓表格。 在写入记录之前，Logger内部的PrivacyLib集成会引用一个执行模块"政策区"(Policy Zones)来检查元数据是否会流到目的表。 这一检查分为两部分：需要哪些投入才能作出决定，以及检查如何在运行时间得到实际评价。 在研究了这两个部分之后，我们审视了PrivacyLib如何跨系统规模，以及如何在不干扰路径中服务的情况下展开变化。
注释元数据
每个 Logger 目的地表都有一个 LoggerConfig 来描述写给它的记录。 元数据Trace LoggerConfig定义了用于信件传送痕迹的这种配置，并将其标记为包含信件元数据：
最终类元数据跟踪LoggerConfig扩展 LoggerConfigBase {
使用MessagingTrafficMetadata PoliticalTrait; (原始内容存档于2018-09-09).
公共函数配置( I)
Logger 外地定义
: Logger 字段定义器 {
$- > 在表级政策下进行注释( T);
/ 外地和目的地定义
}
}
一个伐木目的地可以有很多写法，所以描述其输出数据的注释属于目的地，而不是每个写给它的服务。 如果每个部门分别宣布注释，声明可能会有分歧，或者新作者可以完全省略。 在LoggerConfig中声明一次，让每个编剧对数据都有相同的描述。
对于此目的地，MessagingTrafficMetadata PolicyTrait 提供了 MESSAGING TRAFFIC METADATA 表格级别注释，并在目标上附上了Antropolicy With Tablelevel Policy () 注释。
LoggerConfig是Logger自己的计划，每个系统都用自己的格式描述自己的数据。 要给PrivacyLib一个单一的消费形式，Logger的代码生成管道将配置翻译为PrivacyLib兼容的文物。 生成的元数据TraceLoggerConfigPrivacyLibArtifact来自元数据TraceLoggerConfig，并带有目的地身份和消息注释：
/**
* @ codegen- source 元数据跟踪
*/
类元数据跟踪
扩展 Logger 输入空间LibArtiffact {
最终公用静态函数 get AssetIDImpl ()
: Logger InputAssetID { (英语).
返回 Logger 输入设置ID: get ()
“ metadata Trace 标记符”
);
}
受保护的静态函数获得目的政策变量( P)
：主题政策变量{
返回目的策略：
贸易政策Name:MESSAGING TRAFFIC MEDATA
);
}
/ 省略了其它生成的字段。
}
上面生成的类被Logger的客户端在Hack,Meta的键入PHP语言中使用。 对于它的C++和Python客户端，Logger发布目的地身份和对中心目录的说明。 隐私Lib利用这些信息生成特定语言的文物。 在下一节的C++示例中，PrivacyLib使用LoggerConfig名称为目的地加载C++文物。
运行检查
前一节的文物描述了一个Logger目的地。 隐私 Lib还需要知道Logger写作的意思以及应该运行的模块。 Logger的Hack整合定义
$config - > 图书馆 Config () - > 可操作 PrivacyLib ()
隐私操作：write ()
- >输入( I)
隐私输入( I):
名称 ILogger 输入PrivactyLibPayload 提供程序
),
)
- > 强制执行(
隐私写入执行：zones ()
区数据类型：LOGGER
)
- > 附有说明()
- >启用()
隐私写作执行：:：遥测记录( ) - > 启用( )
// 其他模块
),
);
PrivacyLib Operation::write () 宣布PrivacyLib为 Logger 的写操作，执行() 列出运行在它的模块。 政策区模块可以影响结果，而遥测日志模块记录评价而不影响评价。 在此添加模块不需要更改单个目的地或 Logger 呼叫站点。 ZoneDataType::LOGGER告诉政策区它正在检查Logger目的地。
在C++服务中，Logger构建隐私Lib请求如下：
Const 汽车资产列表 =
std::vector<std::string>{configName};
隐私Lib 构建器( I)
"LOGGER WARITE CLIENT" (英语).
资产列表
LoggerWrite APIenum 帮助器：: api To HashedString ()
政策强制LoggerWriteAPI::log)
打电话者：LOGGER
获取 LoggerWrite 执行 ();
自动结果 =
隐私请求(std: move( 构建者) ) 发送( );
隐私Lib 使用配置Name 来装入目的地的文物。 散列政策强制LoggerWriteAPI::log值映射Logger的API到隐私Lib的写入操作。 剩下的构建者参数确定Logger为调用者，并提供执行写入的实现。 发送 () 引用已配置的模块并记录其结果。 对于政策区模块，PrivacyLib提供了来自生成的文物和当前处理背景的目的地说明。 例如，政策区评价两个条件：
- 当前上下文表示允许使用信息元数据。
- 目的地文物载有MESSAGING TRAFFIC MEDATATA注释。
启用执行时，如果任一条件都失败，政策区模块会拒绝写入。 隐私 Lib将执行模块的结果合并为一个响应。 拒绝在Logger向下游发送记录前停止写入。 观测模块发射遥测而不影响反应。
跨系统扩展隐私检查
隐私Lib使隐私检查能够通过共同的执行模式、轻量级评价和遥测以及覆盖绕过集成客户的请求，在各种不同的系统中进行规模检查。
跨系统共同执行模式
隐私 Lib的集成跨越了非常不同的API的系统：密钥价值存储，图表数据库，ML管道，文件系统，以及Logger等记录系统。 每个集成图显示其访问点为五个共同操作中的一个，PrivacyLib引用其模块：
- 读取：在获取数据时运行。
- 写：存储数据时运行。
- RPC：呼叫另一服务时运行。
- 请求条目：在服务开始处理请求时运行一次。
- 内置执行：将特定计算包在一个更大的请求中，例如评价一个特性。 在计算过程中获得的数据使用其内部隐私环境进行评估。
模块是针对这五个操作而不是针对每个系统的本地API编写的。 读，写，RPC也有操作后的变体，因此观察模块可以在访问决定后运行。 一旦一个系统提供了映射，每个兼容的模块都可以重用它，这就是为什么允许在已经集成的系统上添加一个模块只需要几天的时间。
隐私Lib运行在数据和处理系统的客户端内部，因为客户端已经拥有模块所需的目的身份和请求上下文。 将评价维持在同一进程中，可避免在每次访问时将这些投入跨越进程界限。 这种设计也有两个优点：
- 更丰富的语境：隐私Lib可以使用打电话者的身份、终点和目的，而这些信息仍然可用。 如果在数据存储处对检查进行评估，则经常丢失这些信息。
- 失败隔离：运行该版本代码的服务存在错误变化，将共享的数据存储保存在失败边界之外。
保持计算成本的检查
进行中评价
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
- This latest deep dive into Meta’s Privacy Aware Infrastructure (PAI) takes a look at PrivacyLib.
- A privacy requirement must hold as data moves through Meta’s diverse data and processing systems. Historically, supporting a new requirement meant implementing the same enforcement logic separately in each system. PrivacyLib replaces this with a common library that runs inside the clients of Meta’s data and processing systems.
- A system integrates with PrivacyLib once, and new privacy capabilities plug in as modules that reuse that integration. PrivacyLib performs billions of checks every second, and changes are simulated and rolled out per module.
- For internal AI agents, whose data access paths may not be known in advance, tool calls on supported paths carry agent context to PrivacyLib. The propagated context links each access to its session, so modules can observe data use and block operations that are not allowed.
At Meta, many privacy requirements ultimately come down to the same question: Can we prove that a piece of data goes only where it is allowed? At Meta’s scale, rather than relying solely on manual privacy audits, which can go out of sync as systems change, we have developed Privacy Aware Infrastructure (PAI) to answer this question within the infrastructure itself. PAI offers efficient and reliable first-class privacy constructs embedded in Meta’s infrastructure so that products can keep evolving quickly.
In previous posts, we have covered how data understanding, data lineage, and Policy Zones (our information flow control technology) play a role in PAI. Now, we are diving deep into PrivacyLib, a common library that runs inside the clients of hundreds of our data and processing systems at the point of data access.
PrivacyLib plays an important role in the final stage of the broader PAI workflow: continuously enforcing and monitoring data flows.
Before PrivacyLib, we integrated privacy enforcement separately into the clients of each data and processing system for each requirement. With this approach, implementing a new requirement across systems could take months. Each new requirement, whether purpose limitation or data flow discovery, needed the same set of capabilities: covering a wide range of systems; deploying in weeks rather than months; running on every data access without becoming a latency problem; and rolling out enforcement changes without taking the fleet down. Over time, this approach became harder to scale as requirements and systems evolved.
PrivacyLib separates system integration from requirement implementation. We integrate each system once. When a requirement can be met at the point of data access, we implement it as a PrivacyLib module shared across the systems already integrated, so a new requirement no longer needs a separate implementation in each system.
There are two kinds of modules:
- Enforcing modules can block a data access. For example, Policy Zones enforces purpose limitation.
- Observing modules only record a data access. For example, Privacy Probes collects the runtime signals used to discover data flows.
As internal AI agents became a new way of accessing data, we also added the Agent Observability module to observe agent-related data access events and help safeguard the data that agents can reach. Systems already integrated with PrivacyLib picked it up without new integration work, which is what integrating once buys: each new privacy capability arrives as a module, not as a project.
Engineering Challenges and Approaches of PrivacyLib at Meta’s Scale
PrivacyLib operates across a broad range of data access paths at Meta:
- More than a hundred libraries used to access data services, from key/value stores to AI features.
- Thousands of production services with versions of the deployed library.
- Millions of assets being accessed daily.
- Billions of checks every second.
Operating at this scale creates three engineering challenges:
Walkthrough: Protecting Messaging Metadata
To illustrate how everything comes together at Meta’s scale, let’s walk through a privacy requirement that needs to be upheld when processing messaging metadata.
Users exchange messages through apps like Messenger, which supports end-to-end encryption. As Messenger routes and delivers those messages, subject to applicable law, its backend systems process non-content messaging data (i.e., messaging metadata), such as information about when messages are sent or delivered, to improve product performance, detect abuse, and protect users from harmful conduct.
A variety of these backend services use Logger, a client library that runs in the service process, to write this metadata to data warehouse tables. Before writing a record, the PrivacyLib integration inside Logger invokes Policy Zones, an enforcing module, to check whether the metadata may flow to the destination table. This check is broken down into two parts: what inputs need to be available to make a decision, and how the check is actually evaluated at runtime. After examining these two parts, we look at how PrivacyLib scales across systems and how changes roll out without disrupting the services in the path.
Annotating Messaging Metadata
Every Logger destination table has a LoggerConfig that describes the records written to it. MetadataTraceLoggerConfig defines one such configuration for message-delivery traces and marks it as containing messaging metadata:
final class MetadataTraceLoggerConfig extends LoggerConfigBase {
use MessagingTrafficMetadataPolicyTrait;
public function configure(
LoggerFieldDefiner $fd,
): LoggerFieldDefiner {
$this->annotateWithTableLevelPolicy();
// Field and destination definitions
}
}
A logging destination can have many writers, so the annotation describing its output data belongs to the destination rather than to each service that writes to it. If every service declared the annotation separately, the declarations could disagree, or a new writer could omit it entirely. Declaring it once in LoggerConfig gives every writer the same description of the data.
For this destination, MessagingTrafficMetadataPolicyTrait supplies the MESSAGING_TRAFFIC_METADATA table-level annotation, and annotateWithTableLevelPolicy() attaches it to the destination.
LoggerConfig is Logger’s own schema, and each system describes its data in its own format. To give PrivacyLib a single form to consume, Logger’s code-generation pipeline translates the configuration into a PrivacyLib-compatible artifact. The generated MetadataTraceLoggerConfigPrivacyLibArtifact is derived from MetadataTraceLoggerConfig and carries the destination identity and the messaging annotation:
/**
* @codegen-source MetadataTraceLoggerConfig
*/
class MetadataTraceLoggerConfigPrivacyLibArtifact
extends LoggerInputPrivacyLibArtifact {
final public static function getAssetIDImpl(
): LoggerInputAssetID {
return LoggerInputAssetID::get(
'MetadataTraceLoggerConfig',
);
}
protected static function getPurposePolicyVars(
): TPurposePolicyVars {
return PurposePolicy::makeVars(
TPurposePolicyName::MESSAGING_TRAFFIC_METADATA,
);
}
// Other generated fields omitted.
}
The generated class above is used by Logger’s client in Hack, Meta’s typed PHP language. For its C++ and Python clients, Logger publishes destination identities and annotations to a central catalog. PrivacyLib uses that information to generate language-specific artifacts. In the C++ example in the next section, PrivacyLib uses the LoggerConfig name to load the C++ artifact for the destination.
Running the Check
The artifact from the previous section describes one Logger destination. PrivacyLib also needs to know what a Logger write means and which modules should run. Logger’s Hack integration defines that once for all destinations:
$config->libraryConfig()->enablePrivacyLib(
PrivacyLibOperation::write()
->input(
PrivacyLibInput::instance(
nameof ILoggerInputPrivacyLibPayloadProvider,
),
)
->enforcements(
PrivacyLibWriteEnforcement::zones(
ZoneDataType::LOGGER,
)
->withPropagation()
->enabled(),
PrivacyLibWriteEnforcement::telemetryLogging()->enabled(),
// Other modules
),
);
PrivacyLibOperation::write() declares PrivacyLib’s write operation for Logger, and enforcements() lists the modules that run on it. The Policy Zones module can affect the result, while the Telemetry Logging module records the evaluation without affecting it. Adding a module here requires no changes to individual destinations or Logger call sites. ZoneDataType::LOGGER tells Policy Zones that it is checking a Logger destination.
In C++ services, Logger constructs the PrivacyLib request as follows:
const auto assetList =
std::vector<std::string>{configName};
PrivacyLibBuilder builder(
"LOGGER_WRITE_CLIENT",
assetList,
LoggerWriteAPIEnumHelper::apiToHashedString(
PolicyEnforcedLoggerWriteAPI::log),
Caller::LOGGER,
getLoggerWriteImplementation());
auto result =
PrivacyLibRequest(std::move(builder)).dispatch();
PrivacyLib uses configName to load the destination’s artifact. The hashed PolicyEnforcedLoggerWriteAPI::log value maps Logger’s API to PrivacyLib’s write operation. The remaining builder arguments identify Logger as the caller and provide the implementation that performs the write. dispatch() invokes the configured modules and records their outcomes. For the Policy Zones module, PrivacyLib provides the destination annotations from the generated artifact and the current processing context. For this example, Policy Zones evaluates two conditions:
- The current context represents an allowed use of messaging metadata.
- The destination artifact carries the MESSAGING_TRAFFIC_METADATA annotation.
When enforcement is enabled, the Policy Zones module denies the write if either condition fails. PrivacyLib combines the results from enforcing modules into one response. A denial stops the write before Logger sends the record downstream. Observing modules emit telemetry without affecting the response.
Scaling Privacy Checks Across Systems
PrivacyLib enables privacy checks to scale across heterogeneous systems through a common execution model, lightweight evaluation and telemetry, and coverage for requests that bypass integrated clients.
A Common Execution Model Across Systems
PrivacyLib’s integrations span systems with very different APIs: key-value stores, graph databases, ML pipelines, filesystems, and logging systems like Logger. Each integration maps its access points to one of five common operations, and PrivacyLib invokes its modules at each:
- Read: runs when fetching data.
- Write: runs when storing data.
- RPC: runs when calling another service.
- Request entry: runs once when a service begins handling a request.
- Inline execution: wraps a specific computation inside a larger request, such as evaluating a feature. Data accesses made during that computation are evaluated using its inline privacy context.
Modules are written against these five operations rather than against each system’s native API. Reads, writes, and RPCs also have post-operation variants, so observing modules can run after the access decision has been made. Once a system provides the mapping, every compatible module can reuse it, which is why enabling an additional module on an already-integrated system takes only days.
PrivacyLib runs inside the clients of data and processing systems because the clients already have the destination identity and request context that modules need. Keeping the evaluation in the same process avoids carrying those inputs across process boundaries on every access. This design also has two advantages:
- Richer context: PrivacyLib can use the caller’s identity, endpoint, and purpose while that information is still available. This information is frequently lost if the check is evaluated on the datastore.
- Failure isolation: A faulty change is contained to the service running that version of the code, keeping the shared datastore outside the failure boundary.
Keeping Checks Computationally Inexpensive
In-process evaluati
Auto-fetched and translated by AI Atlas. All rights belong to the original author.