OCT. 20, 2026 (英语).
在过去几年中，Meta的火炬号成为了PyTorch生态系统为大规模基因化AI模型培训提供的首要清洁室参考平台。 TorchTitan完全建在PyTorch-内在缩放原始物上，为训练前旗舰密集和混合专家(MoE)建筑提供了未支配的高模块化展示，如Llama 3,Qwen 3和FLUX.
历史上，想在Google Cloud TPU上训练PyTorch模型的开发者面临着一些挑战：要么进行完整的代码库重写成JAX，要么与遗留的火炬 xla进行拼搏。 这一模式已正式改变。 有了火炬TPU的发布，Cloud TPU现在的功能是直接融入PyTorch核心接口的本土后端(torch.device ("tpu")). 允许培训脚本使用 TPU 现在只需要对设备初始化代码进行最小的修改。 然而，提供设备兼容性只是战斗的一半，开发人员也需要高性能的制作级，分布式的培训配方。 TPU上的火炬号就在这里出现。 TorchTitan TPU是Meta的TorchTitan执行的叉子，经过改造，可以进行Google云TPU的模型训练。 该叉子可在https://github.com/google-pytorch/torchtitan/上查阅。
Google和Meta共同开发的TPU本土执行通过直接在火炬组织内部，而不是要求工程团队发明定制的TPU硬化策略或手动低级XLA通信从头通过。 火炬号作为Google云TPU上缩放PyTorch的定型飞行器。
火炬(TorchTitan)为训练前大型密集型号(如Llama 3 70B/405B和Qwen 3 8B/32B)，专家混合型(如Qwen 3 235B MoE)以及扩散变压器(如FLUX.1 12B)提供经过战斗测试的官方食谱。
除了初始的预训外，火炬公司还充当培训后管道内的可伸缩教练，如VERL用于集团相对政策优化(GRPO)和强化学习(RL)对齐。 在这些工作流程中，火炬处理参数更新步骤，同时与VLLM-TorchTPU等高通量服务引擎同步权重。
平行主义也与模型代码分开：诸如完全硬化的数据平行主义和拉伸平行主义等技术被分层到平行主义-不可知模型执行上。 例外是专家平行主义：MOE层通过一个可插件的符号调度器进行明确的通信，每个后端都可以以自己的方式执行。
火炬泰坦从配置树上构建一个模型，在构建模型之前，该树可以重写。 这提供了一个干净的方法，可以将模型的单个块替换为在TPU上运行较好的版本，而不触及模型定义或共享的教练器循环。
我们积极地使用这个。 TPU配方用Splash Notice取代标准注意，这是一个Pallas内核，它处理因果，滑动窗口和包装文件口罩，而没有实现注意矩阵；将最终投影和跨内向损失替换为一个没有将完整对数的Pallas内核，从未将完整对数转录给HBM；并将MOE专家matmuls替换为分组的Pallas内核。 每个内核是一个JAX函数，由火炬TPU的火炬.tpu.pallas.jax op装饰器转换成PyTorch自定义op，这都是新内核需要的胶水。 每个替换都是一个配置旗，因此一个配方可以打开，一个测试可以将其与其替换的平原PyTorch路径进行比较。 内核可以来自tokamax，用于TPU和GPU的OpenXLA内核库。
FSDP2围绕GPU执行模式构建。 它的全加太和减少散射由模块钩作为明确的集体调用而发布，与计算重叠来自在专用的CUDA流上运行它们与事件同步。 编译者看不到这些：集体坐落在追踪区域之外，因此在TPU上是一把火炬。 完全无法捕捉到每个FSDP钩的编译区域断层和整步图。 火炬TPU只提供非Op流和事件对象，以便FSDP2运行；XLA对其没有用，因为它从数据依赖性来调度工作。
SimpleFSDP取而代之的是编译器的视图。 参数sharding是一种DTensor参数化：读取一个模块的重量进行从sharded到复制的再分配，这是全加法，它的自动分解后是减分法。 没有钩子，也没有明确的集体呼号，所以两个集体在后继图中都是普通的节点，无论该图是每块火炬。 编译区域或图形导师的单个 make fx 步骤。 XLA编译器在需要的matmuls前排出全加太，并将减少的散射器与落后的计算器重叠，在一个程序中。 混合精度是同一机制的一部分，这一政策将集合d型(bf16参数)和减少d型(fp32或bf16梯度)设定在该再分配上。 混合硬化为多晶片是二维网格上的相同代码：在ICI上空的切片内硬化，在DCN上空的切片上跨replicate.
FLUX上，这让火炬与JAX MaxDifuncation达到99%的性能均价：SimleFSDP块的每层HLO是一个单一的模块，其重量全加太，matmuls，注意力和规范融合在一起，这是XLA程序为优化而构建的形状。
许多PyTorch训练员或者用CUDA图表急切模式运行，以减少发布俯冲，或者应用火炬。 编译到孤立区域，一般一次一个变压器块。 火炬TPU尚未支持PyTorch火炬。 流式抽象，这限制了将急切基于FSDP2与层式火炬混合的性能。 编译。 SimpleFSDP帮助通过表达参数的硬化方式与编译兼容，无论是使用图层火炬。 编译或全图抓取。
火炬泰坦的图形导师提供了完整的训练步捕捉(向前，丢失和后向)，用PyTorch的make fx跟踪器提前生成一个FX图，跳过AOTAutograd分区，因此整个计算可以被XLA所见。 GraphTrainer还运行优化通道：执行基于图表的激活内存政策，决定保存，重算或卸载什么。 列车步骤随后作为单XLA程序执行。
这是XLA的型号，它使得PyTorch关于TPU的训练在概念上与JAX相同，因为JAX的整列列车台阶是一个jitted功能。 将整个步骤作为一图而拥有也是使得大模型编译技术成为可能的方法：扫描层上，将the decoder层作为一个单一的循环体运行，这样编译时间和内存会随着深度而停止增长；图形分解器，削减步骤分解器，将削减分解为单独编译的前后程序，其激活器被卸载到中间的主机内存。 在Qwen3 235B上，这条路达到了29°MFU.
优化者也可以加入图表：虽然foldd adam优化器性能非常强，但功能性的AdamW更新可以追溯到列车-step(函数 adamw step)，并确保迭代在第一个前置操作和重量回写之间没有主机侧工作。 在Qwen3-8B上，这条路径达到了51-52%的MFU.
下面的每个杠杆在火炬Titan TPU优化指南中都有描述，几乎所有的都已经是wired进入食谱。 对于大多数模型来说，在TPU上获得一个好的MFU是对现有食谱而非新代码的配置变化序列。 命令很重要，因为每个步骤都会改变下一个措施。 火炬泰坦语中的许多优化可以作为即时使用的构件，可以通过一个旗子翻转来启用(见下文插图).
1. 确定基线
- 选择该步骤如何执行：急切、编译或图形导师。 Eager模式在TPU上运行PyTorch模式，几乎没有任何变化，这使得它成为将一个模型上架并调试的正确地点。 它不是一个性能配置：每个操作都是自己的程序，TPU闲置在发布之间。 火炬TPU的延迟和喷射的急切模式(-tpu config.eager mode=DEFER AND FUSE)通过让XLA的引信跨越业务边界来回收一些没有代码变化的急切模式，而食谱则保持其覆盖编译区域以外的任何内容。 启用每个块的火炬。 compile(-compile.enable)是将调试配置变成一个表演器之一的原因：每个变压器块变成一个单一的XLA程序，所以其操作被连接，发布一次而不是一次一次，并与块的集体一起排好时间。 块必须追踪，它排除了里面依赖数据的Python，这就是模型放弃。 Graph Trainer 走得更远(-tpu config.use graph trainer)，让整个培训步骤，前进，丢失，落后和优化的更新，被捕获为一个程序。 整个步骤必须追踪，你得到的是一个步骤，没有急切的胶水，加上大 -- 第5步中的模型技术。
- Shard with SimpleFSP and a mix-精密政策。 SimpleFSDP在后继图(-simpleism.use simple fsdp)中表示作为集体的硬化，因此编译器可以将全加法和减小散数排在它们周围的矩阵乘法上。 FSDP2本身被优化为急切的性能，但也使得难以追踪成图。 与 bf16 参数和 fp32 梯度递减对等：计算以 bf16 运行，而主重和减量则停留在 fp32 中，这是设置每个 TPU 食谱用途。
- 对于大型模型(~100B+参数)，允许扫描或分割图，以及卸载激活。 一个未滚动的阶梯图随着深度的增大而停止编译，它使每层保存的输入都保持活性，直到后传。 通过层层扫描(-tpu config.use scan layers)将the decoder作为单环体运行，因此编译时间和内存停止随深度增长，并卸载一个堆叠的激活器作为主机内存；这是Qwen3 235B和Llama 3 405B的路径。 图表分解器(-tpu config.use graph split)使图层保持未滚动但向前和向后编译，并且可以选择每个图层组，作为单独的程序，激活器在两者之间卸载。 两者都以图形导师为基础。
2. Tune计算和Kernals
- 打开帕拉斯内核 splash 注意(-splash attention 内核.use splash attention 内核) 从未实现注意矩阵，并处理因果，滑动窗口和打包的文件口罩；在长序列长度时，它最为重要。 引信线性横切损耗(PallasCrossEntropyLoss in the cude)计算LM-头的matmul，软max，一次丢失一个词汇瓦，因此完整的对数的伸缩度从来没有写给HBM. 它的主要价值是内存，下一步变为吞吐量。 对于MOE模型，分组的专家matmul和对称的门/上内核(use pallas paired group mm)用每层一个内核来取代一个绕专家的环。 每个都是食谱中的开关，每个用它取代的平原PyTorch路径测试。
- 内核的瓦片大小闪烁的注意，引信丢失和MOE内核各暴露其块大小(sa block QQ和朋友)，默认是起点而非最佳。 在范围VMEM预算(xla tpu scoped vmem limit kib)范围内，将它们用于模型形状和TPU生成，并期望在Trilium上调制的值不会结转到Ironwood. 食谱中包含每个模型和地形的调值。
3. 优化内存和批量大小
- 提高批量。 在小尺度上，每芯片批量是单大杠杆，因为融资是每个发布程序完成的工作。 将它推到模型是HBM绑定的；引信丢失和注意力内核是释放内存的方法。 在大片上，记住，全球批量是每芯片批量乘以芯片计数，所以大小可以达到模型所能使用的程度，而不是芯片所能持有的程度。
4. Tune编译器、通信和概况
- 统领XLA和Libtpu旗每种食谱都带有它的旗套，如SparseCore latency-hiding schedul
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 20, 2026
Over the past years, Meta’s TorchTitan has emerged as the PyTorch ecosystem’s premier clean-room reference platform for large-scale generative AI model training. Built entirely on PyTorch-native scaling primitives, TorchTitan provides an unencumbered, highly modular showcase for pre-training flagship dense and Mixture-of-Experts (MoE) architectures like Llama 3, Qwen 3, and FLUX.
Historically, developers looking to train PyTorch models on Google Cloud TPUs faced some challenges: either undertake a complete codebase rewrite into JAX, or grapple with legacy torch_xla. That paradigm has officially shifted. With the launch of TorchTPU, Cloud TPUs now feature a native backend (torch.device("tpu")) integrated directly into PyTorch's core interface. Enabling a training script to use TPUs now only requires minimal changes to your device initialization code. Providing device compatibility, however, is only half the battle, developers also need high performance production-grade, distributed training recipes. This is where TorchTitan on TPU comes in. TorchTitan TPU is a fork of Meta’s TorchTitan implementation that is adapted to enable model training on Google Cloud TPUs. The fork is available at https://github.com/google-pytorch/torchtitan/
Rather than requiring engineering teams to invent custom TPU sharding strategies or hand-craft low-level XLA communication passes from scratch, Google and Meta co-developed TPU-native execution passes directly inside TorchTitan. TorchTitan serves as the definitive vehicle for scaling PyTorch on Google Cloud TPUs.
TorchTitan provides official, battle-tested recipes for pre-training large dense models (such as Llama 3 70B/405B and Qwen 3 8B/32B), Mixture-of-Experts (such as Qwen 3 235B MoE), and diffusion transformers (such as FLUX.1 12B).
Beyond initial pre-training, TorchTitan serves as the scalable trainer inside post-training pipelines like VeRL for Group Relative Policy Optimization (GRPO) and Reinforcement Learning (RL) alignment. In these workflows, TorchTitan handles the parameter update step while synchronizing weights with high-throughput serving engines like vLLM-TorchTPU.
Parallelism is also kept separate from the model code: techniques such as fully sharded data parallelism and tensor parallelism are layered onto a parallelism-agnostic model implementation. The exception is expert parallelism: the MoE layer performs explicit communication through a pluggable token dispatcher that each backend can implement in its own way.
TorchTitan builds a model from a configuration tree, and that tree can be rewritten before the model is constructed. This offers a clean way to replace individual blocks of a model with versions that run better on TPU, without touching the model definition or the shared trainer loop.
We use this actively. The TPU recipes replace standard attention with Splash Attention, a Pallas kernel that handles causal, sliding-window and packed-document masks without materializing the attention matrix; replace the final projection and cross-entropy loss with a fused Pallas kernel that never writes the full logits tensor to HBM; and replace MoE expert matmuls with grouped Pallas kernels. Each kernel is a JAX function turned into a PyTorch custom op by TorchTPU's torch.tpu.pallas.jax_op decorator, which is all the glue a new kernel needs. Each replacement is a config flag, so a recipe can turn it on, and a test can compare it against the plain PyTorch path it replaces. Kernels can come from tokamax, the OpenXLA kernel library for TPU and GPU.
FSDP2 is built around the GPU execution model. Its all-gathers and reduce-scatters are issued from module hooks as explicit collective calls, and overlap with compute comes from running them on dedicated CUDA streams synchronized with events. None of that is visible to a compiler: the collectives sit outside the traced region, so on TPU a torch.compile region breaks at every FSDP hook and a whole-step graph cannot be captured at all. TorchTPU provides no-op stream and event objects only so that FSDP2 runs; XLA has no use for them, since it schedules work from data dependencies.
SimpleFSDP takes the compiler's view instead. Parameter sharding is a DTensor parametrization: reading a module's weight performs a redistribute from sharded to replicated, which is the all-gather, and its autograd backward is the reduce-scatter. There are no hooks and no explicit collective calls, so both collectives are ordinary nodes in the traced graph, whether that graph is a per-block torch.compile region or the Graph Trainer's single make_fx step. The XLA compiler schedules the all-gathers ahead of the matmuls that need them and overlaps the reduce-scatters with the backward compute, inside one program. Mixed precision is part of the same mechanism, a policy that sets the gather dtype (bf16 parameters) and the reduce dtype (fp32 or bf16 gradients) on that redistribute. Hybrid sharding for multislice is the same code on a 2-D mesh: shard within a slice over ICI, replicate across slices over DCN.
On FLUX this is what brought TorchTitan to 99% performance parity with JAX MaxDiffusion: the per-layer HLO for a SimpleFSDP block is a single module with the weight all-gathers, matmuls, attention and norms fused together, which is the shape of program XLA was built to optimize.
Many PyTorch trainers either run in eager mode with CUDA graphs to cut launch overhead, or apply torch.compile to isolated regions, typically one transformer block at a time. TorchTPU does not yet support the PyTorch torch.Stream abstraction, which limits the performance of mixing eager-based FSDP2 with layer-wise torch.compile. SimpleFSDP helps by expressing parameter sharding in a way that is compatible with compilation, whether using layer-wise torch.compile or full-graph capture.
TorchTitan's Graph Trainer provides full training-step capture (forward, loss and backward), producing one FX graph ahead of time with PyTorch's make_fx tracer, skipping AOTAutograd partitioning, so the whole computation is visible to XLA). GraphTrainer also runs optimization passes: to implement graph-based activation memory policy that decides what to save, recompute or offload. The train step then executes as a single XLA program.
This is the model XLA was built for, and it makes PyTorch training on TPU conceptually the same as JAX, where the whole train step is one jitted function. Owning the whole step as one graph is also what makes the large-model compilation techniques possible: scan over layers, which runs the decoder layers as a single loop body so that compile time and memory stop growing with depth, and the graph splitter, which cuts the step into separately compiled forward and backward programs with activations offloaded to host memory in between. On Qwen3 235B this path reaches 29%+ MFU.
The optimizer can join the graph as well: While the fused_adam optimizer is very performant, a functional AdamW update can be traced into the train-step (functional_adamw_step) and ensures an iteration has no host-side work between the first forward op and the weight write-back. On Qwen3-8B this path reaches 51-52% MFU.
Every lever below is described in the TorchTitan TPU optimization guide, and nearly all of them are already wired into the recipes. For most models, getting a good MFU on TPU is a sequence of configuration changes on an existing recipe rather than new code. The order matters, because each step changes what the next one measures. A lot of optimizations in TorchTitan are available as ready-to-use building blocks that can be enabled by a flag flip (see illustration below).
1. Establish a Baseline
- Choose how the step executes: eager, compile, or Graph Trainer. Eager mode runs PyTorch models on TPU with virtually no changes, which makes it the right place to bring a model up and debug it. It is not a performance configuration: every op is its own program and the TPU idles between launches. TorchTPU's deferred-and-fused eager mode (--tpu_config.eager_mode=DEFER_AND_FUSE) recovers some of that with no code change by letting XLA fuse across op boundaries, and the recipes keep it on to cover whatever sits outside compiled regions. Enabling per-block torch.compile (--compile.enable) is what turns a debug configuration into a performant one: each transformer block becomes a single XLA program, so its ops are fused, launched once instead of one at a time, and scheduled together with the block's collectives. The block has to trace, which rules out data-dependent Python inside it, and that is all the model gives up. Graph Trainer goes further (--tpu_config.use_graph_trainer) and lets the whole training step, forward, loss, backward and optimizer update, be captured as one program. The whole step has to trace, and what you get back is a step with no eager glue at all, plus the large-model techniques in step 5.
- Shard with SimpleFSDP and a mixed-precision policy. SimpleFSDP expresses sharding as collectives inside the traced graph (--parallelism.use_simple_fsdp), so the compiler can schedule all-gathers and reduce-scatters against the matrix multiplies around them. FSDP2 itself is optimized for eager performance, but that also makes it difficult to trace into a graph. Pair it with bf16 parameters and fp32 gradient reduction: compute runs in bf16 while master weights and reductions stay in fp32, which is the setting every TPU recipe uses.
- For large models (~100B+ parameters), enable scan or split the graph, and offload activations. One unrolled step graph stops compiling as depth grows, and it keeps every layer's saved input live until the backward pass. Scan over layers (--tpu_config.use_scan_layers) runs the decoder as a single loop body, so compile time and memory stop growing with depth, and offloads the one stacked activation to host memory; this is the path Qwen3 235B and Llama 3 405B take. The graph splitter (--tpu_config.use_graph_split) keeps the layers unrolled but compiles forward and backward, and optionally each layer group, as separate programs with activations offloaded in between. Both build on Graph Trainer.
2. Tune Compute and Kernals
- Turn on the Pallas kernels. Splash attention (--splash_attention_kernel.use_splash_attention_kernel) never materializes the attention matrix and handles causal, sliding-window and packed-document masks; it matters most at long sequence lengths. The fused linear cross-entropy loss (PallasCrossEntropyLoss in the recipe) computes the LM-head matmul, softmax and loss one vocabulary tile at a time, so the full logits tensor is never written to HBM. Its main value is memory, which the next step turns into throughput. For MoE models, the grouped expert matmul and the paired gate/up kernel (use_pallas_paired_grouped_mm) replace a loop over experts with one kernel per layer. Each of these is a switch in the recipe, and each is tested against the plain PyTorch path it replaces.
- Tune the kernel tile sizes. Splash attention, the fused loss and the MoE kernels each expose their block sizes (sa_block_* and friends), and the defaults are a starting point rather than an optimum. Tune them for the model shape and the TPU generation, within the scoped VMEM budget (xla_tpu_scoped_vmem_limit_kib), and expect values tuned on Trillium not to carry over to Ironwood. The recipes carry the tuned values for each model and topology.
3. Optimize Memory and Batch Size
- Raise the batch. At small scale the per-chip batch is the single largest lever, because it raises the work done per launched program. Push it until the model is HBM-bound; the fused loss and the attention kernel are what free the memory to do so. On a large slice, remember that the global batch is the per-chip batch times the chip count, so size it to what the model can use, not to what the chips can hold
4. Tune Compiler, Communication and Profile
- Tune the XLA and libtpu flags. Each recipe carries its flag bundle, such as the SparseCore latency-hiding schedul
Auto-fetched and translated by AI Atlas. All rights belong to the original author.