并非所有高速互联都是平等的，使用错误的可以相当昂贵：在InfiniBand上运行同样的分布式训练工作而不是NVLink可以降低43%的性能。
历史上，NVLink仅限于在一个单一节点内的通信，这意味着多节点AI工作量被迫通过InfiniBand或等效云(EFA on AWS,GPUDIrect-TCPX on GCP)进行通信。 兹Q 1Z的NVL72系统通过用统一的NVLink织物将所有72GPU连接在一个架子中，从而消除了这一限制，因此不同节点上的GPU可以以更大的带宽的大小顺序交换数据。
然而，最大限度地发挥NVL72系统的好处，在很大程度上取决于工作量安排：分散在架子上的多节点工作仍然受到InfiniBand的瓶颈。 由于硬件故障导致节点脱线，在不同时间完成的工作将自由能力分散到架子上，保留架子位置变得更加困难。
以及SkyPilot平台如何通过地形意识定位、硬件故障现场恢复以及共享集群的自动分解。
大规模培训是一个沟通问题#
边境培训工作跨越了数千个GPU。 这样做需要回答两个问题：
- 你如何在GPU上压扁你的模型，以确保它能真正运行？
- 根据你的计算预算，你能运行多少份的破碎模型以最大化的培训吞吐量？
第一个问题一般以下列几种方式回答：
- Tensor并行性(TP)：在GPU中分割一层的权重矩阵
- 专家平行主义(EP)：将专家混合型(MoE)层的专家放在不同的全球pu上
- 管道平行主义(PP)：在GPU之间按模式分层
第二题的答案设定了数据平行化(DP)的程度：并行训练的硬化模型的复制件数量，每个复制件在训练数据的不同片段上。
在这些平行主义策略中，拉诺平行主义的全减法和专家平行主义的全减法在训练期间都坐落在关键路径上，使它们成为保持NVLink(§9.1.2)的最重要手段。
日语并行主义
tensor并行主义将每层内部的权重矩阵分成GPU，因此每个GPU只存储并计算每层的一块。
变形器层由两个块组成：自觉和多层受体(MLP). 在其核心，每个块是连续两个矩阵乘法(GEMM). 威震天-LM(§3)显示，通过将一个区块的第一个GEMM列和第二个GEMM列按顺序分割，GPU只需要通过每个区块的全减一次同步，在第二个GEMM之后，而不是每个区块之后。
即使有了这种优化，在前传期间，拉伸平行性仍然需要每个变压器层两个全减速。 这些全部减少的步骤在培训期间处于关键路径上，因为它们不能与计算完全重叠。 因此，在NVLink上通常保持了拉诺平行主义：Hugging Face测量了从NVLink移动到InfiniBand时的训练吞吐量下降了43%.
专家平行主义
在混合型专家(MoE)层中，MLP块被许多更小的MLP所取代，称为专家，以及一个路由器，将每个符号发送给一个小子集的专家。 专家平行主义将教育部的层层分割开来，使这些小型、独立的专家被置于不同的GPU上。
在专家平行主义下，每个MOE层在前传，调度和结合过程中需要两个全程通信步骤。 路由器为每个符物选择顶端-k专家，并将符物发送给有这些专家在记忆中的GPU. 然后，每个GPU通过一个单一的组合矩阵乘法，根据收到的符号计算本地专家。 最后，专家产出被送回到每个令牌的原GPU并合并。
随着模型向更多的专家添加更多的MOE层层和每个符号的路由，全程交通量不断增长，引入了沟通瓶颈。
以64路专家平行方式培训教育部模式，将教育部各层专家分布在64个全球pu。 在标准8-GPU服务器上，将全程流量从高带宽的NVLink上移动，并通过更窄的InfiniBand互联，未优化的通信可以消耗高达60%的培训时间(§1.2). 在NVL72系统中，所有64个GPU都停留在一个单一的NVLink域内，有效消除了通信瓶颈(§9.2.2).
推论也需要快速的节点间联网
上文讨论的平行技术并非培训所独有。 除了使推论引擎能够服务于对单个GPU来说太大的模型外，模型平行主义还可以提高吞吐量。 将一个模型的权重分割到更多GPU中可以释放出KV缓存的内存，因此引擎可以处理更多的并行请求或支持更大的上下文windows. 在某些情况下，释放的内存比GPU数量增长得更快：在一个vLLM 基准测试中，在两个GPU之间分割一个模型的拉诺-平行，使KV缓存块的数量增加了13.9x，并提供了3.9x更高的吞吐量。
专家平行主义遵循同样的逻辑。 由于专家平行性低，每个GPU都持有许多专家，为KV缓存留下的空间更少。 WideEP，指跨越多个节点的专家平行配置，显著减少了每GPU的专家人数。 取舍是网络流量：现在每个全网都跨越更多的GPU. 在标准8-GPU服务器上，一个专家平行组最多可以跨8个GPU，然后才能通过InfiniBand进行通信，其中每GPU带宽降低18x. NVL72通过将所有72 GPU保存在NVLink上的架子中来消除这一限制。 在以交互速度服务DeepSeek-R1时，配有32路专家平行的GB200 NVL72可提供高达4.4x的每个GPU的吞吐量比8-GPU B200服务器高，服务器封顶为8路。
架子是新服务器 #
NVL72将NVLink从一个单一的节点延伸至整个架子，因此整个架子表现得像一个具有共享内存的巨大加速器。
NVL72架上的每个GPU都有18个NVLink端口，每个端口提供100GB/s双向带宽，这些端口汇总到每个GPU总共1.8TB/s. 每个NVLink端口在给定的GPU地图上，都显示为18个NVSwitch芯片之一，这些芯片为机架的织物后飞机提供动力。 由于每个NVSwitch芯片接受72条NVLink线路，每个GPU各一个，因此机架内的总内带宽为130 TB/s.
这种网络地形保证织物完全没有阻塞，意味着架中的任何GPU都可以将数据传送给其他任何具有统一耐久性的GPU，使得大规模分布的AI工作量成为理想。
有时更少是更多：在每个架子上使用较少的节点可大大提高可靠性#
有了NVL72系统，整个18个节点架都用于工作量，以最大限度地利用集群。 在实际操作中，队伍通过在每个架子中留下几个备用节点，从他们的集群中实际获得更多的可用能力。
GPU不可靠。 在其千兆瓦级超级集群上，Meta测量了中断(MTBI)之间的平均时间大约为每4GPU节点490天，在为提高可靠性投入大量工程努力之后。 NVL72集群使得72-GPU架成为计算学的原子单位，与构成架的4-GPU节点相比，断层爆炸半径增加了18x. 机架的MTBI由：
根据上述公式，大约每490 / 18 + 27天就中断一个架子。 如果工作被分配到整个18节点的架子上，那么这些中断都使得整个架子无法排期，直到失败的节点修复。 在128个架子集群上，这意味着一个架子大约每5小时就无法用于全架工作。
每个架子保留两个节点作为备用部分改变这个。 当一个节点失效时，一个在同一机架的备用机取代它，所以一个单一的节点失效就不再使整个机架脱离委托。 成本是每个架子有两个节点闲置，而一切都是健康的。
为了量化这种权衡，NVIDIA模拟不同切片大小如何影响128个架(2,304-节点)集群的可用容量，因为没有节点的比重在增长。
在SkyPilot，我们管理着我们数以万计的客户GPU，在生产NVL72集群时，我们测得的缺货率约为2%。 以这个速度，NVIDIA的型号为显示，16节点切片提供的可用容量比18节点切片(2,037对1,602节点)多27%. 他们还以4%击败9号节点切片，尽管两个9号节点切片精确地填充了一个机架：用9号节点切片，每一次故障都要花费半个机架，而用16号节点切片，一个机架的前两个故障则被完全吸收。 与8节点切片相比，16节点切片给每个工作每NVLink域提供了两倍的GPU，同时放弃了不到1%的可用容量，直到节点不可用率超过约3.3%.
16个节点也是分布式AI工作量的自然大小：64个GPU是2个的功率，这与平行主义组的典型大小相匹配。
现代调度员如何让工作在NVLink上保持
到目前为止，我们已经涵盖了为什么最通信重的平行技术属于NVLink,NVL72如何将NVLink从一个单一的节点扩展到整个支架，为什么每个支架有两个备用节点的16节点切片从生产集群中获得最可用的容量。 SkyPilot通过将每个切片放在一个单一的架子上，从硬件故障中恢复到位置，以及随着工作的来来去，保持整个架子的自由，来去不动，将这一点付诸实践。
调度员需要了解硬件
选择正确的切片尺寸只是第一步。 研究人员如何确保每个多节点训练的片段在单一架上实际运行获得？
直肠排程器将多节点工作分成多个支架，迫使工人通过InfiniBand(每方向400Gb/s)而不是NVLink(1.8 TB/s双向):
SkyPilot的地形-智能调度器(TAS)确保工人在同一机架上合用：
对于多机架工作量，TAS将工作分成16节(可配置)切片，并用两个备用节点各自放在自己的机架上。 然后，当SkyPilot的GPU管理器检测到GPU故障时，SkyPilot Frontier Trainer可以热交换一个保留的健康节点的故障节点，而剩下的节点则继续不间断运行。 如果同一架上没有剩余零件，边疆训练员会将故障架上的切片重排为健康架：
为了量化切片式排程的影响，NVIDIA的GR00T团队比较了两种不同的排程政策在GR00T N1.7的预训期间对横跨128个节点的512 GB300 GPU的影响，这是人造机器人的多式联运基础模型。 第一次运行使用一个天真的排程器，将节点放置在任意数量的架子上。 第二次运行计划采用像图10中那样的具有机架意识的切片式政策，将它放在了精确的8个机架上。 Rack-aware 放置融资吞吐量14.4%，将每个迭代从1.59s削减为1.39s.
在一万-GPU集群上，同样的速度节省了大约每个训练小时16.7K美元。 只要做出更好的安排决定
断裂块
迄今为止，我们探索了如何孤立地进行地貌意识排程。 在实践中，大型集群一般是共享的：不同的工作量，如培训前运行、RL推出和批量推论，都争夺同样的资源。 TAS bin在排程时会在同一架子内打包工作量，但工作在不同的时间完成，留下自由节点分散在架子上。 这是分裂：集群有足够的自由GPU来运行一个架子规模的工作，但是由于它们分布在架子上，无法安排工作。
SkyPilot 的自动破损( Auto Defrag) i
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.
Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.
However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.
This blog covers the communication bottleneck in large-scale training and inference, the NVL72 rack-scale fabric, and slice sizing for resilience to GPU failures, as well as how the SkyPilot Platform keeps multi-node jobs rack-local through topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters.
Training at scale is a communication problem #
Frontier training jobs run across thousands of GPUs. Doing so requires answering two questions:
- How do you shard your model across GPUs to ensure that it can actually run?
- Given your compute budget, how many copies of the sharded model can you run to maximize training throughput?
The first question is typically answered with a combination of:
- Tensor Parallelism (TP): splitting a layer’s weight matrices across GPUs
- Expert Parallelism (EP): placing a mixture-of-experts (MoE) layer’s experts on different GPUs
- Pipeline Parallelism (PP): splitting a model layer-wise across GPUs
The answer to the second question sets the degree of Data Parallelism (DP): the number of copies of the sharded model that train in parallel, each on a different slice of the training data.
Of these parallelism strategies, tensor parallelism’s all-reduce and expert parallelism’s all-to-all communication patterns sit on the critical path during training, making them the most important to keep on NVLink (§9.1.2).
Tensor Parallelism
Tensor parallelism splits the weight matrices inside each layer across GPUs, so every GPU only stores and computes a slice of each layer.
Transformer layers are made up of two blocks: self-attention and a multi-layer perceptron (MLP). At its core, each block is two matrix multiplications (GEMMs) in a row. Megatron-LM (§3) showed that by partitioning a block’s first GEMM column-wise and its second GEMM row-wise, the GPUs only need to synchronize via all-reduce once per block, after the second GEMM, instead of after each one.
Even with this optimization, tensor parallelism still requires two all-reduces per transformer layer during the forward pass. These all-reduce steps sit on the critical path during training because they cannot be fully overlapped with computation. As a result, tensor parallelism is typically kept on NVLink: Hugging Face measured a 43% drop in training throughput when it moved off NVLink onto InfiniBand.
Expert Parallelism
In a mixture-of-experts (MoE) layer, the MLP block is replaced by many smaller MLPs, called experts, and a router that sends each token to a small subset of experts. Expert parallelism splits MoE layers such that these small, independent experts are placed on different GPUs.
Under expert parallelism, each MoE layer requires two all-to-all communication steps during the forward pass, dispatch and combine. A router selects the top-k experts for each token and dispatches tokens to the GPUs that have those experts in memory. Each GPU then computes its local experts on the tokens it received via a single grouped matrix multiplication. Finally, the expert outputs are sent back to each token's original GPU and combined.
As models add more MoE layers and route each token to more experts, the volume of all-to-all traffic grows, introducing a communication bottleneck.
Training an MoE model with 64-way expert parallelism spreads each MoE layer’s experts across 64 GPUs. On standard 8-GPU servers, that moves the all-to-all traffic off high-bandwidth NVLink and through much narrower InfiniBand interconnects where unoptimized communication can consume up to 60% of training time (§1.2). On NVL72 systems, all 64 GPUs stay within a single NVLink domain, effectively eliminating the communication bottleneck (§9.2.2).
Inference needs fast inter-node networking too #
The parallelism techniques discussed above are not unique to training. Beyond enabling inference engines to serve models that are too large for a single GPU, model parallelism can also improve throughput. Splitting a model's weights across more GPUs frees up memory for the KV cache, so the engine can process more concurrent requests or support larger context windows. In some cases, the freed memory grows faster than the number of GPUs: in one vLLM benchmark, splitting a model tensor-parallel across two GPUs increased the amount of KV cache blocks by 13.9x and delivered 3.9x more throughput.
Expert parallelism follows the same logic. With a low degree of expert parallelism, each GPU holds many experts, leaving less room for the KV cache. WideEP, which refers to expert-parallel configurations spanning multiple nodes, significantly reduces the number of experts per GPU. The trade-off is network traffic: every all-to-all now spans more GPUs. On standard 8-GPU servers, an expert-parallel group can span at most eight GPUs before it has to communicate over InfiniBand, where per-GPU bandwidth drops 18x. NVL72 removes this limit by keeping all 72 GPUs in a rack on NVLink. When serving DeepSeek-R1 at interactive speeds, GB200 NVL72 with 32-way expert parallelism delivers up to 4.4x higher throughput per GPU than 8-GPU B200 servers, which are capped at 8-way.
The rack is the new server #
NVL72 extends NVLink from a single node to the entire rack, so the whole rack acts like a gigantic accelerator with shared memory.
Every GPU on the NVL72 rack has 18 NVLink ports each delivering 100 GB/s of bidirectional bandwidth, which sums up to a total of 1.8 TB/s for each GPU. Each NVLink port on a given GPU maps to one of 18 NVSwitch chips that power the rack's fabric backplane. Because each NVSwitch chip accepts 72 NVLink lines, one from each of the GPUs, the aggregate internal bandwidth within the rack is 130 TB/s.
This network topology ensures that the fabric is fully non-blocking, meaning that any GPU in the rack can transmit data to any other GPU with uniform latency, making it ideal for large-scale distributed AI workloads.
Sometimes less is more: using less nodes in each rack significantly improves reliability #
With NVL72 systems, it is tempting to allocate entire 18-node racks to workloads to maximize cluster utilization. In practice, teams actually get more usable capacity out of their cluster by leaving a couple of spare nodes in each rack.
GPUs are unreliable. On its gigawatt-scale supercluster, Meta measured a mean time between interruptions (MTBI) of roughly 490 days per 4-GPU node after putting significant engineering effort into improving reliability. NVL72 clusters make the 72-GPU rack the atomic unit of compute, which increases the fault blast radius by 18x compared to the 4-GPU nodes that make up the rack. The MTBI for the rack is given by:
Based on the formula above, a rack would be interrupted roughly every 490 / 18 ≈ 27 days. If jobs are allocated entire 18-node racks, each of those interruptions makes the entire rack unschedulable until the failed node is repaired. On a 128-rack cluster, this means that a rack would become unusable for full-rack jobs roughly every 5 hours.
Reserving two nodes per rack as spares changes this. When a node fails, a spare in the same rack replaces it, so a single node failure no longer takes the entire rack out of commission. The cost is that two nodes per rack sit idle while everything is healthy.
To quantify this trade-off, NVIDIA modeled how different slice sizes affect usable capacity on a 128-rack (2,304-node) cluster as the share of unavailable nodes grows.
At SkyPilot, we manage tens of thousands of our customers’ GPUs, and on production NVL72 clusters we have measured unavailability rates of around 2%. At that rate, NVIDIA’s model shows that 16-node slices provide 27% more usable capacity than 18-node slices (2,037 vs. 1,602 nodes). They also beat 9-node slices by 4%, even though two 9-node slices fill a rack exactly: with 9-node slices, every failure costs half a rack, while with 16-node slices, the first two failures in a rack are fully absorbed. Compared with 8-node slices, 16-node slices give each job twice as many GPUs per NVLink domain while giving up less than 1% of usable capacity until node unavailability exceeds about 3.3%.
Sixteen nodes is also a natural size for distributed AI workloads: 64 GPUs is a power of two, which matches how parallelism groups are typically sized.
How a modern scheduler keeps jobs on NVLink #
So far, we have covered why the most communication-heavy parallelism techniques belong on NVLink, how NVL72 extends NVLink from a single node to the whole rack, and why 16-node slices with two spare nodes per rack get the most usable capacity out of production clusters. SkyPilot puts this into practice by placing each slice on a single rack, recovering from hardware failures in-place, and keeping whole racks free as jobs come and go.
Schedulers need to understand the hardware
Choosing the right slice size is only the first step. How do researchers ensure that each slice of a multi-node training run actually lands on a single rack?
Naive schedulers split multi-node jobs across racks, forcing workers to communicate over InfiniBand (400 Gb/s per direction) instead of NVLink (1.8 TB/s bidirectional):
SkyPilot’s Topology-Aware Scheduler (TAS) ensures that workers are colocated on the same rack:
For multi-rack workloads, TAS splits the job into 16-node (configurable) slices, and places them each on their own rack with two spare nodes. Then, when SkyPilot’s GPU Manager detects GPU failures, SkyPilot Frontier Trainer can hot swap the faulty node with one of the reserved healthy nodes while the remaining nodes continue running without interruption. If there are no remaining spares in the same rack, Frontier Trainer will reschedule the slice on the faulty rack to a healthy rack:
To quantify the impact of slice-based scheduling, NVIDIA’s GR00T team compared the effect of two different scheduling policies during pre-training of GR00T N1.7, a multimodal foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes. The first run used a naive scheduler that placed nodes across an arbitrary number of racks. The second run was scheduled with a rack-aware, slice-based policy like the one in Figure 10, which placed it on exactly 8 racks. Rack-aware placement raised throughput by 14.4%, cutting each iteration from 1.59 s to 1.39 s.
On a 10,000-GPU cluster, the same speedup saves roughly $16.7K per training hour. That’s about $400K saved for every training day, just by making better scheduling decisions.
Fragmentation blocks rack-scale jobs
So far we have explored how topology-aware scheduling works in isolation. In practice, large clusters are typically shared: heterogeneous workloads such as pre-training runs, RL rollouts, and batch inference all compete for the same resources. TAS bin packs workloads within the same rack at scheduling time, but jobs finish at different times, leaving free nodes scattered across racks. This is fragmentation: the cluster has enough free GPUs to run a rack-scale job, but because they’re spread across racks, the job can’t be scheduled.
SkyPilot’s Automatic Defragmentation (Auto Defrag) i
Auto-fetched and translated by AI Atlas. All rights belong to the original author.