与Ray Train DLCs和EKS普及教育组织分配培训

人工智能

与Ray Train DLCs和EKS普及教育组织分配培训

通常阻止PyTorch训练从缩放到集群的是容器，而不是模型代码。 每个容器都需要相同的CUDA运行时间，PyTorch构建基于它编译，NCCL(NVIDIA集体通信库)，弹性Fabric适配器(EFA)用户空间库，AWS OFI NCCL插件将NCCL连接到全民教育，集群运营商同意的Ray版本。 建造和重塑这幅图像每一次这些棋子的移动都是一个自己的项目，这和你开始训练的模型无关。 雷电列车深层学习容器(DLC)就是这个图像，由AWS建造。 AWS组装堆栈，作为单元进行测试，在Amazon弹性容器登记簿(Amazon ECR)公开发布，并保持补丁。

雷作为两个角色的单一集群，在机器之间运行一个工作。 雷头协调和运行跟踪集群状态的全局控制商店，您提交任务时的仪表板和工作服务器。 雷工节点提供GPU，并运行培训流程。 每个工人加入通过在头部开始时连接到它的地址。 在Kubernetes上，每个都是一个舱。 Kubernetes对于一个雷星团没有内在的想法，所以一定有些东西要创造头和工人的吊舱，让雷在每一个角色中开始正确的角色，把工人指在头部，替换失败的吊舱。 库贝雷这样做。 开源 Kubernetes是一个操作员，新增是一个RayCluster资源：你用一个标语描述一个头组和一个工人组，KubeRay从中构建并维护集群。

一个被标记的图像既作为雷头运行，又作为雷工人运行，无论是KubeRay在Amazon弹性Kubernetes服务(Amazon EKS)上启动，还是你自己在Amazon弹性计算云(Amazon EC2)上启动。 您添加的是配置：哪个实例，每个 pock 得到多少 GPU，哪个网络运输 NCCL 使用，以及哪些额外的 Python 包需要工作。 皆从影像外设。

本帖通过一个样本，在两个g6.12x大例上，精确地使用发布的DLC到微调 Qwen2.5-1.5B，并带有PyTorch Fully Sharted Data Affairs (FSDP)的8个NVIDIA L4 GPU. 训练代码是一个函数，集群是一个RayCluster Express 加上一个变量文件，任何一点都不构建图像。 由于DLC已经承载了全民教育的堆栈，一旦吊舱提出要求，两个实例之间的集体就会冲过全民教育。 整个解决方案都在这个样本库中，它使用与我们多节点的Ray Service走过相同的KubeRay-on-EKS基础。

你得到什么，没有建造任何东西

雷电列车DLC是PyTorch训练形象，雷电的训练图书馆位于顶部。 标记这个样本牵引，public. ecr. aws/deplearning-containers/ray:train-ml-cuda-v1.1

网络排队是最不想拥有的错配不会产生错误。 NCCL降为TCP套接字，任务以速度的一小部分完成。 在DLC中，这些图书馆是一起建造和测试的，AWS闸门在多节点全民教育和多节点KubeRay培训测试上各发布，这些正是本样本使用的路径。

更多属性去除您在每组中要完成的工作：

- 不设登记图像是公开的，因此节点在没有证书的情况下拉动它，没有任何东西建造或推向路上。

- 没有匹配的驱动程序启动时，切入点会检查主机的 NVIDIA 驱动程序，如果它比图像的 CUDA 年长，则自动打开捆绑的 CUDA 前置兼容库。

- 没有头对头的图像 - 图像没有固定的角色。 不管什么开始容器提供射线开始命令，所以一个标签服务每个舱。

- 没有单独的推论图像家族 – Ray Train与Ray Serve DLC和Ray LLM DLC共享射线ECR寄存器，在列车-ml标记前缀下。

配置它，不要重建它

一个培训工作与下一个培训工作之间的所有差异都是围绕图像设定的。 样本使用其中的每个机制：

包每个工作。 图像载着HuggingFace训练堆，因此这个样本的工作不需要任何添加。 当一个实验确实需要图像不携带的库时，Ray在提交时通过runtime env为该任务安装，图像不变。 没有新图像，没有新标签。

运输每份工作。 图像默认为FI PROVIDER=efa，其NCCL配置排除回转和Docker桥接口。 在 EKS 上，您可以将 NCCL SOCKET IFNAME 指向 room 的 eth0。 样本将所有四个网络变量都设置在清单中，因此，行为在一个文件中可见，因为这些是简单的环境变量，单项工作可以通过运行时间 env来覆盖它们。

您选择的版本。 样针为列车-ml-cuda-v1.1，即目前微量释放，因此一个集群从一个运行到另一个运行都是可复制的。 vX.Y标签随安全重建而移动。 完全合格的vX. Y.Z标记在最初的发行建筑时保持冻结，所以当需要字节相同的运行时，请固定一个摘要。 也可以从私人ECR URI中获取同样的图像。

A. 如何划分分组

所有节点都坐落在一个虚拟私人云(VPC)的私人子网中，并且只能通过一个NAT网关进入互联网，这就是每个节点如何从Amazon ECR Public中拉出DLC，以及工人如何从Hugging Face下载模型重量。 一个m7i.x大构成系统节点组，托管kube-system加载，KubeRay操作员和Ray头。 头部在8265号港口和6379号雷的控股商店运行工作服务器，它广告中没有GPU，因此没有培训工作获得在上面。 工作从头舱内通过kubectl exec提交，通过EKS API，因此8265端口从未在集群外暴露。

培训涉及2个g6.12x大实例，每个实例有4个NVIDIA L4 GPU、48个vCPU、192个GB内存、40个Gbps联网和一个全民教育接口。 Ray Train将四个工人放在每个舱内：一次排名0-3，另一次排名4-7。 因为全民教育流量不会跨越可用区

雷列车在集装箱里做什么

DLC提供堆栈。 Ray Train提供了本来会成为发射装置代码的部分。 多节点 PyTorch通常是指在每个主机上开始每个GPU一个进程，并给每个主机一个等级，世界大小和一个会合地址。 通过Ray Train，您将训练循环作为函数写入，并用缩放Config传递给火炬Trainer. 雷在集群有GPU的地方每GPU开始一个工人，形成火炬。 分布过程组，并在每个工人身上运行功能。

这个样本向Ray Train询问FSDP，它将模型的权重，梯度和优化状态分成了所有8个GPU，而不是把它们复制到每个单元。 bfloat16中的Qwen2.5-1.5B需要约12GB的这4个角。 硬化了八种方式，每个L4 GPU持有大约1.5GB. 价格是通信：FSDP在使用前收集每层的权重，然后减少散射梯度，因此整个模型在每一步的排名之间交叉。 在这两个例子之间，交通是DLC的全民教育堆载的。

先决条件

开始前必须具备以下条件：

- 工具：AWS 命令行界面(AWS CLI)v2 签名到目标账户，eksctl, kubectl, Helm，以及 GNU 的 envsubsite.

- 配额：在AWS地区，至少有一个区提供两例G-Family On-Demand能力。 如果您已经为实例保留了状态，请设置 CAPACITY RESERVATION ID。 脚本将节点组发布插入其中，并跳过容量搜索。

- 预算：在GPU实例出现时，大约每小时9.50美元，涵盖EKS控制平面和NAT Gateway。

快速确认CLI指向您期望的账户：

培训守则：一个职能和一个训练员

代码/train.py是整个工作，它并不包含容器，集群或网络的特定内容。 雷运行列车时每GPU一次。 它问雷的级别，加载模型，手为FSDP准备模型，运行一个普通的PyTorch循环：

代码/train.py——每GPU功能(简写)

驱动程序加入运行的集群，计数其GPU并启动导师。 由于工人的计数来自集群，同一个脚本在一个节点或许多节点上运行不变：

代码/train.py——司机(简写)

批次是随机的代号ID，这样运行仅依赖于集群。 样本是根据真正的FSDP集体是否跨越节点和完成的工作来判断的。 对于真实的微调，用Ray Data或DataLoader输油管替换随机变速器。 函数中没有任何变化。

名单：DLC 唯一的配置地点

express/ray cluster.yaml是图像与集群相遇的地方，而且由于图像需求如此之少而短。 头部唯一的特殊设置是NVIDIA VISIBLE DEVICES=void，它阻止了容器运行时间的NVIDIA钩子在一个没有GPU的节点上发生故障。 工人团体拥有其他一切：

明细/射线组群.yaml——工人组群(精简)

读为三决定。

哪个图像：${DLC IMAGE}，未修改，既适用于头部，也适用于工人。

哪个硬件：4个GPU和节点的每个吊舱的单一全民教育接口。 要求vpc.amazonaws.com/efa是让全民教育设备插件将接口挂入容器的原因。 NCCL看不到普及教育

也就是四个环境变量所以没有其他东西可以自动检测

两个较小的细节保持了多节点工作的稳定。 首先，要求应同等限制，使工人处于Kubernetes的保证服务质量(QoS)等级。 这很重要，因为单节奏或被逐出的军衔会拖累其他七位。 第二，num-gpus是来源于GPU请求的同一变量，因此Ray对容量的看法总是与Kubernetes一致。

提出来：一个负责基础设施的指令，然后是工作

配置生活在一个地方，脚本/env.sh：集群名称，区域，Kubernetes版本，节点类型和计数，图像，绑定版本的KubeRay和两个设备插件，以及训练参数。 将其中任何一种导出以改变它运行。 五个剧本都具有特异性，能够安全地重新运行。 部署 croup.sh，部署 node group.sh，在创建或触碰集群资源前，部署 ray train job.sh。 安装 gpu plugins.sh 并安装 kuberay.sh 是调和器并直接通过。

./dept all.sh – 发现实际容量所在，然后在那里建设基础设施。

g6.12x 地区与可提供区之间的大容量日间转移，因此这是起点，而不是在env.sh中被预先猜到的地区。 它首先运行 find gpu capacity.sh，检查每个候选区域：是否在每个可用区提供g6.12x大，该账户的On-Demand G/VT-family vCPU配额是否为所要求的节点计数留有空间，EC2点投放分数作为容量信号。 使用 -- probe 默认部署 all.sh 通过，它通过在每个候选区创建并立即取消真正的容量保留区，得到一个明确的答案。 然后显示计划，并问一次：

回答 y 运行，顺序是部署 croup.sh，部署 node group.sh，安装 gpu plugins.sh，在推荐的区域安装 kuberay.sh，每人跳过自己的重复确认，因为这个区域已经要求 (-y 跳过此提示，并且 -- plan 显示计划没有创建任何东西)。 如果GPU节点组不能在最佳区域中发布，则脚本将其删除并尝试探测器的下一个区域

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Distributed training with Ray Train DLCs and EFA on EKS

What usually stops PyTorch training from scaling out to a cluster is the container, not the model code. Every container needs the same CUDA runtime, a PyTorch build compiled against it, NCCL (NVIDIA Collective Communications Library), the Elastic Fabric Adapter (EFA) userspace libraries, the AWS OFI NCCL plugin that connects NCCL to EFA, and a Ray version the cluster operator agrees with. Building that image and rebuilding it each time one of those pieces moves is a project of its own, and it has nothing to do with the model you set out to train. The Ray Train Deep Learning Container (DLC) is that image, built by AWS. AWS assembles the stack, tests it as one unit, publishes it in Amazon Elastic Container Registry (Amazon ECR) Public and keeps it patched.

Ray runs a job across machines as a single cluster with two roles. The Ray head coordinates and runs the global control store that tracks cluster state, the dashboard and job server you submit jobs to. Ray worker nodes supply the GPUs and run the training processes. Each worker joins the cluster by connecting to the head’s address when it starts. On Kubernetes, each of these is a pod. Kubernetes has no built-in idea of a Ray cluster, so something must create the head and worker pods, start Ray in the right role in each, point the workers at the head, and replace pods that fail. KubeRay does this. It’s an open source Kubernetes operator that adds a RayCluster resource: you describe a head group and a worker group in one manifest, and KubeRay builds and maintains the cluster from it.

A single tagged image runs as both the Ray head and the Ray workers, whether KubeRay starts it on Amazon Elastic Kubernetes Service (Amazon EKS) or you start it yourself on Amazon Elastic Compute Cloud (Amazon EC2). What you add is configuration: which instances, how many GPUs each pod gets, which network transport NCCL uses, and which extra Python packages a job needs. All of it is set from outside the image.

This post walks through a sample that uses the DLC exactly as published to fine-tune Qwen2.5-1.5B with PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs on two g6.12xlarge instances. The training code is one function, the cluster is one RayCluster manifest plus a file of variables, and no image is built at any point. Because the DLC already carries the EFA stack, collectives between the two instances run over the EFA as soon as the pods request it. The entire solution is in this sample repository, and it uses the same KubeRay-on-EKS foundation as our multi-node Ray Serve walkthrough.

What you get without building anything

The Ray Train DLC is the PyTorch training image with Ray’s training libraries on top. The tag this sample pulls, public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1, replaces a build that would otherwise look like this:

The networking row is the one most teams would least like to own. A mismatch there doesn’t produce an error. NCCL drops to TCP sockets, and the job runs to completion at a fraction of the speed. In the DLC, those libraries are built and tested together, and AWS gates each release on multi-node EFA and multi-node KubeRay training tests, which are exactly the paths this sample uses.

A few more properties remove work that you would otherwise do per cluster:

- No registry setup – The image is public, so nodes pull it without credentials and nothing is built or pushed on the way.

- No driver matching – On startup the entrypoint checks the host’s NVIDIA driver and, if it is older than the image’s CUDA, turns on the bundled CUDA forward-compatibility libraries automatically.

- No head-versus-worker images – The image has no fixed role. Whatever starts the container supplies the ray start command, so one tag serves every pod.

- No separate inference image family – Ray Train shares the ray ECR repository with the Ray Serve DLC and Ray LLM DLCs, under the train-ml tag prefix.

Configure it, don’t rebuild it

Everything that differs between one training job and the next is set around the image. The sample uses each of these mechanisms:

Packages per job. The image ships the HuggingFace training stack, so this sample’s job needs nothing added. When an experiment does need a library the image does not carry, Ray installs it for that job at submit time through runtime_env and the image is unchanged. No new image, no new tag.

Transport per job. The image defaults to FI_PROVIDER=efa, and its NCCL configuration excludes the loopback and Docker bridge interfaces. On EKS you point NCCL_SOCKET_IFNAME at the pod’s eth0. The sample sets all four networking variables in the manifest anyway, so the behavior is visible in one file and doesn’t depend on image defaults. Because these are plain environment variables, a single job can override them through its runtime_env.

Versions you choose. The sample pins train-ml-cuda-v1.1, the current minor release, so a cluster is reproducible from one run to the next. The vX.Y tags move with security rebuilds. The fully qualified vX.Y.Z tags stay frozen at the original release build, so pin a digest when you need a byte-identical run. The same image is also available from private ECR URIs.

How the cluster is laid out

All nodes sit in private subnets of one virtual private cloud (VPC) and reach the internet only through a NAT Gateway, which is how every node pulls the DLC from Amazon ECR Public and how workers download model weights from Hugging Face. One m7i.xlarge forms the system node group, hosting kube-system add-ons, the KubeRay operator and the Ray head. The head runs the job server on port 8265 and Ray’s control store on 6379, and it advertises no GPUs, so no training work lands on it. Jobs are submitted from inside the head pod with kubectl exec, through the EKS API, so port 8265 is never exposed outside the cluster.

Training runs on two g6.12xlarge instances, each with four NVIDIA L4 GPUs, 48 vCPUs, 192 GiB of memory, 40 Gbps of networking and one EFA interface. Each runs one Ray worker pod that claims all four GPUs and the EFA interface, and Ray Train places four workers in each pod: ranks 0–3 on one instance, 4–7 on the other. Because EFA traffic doesn’t cross Availability Zones, the scripts place both instances in a single zone that has been checked to offer g6.12xlarge.

What Ray Train does inside the container

The DLC provides the stack. Ray Train provides the part that would otherwise be launcher code. Multi-node PyTorch normally means starting a process per GPU on every host and giving each one its rank, the world size and a rendezvous address. With Ray Train you write the training loop as a function and pass it to TorchTrainer with a ScalingConfig. Ray starts one worker per GPU wherever the cluster has GPUs, forms the torch.distributed process group, and runs your function on each worker.

This sample asks Ray Train for FSDP, which splits the model’s weights, gradients and optimizer state across all eight GPUs rather than copying them to each one. Qwen2.5-1.5B in bfloat16 needs about 12 GB for those four tensors. Sharded eight ways, each L4 GPU holds roughly 1.5 GB. The price is communication: FSDP gathers each layer’s weights before using it and reduce-scatters gradients afterwards, so the whole model crosses between ranks on every step. Between the two instances, that traffic is what the DLC’s EFA stack carries.

Prerequisites

You must have the following in place before you start:

- Tools: AWS Command Line Interface (AWS CLI) v2 signed in to the target account, eksctl, kubectl, Helm, and envsubst from GNU gettext.

- Quota: 96 vCPUs of G-family On-Demand capacity for the two g6.12xlarge instances, in an AWS Region where at least one zone offers them. If you already hold an On-Demand Capacity Reservation for the instances, set CAPACITY_RESERVATION_ID instead. The scripts launch the node group into it and skip the capacity search.

- Budget: about $9.50 an hour while the GPU instances are up, covering both instances, the EKS control plane and the NAT Gateway.

A quick way to confirm that the CLI is pointed at the account that you expect:

The training code: One function and a trainer

code/train.py is the whole job, and it contains nothing specific to the container, the cluster or the network. Ray runs train_func once per GPU. It asks Ray for its rank, loads the model, hands it to prepare_model for FSDP, and runs an ordinary PyTorch loop:

code/train.py — the per-GPU function (abridged)

The driver joins the running cluster, counts its GPUs and starts the trainer. Because the worker count comes from the cluster, the same script runs unchanged on one node or many:

code/train.py — the driver (abridged)

The batches are random token IDs so that the run depends on nothing but the cluster. The sample is judged on whether real FSDP collectives crossed both nodes and the job completed. For a real fine-tune, replace the random tensor with a Ray Data or DataLoader pipeline. Nothing else in the function changes.

The manifest: The only place the DLC is configured

manifest/raycluster.yaml is where the image meets the cluster, and it is short because the image needs so little. The head’s only special setting is NVIDIA_VISIBLE_DEVICES=void, which stops the container runtime’s NVIDIA hook from failing on a node that has no GPU. The worker group holds everything else:

manifest/raycluster.yaml — worker group (abridged)

Read it as three decisions.

Which image: ${DLC_IMAGE}, unmodified, for both head and workers.

Which hardware: four GPUs and the node’s single EFA interface per pod. Requesting vpc.amazonaws.com/efa is what makes the EFA device plugin mount the interface into the container. Without it, NCCL can’t see EFA and falls back to sockets.

Which transport: four environment variables, so nothing is left to auto-detection.

Two smaller details keep a multi-node job stable. First, requests should equal limits, which places the workers in Kubernetes’ Guaranteed Quality of Service (QoS) class. This matters because a single throttled or evicted rank stalls the other seven. Second, num-gpus is derived from the same variable as the GPU request, so Ray’s view of capacity always matches Kubernetes’.

Bringing it up: One command for the infrastructure, then the job

Configuration lives in one place, scripts/env.sh: cluster name, Region, Kubernetes version, node types and counts, the image, pinned versions of KubeRay and both device plugins, and the training parameters. Export any of them to change it for a run. All five scripts are idempotent and safe to re-run. deploy_cluster.sh, deploy_node_group.sh, and deploy_ray_train_job.sh each prompt before creating or touching cluster resources. install_gpu_plugins.sh and install_kuberay.sh are reconcilers and run straight through.

./deploy_all.sh – Finds where the capacity actually is, then builds the infrastructure there.

g6.12xlarge capacity shifts between Regions and Availability Zones day to day, so this is the starting point rather than a Region guessed up front in env.sh. It first runs find_gpu_capacity.sh, which checks, per candidate Region: whether g6.12xlarge is offered in each Availability Zone, whether the account’s On-Demand G/VT-family vCPU quota has room for the requested node count, and the EC2 Spot placement score as a capacity signal. With --probe, which deploy_all.sh passes by default, it gets a definitive answer by creating and immediately canceling a real Capacity Reservation in each candidate zone. It then shows the plan and asks once:

Answering y runs, in order, deploy_cluster.sh, deploy_node_group.sh, install_gpu_plugins.sh, and install_kuberay.sh in the recommended Region, each skipping its own repeat confirmation because this one already asked (-y skips this prompt too, and --plan shows the plan without creating anything). If the GPU node group can’t launch in the best zone, the script removes it and tries the next zone the probe confi

Auto-fetched and translated by AI Atlas. All rights belong to the original author.