本章目标：搞清楚 Ray Train 在分布式训练里到底负责哪一段、 V2 默认开启之后写法变了什么、容错和抢占怎么工作， 以及什么时候该直接用 torchrun 而不是 Ray。 读完你应该能回答：Ray 是"训练框架"还是"进程编排器"？
13.1 定位：Ray Train 封装了什么
先把话说清楚，因为这里最容易产生误解：
Ray Train 不是训练框架，是训练的"声明式启动器 + 进程编排 + 容错外壳"。
它不实现反向传播、不实现集合通信、不实现优化器。分工是：
这张表就是本章的主线，加粗的两行是 Ray Train 的全部价值。
声明式是什么意思
用 torchrun，你要自己算 WORLD_SIZE、RANK、MASTER_ADDR，自己决定在哪台
机器上起几个进程。用 Ray Train，你写的是声明：
ScalingConfig(num_workers=8, use_gpu=True) —— 意思是"给我 8 个 worker，
每个 1 张 GPU"。至于这 8 个 worker 落在哪、怎么起进程、怎么注入环境变量、
挂了怎么拉起来，都是 Ray 的事。
代价：你对进程和拓扑的控制变弱了。比如"节点内 8 卡 NVLink 全连、跨节点
走 IB"这种拓扑敏感的手工排布，Ray Train 只给了粗粒度开关
（ScalingConfig(placement_strategy="PACK"/"SPREAD")，见 13.3）；要精确控制
就得回到 Ray Core 的放置组（placement_group +
PlacementGroupSchedulingStrategy）自己摆 —— 那是第 08 章的调度原语，不是
Ray Train 的 API。
⚠️ 别把"放置组"和"给 bundle 打标签"混为一谈：第 08 章讲的是放置组本身
（bundle、STRICT_PACK、placement_group_bundle_index），没有涉及
bundle 级别的标签（bundle_label）用法 —— 本教程不展开它。用之前先查当前
版本 Ray Core 文档里 scheduling_strategy / bundle label 的说明。
已核实：ScalingConfig 里没有能指定 bundle 的参数 ——
2.58 的完整字段清单见 §13.3 的那张表（num_workers / use_gpu /
resources_per_worker / placement_strategy / accelerator_type /
use_tpu / topology / label_selector / elastic_resize_monitor_interval_s，
外加已弃用的 trainer_resources），没有任何 bundle / placement group 入口。
Ray Train 的放置控制到 placement_strategy（PACK / SPREAD）这一层为止。
13.2 Ray Train V2：默认开启与迁移
- Ray Train V2 自 2.51.0 起默认开启（PR #57857，把 is_v2_enabled() 的
默认值从 False 翻成 True）；在此之前（2.43–2.50）是 opt-in： RAY_TRAIN_V2_ENABLED=1。
- 回退到 V1 要显式设 RAY_TRAIN_V2_ENABLED=0，但 V1 API 已归档到
deprecated 目录，回退是过渡手段而不是长期方案。
- ⚠️ 官方文档有部分页面措辞过时，仍写着"需设 RAY_TRAIN_V2_ENABLED=1"。
以 release notes 与源码为准 —— 版本切换期留下没改干净的段落，在 Ray 的
文档里不是孤例。
V1 与 V2 的写法差异
V2 新增的配置类与 API（2.58 一手清单）：
- LoggingConfig —— 结构化日志配置。2.58 里它只有一个字段： log_level: str = "INFO"（ray/train/v2/api/config.py）；
- ValidationConfig —— 训练中途的验证/评估钩子
（源码里 TrainController 会据此建一个 ValidationManager 并注册成 report/controller/worker 回调）；
- PreemptionError / get_preemption_info() —— 抢占处理（见 13.6）。
迁移建议：如果你的代码库是 2.45 之前写的，先做的不是改 API，
而是确认自己有没有依赖 session 命名空间。那是 V1 的核心特征。
还要确认的第二件事：代码里实例化的是哪个 XxxTrainer。框架专属 Trainer
里有三个已经被删掉了（LightningTrainer / TransformersTrainer /
AccelerateTrainer，2.9 起移除），而 XGBoostTrainer / LightGBMTrainer
仍在且已 V2 化。V2 的推荐路径是 TorchTrainer + 框架自己的胶水 ——
见 13.3 的「已移除的 Trainer 在 V2 下怎么办」。
13.3 现代写法：ScalingConfig + TorchTrainer
先澄清三个高频困惑
① 没有 ray.train.Trainer 这个类。
ray.train 的 __all__ 里不存在通用的 Trainer —— 你要用的是
后端专属的 XxxTrainer：
from ray.train.torch import TorchTrainer # 最常用
from ray.train.xgboost import XGBoostTrainer
from ray.train.lightgbm import LightGBMTrainer # 表格数据另一条路
🔴 别再抄 LightningTrainer / TransformersTrainer / AccelerateTrainer
—— 它们已经从 Ray 里删掉了。
生命周期是：2.7 标记弃用 → 2.8 调用即报错 → 2.9 移除
（REP "Unify Torch based Trainers on the TorchTrainer API"）。
本教程面向 2.58，这三个名字import 就会失败。
它们的功能统一收敛到 TorchTrainer：把框架自己的 Trainer
建在训练函数内部，再用 prepare_trainer / RayTrainReportCallback
把指标与 checkpoint 接回 ray.train.report（§13.7 有完整例子）。
⚠️ ray.train.lightning 这个模块还在，但 __all__ 里剩下的是
prepare_trainer / RayDDPStrategy / RayFSDPStrategy /
RayDeepSpeedStrategy / RayLightningEnvironment / RayTrainReportCallback
—— 没有 LightningTrainer。
（网上和部分教程里写的 ray.train.Trainer(...) 是错的，会 ImportError。
本教程附录 A 早期版本也写错过，已修正。）
② 同名不同类：TorchTrainer 有两个导入路径。
from ray.train.torch import TorchTrainer # ← 用这个(会按开关指向 V1 或 V2)
from ray.train.torch.torch_trainer import TorchTrainer # V1 实现
from ray.train.v2.torch.torch_trainer import TorchTrainer # V2 实现
ray.train.torch.TorchTrainer 会根据 V2 开关帮你选对实现；
直接 import 具体的 .torch_trainer / .v2.torch_trainer 是 V1/V2 混用报错
最常见的原因 —— 别那么写。
③ 容错默认是关的。 FailureConfig.max_failures 默认 0，
意思是"失败一次就放弃"。所以"我配了 FailureConfig 但任务失败后没有恢复"
不是 bug —— 你多半没显式给 max_failures：
from ray.train import FailureConfig, RunConfig
RunConfig(failure_config=FailureConfig(max_failures=3)) # 至少要给这个
（-1 = 无限重试。⚠️ fail_fast 在 V2 下已经不可用 —— 传了就报错，
不是「与 max_failures 互斥」那么温和。）
⚠️ 本书第四轮在这里漏了一个字段，与 §13.5 冲突：V2 的容错预算由 三个字段共同构成 ——
早先这里写的是"只由 max_failures 加 controller_failure_limit 控制"，
漏掉了 max_preemption_failures —— 而抢占与真实故障分开计数正是 V2 的
关键设计（见 §13.5），漏掉它会让读者以为"抢占会吃掉我的重试额度"。
已移除的 Trainer 在 V2 下怎么办
上面①里列的三个"框架专属 Trainer"是 V1 时代的产物：它们把某个框架的
Trainer 整体包进来，再透传一堆 xxx_init_args / xxx_init_kwargs。
V2 里推荐的主路径不是它们，而是 TorchTrainer + 框架自己的胶水。
V2 下的推荐写法（以 Lightning 为例，Transformers 同理 —— 把框架的 Trainer
建在训练函数内部，用 Ray 提供的 callback/工具把指标和 checkpoint 接回
ray.train.report）：
import pytorch_lightning as pl
from ray.train.lightning import RayTrainReportCallback, prepare_trainer
from ray.train.torch import TorchTrainer
def train_loop_per_worker(config):
model = MyLightningModule(...)
ltrainer = pl.Trainer(max_epochs=config["epochs"],
callbacks=[RayTrainReportCallback()]) # 指标/checkpoint → Ray
ltrainer = prepare_trainer(ltrainer) # 换掉与 Ray 冲突的 DDP 包装、logger 等
ltrainer.fit(model, datamodule=MyDataModule(...))
trainer = TorchTrainer(train_loop_per_worker,
scaling_config=ScalingConfig(num_workers=4, use_gpu=True))
这样写的好处是：进程编排、容错、抢占处理全部走 13.2–13.5 讲的那套 V2 机制， 而不是被夹在"框架专属 Trainer"的中间层里 —— 那一层的参数透传和 Ray 版本耦合， 是迁移时最容易碎的地方。
⚠️ 名称已核实，签名未核实：2.58 的 ray/train/lightning/__init__.py
里 __all__ 就是
["prepare_trainer", "RayDDPStrategy", "RayFSDPStrategy", "RayDeepSpeedStrategy", "RayLightningEnvironment", "RayTrainReportCallback"]
—— 上面用到的两个名字都在，且没有 LightningTrainer。
但这两个东西的签名（参数名与顺序）本书未逐个核对，
迁移前请查当前版本的 Train 文档。
另外"框架专属 Trainer 哪些能在 V2 里用"这个问题已经不成立了 ——
它们在 2.9 就从 Ray 里移除了（见上面①），所以不存在"哪些仍能配 V1"。
最小可跑的 TorchTrainer：
import ray.train
from ray.train import RunConfig, ScalingConfig
from ray.train.torch import TorchTrainer
def train_loop_per_worker(config):
import torch, torch.nn as nn
# 1) 模型：交给 Ray 包一层 DDP
model = ray.train.torch.prepare_model(
nn.Sequential(nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, 1)))
# 2) 数据：交给 Ray 分片 + 对齐 DistributedSampler
loader = ray.train.torch.prepare_data_loader(
ray.train.get_dataset_shard("train").iter_torch_batches(batch_size=64))
# 3) 训练循环：抄一份 PyTorch 的就行
opt = torch.optim.Adam(model.parameters(), lr=config["lr"])
for _ in range(config["epochs"]):
for batch in loader:
...
opt.zero_grad()
loss.backward() # 梯度同步发生在这一行（NCCL AllReduce）
opt.step()
ray.train.report({"loss": loss.item()})
trainer = TorchTrainer(
train_loop_per_worker,
# 上面的 get_dataset_shard("train") 取的就是这里声明的数据（见「数据入口」）
datasets={"train": ds}, # ← 不要注释掉,否则训练函数取不到分片
train_loop_config={"epochs": 3, "lr": 1e-3},
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
run_config=RunConfig(name="demo-run",
storage_path="/mnt/shared/ray_results"), # 全节点可访问
)
result = trainer.fit()
关键点
（1）prepare_model 做了什么？ 把模型搬到正确的 device，并按你指定的
并行策略换成对应的包装：默认是 DDP（DistributedDataParallel），
要 FSDP 得显式给策略（prepare_model(model, parallel_strategy=...)）。
⚠️ 它不是"看一眼 rank/world_size 自动选包装"的 —— 选哪种并行由参数决定，
rank/world_size 只用来决定"这个 worker 该拿哪一片"。
它不做的是：不改你的模型结构、不改优化器、不改 loss。所以
loss.backward() 触发的是标准 NCCL AllReduce —— 数据面不是 Ray 的。
（2）prepare_data_loader 做了什么？ 给 DataLoader 加分布式采样器，让每个
worker 拿到不重叠的分片，并把 device 设好。配合
ray.train.get_dataset_shard("train")（Ray Data 的 Dataset 会被自动切分）。
好处是数据从对象存储直连训练 worker、不经过 driver；用普通 DataLoader
也行，但要自己处理 sampler 和分片。
（3）ray.train.report 与 Result。 report(metrics, checkpoint=...)
把指标和 checkpoint 上报给 controller。trainer.fit() 返回的 Result：
注意 RunConfig(storage_path, name) 这两项在恢复时必须一致，否则找不到
上次的 run 状态（官方 fault-tolerance 文档明确写了这一点）。
数据入口：datasets= 与 DataConfig 的分工
先把最容易搞反的一件事说清楚：
datasets={"train": ds} 是 V2 的现行写法，不是旧写法。
很多教程（包括本教程的早期版本）把它标成"V1 时代的写法"，那是错的。
V2 改的是别的东西 —— 数据集仍然走 datasets=。
# ✓ V2 现行写法
trainer = TorchTrainer(
train_loop_per_worker,
datasets={"train": train_ds, "valid": valid_ds}, # key 就是分片名
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
)
那 DataConfig 是干什么的？它管的是「怎么切分」，不是「有哪些数据」：
from ray.train import DataConfig
trainer = TorchTrainer(
train_loop_per_worker,
datasets={"train": train_ds, "valid": valid_ds},
# 只控制 ingestion / 切分行为,不承载数据本身
dataset_config=DataConfig(
datasets_to_split="all", # "all"（默认） | ["train", ...]；想关掉切分传 []
execution_options=None, # 透传给 Ray Data 的 ExecutionOptions
enable_shard_locality=True, # 默认 True：分片尽量贴近 worker 所在节点
),
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
run_config=RunConfig(name="demo-run", storage_path="/mnt/shared/ray_results"),
)
训练函数侧按名字取分片（名字就是 datasets 的 key）：
def train_loop_per_worker(config):
train_shard = ray.train.get_dataset_shard("train")
valid_shard = ray.train.get_dataset_shard("valid") # 可能为 None，先判空
for batch in train_shard.iter_torch_batches(batch_size=256, device="cuda"):
...
三点值得说清楚：
- 为什么会混淆：⚠️ DataConfig 不是 V2 新增的 —— 它早在
Ray 2.6/2.7（PR #35236）就存在了，当时是来替代弃用的 DatasetConfig，
比 Train V2（2.43+）早了约一年半。所以 V1 教程里 dataset_config=DataConfig(...) 本来就是常见写法。
真正变的是：V2 把「切分策略」从 Trainer 的隐式行为提升成了显式配置，
并且 datasets= 明确为数据入口；
- 切分发生在 worker 侧，不是 driver —— 底层就是第 12 章 §12.2 讲的
streaming_split（每个 worker 拿一个互不重叠的流，数据直连 worker，不经过
driver 的 Python 堆）；
- get_dataset_shard() 只能在训练函数里调，driver 上调不到东西。
⚠️ DataConfig 的精确字段名以你所用版本为准：本书写作时
datasets_to_split 与 execution_options 是核对过的形参。
第六轮标"未确认"的第三个字段现已核实：2.58 的
ray/train/_internal/data_config.py 里，DataConfig.__init__ 的完整签名是
(datasets_to_split="all", execution_options=None, enable_shard_locality=True)
—— 除了上面两个，还有一个 enable_shard_locality（默认 True，
让分片尽量贴近持有它的 worker 所在节点）。
而 datasets= 这个入口本身是稳定的 —— 拿不准时就用它。
ScalingConfig 的其他字段
num_workers / use_gpu 只是最常用的两个：
这一版 ScalingConfig 的完整公开字段（2.58 实测，可用于自查）：
num_workers / use_gpu / resources_per_worker / placement_strategy /
accelerator_type / use_tpu / topology / label_selector /
elastic_resize_monitor_interval_s —— 外加一个已弃用的
trainer_resources。（num_workers 还接受 (min, max) 二元组，见 §13.6。）
训练函数里的运行时助手
训练函数（train_loop_per_worker）里能用的东西其实就这几类：
⚠️ get_world_size() / get_world_rank() / get_local_rank() 这些
不是 ray.train 的模块级函数 —— 它们是 TrainContext 的方法。
ray/train/__init__.py 的 __all__ 里的函数只有
get_context / get_dataset_shard / get_checkpoint / report
（V2 另有 get_all_reported_checkpoints / get_preemption_info）——
其余导出项都是类（RunConfig、ScalingConfig、Checkpoint…），
没有任何 rank 类助手。
写 ray.train.get_local_rank() 会直接 AttributeError；
正确写法是 ray.train.get_context().get_local_rank()。
本书早先把它列成模块级函数，是抄了非官方的记忆版本 —— 以你的版本上
的 Train API 页为准再核一次（若某版本补了模块级别名，以那个版本为准）。
get_device() 不要自己拼（完整路径是 ray.train.torch.get_device()）。
它会返回 prepare_model 真正用的那个 device
（cuda:0 / cuda:1… / cpu），你手写 torch.device(f"cuda:{rank}") 在多机
场景下会把卡对错。V1 里的 session.get_world_rank() 与 V2 的
ray.train.get_context().get_world_rank() 是同一件事的新旧两种写法
—— 但只有后者存在：ray.train.get_world_rank() 这个模块级写法
不存在（写了是 AttributeError）。
⚠️ 本书早先在这里留了半句"模块级快捷函数与 get_context() 方法都在"，
与本章 §13.3 的警告、§13.11 的小结直接冲突 —— 第五轮已删除。
checkpoint 的正确姿势
import os, tempfile
from ray.train import Checkpoint
with tempfile.TemporaryDirectory() as tmpdir:
torch.save(model.state_dict(), os.path.join(tmpdir, "model.pt"))
ckpt = Checkpoint.from_directory(tmpdir)
ray.train.report({"loss": loss}, checkpoint=ckpt)
不要自己 torch.save 到固定路径然后不上报 —— 那样 Ray Train 不知道
checkpoint 的存在，失败重试时不会拿它恢复。这是最常见的"用了 Ray 但容错
没生效"的原因。
Checkpoint 的造法与读取姿势
🔴 先说一条会直接报错的：2.58 的 ray.train.Checkpoint
（python/ray/train/_checkpoint.py）是一个目录即一切的新类，
只剩两个类方法 + 一个构造函数。老教程里的
Checkpoint.from_dict(...) 与 ckpt.to_dict() 都已经不存在了 ——
它们不是"还有但弃用"，而是会抛 AttributeError：
AttributeError: The new `ray.train.Checkpoint` class does not support `from_dict()`.
Instead, only directories are supported.
_checkpoint.py 里的 _CheckpointMetaClass.__getattr__（第 30 行起）
对这些名字做了拦截并抛迁移错误：
要存一个字典怎么办：自己 pickle 落到目录里，再 from_directory
（这正是官方迁移提示给的写法）：
with tempfile.TemporaryDirectory() as d:
with open(os.path.join(d, "data.pkl"), "wb") as fp:
pickle.dump({"some": "value"}, fp)
ray.train.report(..., checkpoint=Checkpoint.from_directory(d))
保存权重（大模型版）：
import os, tempfile, torch
from ray.train import Checkpoint
def train_loop_per_worker(config):
model = ray.train.torch.prepare_model(...)
# ① 恢复：有 checkpoint 就先加载
ckpt = ray.train.get_checkpoint()
if ckpt is not None:
# as_directory() 永远返回上下文管理器 —— 不要写 isinstance(x, str)
with ckpt.as_directory() as d:
model.load_state_dict(torch.load(os.path.join(d, "model.pt")))
...
# ② 上报：先写进目录，再整体交给 Ray
with tempfile.TemporaryDirectory() as tmpdir:
torch.save(model.state_dict(), os.path.join(tmpdir, "model.pt"))
ray.train.report({"loss": loss.item()
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The objective of this chapter is to find out which part of Ray Train is in charge of in-situ training, what the V2 writing has changed after the default start, what the tolerance of error and what the grab is, and when torchrun instead of Ray. You should be able to say, "Ray, is it a training framework or a process organizer?"
13.1 Position: what's Ray Train sealed?
Let's be clear, because it's the easiest place for misunderstanding:
Ray Train is not a training framework, it's a training "declaration starter + process layout + error-tolerant shell".
It does not achieve reverse transmission, does not achieve aggregate communication and does not achieve optimization. The division of labour is:
This table is the main line of this chapter, and the two thick lines are the full value of Ray Train.
What does that mean?
With Torchrun, you're gonna have to figure out which channel you're on.
A few processes on the machine. With Ray Train, you wrote a statement:
ScalingConfig (num workers=8, use gpu=True) - means "Give me eight workers,
Each one GPU." And as for these eight worker places, how to start the process, how to inject environmental variables, where to put them.
Hanging up is all Ray's business.
The price: Your control over the process and the expansion has weakened. For example, within the node, 8 cards, NVLink, all company, across node.
Let's go. IB's sensitive hand-carving, Ray Train. Off
(ScalingConfig (placement strategy= "PACK"/ "SPREAD"), see 133); for precise control
Other Organiser
PlacementGroup SchedulingStrategy) -- that's chapter 08, no.
Ray Train's API.
Don't mix "place the team" with "label the bookle": Chapter 08 is about the placement group itself.
(bundle, STRIT PACK, place group bundle index), not relevant
Bundle level label (bundle label) is used - this course does not expand it. Check the current situation before using it.
A description of the scripting strategy/bundle label in the Ray Core document.
Verified: no arguments in ScalingConfig to specify a bundle --
2.58 For a complete list of field, see table in § 13.3 (num workers / use gpu /
resources_per_worker / placement_strategy / accelerator_type /
use_tpu / topology / label_selector / elastic_resize_monitor_interval_s，
Plus abandoned tracker resources), there is no entry point for bookle/placement group.
Ray Train's placement is controlled until the level of place strategy (PACK/SPREAD).
13.2 Ray Train V2: Default Opening and Migration
- Ray Train V2 started defaultly from 2.51.0 (PR #57857, is v2 enabled()
The default value is converted from False to True; before that (2.43-2.50) is opt-in: RY TRAIN V2 ENABLED=1.
- rollback to V1 to highlight RAY TRAIN V2 ENABLED=0, but V1 API has been archived
rollback is a transitional tool rather than a long-term programme.
- The official document has some page wording that is outdated and still says "Ray TRAIN V2 ENABLED=1".
Releasing notes and source code - version transition leaves clean paragraphs in Ray
The document is not unique.
Difference between V1 and V2
V2 Added Configuration Class and API (2.58 list):
- LoggingConfig - Structured Log Configuration. 2.58 It has only one field: log level:str = "INFO" (ray/train/v2/api/config.py);
- ValidationConfig - Training mid-course validation/assessment hooks
(TrainController will build and register as report/controller/worker back)
- PreptionError / get preption info() — seizure and disposal (see 136).
Migration proposal: If your code library is written before 2.45, it's not about changing API,
It's to make sure you don't rely on the session to name space. That's the core feature of V1.
And the second thing to be confirmed: which XxTrainer is the example in the code. Frame Exclusive Trainer
Three of them have been deleted.
AccelerateTrainer, 2.9 Start removal), and XGBoostTrainer / LightGBMTrainer
Still and already V2 is. The recommended path for V2 is TorchTrainer + frame its own glue --
See 13.3 "What about removed Trainer under V2"
13.3 Modern writing: ScalingConfig + TorchTrainer
Let's clear up three high frequency confusions.
1 No Ray.train.Trainer category.
There is no common Trainer in all of Ray.train - what you want to use
XxxTrainer:
# From Ray.train.toch import TorchTrainer #
from ray.train.xgboost import XGBoostTrainer
From Ray.train.lightgbm import LightGBMTrainer# another path for table data
Stop copying Lightning Trainer / Transformers Trainer / Accelerate Trainer
They've been removed from Ray.
Life cycle is 2.7 detached mark 2.8 Call is reported error 2.9 removed
（REP "Unify Torch based Trainers on the TorchTrainer API"）。
This program is oriented towards 2.58, and these three names will fail.
Their functions are consolidated into TorchTrainer: Putting the frame in its own Trainer
Build inside the training function and use prepare trainer / RayTrainReportCallback
Take the indicator back to checkpoint ray.train.report (§13.7 with full examples).
⚠️.train.lightning is still there, but the rest of all...
prepare_trainer / RayDDPStrategy / RayFSDPStrategy /
RayDeepSpeedStrategy / RayLightningEnvironment / RayTrainReportCallback
- No Lightning Trainer.
It's wrong. It's gonna be Import Error.
The earlier version of appendix A to this curriculum was also written out and amended.
2 Different categories: TorchTrainer has two import paths.
# Use this (which points to V1 or V2)
#V1
#V2
Ray.train.toch.TorchTrainer will help you to achieve it according to the V2 switch;
Direct IMP specific.toch trainer /.v2.torch trainer is V1/V2 mixed Wrong.
The most common reason -- don't write that.
3 Error tolerance default is off. FailureConfig.max failures default 0,
It means "fail at once." So, "I got it, FairureConfig, but it didn't work out."
Not bug -- you probably didn't show up to max failures:
from ray.train import FailureConfig, RunConfig
RunConfig
(1 = infinite retry. ⚠️ fail fast is no longer available under V2 - if passed, it is wrong.
It’s not as soft as “max failures”.
⚠️ The fourth round of this book is missing a field, which is in conflict with § 13.5: the wrong budget for V2 is made up of three fields -
Earlier it was written here, "only controlled by max failures plus controller failure list,"
Missing max prescription failures -- the number of thefts separated from real failures is V2
Key design (see § 13.5), missing it will make readers think "the grab will eat my retryquota".
Removed Trainer What to do under V2
Three of the "Framework Exclusive" listed above is the product of the V1 era: They put a frame in it.
Trainer packs it in its entirety, and passes through a bunch of xx init args / xx init kwargs.
The main path recommended in V2 is not them, but TorchTrainer+ frames its own glue.
Recommended for V2 (Lightning, for example, Transformers - Framed Trainer)
Build inside the training function to retrieve indicators and checkpoints with a callback/tool provided by Ray
ray.train.report）：
import pytorch_lightning as pl
from ray.train.lightning import RayTrainReportCallback, prepare_trainer
from ray.train.torch import TorchTrainer
def train_loop_per_worker(config):
model = MyLightningModule(...)
ltrainer = pl.Trainer(max_epochs=config["epochs"],
# Indicator/checkpoint # Ray
ltrainer = prepare trainer (ltrainer) # Replace DDP package with Ray
ltrainer.fit(model, datamodule=MyDataModule(...))
trainer = TorchTrainer(train_loop_per_worker,
scaling_config=ScalingConfig(num_workers=4, use_gpu=True))
The advantage of this is that process organization, error tolerance, and capture of all the V2 mechanisms described in 13.2-13.5, instead of being caught in the middle of the "Framework Exclusive Trainer" -- the one layer that connects the parameters with Ray's version, is the easiest place to break when moving.
⚠️ Name verified, signature unverified: Ray/training/lightning/ init.py
That's right.
["prepare_trainer", "RayDDPStrategy", "RayFSDPStrategy", "RayDeepSpeedStrategy", "RayLightningEnvironment", "RayTrainReportCallback"]
— Both of the names used above, and no Lightning Trainer.
But these two signatures (parameters and sequences) are not checked individually.
Check the Train document for the current version before moving.
And the "Framework Exclusive Trainer, which can be used in V2" is no longer a problem.
They were removed from Ray in 2.9 (see 1 above), so there was no "what still fits V1".
The smallest runaway:
import ray.train
from ray.train import RunConfig, ScalingConfig
from ray.train.torch import TorchTrainer
def train_loop_per_worker(config):
import torch, torch.nn as nn
#1) Model: Hand over to Ray for one floor DDP
model = ray.train.torch.prepare_model(
nn.Sequential(nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, 1)))
#2) Data: Hand over to Ray's + Alignment DistributedSampler
loader = ray.train.torch.prepare_data_loader(
ray.train.get_dataset_shard("train").iter_torch_batches(batch_size=64))
Training cycle: just copy a PyTorch
opt = torch.optim.Adam(model.parameters(), lr=config["lr"])
for _ in range(config["epochs"]):
for batch in loader:
...
opt.zero_grad()
# Gradient Synchronization occurs in this line (NCCL AllReduce)
opt.step()
ray.train.report({"loss": loss.item()})
trainer = TorchTrainer(
train_loop_per_worker,
# Up here, get dataset shard ("train") takes the data that is stated here
# Do not comment, otherwise the training function does not get a fraction
train_loop_config={"epochs": 3, "lr": 1e-3},
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
run_config=RunConfig(name="demo-run",
# Full node accessible
)
result = trainer.fit()
Key points
(1) Prepare mode what? Move the model to the right device and do as you specify.
The parallel strategy was changed to the corresponding package: the default is DDP (Distributed DataPaallel), which is the first time that the government has been able to get the data from the government.
For FSDP to be visible to policy (prepare model, parallel strategy=...).
⚠️ It's not "a glance at rank/world size autopackage" -- choosing which parallel parameters determine,
rank/world size is only used to decide which piece this workingr should take.
What it doesn't do is it doesn't change your model structure, it doesn't change the optimizer, it doesn't change the loss. So...
loss.backward() triggers the standard NCCL AllReduce - the data face is not Ray.
(2) Prepare data loader did what? Add distributed sampler to DataLoader for each
{\cHFFFFFF}{\cH00FFFF}{\cH00FFFF} Worker gets a non-overlapping segment and sets device. Cooperation
Ray.train.get dataset shard (“train”) (Dataset of Ray Data is automatically cut).
Advantages are that data store straight-link training from object workker, without driver; common DataLoader
Fine, but handle the sampler and the split.
(3) Ray.train.report with Result. Report (Metrics, checkpoint=...)
Report the indicators and checkpoints to controller. Return:
Note that RunConfig(storage path, name) must be consistent when it recovers or not found
The last run state (this is clearly written in the official file).
Data portal: datasets = division of labour with DataConfig
Let's get this straight:
Data=("train":ds} is the current version of V2, not the old version.
A lot of curricula (including an early version of the curriculum) label it "V1" -- that's wrong.
V2 changed something else -- data sets still go datasets=.
# V2, as it stands
trainer = TorchTrainer(
train_loop_per_worker,
# Key is the part First Name
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
)
So, what does DataConfig do? It's about how to split, not what data:
from ray.train import DataConfig
trainer = TorchTrainer(
train_loop_per_worker,
datasets={"train": train_ds, "valid": valid_ds},
# Controls only inglement / splits, does not load the data itself
dataset_config=DataConfig(
Datats to split=`all', #all' (default) |["train",...]; trying to turn off the cut-off
Access options=None, # ExchangeOptions
# Default True: the fraction is as close as possible to the node that works
),
scaling_config=ScalingConfig(num_workers=4, use_gpu=True),
run_config=RunConfig(name="demo-run", storage_path="/mnt/shared/ray_results"),
)
Training function side by side to draw fractions by name (name is the key of datasets):
def train_loop_per_worker(config):
train_shard = ray.train.get_dataset_shard("train")
# May be None, emptied first
for batch in train_shard.iter_torch_batches(batch_size=256, device="cuda"):
...
Three points deserve to be made clear:
- Why is it confused: DataConfig is not V2 added -- it's long gone.
Ray 2.6/2.7 (PR #35236) existed as a substitute for abandoned DatasetConfig,
About a year and a half before Train V2 (2.43+). So the V1 tutorial, the dataset config=DataConfig(...) is a common writing.
The real change is that V2 has raised the "severation strategy" from the hidden behaviour of Trainer to a visible configuration.
and datasets = explicitly as data entry;
- Severation occurs on the side of the worker, not driver -- bottom or chapter 12 § 12.2
Streaming spit (each workper gets a stream that doesn't overlap, data goes straight to worker, without passing)
A Python stack of driver;
- get dataset shard() can't get anything up in the training function.
The exact field name for DataConfig is based on the version you used:
Data to spits are checked against scenarios options.
The third field of the sixth round mark "unrecognised" has been verified: 2.58
ray/train/ international/data config.py, the full signature of DataConfig. init is
(datasets_to_split="all", execution_options=None, enable_shard_locality=True)
— In addition to the two above, there's one other (default True,
Let the fractions be as close as possible to the node where the worker holds it.
And datasets = the portal itself is stable -- use it when it's not possible.
ScalingConfig's Other field
Nom workers / use gpu is just two of the most common:
The full public version of ScalingConfig is field (2.58 measurements, available for self-checking):
num_workers / use_gpu / resources_per_worker / placement_strategy /
accelerator_type / use_tpu / topology / label_selector /
ELSTIC resize monitor interval s -- plus one abandoned
Trainer resources. (num workers also accept (min, max) binary group, see §13.6.
Runtime Assistant in Training Functions
These are the types of things that can be used in training functions:
⚠️ get world size() / get world rank() / get local rank() these
Not a module level function for Ray.training - these are TrainContext methods.
all functions for Ray/training/ init.py only
get_context / get_dataset_shard / get_checkpoint / report
(V2 get all reported checkpoints / get prevention info)
The rest of the export items are in categories (RunConfig, ScalingConfig, Checkpoint...).
No rank type assistant.
Write ray.train.get local rank() directly AttributeError;
The correct formulation is Ray.train.get context().get local rank().
This book was previously listed as a modular function, copying an unofficial version of memory -- in your version. Move!
The Train API page should be rechecked accordingly (if a version is supplemented with a module level name, whichever is the).
Get device() do not spell it yourself (the complete path is Ray.train.toch.get device()).
It returns the real use of prepare model
(cuda: 0 / cuda: 1... / cpu), handwritten torch.device (f "cuda:urk" )
The scene will get the cards right. V1 session.get world rank() with V2
Ray.train.get content().get world rank() is both old and new.
— but only the latter exists: Ray.train.get world rank() This modular type
It doesn't exist (it says Attribute Error).
⚠️ The book left a half sentence here earlier, "Modular shortcuts and get context() methods"
Direct conflict with the warnings of § 13.3 of this chapter, and the knot of § 13.11 - the fifth round has been deleted.
Checkpoint's correct position
import os, tempfile
from ray.train import Checkpoint
with tempfile.TemporaryDirectory() as tmpdir:
torch.save(model.state_dict(), os.path.join(tmpdir, "model.pt"))
ckpt = Checkpoint.from_directory(tmpdir)
ray.train.report({"loss": loss}, checkpoint=ckpt)
Don't go on your own torch.save to a fixed path and don't report -- so Ray Train doesn't know.
Checkpoint exists and fails retry will not take it back. It's the most common "used" thing, Ray, but wrong.
'Cause it didn't work out.
Checkpoint's construction and reading posture
First of all, there's one thing that can be reported directly wrong: ray.train.Checkpoint.
(python/ray/train/ checkpoint.py) is a new category of catalogue, everything.
There are only two types of methods + a tectonic function. From the old school.
Checkpoint. frum dict(...) and ckpt.to dict() are no longer available...
They're not "there are others, but they're discarded," but they're gonna throw Attribut Error:
AttributeError: The new `ray.train.Checkpoint` class does not support `from_dict()`.
Instead, only directories are supported.
Checkpoint.py CheckpointMetaClass. getattr (line 30)
The names were intercepted and the displacement error dropped:
What do you want to do with a dictionary? Put yourself in a directory, from directory
(This is exactly what the official migration suggests:
with tempfile.TemporaryDirectory() as d:
with open(os.path.join(d, "data.pkl"), "wb") as fp:
pickle.dump({"some": "value"}, fp)
ray.train.report(..., checkpoint=Checkpoint.from_directory(d))
Save weight (LLM version):
import os, tempfile, torch
from ray.train import Checkpoint
def train_loop_per_worker(config):
model = ray.train.torch.prepare_model(...)
#1 Restore: checkpoint load first
ckpt = ray.train.get_checkpoint()
if ckpt is not None:
# as directory() always returns context Manager - do not write isism(x,str)
with ckpt.as_directory() as d:
model.load_state_dict(torch.load(os.path.join(d, "model.pt")))
...
#2 Report: Write in the directory and hand it all over to Ray
with tempfile.TemporaryDirectory() as tmpdir:
torch.save(model.state_dict(), os.path.join(tmpdir, "model.pt"))
ray.train.report({"loss": loss.item()
Auto-fetched and translated by AI Atlas. All rights belong to the original author.