人工智能
为更快的 SageMaker AI 任务发布 SOCI 索引图像
你深层学习的容器图像每一分钟要花一分钟的下载时间，就是花一分钟的费用，还有一分钟的团队等待。 集装箱图像下载纯粹是间接费用：可收费的工作时间不会产生模型进度。 对于现代的深层学习图像，管理费是每个工作的几分钟。 SageMaker AI Training 和 Amazon SageMaker AI 的处理工作
通过SOCI索引，SageMaker AI懒惰地装入了你的容器图像。 它只获取启动容器所需的文件，并且在您代码运行时会流出背景中的其余文件，所以您的工作以秒而不是分钟开始。 您通过指向一个预索引的 AWS 深层学习容器( DLC) 或者在您自己的图像中添加一个索引来通过它。 两种方法都要求修改您的工作定义、切入点或数据通道。 其结果是更快地开始工作，降低工作成本，花费更多的时间建造模型，而不是等待基础设施。
SOCI-index化的图像以秒数开始Amazon SageMaker AI训练容器，对训练性能没有影响。 正如我们测量到的图像拉动时的GPU闲置时间减少了90%
显示如何通过 SOCI 索引：是什么，我们测量的结果，以及如何为您的工作打开它。
挑战
SageMaker AI培训和处理工作在Amazon弹性集装箱登记册(Amazon ECR)储存的集装箱图像内进行。 在任务可以执行代码之前，服务会将整个图像下载到实例中。 它解压后将每一层写成磁盘。
对于小图像，这需要几秒钟。 但深层学习图像通常超过10GB或以上。 下载新增分钟给每个工作，你支付它作为工作时间。 其影响是最大的，具体来说，迭代速度最为重要：超参数调试扫码为数百个工作，快速实验，以及短处理工作，其中图像下载可以主导总运行时间。
SOCI 如何编制索引
可搜索的OCI(SOCI)是一种开源技术，它懒惰地装载容器图像。 运行时间不是在开始前下载整个图像，而是使用SOCI索引，只获取容器发布所需的文件。 然后，它流其余的背景，因为您的代码访问它。
一个 SOCI 索引是映射您图像中每个文件的元数据。 你把它储存在Amazon ECR中，作为符合OCI的文物，与你的原始图像相连。 由于索引是单独的文物，你的图像摘要和任何现存的签名仍然有效。
SageMaker AI 自动检查索引
当您创建培训或处理工作时，SageMaker AI 检查您指定的图像是否有 SOCI 索引，并相应行事：
因为工作总是在有需要的时候会回到标准的下载，所以添加一个SOCI指数永远不会妨碍你的工作运行。 你只得到更快的启动。 你永远不会冒险打破现有的工作流程。
我们衡量的
我们在真正的 SageMaker AI 训练工作上，然后，我们从两个方面开展全面培训工作，以衡量对培训速度的影响。
设置
- 8幅图像，1.8–35.4 GB压缩。
- GPU和CPU实例：ml.g4dn、ml.g5、ml.g6、ml.g6e、ml.g7、ml.g7e、ml.p4d、ml.p5和ml.m7i，从x大到48大，分布在美国西部(奥雷贡)地区(西-2)和美国东部(北弗吉尼亚)地区(东-1)。
- 从图像拉动开始到容器准备运行为止的下载时间。 所有数字均为重复运行的中位数。
结果1：在每个图像大小下下载时间从分钟到秒
以下图表比较了标准Docker拖动的容器图像下载时间，以对抗超8个图像大小的SOCI懒惰装载，从1.8GB到35.4GB.
每个图像大小显示两条。 Docker下载时间栏(蓝色)随图像大小稳步增长，从1.8GB的48秒增加到35.4GB的7分17秒。
每个对齐的显示上方有一个速度标签，Docker时间与SOCI时间的比例。 速度从最小图像大约10x到最大图像超过35x不等。
标准下载时间随着图像大小的增加而增加，因为每个文件都被解压和写入。 懒散的装载不管大小都停留在3–11秒，一个8x–46x的改进。 图像越大，你节省的时间就越多。
结果2：保存在每一个实例类型和大小上
运行13.2GB的图像横跨13个实例类型/大小组合，标准下载范围从3min 16 s到4min 20 s不等，而懒惰的加载则维持在6-8 s. 每个配置都快34倍了您不需要更改实例类型或大小才能受益。
成果3：培训工作未受影响，超过头一个时期的惩罚
我们从Docker和SOCI指数化的图像完成了两种完整的训练工作并且比较每个时代：
- 关于图像网的ResNet-50的图像分类(9 469幅图像)
- 集装箱图像：PyTorch(13.2 GB DLC)和TensorFlow(9.9 GB DLC)
- 实例：ml.p5.48x大，ml.p4d.24x大，ml.g7e.48x大，ml.g7.48x大，ml.g6e.12x大，ml.g6e.24x大，ml.g6e.48x大。
- 大型语言模型(LLM) 微调：全参数监制微调(SFT) Qwen2.5-0.5B在羊座清洁(51,760例)上配有Hugging Face Trainer.
- 集装箱图像：9.4 GB Hugging Face PyTorch DLC.
- 实例：ml.p5.48x大，ml.p4d.24x大，ml.g6.48x大。
以下图表将任务启动分解为图像下载时间和第一个训练时段，将标准Docker拉动与跨GPU和CPU配置的SOCI索引图像进行对比，以完成这两个工作量。
- 从第二个时期开始，每个时期的时间(在运行中的变化范围内)都相同，对两种工作量来说都是如此。
- 第一个纪元需要从懒惰的图像中 3 - 16 s 更长的时间，而文件框架第一次被获取。 这种头一个时期的成本在每一个实例大小上都是一样的。
- 网：每份工作提前1-6分钟完成。 这就是下载节省减去一次性的第一个时代成本，每个工作固定节省。
测试你自己的工作量
这些数字来自上述两项工作量。 第一个时代的成本取决于您的形象和您的代码在启动时的触摸：
- 懒惰加载在第一训练步骤只需要图像的一小部分时获胜，其余的溪流在训练时获胜。
- 如果您的切入点在训练开始前读取了大部分图像( 例如扫描、 检查或预装其内容)，第一个时代将等待大约相同的字节，一个完整的拉动。 净改善不大。
- 我们的测试中仍然有前置开始：
- 火炬。 编译和一个刚刚建成的 CUDA 扩展器在第一个时代增加了 7-15 秒(净节省 3 分钟 25 秒 - 3 分钟 56 秒)
- 6GB型号在加载时间烤入图像时，其重量会流出，增加了2-3.5分钟，但总体工作仍然在索引图像上提前完成2-2.5分钟。
开始
您可以采用 SOCI 索引两种方式：使用预索引的 AWS 深层学习容器，或者为自己的自定义图像创建索引。
备选案文1：使用预索引的AWS深层学习容器
AWS发布一些流行的深层学习容器的SOCI索引变体，由图像标签上的-soci后缀识别：
PyTorch训练，TensorFlow训练，vLLM，和SGLang都有预索引变体，包括一些框架的arm64变体。 要采用懒惰的加载，请将您工作定义中的图像标记更新到 -soci 变体。 浏览 AWS 深层学习容器图片库中的完整列表。
注意：如果您从 DLC 碱性图像构建自己的图像，您必须为您的最终图像创建 SOCI 索引。 来自基图像的索引不会延续到衍生图像。
选项2：为您自己的图像创建 SOCI 索引
要索引一个自定义图像，您需要Linux主机(如Amazon弹性计算云(Amazon EC2))实例运行 Amazon Linux 2023，并装有容器，SOCI快照器，同时安装芬奇或书呆子ctl.
- 设定环境变量。
- 安装所需的工具。
- 认证到Amazon ECR。
- 拉你的原始图像。
- 创建SOCI索引图像。
- 按下索引图像
这在您的寄存器中创建了两个文物：SOCI索引和符合OCI的图像索引表，将您的图像与索引联系起来。 您的原始图像及其摘要保持不变。
CI/CD 自动创建索引
为了在每次推力上自动生成索引，部署CFN ECR SOCI索引构建器。 它使用Amazon EventBridge规则检测Amazon ECR图像推力事件并构建索引。 您的建设管道无需更改。
在 SageMaker AI 任务中使用索引图像
参考你SOCI索引的图像和任何其他图像一样不需要修改 API。 只有标记不同。
培训实例( SageMaker Python SDK)
校验您的图像是否包含 SOCI 索引
在发布之前，请确认您的标签解析到索引图像：
寻找儿童名单，其文物Type从应用/vnd.amazon.soci.index开始。 与您的平台显示。 如果它不在那里，标记不会指向有索引的图像。
要记住的事
- Amazon ECR 仅限私人存储器。 其他登记册中的图像，或者通过一个私有的Docker登记册(Training Repository Access Mode set to Vpc)拉动，不具备懒惰加载的资格。
- 改进带有图像大小的尺度。 图像小于大约4GB，可见相对较小的改进。
- 检查Docker的具体假设。 SOCI索引的图像运行在一个基于容器的运行时间上。 审查代码的切入点和设置脚本，这些代码称为Docker CLI，读作/var/run/docker.sock，或者从cgroup路径获取标识符。 SageMaker AI为后向兼容性挂载了 /.dockerenv 标记，但避免添加新的依赖性。
- 在所有AWS地区都有SageMaker AI工作的SOCI索引。
结论
通过 SOCI 索引，您可以从您的 SageMaker AI 操作中将容器图像下载时间最小化。 在您的图像中添加 SOCI 索引，或者切换到预索引的 AWS 深层学习容器。 无论哪种方式，你都将启动时间从分钟缩短到秒，降低总的工作成本，并加快机器学习(ML)迭代周期。 而您现有的工作定义，API的调用，和数据管道完全一样。
为了更多了解Amazon SageMaker AI，并开始使用SOCI索引，探索以下资源：
- 选一个SOCI索引的深层学习容器。 浏览深层学习容器图片库和图像标签中的附件-soci。 关于具体框架的指导，见AWS深层学习容器的SCI指数。
- 索引你自己的定制图像。 遵循 SageMaker AI 开发者指南中的一步步走过，或者参考 SOCI shapter 正在启动的高级选项指南。
- 在你的CI/CD管道中自动索引。 部署 CFN ECR SOCI 索引构建器，在每次图像推力上自动生成索引。 不需要改变管道。
- 和芬奇进行局部测试使用芬奇懒散的穿行验证你的形象在基于容器的运行时间在你提交工作之前。
- 潜入科技深处使用DLAMI上的SOCI索引和AWS机器学习博客上的DLC来读取容器冷启动时间，或在引擎盖下探索：在AWS容器博客上
更快的容器启动只差一个图像标记。 试试你的下一个SageMaker AI的工作，看看区别。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Announcing SOCI-indexed images for faster SageMaker AI jobs
Every minute your deep learning container image spends downloading is a minute you pay for and a minute your team waits. Container image download is pure overhead: billable job time that produces no model progress. For modern deep learning images, that overhead is several minutes on every single job. Today, we’re excited to announce Seekable OCI (SOCI) indexing for Amazon SageMaker AI Training and Amazon SageMaker AI processing jobs, which takes full container image download off the critical path of every job.
With SOCI indexing, SageMaker AI lazy-loads your container image. It fetches only the files needed to start the container and streams the rest in the background as your code runs, so your job begins in seconds instead of minutes. You adopt it by pointing to a pre-indexed AWS Deep Learning Container (DLC) or adding an index to your own image. Neither approach requires changes to your job definition, entry point, or data channels. The result is faster job starts, lower job cost, and more time spent building models rather than waiting on infrastructure.
SOCI-indexed images start Amazon SageMaker AI training containers in seconds, with no impact on training performance. As we show in the What we measured section, this reduced GPU idle time during image pulls by up to 90 percent in our benchmarks.
This post shows how to adopt SOCI indexing: what it is, the results we measured, and how to turn it on for your own jobs.
The challenge
SageMaker AI training and processing jobs run inside container images stored in Amazon Elastic Container Registry (Amazon ECR). Before a job can execute your code, the service downloads the entire image to the instance. It decompresses and writes every layer to disk.
For small images this takes seconds. But deep learning images routinely exceed 10 GB or more. That download adds minutes to every job, and you pay for it as job time. The impact is largest exactly where iteration speed matters most: hyperparameter tuning sweeps for hundreds of jobs, quick experiments, and short processing jobs, where image download can dominate total runtime.
How SOCI indexing works
Seekable OCI (SOCI) is an open source technology that lazy loads container images. Instead of downloading the whole image before starting, the runtime uses a SOCI index to fetch only the files needed to launch the container immediately. It then streams the rest in the background as your code accesses it.
A SOCI index is metadata that maps every file inside your image. You store it in Amazon ECR as an OCI-compliant artifact linked to your original image. Because the index is a separate artifact, your image digest and any existing signatures remain valid.
SageMaker AI checks for an index automatically
When you create a training or processing job, SageMaker AI checks whether the image you specified has a SOCI index and behaves accordingly:
Because jobs always fall back to a standard download when needed, adding a SOCI index never prevents your job from running. You only gain faster startup. You never risk breaking an existing workflow.
What we measured
We pulled the same container images two ways on real SageMaker AI training jobs: standard download and SOCI-indexed lazy loading. We then ran full training jobs from both to measure the effect on training speed.
Setup
- Eight images, 1.8–35.4 GB compressed.
- GPU and CPU instances ml.g4dn, ml.g5, ml.g6, ml.g6e, ml.g7, ml.g7e, ml.p4d, ml.p5, and ml.m7i, from xlarge to 48xlarge, in the US West (Oregon) Region (us-west-2) and the US East (N. Virginia) Region (us-east-1).
- Download time measured from the start of the image pull until the container is ready to run. All figures are medians of repeated runs.
Result 1: Download time drops from minutes to seconds at every image size
The following chart compares container image download time for a standard Docker pull against SOCI lazy loading across eight image sizes, from 1.8 GB to 35.4 GB.
Each image size shows two bars. The Docker download time bar (blue) grows steadily with image size, from 48 seconds at 1.8 GB to 7 minutes 17 seconds at 35.4 GB.
A speedup label above each pair shows the ratio of Docker time to SOCI time. Speedups range from approximately 10x for the smallest image to over 35x for the largest.
Standard download time grows with image size because every file is decompressed and written. Lazy loading stayed at 3–11 seconds regardless of size, an 8x–46x improvement. The bigger the image, the more minutes you save.
Result 2: The saving holds on every instance type and size
Running the 13.2 GB image across 13 instance type/size combinations, standard download ranged from 3 min 16 s to 4 min 20 s, while lazy loading held at 6-8 s. That is about 34x faster on every configuration. You don’t need to change instance type or size to benefit.
Result 3: Training performance is unaffected beyond a first-epoch penalty
We ran two kinds of full training jobs from the Docker-pulled and from the SOCI-indexed image and compared every epoch:
- Image classification on ResNet-50 on Imagenette (9,469 images)
- Container images: PyTorch (13.2 GB DLC) and TensorFlow (9.9 GB DLC)
- Instances: ml.p5.48xlarge, ml.p4d.24xlarge, ml.g7e.48xlarge, ml.g7.48xlarge, ml.g6e.12xlarge, ml.g6e.24xlarge, and ml.g6e.48xlarge.
- Large language model (LLM) fine-tuning: full-parameter supervised fine-tuning (SFT) of Qwen2.5-0.5B on alpaca-cleaned (51,760 examples) with the Hugging Face Trainer.
- Container image: 9.4 GB Hugging Face PyTorch DLC.
- Instances: ml.p5.48xlarge, ml.p4d.24xlarge, ml.g6.48xlarge.
The following chart breaks job startup into image download time and first training epoch time, comparing a standard Docker pull with a SOCI-indexed image across GPU and CPU configurations for both workloads.
- From the second epoch on, time per epoch is the same (within run-to-run variation) on every instance, for both workloads.
- The first epoch takes 3–16 s longer from the lazy-loaded image, while files the framework touches for the first time are fetched. This first-epoch cost is the same on every instance size.
- Net: each job finished 1-6 min sooner. That is the download saving minus the one-time first-epoch cost, a fixed saving per job.
Test this on your own workload
These numbers are from the two workloads described earlier. The first-epoch cost depends on your image and what your code touches at startup:
- Lazy loading wins when the first training step needs only a fraction of the image, the rest streams in while you train.
- If your entry point reads most of the image before training starts (for example, scanning, checksumming, or preloading its contents), the first epoch waits for roughly the same bytes a full pull would have fetched. The net improvement would be small.
- Front-loaded starts still came out ahead in our tests:
- torch.compile and a just-in-time (JIT) built CUDA extension added 7-15 s to the first epoch (net saving still 3 min 25 s–3 min 56 s)
- A 6 GB model baked into the image streamed its weights at load time, adding 2-3.5 min, but the overall job still finished 2-2.5 mins sooner on the indexed image.
Getting started
You can adopt SOCI indexing two ways: use a pre-indexed AWS Deep Learning Container or create an index for your own custom image.
Option 1: Use a pre-indexed AWS Deep Learning Container
AWS publishes SOCI-indexed variants of several popular Deep Learning Containers, identified by the -soci suffix on the image tag:
Pre-indexed variants are available for PyTorch Training, TensorFlow Training, vLLM, and SGLang, including arm64 variants for some frameworks. To adopt lazy loading, update the image tag in your job definition to the -soci variant. Browse the full list in the AWS Deep Learning Containers Gallery.
Note: If you build your own image FROM a DLC base image, you must create a SOCI index for your final image. The index from the base image does not carry forward to derived images.
Option 2: Create a SOCI index for your own image
To index a custom image, you need a Linux host (such as an Amazon Elastic Compute Cloud (Amazon EC2) instance running Amazon Linux 2023) with containerd, the SOCI snapshotter, and either Finch or nerdctl installed.
- Set your environment variables.
- Install the required tools.
- Authenticate to Amazon ECR.
- Pull your original image.
- Create the SOCI-indexed image.
- Push the indexed image.
This creates two artifacts in your repository: the SOCI index and an OCI-compliant image index manifest linking your image to the index. Your original image and its digest remain unchanged.
Automate index creation in CI/CD
To generate indexes automatically on every push, deploy the CFN ECR SOCI Index Builder. It uses an Amazon EventBridge rule to detect Amazon ECR image push events and build the index. No changes to your build pipeline are required.
Use the indexed image in your SageMaker AI job
Reference your SOCI-indexed image exactly as you would any other image. No API changes are needed. Only the tag differs.
Training example (SageMaker Python SDK)
Verify that your image includes a SOCI index
Before launching, confirm your tag resolves to an indexed image:
Look for a child manifest whose artifactType begins with application/vnd.amazon.soci.index. alongside your platform manifest. If it isn’t there, the tag does not point to an indexed image.
Things to keep in mind
- Amazon ECR private repositories only. Images in other registries, or pulled through a private Docker registry (TrainingRepositoryAccessMode set to Vpc), are not eligible for lazy loading.
- Improvement scales with image size. Images smaller than roughly 4 GB see relatively small improvements.
- Check for Docker-specific assumptions. SOCI-indexed images run on a containerd-based runtime. Review entry points and setup scripts for code that calls the Docker CLI, reads /var/run/docker.sock, or derives identifiers from cgroup paths. SageMaker AI mounts a /.dockerenv marker for backward compatibility, but avoid adding new dependencies on it.
- SOCI indexing for SageMaker AI jobs is available across all AWS Regions.
Conclusion
With SOCI indexing, you can minimize container image download time from your SageMaker AI training and processing jobs. Add a SOCI index to your images, or switch to a pre-indexed AWS Deep Learning Container. Either way, you cut startup time from minutes to seconds, reduce total job cost, and accelerate your machine learning (ML) iteration cycle. And your existing job definitions, API calls, and data pipelines stay exactly the same.
To learn more about Amazon SageMaker AI and start using SOCI indexing, explore the following resources:
- Pick a SOCI-indexed Deep Learning Container. Browse the Deep Learning Containers Gallery and append -soci to your image tag. For framework-specific guidance, see SOCI index for AWS Deep Learning Containers.
- Index your own custom image. Follow the step-by-step walkthrough in the SageMaker AI Developer Guide, or refer to the SOCI snapshotter getting-started guide for advanced options.
- Automate indexing in your CI/CD pipeline. Deploy the CFN ECR SOCI Index Builder to generate indexes automatically on every image push. No pipeline changes are required.
- Test locally with Finch. Use the Finch lazy-loading walkthrough to validate your image on a containerd-based runtime before you submit a job.
- Dive deeper into the technology. Read Reducing container cold start times using SOCI index on DLAMI and DLC on the AWS Machine Learning Blog for benchmarks, or explore Under the hood: Lazy loading container images with Seekable OCI and AWS Fargate on the AWS Containers Blog to understand the lazy-loading mechanism in detail.
Faster container startup is only one image tag away. Try it on your next SageMaker AI job and see the difference.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.