AWS Batch的配额管理为SageMaker培训工作提供跨团队借款和资源

人工智能

AWS Batch的配额管理为SageMaker培训工作提供跨团队借款和资源

AWS批次的发布支持了Amazon SageMaker训练工作，客户获得了全方位管理的新方式来排队，排班，并重试机器学习(ML)训练工作量而不建立自己的基础设施。 相反，它们使用AWS批次的排队和排程功能。 随着各组织扩大人工智能和机器学习(AI/ML)业务规模，并增加机上团队，协调共享图形处理单元(GPU)能力的访问，可以为管理员带来额外的通信、协调和配置间接费用。 这种间接费用包括设置单独的队列，调整容量限制，以及精心安排合适的工作量，以在合适的时间获得合适的资源。 AWS Batch的一个核心价值命题是帮助客户增加其计算利用率，随着培训环境的增长，利用现有能力做更多工作的机会也随之增加。

从管理多团队训练环境的客户向后工作，我们听到一个一致的要求：给我们一个方法来定义每个团队或项目的拨款，自动分享闲置能力，并首先管理我们的最高优先工作量。 这一请求导致我们建立了配额管理，即AWS批量的新调度能力。 管理者可以利用它将计算能力分为定额份额，按团队，项目，或工作量排列。 每个配额份额都采用可配置的资源共享战略，最大限度地利用GPU，同时尊重每个团队最重要的分配。

这篇文章显示配额管理如何运作，包括如何用贷款和借贷策略配置配额份额，为优先工作量设定共享优先，并开始使用SageMaker Python SDK.

配额份额和预扣

管理员可以通过为单个团队或项目等单独的实体规定配额份额来分配和分享受限制的GPU资源。 每个配额份额包括计算配额(容量限制)和闲置计算的资源共享战略。 这些策略决定了闲置计算是保留到工作到来(RESERVE)，借给其他有工作的配额股份(LEND)，还是同时借给闲置和借出，当该股份有可能会受益的工作(LEND AND BORROW).

配额份额作为虚拟队列在关联的任务队列中依次嵌入。 在为工作队列安排任务时，AWS批次会优先处理其容量范围内的配额份额，因为它通过相关的配额份额进行迁移。 如果工作达到一定的配额份额，则AWS批量将进行交叉分享。 如此一来，管理者就不需要追踪工作借款能力，AWS批量检查是否有从另一批配额借款的能力。 如果是这样，工作量可以使用其他零星能力来承担。

您可以在配置配额时打开基于优先级的共享预置。 如果交叉分享先发制人在工作达到贷款配额时恢复了被借出的能力，则分享先发制人将更低的优先工作置于计划D、STARING或RUNNING状态。 这样可以让队列前方的更高优先级的工作更快地开始运行。

先决条件

在此示例之前，您必须拥有 AWS 账户，并有权限创建和管理 AWS 批量资源。 例如，请参见引用 AWS Batch 和 SageMaker APIs 所需的 IAM 权限样本。

该示例使用SageMaker Python SDK v3. 完成安装 SageMaker Python SDK 兼容版本所需的设置：

配置配额管理资源。

利用新的配额管理能力，需要制定包含配额共享政策的时间安排政策，明确其在特定情况下的能力限制的服务环境，以及与两者相连的SageMaker培训工作队列。 之后，您可以将配额股份与工作队列联系起来。 用户指南有AWS命令线接口(AWS CLI)，并有配置配额管理资源和配额份额的控制台实例。 SageMaker Python SDK v3中与aws batch模块整合的端到端配额管理实例包含在一个样本笔记本中。 在这个例子中，我们跨过代码片段来总结关键组件，并使用笔记本中的帮助器功能来创建一个调度政策，服务环境，工作队列，以及三个配额共享。

在此配置的三个配额份额代表三个不同的项目。 每个都具有相同的容量限制：一个ml.g5.x大实例。 QS1配额份额代表背景工作。 对于这个项目，您想要先进行最重要的背景工作，并尽量借用可用的剩余容量。 在这个配额份额内，没有紧迫的工作，而且由于所有的工作都使用检查站，因此先发制人是可以接受的。 这个配额份额使用 LEND AND BORROW 资源共享策略，并开启了共享优先级。 在同一份额范围内的较高优先工作可以预先阻止优先程度较低的工作。

QS2是中等优先工作量。 选择这种配置的团队宁愿等待能力到位，而不是忍受额外的先发制人。 这些先发制人来自被收回的借贷能力，或来自优先就业岗位的先发制人。 这适用于较短的工作，或大多数提交的工作具有类似的优先次序。 配额份额仍然受益于在派遣工作时考虑的工作优先事项。 提交的工作触发了AWS批量，以收回任何开展工作所需的出借能力。

QS3是需要以尽可能少的中断时间尽快开始的工作量。 且不向其他配额股借或借。 因为它的工作量不会超过它们的能力限度。 通过同时关闭股份预置，提交这一配额份额的工作量不会被AWS批次预先抵销，只要容量限制不变。

这些配额共享配置是对样本笔记本中创建 quota management resources函数的输入，该功能还配置了相应的列表策略，服务环境，以及所选择的恒定名称的作业队列。

向 QS1 提交 SageMaker 培训任务

SageMaker Python SDK内部的aws batch模块已经扩展，允许将SageMaker培训任务的程序提交到配额管理驱动的AWS批次任务队列。 我们使用Hello World ModelTrainer，以下代码片段根据所选择的区域和实例类型确定一个图像URI来使用。

我们用这个图像 URI 在一个基本的问候世界模型教练的例子：

要在 SageMaker Python SDK 内使用任务队列，请向 aws batch 模块内的训练队列构造器提供队列名称。

然后，我们可以向QS1提交三个工作，我们的背景工作配额份额。 提交的第一份工作具有最高优先级，其次是中等优先级，然后是最低优先级。 由于QS1或QS2目前没有运行任何工作，所以应该开始两个实例，与提交的第一和第二个工作相对应(因为它们具有最高的相对优先地位)。 请注意，因为这仅仅是一个问候世界训练工作

向QS2提交工作，触发跨份额先发制人收回借出的能力

拥有QS2的团队可能会发现他们有工作来表演，并向QS2提交工作。 如此一来就触发了QS1的先发制人能力回放，而QS1目前借用的是这种能力(QS3的容量没有配置为借出). 此是横分先释。

更新QS1中的工作优先顺序触发共享优先操作最高优先工作

如果科学家意识到QS1中的 qs1 job low job 在 QS1 中仍然可以运行，由于优先事项的变化，现在需要开始运行，他们可以使用新的更新ServiceJob API来提高已经提交的QS1中的工作的优先级。 因为没有可用的借贷能力，这使得共享先发制人

(可选)清理已创建的资源

为清理此帖子创造的资源，运行以下代码：

结论

以及如何与SageMaker训练工作一起使用，并开始为自己的项目设定配额份额。 详见AWS批量配额管理用户指南。

兹AWS 配额管理可以帮助您的组织提高计算利用率，并更有效地确定团队和工作量的优先次序，为您的基础设施管理员和ML科学家腾出时间。 通过实行配额管理，你们的团队可以专注于工作量，减少管理和协调基础设施的时间。 这种能力在Amazon SageMaker AI的培训计划中特别强大，这样你的组织就可以在需要的时候按所需数量保留能力。 通过使用与SageMaker AI的AWS批量，可以充分利用训练计划。 我们鼓励你们尝试这种新能力，以便在你们的行动中产生有意义的影响。

本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Lend and borrow resources across teams with AWS Batch’s quota management for SageMaker Training jobs

With the launch of AWS Batch support for Amazon SageMaker Training jobs, customers gained a new fully managed way to queue, schedule, and retry machine learning (ML) training workloads without building their own infrastructure. Instead, they use the queuing and scheduling features of AWS Batch. As organizations scale their artificial intelligence and machine learning (AI/ML) operations and onboard more teams, coordinating access to shared graphics processing unit (GPU) capacity can introduce additional communication, coordination, and configuration overhead for administrators. This overhead includes setting up separate queues, adjusting capacity limits, and carefully orchestrating the right workloads to get the right resources at the right time. A core value proposition of AWS Batch is helping customers increase utilization of their compute, and as training environments grow, so does the opportunity to do more with existing capacity.

Working backward from customers managing multi-team training environments, we heard a consistent request: give us a way to define allocations per team or project, share idle capacity automatically, and run our highest-priority workloads first. This request led us to build quota management, a new scheduling capability for AWS Batch. Administrators can use it to divide compute capacity into quota shares, organized by team, project, or workload. Each quota share uses configurable resource-sharing strategies that maximize GPU utilization while honoring each team’s allocation when it matters most.

This post shows how quota management works, including how to configure quota shares with lending and borrowing strategies, set up in-share preemption for priority workloads, and get started using the SageMaker Python SDK.

Quota shares and preemption

Administrators can allocate and share constrained GPU resources by defining a quota share for separate entities, such as individual teams or projects. Each quota share includes compute quotas (capacity limits) and resource-sharing strategies for idle compute. These strategies determine whether idle compute is reserved until jobs arrive (RESERVE), lent to other quota shares that have work available (LEND), or both lent when idle and borrowed when the share has work that might benefit (LEND_AND_BORROW).

A quota share operates as a virtual queue nested hierarchically within an associated job queue. When scheduling jobs for a job queue, AWS Batch prioritizes quota shares that are under their capacity limits as it iterates through the associated quota shares. If jobs arrive in a quota share that has lent out its capacity, AWS Batch performs cross-share preemption. This reclaims resources for the lending quota share, so administrators don’t need to track down the jobs borrowing capacity and terminate them on their own. If jobs arrive in a quota share that is at or above its capacity limit, AWS Batch checks whether there’s available capacity to borrow from another quota share. If so, workloads can run using otherwise-idle capacity.

You can turn on priority-based in-share preemption when you configure a quota share. Where cross-share preemption reclaims lent capacity as jobs arrive in a lending quota share, in-share preemption preempts lower priority jobs that are in the SCHEDULED, STARTING, or RUNNING state. This allows higher priority jobs at the front of the queue to start running sooner.

Prerequisites

Before you proceed with this example, you must have an AWS account with permissions to create and manage AWS Batch resources. For this example, see the sample IAM permissions needed to invoke AWS Batch and SageMaker APIs.

The example uses the SageMaker Python SDK v3. Complete the setup required to install a compatible version of the SageMaker Python SDK:

Configuring quota management resources

Using the new quota management capabilities requires creating a Scheduling Policy that contains a quotaSharePolicy, a Service Environment that specifies its capacity limits in terms of specific instances, and a SageMaker Training job queue that is connected to both. After that, you can associate quota shares to a job queue. The user guide has AWS Command Line Interface (AWS CLI) and console examples for configuring quota management resources and quota shares. An end-to-end example of quota management integrated with the aws_batch module in the SageMaker Python SDK v3 is contained in a sample notebook. In this example, we step through code snippets that summarize the key components, and use the helper functions from the notebook to create a Scheduling Policy, Service Environment, Job Queue, and three Quota Shares.

The three quota shares configured here represent three different projects. Each has the same capacity limit set: a single ml.g5.xlarge instance. The QS1 quota share represents background jobs. For this project, you want to run the most important background job first and borrow as much spare capacity as is available. Within this quota share, no jobs are urgent, and preemptions are acceptable because all jobs use checkpointing. This quota share uses the LEND_AND_BORROW resource-sharing strategy and has in-share preemption turned on. Higher priority jobs within the same quota share can then preempt running jobs that are lower priority.

QS2 represents medium priority workloads. Teams choosing this configuration would prefer to wait for capacity to become available rather than endure additional preemptions. Those preemptions come from borrowed capacity being reclaimed or from in-share preemptions because of higher priority job submission. This is appropriate for shorter jobs, or when most submitted jobs have similar priorities. The quota share still benefits from job priority being taken into consideration at job dispatch. Submitted jobs trigger AWS Batch to reclaim any lent capacity needed to run the jobs.

QS3 represents workloads that need to start as quickly as possible with minimal interruptions. It’s configured to reserve capacity and neither lends nor borrows to other quota shares. It doesn’t experience cross-share preemptions, because its workloads don’t exceed their capacity limits. By also turning off in-share preemption, workloads submitted to this quota share are not preempted by AWS Batch as long as the capacity limit is unchanged.

These quota share configurations are inputs to the create_quota_management_resources function in the sample notebook, which also configures the corresponding Scheduling Policy, Service Environment, and Job Queue with the constant names chosen.

Submit SageMaker Training jobs to QS1

The aws_batch module within the SageMaker Python SDK has been extended to allow programmatic submission of SageMaker Training jobs to a quota-management-enabled AWS Batch job queue. To demonstrate these new capabilities, we use a Hello World ModelTrainer that’s submitted to the job queue. The following code snippet identifies an image URI to use based on the Region and instance type selected.

We use this image URI within a basic hello world model trainer example:

To use the job queue within the SageMaker Python SDK, provide the queue name to the TrainingQueue constructor within the aws_batch module.

We can then submit three jobs to QS1, our background jobs quota share. The first job submitted has the highest priority, followed by the medium priority job, and then the lowest priority job. As nothing is currently running in QS1 or QS2, two instances’ worth of jobs should start, corresponding to the first and second jobs submitted (as they had the highest relative priority). Note that because this is only a hello world training job, it doesn’t require data, but typically a ModelTrainer requires data to perform training.

Submitting jobs to QS2 triggers cross-share preemption to reclaim lent capacity

The team that owns QS2 might discover they have work to perform and submit a job to QS2. Doing so triggers a preemption of capacity back from QS1, which is currently borrowing that capacity (QS3’s capacity isn’t configured to be borrowed). This is a cross-share preemption.

Updating job priorities in QS1 triggers in-share preemption to run highest priority jobs

If a scientist realizes that the qs1_job_low job in QS1, which is still RUNNABLE, needs to begin running now because of changing priorities, they can use the new UpdateServiceJob API to increase the priority of an already submitted job in QS1. Because there’s no available capacity to borrow, this kicks off an in-share preemption, so that the job currently running is preempted and qs1_job_low can begin.

(Optional) Clean up created resources

To clean up the resources created in this post, run the following code:

Conclusion

In this post, we covered quota management, a new scheduling capability in AWS Batch, and how to use it with SageMaker Training jobs and get started setting up quota shares for your own projects. For more details, see the AWS Batch quota management user guide.

AWS Batch with quota management can help your organization increase compute utilization and more effectively prioritize your teams and workloads, freeing up time for your infrastructure admins and ML scientists. By implementing quota management, your teams can focus on their workloads and reduce time spent managing and coordinating infrastructure. This capability is especially powerful with Amazon SageMaker AI training plans, so that your organization can reserve capacity in the quantity you need, during the time you need it. By using AWS Batch with SageMaker AI, you can make full use of the training plan. We encourage you to try out this new capability so it can make a meaningful impact in your operations.

Auto-fetched and translated by AI Atlas. All rights belong to the original author.