人工智能
在Amazon EKS和SageMaker HyperPod EKS上使用NVIDIA Nemotron模型的加速代理AI
NVIDIA 尼莫特龙 3.5 Super VL 加入尼莫特龙 3 Nano, Nano Omni, Super, Ultra，和尼莫特龙 3.5 AWS上已有闪电模型。 它是一个120B参数的专家混合模型，每个符号激活12B参数。 它将推理和工具使用与文本，图像，文档，和视频理解结合起来，因此一个模型既服务于文本，也服务于视觉任务。 这个帖子探索其架构并使用大小写，然后显示如何在Amazon弹性Kubernetes服务(Amazon EKS),Amazon EKS自动模式和Amazon SageMaker HyperPod EKS上部署。
关于尼莫特龙3.5超级VL
NVIDIA Nemotron 3.5 Super VL是一种开放式的多式联运模型，由Nemotron 3 Ultra蒸馏而成。 NVIDIA发布权重，培训数据集，以及培训后的食谱，这样您就可以自定义特定域工作量的模型，并在您控制的基础设施上运行。 模型细节：
- 120B总参数，每个符号为12B。
-混合Mamba-2,MOE，和注意层与RADIO视觉编码器。
- 上下文长度可达1M令牌。
- 输入：文本、图像、文档和视频。
- 输出：文本。
- BF16和NVFP4检查站。
NVIDIA报告赫尔梅斯法官座席为63.44%,LangChain为87.05%,CharXiv图表推算基准测试为80.8%. (技术报告)
示范建筑
Nemotron 3.5 Super VL使用混合的Mamba-Transformer MoE架构，将Mamba-2，注意力，以及专家层相互连接。 Sparse的路由激活了120B参数中的12B每令牌，因此每个令牌的计算匹配一个更小的模型，而GPU内存仍然持有全重。 Mamba-2层在生成时会携带一个恒定大小的状态，因此它们的内存不会像KV缓存那样以序列长度生长，这让长长的上下文可以负担得起。 递归共享的多托肯预测(MTP)头支持本土投机解码。
使用大小写
Nemotron 3.5 Super VL支持agent工作流程将推理与视觉理解结合起来：
- 编码：测试和验证代码，分析错误，建议修正。
- 网络研究：相互参照，通过搜索工具核实返回的来源。
- 文件情报：阅读表格、图表和页面布局，以编制摘要。
- 网络安全：查明脆弱性和分类威胁。
- 视频理解：搜索、内容、审查和事件分析。
尼莫特龙 3.5 Super VL通过NVIDIA参考实施，烹饪本，和NVIDIA Agent工具包与主要开放的agent框架和管弦系统集成。 关于支持框架的全部列表，见NVIDIA开发者博客。
最小 GPU 配置
在AWS部署
本节概述了结构、验证和绩效评价办法。 aws-do-eks项目载有部署Nemotron 3.5 Super VL和Nemotron 3 Ultra在AWS的详细走法，包括先决条件、集群创建、模型重量下载、部署、测试、基准和清理。 同样的模式也可以应用于NVIDIA尼莫特龙家族中的其他模型。
部署结构
参考执行运行于标准Amazon EKS,EKS Auto Mode，以及SageMaker HyperPod EKS. 使用 EKS 自动模式，AWS 代表您在集群中管理缩放，GPU，和全民教育支持。 用于模型部署的Kubernetes表单对每个管弦乐器选项都是相同的。 每个集群都有一个主机Kubernetes API的控制平面和一个CPU和GPU节点的数据平面。 CPU节点运行系统和测试舱，而GPU节点运行模型。 Amazon弹性容器注册(Amazon ECR)存储容器图像，Lustre卷共用的Amazon FSx持有模型重量，弹性纤维适配器(EFA)连接GPU节点。 下图显示为标准Amazon EKS集群。
兹NVIDIA Dynamo服务于在一个单一节点上汇总或分解的模型，在不同的节点上进行预填和解码。 对于视觉语言模型，图像/视频编码阶段还可以按照自己的节点进行分类。
审定和业绩
这个架构在p6-b200.48x大(8×NVIDIA B200 GPU)和p5en.48x大(8×NVIDIA H200 GPU)两个实例上都得到了验证。 您可以通过 NVIDIA 集体通信库(NCCL)验证集群及其网络全减微-基准测试。
在NCCL测试运行期间，看到所有GPU的利用率和所有全民教育适配器的流量，就证明分组如预期的那样工作。
您可以用 NVIDIA aiperf 工具测量集群上部署的模型的性能。
aiperf输出在左上方的面板显示键性能指标(KPIs)中，喜欢时间到第一代号(TTFT)，间托式纬度(ITL)，以及每秒的吞吐量(TPS). 通过在不同货币水平上进行这些测量，我们可以研究长期性和吞吐量如何取决于同时提出的请求的数量。 这些信息对于周密规划能力和模型部署地形至关重要。 aws-do-eks容器提供可视化GPU和全民教育利用的自动化，以及使用单一命令进行Aiperf和货币扫描。
前面的货币扫描图揭示认为，Nemotron 3.5 Super VL的推论具有较低的TTFT，在两个节点的汇总部署和分类部署之间大约是相同的国际交易日志。
这样的图表在作出部署决定时是有用的。 绘制您作为水平线的延迟预算，例如TTFT低于600毫秒(约1秒)，国际交易日志低于100毫秒每令牌。 两条线下最右侧的点是这一地形学的一个复制品的吞吐量可以在服务级目标(SLO)内服务，这里的c32，或每秒约320个令牌。 要规划容量，将目标符号每秒除以这个每个replica的吞吐量，并找到要部署的预填解码对的数量。 改变后重新运行同样的扫码(精度，TP/PP形状，汇总与分类，新的Dynamo或vLLM发布)，使权衡立即可见。 一个左右移动的曲线就是胜利。 由于TTFT和ILFT是分开绘制的，它也告诉你要放大哪一边：上升的ILFT 称添加解码能力，上升的TTFT 称添加预填能力。
结论
这个帖子涵盖了在Amazon EKS,EKS Auto Mode，以及SageMaker HyperPod EKS与NVIDIA Dynamo和vLLLM上服务NVIDIA Nemotron模型的架构和性能评价模式。 要开始，在自己的账户中部署Nemotron 3.5 Super VL，使用参考执行。 关于EKS项目的AI中也有更多的推论蓝图。 为了获得更多关于EKS推论的亲身经验，登记参加我们即将举行的讲习班。
鸣谢
提交人特别感谢参与编写、验证、审查和出版这些内容的所有人，包括但不限于：Eliuth Triana、刘家红、Kshitiz Gupta、Abhijeet Patil、Mike Stefaniak、Chris Splinter、Shidal Joshi、Aditya Ramakrishnan、Natarajan Kumar、Anish Maddipoti。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS
NVIDIA Nemotron 3.5 Super VL joins the NVIDIA Nemotron 3 Nano, Nano Omni, Super, Ultra, and Nemotron 3.5 Lightning models already available on AWS. It’s a 120B-parameter mixture-of-experts (MoE) model that activates 12B parameters per token. It combines reasoning and tool use with text, image, document, and video understanding, so one model serves both text and visual tasks. This post explores its architecture and use cases, then shows how to deploy it on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon EKS Auto Mode, and Amazon SageMaker HyperPod EKS.
About Nemotron 3.5 Super VL
NVIDIA Nemotron 3.5 Super VL is an open multimodal model distilled from Nemotron 3 Ultra. NVIDIA publishes the weights, training datasets, and post-training recipes, so you can customize the model for domain-specific workloads and run it on infrastructure you control. Model details:
- 120B total parameters, 12B active per token.
- Hybrid Mamba-2, MoE, and attention layers with a RADIO vision encoder.
- Context length up to 1M tokens.
- Input: text, image, document, and video.
- Output: text.
- BF16 and NVFP4 checkpoints.
NVIDIA reports 63.44 percent on Hermes Bench and 87.05 percent on LangChain agent evaluations, and 80.8 percent on the CharXiv chart-reasoning benchmark. (technical report)
Model architecture
Nemotron 3.5 Super VL uses a hybrid Mamba-Transformer MoE architecture, interleaving Mamba-2, attention, and expert layers. Sparse routing activates 12B of the 120B parameters per token, so compute per token matches a much smaller model while GPU memory still holds the full weights. Mamba-2 layers carry a constant-size state during generation, so their memory does not grow with sequence length the way KV cache does, which keeps long contexts affordable. A recursively shared Multi-Token Prediction (MTP) head supports native speculative decoding.
Use cases
Nemotron 3.5 Super VL supports agent workflows that combine reasoning with visual understanding:
- Coding: test and validate code, analyze errors, recommend fixes.
- Web research: cross-reference and verify sources returned by search tools.
- Document intelligence: read tables, charts, and page layouts to produce summaries.
- Cyber security: identify vulnerabilities and triage threats.
- Video understanding: search, content, review, and incident analysis.
Nemotron 3.5 Super VL integrates with major open agent frameworks and orchestration systems through NVIDIA reference implementations, cookbooks, and the NVIDIA Agent Toolkit. For the full list of supported frameworks, see the NVIDIA developer blog.
Minimum GPU configurations
Deployment on AWS
This section gives an overview of architecture, validation, and performance evaluation approaches. A detailed walkthrough of deploying Nemotron 3.5 Super VL and Nemotron 3 Ultra on AWS is available in the aws-do-eks project, including prerequisites, cluster creation, model weights download, deployment, testing, benchmarking, and cleanup. The same pattern can also be applied to other models in the NVIDIA Nemotron family.
Deployment architecture
The reference implementation runs on standard Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. With EKS Auto Mode, AWS manages scaling, GPU, and EFA support in the cluster on your behalf. The Kubernetes manifests for model deployment are the same for each of the orchestrator options. Each cluster has a control plane hosting the Kubernetes API and a data plane of CPU and GPU nodes. CPU nodes run system and test pods, while the GPU nodes run the model. Amazon Elastic Container Registry (Amazon ECR) stores the container images, a shared Amazon FSx for Lustre volume holds the model weights, and Elastic Fabric Adapter (EFA) connects the GPU nodes. The following figure shows a standard Amazon EKS cluster.
NVIDIA Dynamo serves the model aggregated on a single node or disaggregated with prefill and decode on different nodes. For vision-language models, the image/video encoding stage can in addition run disaggregated on its own node.
Validation and performance
This architecture has been validated on both p6-b200.48xlarge (8 x NVIDIA B200 GPUs) and p5en.48xlarge (8 x NVIDIA H200 GPUs) instances. You can validate the cluster and its network with an NVIDIA Collective Communications Library (NCCL) all-reduce micro-benchmark.
Seeing utilization on all GPUs and traffic on all EFA adapters while the NCCL test is running, verifies that the cluster is working as expected.
You can measure the performance of a deployed model on the cluster with the NVIDIA aiperf tool.
The aiperf output in the upper left pane shows key performance indicators (KPIs) like time to first token (TTFT), inter-token latency (ITL), and throughput in tokens per second (TPS). By taking these measurements at various levels of concurrency, we can study how latency and throughput depend on the number of concurrent requests. This information is essential to thoughtfully plan capacity and model deployment topology. The aws-do-eks container provides automation for visualizing GPU and EFA utilization as well as running aiperf and concurrency sweeps with single commands.
The preceding concurrency sweep diagram reveals that the inference of Nemotron 3.5 Super VL has lower TTFT and about the same ITL between aggregated and disaggregated deployments on two nodes.
A chart like this is useful when making deployment decisions. Draw your latency budget as horizontal lines, for example TTFT under 600 ms (about 1 second) and ITL under 100 ms per token. The rightmost point under both lines is the throughput one replica of this topology can serve within the service level objective (SLO), here c32, or about 320 tokens per second. To plan capacity, divide your target tokens per second by this per-replica throughput and find the number of prefill-decode pairs to deploy. Re-running the same sweep after a change (precision, TP/PP shape, aggregated versus disaggregated, a new Dynamo or vLLM release) makes the trade-off visible immediately. A curve that moves right and down is a win. Because TTFT and ITL are plotted separately, it also tells you which side to scale: rising ITL says add decode capacity, and rising TTFT says add prefill capacity.
Conclusion
This post covers the architecture and performance evaluation patterns for serving NVIDIA Nemotron models on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM. To get started, deploy Nemotron 3.5 Super VL in your own account, using the reference implementation. More inference blueprints are also available in the AI on EKS project. To get further hands-on experience with inference on EKS, register for our upcoming workshops.
Acknowledgements
The authors would like to convey special thanks to everyone involved in developing, validating, reviewing, and publishing this content, including but not limited to: Eliuth Triana, Jiahong Liu, Kshitiz Gupta, Abhijeet Patil, Mike Stefaniak, Chris Splinter, Sheetal Joshi, Aditya Ramakrishnan, Natarajan Kumar, Anish Maddipoti.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.