本地和分配的大型 LLM Mac Studio上的MLX和RDMA推论
使用Apple硅的Mac上的MLX让AI研究人员和开发者在私人环境中对最新的模型和方法进行局部实验。 开源阵列框架利用Apple硅的统一内存架构，使任何人都能在自己的硬件上运行新的推论和微调方法，并探索其他AI技术。 在之前的一篇文章中，我们强调了M5架构及其GPU神经加速器的优点。
在本文中，我们展示了如何运行最新的前沿级模型——Muse Spark Open，近万亿参数——从您的办公桌右转，使用最新版本的MLX，发布今天。 在四个Mac Studio的集群上，Muse Spark Open每秒生成超过36个令牌(见图1)，远高于人类平均阅读速度。
从一个单一的Mac Studio(最多512GB 统一内存)开始，然后用RDMA在Sunderbolt驱动器上跨过多个Macs，下面我们分享了现在在Mac Studio上可以运行的模型范围的概况。
用 MLX 运行大语言模型
我们在Mac Studio上使用MLX LM软件包，我们比较了两代的Mac Studio：近代的发布 M5 Ultra和前代的M3 Ultra. M5生成具有GPU神经加速器的特点，它支持专用矩阵乘法操作，对许多机器学习工作量至关重要。
我们评价了三种跨越各种大小的前沿级模型(表1)(以数十亿参数表示，B)：新发布 Muse Spark Open(816 B),Mistral中3.5(128 B)和Qwen 3.8(27 B)。 Muse Spark Open是专家(MoE)模型的混合体，它只保留了一小部分专家——因此也是参数——活性，在推论中减少计算，并以NVFP4精度评价。 Mistral Media 3.5和Qwen 3.8是密集的建筑，所有参数在推论中都是活跃的，并且用其原生浮点精度(分别为FP8和BF16)来评价。
我们的评价在一系列上下文中衡量三种度量：第一兆(TTFT，以秒计)、发电吞吐量(以兆/秒计)和峰值功率使用量(以瓦特计)。 本文中的所有性能编号都是用mlx lm获得的。 基准测试 (英语). 例如，运行带有8,192个上下文符号的Qwen 3.8，并生成128个令牌时，会使用命令行执行：
mlx lm.基准测试 - 型号为Qwen/Qwen3.8-27B -p 8192 - g 128 - b 1
Mac Studio上流行使用本地LLMs，正在私人环境下运行代理体验，包括代理编码应用程序，这些应用程序往往需要大的背景。 因此，我们以8 192和65 536个背景规模报告执行情况。
表2和表3分别报告了原始TTFT和解码吞吐量。
单个Mac工作室的推论性能
Qwen 3.8 27B和Mistral Medical 3.5 128B在512GB的单一Mac Studio上舒适地搭配，在大上下文尺寸(65,536个令牌)设置中搭载了最多68GB和158GB的RAM. 在图2中，我们比较了他们在单一Mac Studio上的推论性能与RAM的512GB，无论是M3 Ultra还是M5 Ultra.
到第一令牌的时间(TTFT)本质上是计算绑定的，这就是M5 Ultra GPU中的神经加速器闪耀的地方：GPU神经加速器快速矩阵倍增的专用硬件支持直接转化为TTFT的重大加速，比M3 Ultra快达4.6×(见图2). 这使得带有M5 Ultra的Mac Studio成为处理涉及非常大背景的任务时所选择的桌面。
生成速度(每秒令牌)是内存-带宽绑定。 M5 Ultra芯片提供1.2 TB/s的统一内存带宽，比M3 Ultra的819 GB/s更高。 在这里，这再次直接转化为生成速度(见图3)，由M5 Ultra的Mac Studio报告1.6×更快的符号/s.
Mac Studio对LLM推论也具有节能性(见图4)：峰值功率消耗约为M3 Ultra的Mac Studio200W，而M5 Ultra的Mac Studio300W——是普通烤面包机使用的功率的一小部分。 更重要的是，这种有限的耗电量意味着可以部署一个Mac Studio，用于任何家庭或办公环境中的AI工作。
虽然Qwen和Mistral两个模型都适合一个带有RAM的512GB的Mac Studio上，但Muse Spark Open仅对模型参数的原生精度(BF16)需要超过1.6TB的RAM. 在4位化的(NVFP4)格式中，模型内存足迹被降低到约493GB. 在推论中加入中间计算所需的内存，模型不能适应单机。 它需要跨多个Mac Studio计算机的并行。
横跨两个 Mac Studio 计算机的推论
Muse Spark Open等非常大型的前沿级型号将不适合单机。 然而，可以将一半的模型存储在一个Mac Studio上，另一半存储在另一个工作室上。 然后我们可以把模型的推论通过两台计算机进行：第一个Mac Studio执行模型的上半层，然后将输出送到第二个Mac Studio，后者执行下半层。 这种平行化方案称为管道平行化(见图5)，实施极为简单，只需要最小的代码修改。
MLX支持不同的平行计划，如MLX分布的文件所描述的。 这些系统依靠快速机际通信：macOS 26.2 推出是Sunderbolt 5上空的低常态RDMA驱动程序，MLX(v0.30.1及以后)基于此驱动程序，为不同的平行化计划提供新的通信集体(jaccl，宣告为"jackal"). 在我们的设置中，两个Mac Studio连接了两个Sunderbolt电缆(将链接带宽增加一倍)，提供了Jaccl使用的RDMA-over-Thunderbolt链接。 一旦建立主机，MLX提供了mlx.发布，一个帮助者命令，用于跨越多个节点的发布任务：
mlx.发布 - hostfile hostfile.json - 后端 jaccl - env MLX METAL FAST SYNCH=1 - \
mlx lm.基准测试 - 型号误差/Mistral-中度-3.5-128B -p 8192 - g 128 -b 1 - 预填步长 8192 - 管道
在此命令中，-backend jacl指定了 RDMA 霹雳驱动程序后端。 e-env MLX METAL FAST SYNCH=1旗启用了用于快速CPU-GPU同步的实验性MLX特性，减少了CPU(启动霹雳通信)和GPU(进行计算)之间的通信间接。
一般而言，与管道平行化的解码推导速度(见图7)与单机非常相似，因为层仍按顺序评估。 通信间接费用也非常低，因为两层之间只需交换中间产出。
然而，值得注意的是，对于非常大的情况(我们基准测试的65 536个符号)，管道平行化的完成速度快于一个用于预填的单一节点——最高为~1.8×更快(见图6)。 这是因为MLX将大上下文分割成小块(这里，8,192个令牌，由 -- prefill-step-size 8192设置)，并将预填管道贯穿于机器之间：而第二台机器计算后几层块\(k\)，而第一台机器计算前几层块\(k+1\)，因此连续的块在时间上重叠。 为了更好地计算平行，我们接下来将转向拉诺平行。
加速推论的十进制并行
Tensor平行化(见图5)将LLM的宽度分布在几台计算机上，与管道平行化形成对比，管道平行化将模型分布在它的深度。 现在每台机器都持有模型各层的一小部分，这样每个层的计算都是平行的。 在高效实施时，拉伸平行化会提高推论速度。 然而，拉诺平行化比管道平行化更难执行：每层必须平行，这增加了代码的复杂性。 该计划还涉及许多小的通信业务：对于每个输入符和每一层，必须广播投入和总产出。 因此，在MLX jacl的支持下，雷霆RDMA等非常低常态的通信技术对于这种方法产生任何速度都是至关重要的。
有了MLQQs高水平的mlx.发布帮助器，很容易切换到拉诺平行化。 Mac Studio电脑以雷电网连接，每对机器由两台雷电网连接对等电缆。 假设主机被设置，我们只是发布，基准测试跨越多个机器，其：
mlx.发布 - hostfile hostfile.json - 后端 jaccl - env MLX METAL FAST SYNCH=1 - \
mlx lm.基准测试-型号meta-型号/Muse-Spark-Open-816B-A42B-NVFP4-p 8192-g 128-b 1
图8和图9显示使用带有M5 Ultra的两台或四台Mac Studio计算机获得的速度，相对单机基线(或指Muse Spark Open，即管道平行基线). 值得注意的是，密集结构在多台计算机中的规模更好。
使用少量的机器，拉伸平行化的通信间接费用可以在非常大的情况下抵消其一些计算效益，尽管这只适用于预填。 显示与Muse Spark Open，其中专家输出的单向正常化新增一些预填通信，足够在非常大的情况下(65,536令牌)，然而，这一方法随着机器的加入而有所收效，这从四台M5超强机可以看出。 解码不受影响：Muse Spark Open即使在大语境下也能在拉诺平行化下保持显著的解码速度，与Qwen 3.8和Mistral Media 3.5等密度模型相当。
双线网在预填时也有所回报，其中额外的链接带宽被感觉。 对于Muse Spark Open on 4 M5 Ultra，在65,536-token上下文，每条链接有两个电缆将TTFT从约50.5s(1条电缆)切到40.6s——表2中报告的双电缆值，单电缆运行是额外的实验。
Mac 上的本地代理AI
最近Mac上的代理AI获得了相当的牵引力，代理编码工具在开发者中看到广泛采用。 有了MLX，将这些工具本地化的动力就和发布一样简单，它是一个带有mlx lm的服务器。 服务器工具。 例如，一个运行在端口8080上的Muse Spark Open的服务器开始于：
mlx.发布 - hostfile hostfile.json - 后端 jaccl - env MLX METAL FAST SYNCH=1 - \
mlx lm.server - 型号为meta-型号/Muse-Spark-Open-816B-A42B-NVFP4 - 港8080
就像Mlx lm一样。 基准测试，mlx lm.server帮助器可以使用mlx.发布的前缀，以并行计算跨多个Mac Studio. 从那里，代理AI可以启用OpenCode等工具，这些工具可以很容易地配置到使用本地的LLM服务器：
{
"$schema":"https://opencode.ai/config.json" (中文(简体) ).
"模型":"meta-型号/Muse-Spark-Open-816B-A42B-NVFP4".
"提供方":{
"mlx": {
"npm":"@ai-sdk/openai-兼容".
"姓名":"当地MLX"
"选项":{
"BaseURL":"http://127.0.0.1:8080/v1" 互联网档案馆的存檔，存档日期2013-07-02.
},
"模型":{
"meta-型号/Muse-Spark-Open-816B-A42B-NVFP4":{
"姓名":"Muse Spark Open 816B(MLX)" (中文(简体) ).
}
}
}
}
}
许多其他本地代理人工智能经验可以通过类似方法实现。
结论
在过去的一年里，我们致力于MLX的几个关键创新，以使最先进 LLMs在近期的Apple硅上就地运行更加容易和高效。 其中包括对M5芯片家族GPU中的神经加速器的软件支持，使用RDMA在Sunderbolt上跨多个Mac的低纬度通信，以及最近流行的架构的高效实施，如专家混合(MoE). 每一个都是靠自己来帮助的，但都是它们与最新的Apple硅的结合——Mac Studio与。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private environment. The open-source array framework leverages the unified memory architecture of Apple silicon, enabling anyone to run new inference and fine-tuning methods on their own hardware and explore other AI techniques. In a previous post, we highlighted the advantages of the M5 architecture and its GPU Neural Accelerators.
In this post, we show how to run the latest frontier-class model — Muse Spark Open, with nearly a trillion parameters — right from your desk, using the latest version of MLX, released today. On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second (see Figure 1), well above average human reading speed.
Beginning on a single Mac Studio (up to 512GB of unified memory), then scaling out across multiple Macs with the RDMA over Thunderbolt driver, below we share an overview of the range of models you can now run on Mac Studio.
Running Large Language Models with MLX
We benchmark several popular open-weight LLMs on Mac Studio, using the MLX LM package, as described in our prior post. We compare two generations of Mac Studio: the recently released M5 Ultra and the previous generation, M3 Ultra. The M5 generation features the GPU Neural Accelerator, which supports dedicated matrix multiplication operations, essential for many machine learning workloads.
We evaluate three frontier-class models (Table 1) spanning a wide range of sizes (in billions of parameters, B): the newly released Muse Spark Open (816 B), Mistral Medium 3.5 (128 B), and Qwen 3.8 (27 B). Muse Spark Open is a mixture of experts (MoE) model, which keeps only a fraction of its experts — and thus parameters — active, reducing compute at inference, and it is evaluated in NVFP4 precision. Mistral Medium 3.5 and Qwen 3.8 are dense architectures, where all parameters are active at inference, and they are evaluated in their native floating-point precision (FP8 and BF16, respectively).
Our evaluation measures three metrics across a range of context sizes: time to first token (TTFT, in seconds), generation throughput (in tokens/second), and peak power usage (in watts). All the performance numbers in this post were obtained with mlx_lm.benchmark. For example, running Qwen 3.8 with 8,192 tokens of context and generating 128 tokens is performed with the command line:
mlx_lm.benchmark --model Qwen/Qwen3.8-27B -p 8192 -g 128 -b 1
A popular use of local LLMs on Mac Studio is running agentic experiences in a private environment, including agentic coding applications that often need large contexts. We therefore report performance at both 8,192 and 65,536 context sizes.
Raw TTFT and decode throughput are reported in Table 2 and Table 3, respectively.
Inference Performance on a Single Mac Studio
Qwen 3.8 27B and Mistral Medium 3.5 128B fit comfortably on a single Mac Studio with 512GB, taking up to 68GB and 158GB of RAM in the large context size (65,536 tokens) setup. In Figure 2, we compare their inference performance on a single Mac Studio with 512GB of RAM, both on M3 Ultra and M5 Ultra.
Time to first token (TTFT) is essentially compute bound, and that is where the Neural Accelerators in the M5 Ultra GPU shine: the specialized hardware support for fast matrix-multiplication of the GPU Neural Accelerator directly translates into significant speedups for TTFT, up to 4.6× faster compared to an M3 Ultra (see Figure 2). That makes the Mac Studio with M5 Ultra the desktop of choice when it comes to handling tasks involving very large contexts.
Generation speed (in tokens per second) is memory-bandwidth bound. The M5 Ultra chip delivers 1.2 TB/s of unified memory bandwidth, up from 819 GB/s for the M3 Ultra. Here again, that translates directly into generation speed (see Figure 3), with the Mac Studio with M5 Ultra reporting 1.6× faster tokens/s.
The Mac Studio is also energy-efficient for LLM inference (see Figure 4): peak power consumption is about 200W for the Mac Studio with M3 Ultra, and 300W for the Mac Studio with M5 Ultra — a fraction of the power used by a common toaster. More importantly, that modest amount of power consumption means a Mac Studio can be deployed for AI work in any home or office environment.
While both the Qwen and Mistral models fit on a single Mac Studio with 512GB of RAM, Muse Spark Open would require more than 1.6TB of RAM in its native precision (BF16) for the model parameters alone. In a 4-bit quantized (NVFP4) format, the model memory footprint is lowered to about 493GB. Add the memory needed for intermediate computations during inference, and the model cannot fit on a single machine. It requires parallelization across multiple Mac Studio computers.
Pipelining Inference Across Two Mac Studio Computers
Very large frontier-class models like Muse Spark Open will not fit on a single machine. It is, however, possible to store half of the model on one Mac Studio and the other half on another. We can then pipeline the model’s inference across two computers: the first Mac Studio executes the first half of the model’s layers, then sends the output to the second Mac Studio, which executes the second half. This parallelization scheme is called pipeline parallelization (see Figure 5) and is extremely simple to implement, requiring only minimal code changes.
MLX supports different parallelization schemes, as described in the MLX distributed documentation. These rely on fast inter-machine communication: macOS 26.2 introduced a low-latency RDMA driver over Thunderbolt 5, and MLX (v0.30.1 and later) builds on top of this driver and provides new communication collectives (jaccl, pronounced “jackal”) for different parallelization schemes. In our setup, the two Mac Studios are connected with two Thunderbolt cables (to double the link bandwidth), providing the RDMA-over-Thunderbolt link that jaccl uses. Once the hosts are set up, MLX provides mlx.launch, a helper command to launch jobs across multiple nodes:
mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.benchmark --model mistralai/Mistral-Medium-3.5-128B -p 8192 -g 128 -b 1 --prefill-step-size 8192 --pipeline
In this command, --backend jaccl specifies the RDMA Thunderbolt driver backend. The --env MLX_METAL_FAST_SYNCH=1 flag enables an experimental MLX feature for fast CPU-GPU synchronization, reducing communication overhead between the CPU (which initiates the Thunderbolt communication) and the GPU (which does the compute).
In general, decode inference speed with pipeline parallelization (see Figure 7) is very similar to that of a single machine, since the layers are still evaluated sequentially. Communication overhead is also very low, as only the intermediate outputs between the two sets of layers need to be exchanged.
It is however notable that for very large contexts (65,536 tokens in our benchmark), pipeline parallelization ends up faster than a single node for prefill — up to ~1.8× faster (see Figure 6). This is because MLX segments large contexts into smaller chunks (here, 8,192 tokens, as set by --prefill-step-size 8192) and pipelines the prefill across the machines: while the second machine computes the later layers of chunk \(k\), the first machine computes the earlier layers of chunk \(k+1\), so consecutive chunks overlap in time. For even better compute parallelization, we next turn to tensor parallelization.
Tensor Parallelization for Faster Inference
Tensor parallelization (see Figure 5) distributes the LLM’s width across several computers, in contrast to pipeline parallelization, which distributes the model across its depth. Each machine now holds a fraction of each of the layers of the model, such that per-layer computation is parallelized. When efficiently implemented, tensor parallelization improves inference speed. However, tensor parallelization is more complex to implement than pipeline parallelization: each layer must be parallelized, which increases code complexity. The scheme also involves many small communication operations: for each input token and for each layer, one must broadcast inputs and aggregate outputs. As a result, a very low-latency communication technology like Thunderbolt RDMA, supported by MLX jaccl, is critical for this approach to yield any speedup.
With MLX’s high-level mlx.launch helper, it is easy to switch to tensor parallelization. The Mac Studio computers are connected in a Thunderbolt mesh, with each pair of machines linked peer-to-peer by two Thunderbolt cables. Assuming the hosts are set up, we simply launch the benchmark across multiple machines with:
mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.benchmark --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 -p 8192 -g 128 -b 1
Figure 8 and Figure 9 show the speedups obtained with two or four Mac Studio computers with M5 Ultra, relative to a single-machine baseline (or, for Muse Spark Open, the pipeline parallelization baseline). Notably, dense architectures scale better across multiple computers.
With a small number of machines, the communication overhead of tensor parallelization can offset some of its compute benefits at very large contexts, though this applies to prefill only. This shows up with Muse Spark Open, where separate normalization of the experts’ outputs adds some prefill communication, enough that at very large contexts (65,536 tokens), two-machine tensor parallelization lags behind two-machine pipeline parallelization. The approach however pays off as machines are added, as seen with four M5 Ultras. Decode is unaffected: Muse Spark Open sustains a significant decode speedup under tensor parallelization even at large contexts, comparable to dense models like Qwen 3.8 and Mistral Medium 3.5.
The two-cable mesh also pays off during prefill, where the extra link bandwidth is felt. For Muse Spark Open on four M5 Ultras at a 65,536-token context, two cables per link cut TTFT from about 50.5s (one cable) to 40.6s — the two-cable value reported in Table 2, the one-cable run being an additional experiment.
Local Agentic AI on Mac
Recently, agentic AI on Mac has gained considerable traction, with agentic coding tools seeing widespread adoption among developers. With MLX, powering these tools locally is as simple as launching a server with the mlx_lm.server tool. For example, a server running Muse Spark Open on port 8080 is started with:
mlx.launch --hostfile hostfile.json --backend jaccl --env MLX_METAL_FAST_SYNCH=1 -- \
mlx_lm.server --model meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --port 8080
As with mlx_lm.benchmark, the mlx_lm.server helper can be prefixed with mlx.launch to parallelize computation across multiple Mac Studios. From there, agentic AI can be enabled with tools like OpenCode, which can be easily configured to use that local LLM server:
{
"$schema": "https://opencode.ai/config.json",
"model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
"provider": {
"mlx": {
"npm": "@ai-sdk/openai-compatible",
"name": "Local MLX",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"meta-models/Muse-Spark-Open-816B-A42B-NVFP4": {
"name": "Muse Spark Open 816B (MLX)"
}
}
}
}
}
Many other local agentic AI experiences can be enabled through a similar approach.
Conclusion
Over the past year, we have worked on several key innovations in MLX to make it easier and more efficient to run state-of-the-art LLMs locally on recent Apple silicon. These include software support for the Neural Accelerators in the GPU of the M5 family of chips, low-latency communication across several Macs using RDMA over Thunderbolt, and efficient implementations of recent popular architectures such as mixture of experts (MoE). Each of these helps on its own, but it is their combination with the latest Apple silicon — Mac Studio with
Auto-fetched and translated by AI Atlas. All rights belong to the original author.