Prime Intellect拥有发布 Prime Mega MoE，是DeepSeek熔化的Mega MoE内核的训练扩展。 兹Q 1Z内核只跑过前方传球。 Prime Intellect增加了快速，重叠的后传。 这使得引信内核可用于训练，而不仅仅是推论。 在其精品-rl教练器中，Qwen3 MoE模型的全面训练步骤运行速度高达34%. 反对火炬，收益达到55%. 对于训练混合型专家的团队，这是在GPU时数的直接削减。
TL；数据
- 大小：GPU内核，不是模型。 在Qwen3-30B-A3B和Qwen3-235B-A22B(128位专家，前8位路线)上试验。
- 运行在：NVIDIA Hopper(blockwise FP8)和Blackwell(MXFP8)GPUs,BF16两个上。 公布的编号使用8×H200节点，32至128GPU.
- 业绩：32%至34%的全程培训步骤比初级水平基线更快。 比火炬快48%到55%.
- 最佳：3.11x更快的前进通过Qwen3-30B-A3B MoE层(1.2万米到3.9米).
- 最差：1.24x后传Qwen3-235B-A22B MoE层(30.5米至24.6米).
何为"超级精英"?
Prime Mega MoE是一款由Prime Intellect提供训练的，相互重叠的MOE内核。 它基于DeepSeek的开源 DeepGEMM库中的Mega MoE内核。
教育部培训的通信为何有限制？
在教育部的模型中，每个符号都交给一小部分专家。 这些专家坐在不同的GPU上。 所以每一层都发出信号，运行专家数学，并发送结果回来。 这发生在前传和后传。
在标准设置中，这些都是单独的操作。 GPU排序符和等待在运输层。 然后它计算，手结果回来，再等待。 这些等待是闲置的GPU时间。
熔化的内核是如何工作的？
DeepSeek的Mega MoE于2026年4月加入DeepGEMM，用一个内核呼叫取代了这个链条。 它将专家平行的发送，2个专家线性层，SwiGLU，并结合。 虽然一个象征团体正在过境，但内核计算了已经抵达的团体。 它还通过使用只使用GPU的刮痕缓冲来跳过中间伸缩器。 但是，它只是提前执行。
Prime Intellect写了缺失的落后内核，同样重叠。 支持的类型是BF16,MXFP8(仅限布莱克威尔)和阻断式FP8(仅限霍珀). prime-rl配置还曝光了fp8 bwd模式：BF16向前，FP8向后。
超级大魔王的动作有多快？
以下所有数字均由Prime Intellect自报。 训练步骤使用8×H200节点，FP8,16,384-token序列并全面启动检查。 每个结果都是步骤6至20的平均值。
完整的训练步骤，秒( 下方更好):
针对火炬，30B型机车的吞吐量从5,337升至7,886令牌/s/GPU. 在235B型机车上，由1,094升至1,697. Prime-rl基线使用火炬全程加DeepGEMM FP8专家。
单 MoE 层，毫秒(1个节点，8×H200,EP=8,128位专家，前8，随机路由):
规律明了。 前进收益较大，落后收益较小。 标题“最多3.11x”仅指前方通行证。
如何与其他已安装引信的教育部选项进行比较？
火炬泰坦41%的收益来自不同的设置(B200,671B模型，BF16基线). 它不具有直接可比性。
开发者如何使用？
Prime Mega MoE是Preme-rl的调度后端。 设置类型= [trainer.model.moe.dispatch]下的"mega moe". 从 bf16 选择精度，mxfp8, fp8 或 fp8 bwd. 训练内核飞船在从质内核预先建造的DeepGEMM轮中。 股票DeepGEMM的建设缺乏它们。
关键外卖
- Prime Mega MOE 新增是DeepSeek的Mega MOE的一款被熔化，重叠的后传。
- 完整的培训步骤比基础基线快32%至34%。
在Qwen3MOE型号上，步数48%至55%的速度更快。
- 前期收益达到3.11x，但后期收益下降到大层1.24x。
- 它以最原始的Rl作为超级摩尔发送后端
请检查access-date=中的日期值 (帮助) Prime Mega MoE博客文章，并加入我们的通讯。 等等你在电报里吗现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar 等吗？联系我们
Asif Razzaq是Marktechpost AI Media Inc.的首席执行官。 作为一个有远见的企业家和工程师，阿西夫致力于利用人工智能的潜力促进社会福祉。 他最近的努力是人工智能媒体平台的发布，该平台以深入报道机器学习和深层学习新闻而著称，这些新闻在技术上既可靠又为广大观众所容易理解。 该平台每月有超过200万次观看，表明其受观众欢迎。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Prime Intellect has released Prime Mega MoE, a training extension of DeepSeek’s fused Mega MoE kernel. DeepSeek’s kernel only ran the forward pass. Prime Intellect added a fast, overlapping backward pass. That makes the fused kernel usable for training, not just inference. In its prime-rl trainer, full training steps on Qwen3 MoE models run up to 34% faster. Against TorchTitan, the gain reaches 55%. For teams training mixture-of-experts (MoE) models, that is a direct cut in GPU hours.
TL;DR
- Size: A GPU kernel, not a model. Tested on Qwen3-30B-A3B and Qwen3-235B-A22B (128 experts, top-8 routing).
- Runs on: NVIDIA Hopper (blockwise FP8) and Blackwell (MXFP8) GPUs, BF16 on both. Published numbers use 8×H200 nodes, 32 to 128 GPUs.
- Performance: 32% to 34% faster full training steps than the prime-rl baseline. 48% to 55% faster than TorchTitan.
- Best: 3.11x faster forward pass on a Qwen3-30B-A3B MoE layer (12.0 ms to 3.9 ms).
- Worst: 1.24x backward pass on a Qwen3-235B-A22B MoE layer (30.5 ms to 24.6 ms).
What is Prime Mega MoE?
Prime Mega MoE is a fused, overlapping MoE kernel for training from Prime Intellect. It builds on the Mega MoE kernel in DeepSeek’s open-source DeepGEMM library.
Why is MoE training communication-bound?
In an MoE model, each token goes to a small subset of experts. Those experts sit on different GPUs. So every layer sends tokens out, runs expert math, and sends results back. This happens in the forward pass and again in the backward pass.
In a standard setup, these are separate operations. The GPU sorts tokens and waits on the transport layer. Then it computes, hands results back, and waits again. Those waits are idle GPU time.
How does the fused kernel work?
DeepSeek’s Mega MoE, added to DeepGEMM in April 2026, replaces that chain with one kernel call. It fuses expert-parallel dispatch, the 2 expert linear layers, SwiGLU, and combine. While one token group is in transit, the kernel computes on a group that already arrived. It also skips intermediate tensors by using GPU-only scratch buffers. But it only shipped a forward implementation.
Prime Intellect wrote the missing backward kernel, also overlapping. Supported types are BF16, MXFP8 (Blackwell only) and blockwise FP8 (Hopper only). The prime-rl config also exposes an fp8_bwd mode: BF16 forward, FP8 backward.
How fast is Prime Mega MoE?
All numbers below are self-reported by Prime Intellect. Training steps use 8×H200 nodes, FP8, 16,384-token sequences and full activation checkpointing. Each result is the mean of steps 6 to 20.
Full training step, seconds (lower is better):
Against TorchTitan, throughput rose from 5,337 to 7,886 tokens/s/GPU on the 30B model. On the 235B model, it rose from 1,094 to 1,697. The prime-rl baseline uses torch all-to-all plus DeepGEMM FP8 experts.
Single MoE layer, milliseconds (1 node, 8×H200, EP=8, 128 experts, top-8, random routing):
The pattern is clear. Forward gains are large, and backward gains are smaller. The headline “up to 3.11x” refers to the forward pass only.
How does it compare with other fused MoE options?
TorchTitan’s 41% gain comes from a different setup (B200, 671B model, BF16 baseline). It is not directly comparable.
How do developers use it?
Prime Mega MoE is a dispatch backend in prime-rl. Set type = "mega_moe" under [trainer.model.moe.dispatch]. Choose precision from bf16, mxfp8, fp8 or fp8_bwd. The training kernels ship in a prebuilt DeepGEMM wheel from prime-kernels. The stock DeepGEMM build lacks them.
Key Takeaways
- Prime Mega MoE adds a fused, overlapping backward pass to DeepSeek’s Mega MoE.
- Full training steps run 32% to 34% faster than the prime-rl baseline.
- Against TorchTitan, steps run 48% to 55% faster on Qwen3 MoE models.
- Forward gains reach 3.11x, but backward gains drop to 1.24x on large layers.
- It ships in prime-rl as the mega_moe dispatch backend.
Check out the Prime Mega MoE blog post, the announcement on X, and the prime-rl repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.