Nvidia开放尼莫特龙3.5 译名结合了抄写、翻译和语音识别
关键点
- Nvidia有发布尼莫特龙3.5 转写(Transtics)，一种遵循文本命令的开放式语音模式，并结合了最多40种语言的转写，翻译，语音识别。
- 该模型有300亿个参数，但每步只有40亿个活跃，因此运行在一个单一的GPU上。 它的排名接近基准测试的顶部，以获得准确性和扬声器归属。
- Nvidia还发布Granary v2，一个有84.4万小时音频的数据集，以及高效调整模型以适应医学等专门领域的工具。
Nvidia有发布的开放式语音识别模型，它接受文本指示，公司称带领几个基准测试。 该版还包括一个大型的培训数据集和使模型适应专门领域的工具。
简单的笔录不会告诉你称是什么它也不会翻译任何东西，或者拼写技术术语正确。 尼莫特龙 3.5 Transtics旨在用一个单一的模型处理所有这一切，它也可以回答关于录音内容的问题。 Nvidia 称模式转录40种语言，将其中31种语言译成英文或从英文译出，并用13种语言识别扬声器。 它处理长达一小时的录音。 发言者最多可达8人。
用户使用简单的指令控制模型。 视情况而定，同一会议录音可成为带有发言者标签的录音记录、英文译本或所作决定的摘要。 用户也可以通过诸如药物名称等技术术语作为提示。 模型只有在录音中真正出现时才应该使用。
单个 GPU 上的近端精度
该模型有300亿参数，但每个处理步骤只有约40亿活跃。 因此，它运行在单一的Nvidia GPU上。 这涵盖了从紧凑DGX Spark到数据中心硬件的所有内容。 Nvidia 称该型号以NVFP4精度在一台RTX PRO 6000 Blackwell服务器版上在不到3分钟的时间内处理100小时的音频。
在Hugging Face的英语Open ASR Leaderboard上，Transis在开放量级模型中排名第一，平均字差率为5.33%. 在人工分析的比较中，它的得分是2.81%. 这接近最佳的开放模型，Voxtral Small 以2.77%的成绩，但Trancis只使用了大约六分之一的有效参数。 Nvidia表示，后面有几条被广泛使用的商业语音服务线索。
领先者在演讲者识别方面是最大的。 VoiceArena测试了14个对话系统，其中发言者有时相互交谈，Transis排在第一位。 同时，它也减少了最少的字数，少了8.32%，而中位数约为31%。 在大卫AI的基准测试上，它测量了21种语言的演讲者归属，它贴出了最低的误差率。 它刚刚在Nvidia自己专为任务设计的模型前完成。
开放的数据集允许开发人员为自己的领域提供培训
除了模型，Nvidia正在发布Granary v2数据集。 它包含约84.4万小时的48种语言的语音，比上一个版本的约64.3万小时的25种语言的语音有所上升。 每张录音都以几种抄录风格出现。 一个版本是逐字的，有"uh"等填词和拼写数字。 另一个被清理了，没有填充器和数字，而不是文字。 Nvidia也正在分享它用来构建数据集的工具，因此开发者可以在自己的录音上运行。 称公司的数据集虽然并不完全符合模型的训练数据。
Nvidia测试的目标是微调对印度医生和医科学生的录音。 在完整的基准测试上，更精确的迅速将误差率从30.8%降低到23.8%. 在微调之后，该比率下降到11.8%. 培训数据大多是合成语言。 对于演讲者来说，这个模型以前没有听说过，Nvidia分别报告从22.44%下降到12.17%，相对改善45.8%. 这使得调整后的模型领先于开放的医学演讲模型，其得分达到13.79%. Nvidia 称整个过程在单张Nvidia GB300上花了大约14个GPU小时。
专门化要付出代价的一般英语语音误差率略有上升，医学模型在品牌药物上保持更准确。 Nvidia建议在构建自定义版本时至少预留100张真实，经核实的录音进行测试。
基础模型也有限度。 特征因语言而异，Nvidia 称音频问答功能则接受英语培训。 该模型也仅用于录制音频。 Nvidia 称用户应当抽查姓名，扬声器更改，回答。
该模型和数据集位于Hugging Face上。 还有一个可以尝试的演示。 对于微调，Nvidia提供指南和扩展，让AI编码助理运行培训工作流程。
Nvidia直到最近才于9月将专用的扬声器识别模型推出 Nemotron 3 Diarization. 它可以分辨出多达8个扬声器，甚至在直播中也可以分辨出来，但是它需要像Nvidia的Parakeet这样的单独的模型来进行实际的转录。 尼莫特龙 3.5 Transis既结合了工作，又结合了新增的翻译和音频理解。 虽然它并不支持实时处理。 Nvidia已经将Granary数据集用于其多式联运Nemotron 3 Nano Omni。
竞争者正在确定不同的优先次序。 Mistral的Voxtral家族也结合了转录和音频理解。 随着近期的宣布 Voxtral 3，公司主要关注实时语音agents.
AI News Without the Hype – 由人类治愈
订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。
现在订阅
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Nvidia's open Nemotron 3.5 Transcribe combines transcription, translation, and speaker recognition
Key Points
- Nvidia has released Nemotron 3.5 Transcribe, an open speech model that follows text commands and combines transcription, translation, and speaker recognition for up to 40 languages.
- The model has 30 billion parameters, but only 4 billion are active per step, so it runs on a single GPU. It ranks near the top of benchmarks for accuracy and speaker attribution.
- Nvidia is also releasing Granary v2, a dataset with 844,000 hours of audio, along with tools for efficiently adapting the model to specialized fields like medicine.
Nvidia has released an open speech recognition model that takes text instructions, and the company says it leads several benchmarks. The release also includes a large training dataset and tools for adapting the model to specialized fields.
A plain transcript doesn't tell you who said what. It also won't translate anything or spell technical terms correctly. Nemotron 3.5 Transcribe is designed to handle all of that in a single model, and it can answer questions about a recording's content as well. Nvidia says the model transcribes 40 languages, translates 31 of them to or from English, and identifies speakers in 13. It handles recordings up to an hour long. They can include up to eight speakers.
Users control the model with simple instructions. Depending on the prompt, the same meeting recording can become a transcript with speaker labels, an English translation, or a summary of the decisions made. Users can also pass along technical terms like drug names as hints. The model is only supposed to use them if they actually appear in the recording.
Near-top accuracy on a single GPU
The model has 30 billion parameters, but only about 4 billion are active per processing step. As a result, it runs on a single Nvidia GPU. That covers everything from the compact DGX Spark to data center hardware. Nvidia says the model processes 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition at NVFP4 precision.
On Hugging Face's English Open ASR Leaderboard, Transcribe ranks first among open-weight models with an average word error rate of 5.33 percent. In Artificial Analysis' comparison, it scores 2.81 percent. That's close to the best open model, Voxtral Small at 2.77 percent, but Transcribe uses only about a sixth of the active parameters. According to Nvidia, several widely used commercial speech services trail behind.
The lead is biggest in speaker recognition. VoiceArena tested 14 systems on conversations where speakers sometimes talk over each other, and Transcribe took first place. It also dropped the fewest words, missing 8.32 percent compared to a median of about 31 percent. On David AI's benchmark, which measures speaker attribution across 21 languages, it posted the lowest error rate. It finished just ahead of Nvidia's own dedicated model for the task.
An open dataset lets developers train for their own fields
Alongside the model, Nvidia is releasing the Granary v2 dataset. It contains about 844,000 hours of speech in 48 languages, up from roughly 643,000 hours in 25 languages in the previous version. Each recording comes in several transcription styles. One version is verbatim, with filler words like "uh" and spelled-out numbers. Another is cleaned up, with no fillers and digits instead of words. Nvidia is also sharing the tools it used to build the dataset, so developers can run them on their own recordings. The company says the dataset doesn't fully match the model's training data, though.
Nvidia tested targeted fine-tuning on recordings of Indian doctors and medical students. On the full benchmark, a more precise prompt cut the error rate from 30.8 to 23.8 percent. After fine-tuning, the rate fell to 11.8 percent. The training data was mostly synthetic speech. For speakers the model hadn't heard before, Nvidia separately reports a drop from 22.44 to 12.17 percent, a relative improvement of 45.8 percent. That put the adapted model ahead of an open medical speech model, which scored 13.79 percent. Nvidia says the whole process took about 14 GPU hours on a single Nvidia GB300.
The specialization comes at a cost. The error rate rose slightly on general English speech, and the medical model stayed more accurate on brand-name drugs. Nvidia recommends setting aside at least 100 real, verified recordings for testing when building custom versions.
The base model has limits too. Features vary by language, and Nvidia says the audio question-answering feature was trained in English. The model is also built only for recorded audio. Nvidia says users should spot-check names, speaker changes, and answers.
The model and dataset are on Hugging Face. There's also a demo you can try. For fine-tuning, Nvidia offers guides and an extension that lets an AI coding assistant run the training workflow.
Nvidia only recently introduced Nemotron 3 Diarization, a dedicated speaker recognition model, in September. It can tell apart up to eight speakers, even in live conversations, but it needed a separate model like Nvidia's Parakeet for the actual transcription. Nemotron 3.5 Transcribe combines both jobs and adds translation and audio understanding. It doesn't support real-time processing, though. Nvidia had already used the Granary dataset for its multimodal Nemotron 3 Nano Omni.
Competitors are setting different priorities. Mistral's Voxtral family also combines transcription and audio understanding. With the recently announced Voxtral 3, the company is focusing on real-time voice agents.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
Auto-fetched and translated by AI Atlas. All rights belong to the original author.