NVIDIA 尼莫特龙 3.5 转写和 Granary v2：智能多语种转写您可以提示和适应
一个多语种转录，扬声器归属，翻译，音频QQA的开放模式，具有顶级领导板结果，外加开放的Granary v2数据管道和微调配方来适应。
笔录可以捕捉到每个口语单词，仍然会留下用户或应用程序想知道是谁的称什么，如何翻译，或者如何格式化特定域的词汇。 多语种电话、节奏快的会议和医疗口述需要不同的产出：翻译、演讲者标签或正确拼写的医疗术语。
NVIDIA 尼莫特龙 3.5 Translax是一个开放的，可快速的模型，将录制的语音变为有用的文本，现在可以在Hugging Face和ModelScope上找到。 它将带来转录，扬声器归属(Diariation)，翻译，音频问题回答为一个检查点：它将40种语言转录，将其中31种语言转译为英语或从英语转译为英语，将扬声器归属为13种。 它处理长达1小时的录音，并给最多8名发言者贴标签。 开发者可以用自然语言指导输出，选择其音频处理的地方，并根据域名词汇和口音调整识别。
Transis在Hugging Face Open ASR Leaderboard上的开放量级模型中，在David AI的21种语言对称化领导板上，它运行在单一的NVIDIA GPU上，从DGX Spark到Blackwell数据中心系统。
Granary v2的多语言数据(可在Hugging Face和ModelScope上获得)和可重复使用的曲解管道，连同NeMo Speech 微调工具，将即时控制扩展至培训：开发者可以定义他们需要的书面形式，并根据他们的应用程序遇到的语音调整模型。 在一项医疗案例研究中，微调将被搁置的扬声器的字数误差率相对于最佳速率削减了45.8%.
使用自然语言进行翻译和分析
同一检查站可以根据以下指示，将一个会议录音变为不同的产出：
- “记录和标注每个发言者。”
- “将此音频翻译成英文。”
- “这次会议作出了什么决定？”
在下面的影片中体验一下这个模式的动作，它处理1940年的螺旋球喜剧"他的女孩星期五"(His Girl Friday)的经典场景——这部电影以其快速火力而闻名，对话高度重叠。
尼莫特龙 3.5 转录一个来自他女友星期五(1940年)的场景，并给发言者贴上标签，包括重叠的对话。
对于不熟悉的术语，在训练前先尝试背景。 例如，在保健方面：
用原文转述这个医学讨论。 可能的条件：HbA1c，元素。 仅在音频支持时使用这些术语。 保留标准的医疗拼写和传统写法编号。
模型卡推荐了预设，包括音频问题的推理，加上完整的即时指南和逐项任务矩阵：语言支持因任务而异，音频问题回答用英语训练。 文字层面的时间戳也得到支持。 与任何语音模型一样，抽查姓名，扬声器更改，并回答录音。
从现场课到持久课 Agora，他的实时语音和视频平台在应用中授权直播对话，在课堂演示中使用了单词级别的时间戳：Nemotron 3.5 Trancis将一个录制的班级转换成在重播的同时显示的笔录，这样学生就可以看到什么是称，什么时候。 Agora的博客贯穿了他们是如何构建的。
长板从转录到扬声器归属获胜
在开放量级模型中，Transis领先称；在谁的称上，它是超过两个扬声器-分配基准测试直接。
Hugging Face开放ASR导板。 领导板在公共测试集中以字误率(WER)衡量抄写精度，分为三条轨道：
- 英语：在开放重量型号中以5.33%的平均WER(图1)居首位。
多语种：4.62%是七种语言的WER，在模型卡的开放ASR比较中，在评估的开放量模型中最低。
- 长式：10.98%是WER，也是这一比较中最低的。
图1. 内莫特龙 3.5 Transis在Hugging Face Open ASR Leaderboard上的开放量级模型中排名第一(英文)，平均为5.33% WER，不包括API模型。 更低的WER表示较少的抄录错误。
人工分析语音转文本导板。 人工分析独立按照AA-WER将其字错误率指数语音转文本模型排行。 尼莫特龙 3.5 以2.8%的AA-WER在开放量级模型中转换为第一位的关联，其关联的模型有约4B活性参数与24B. 在AA-AgentTalk上，占AA-WER指数50%的被搁置的语音-agent语音设置，Transis评分1.8%WER，对次佳开放量级模型的评分2.3%. 转写其语音编码器与混合的Mamba - Transformer 混合专家语言模型成对，因此其30B参数中只有约4B个是每个符号活动。 这说明人工分析的开放式Pareto 精确度和有效参数的前沿。 在AA的全模图上，它排在几个被广泛使用的专有言论API之前。
图2. 活性参数六分之一的AA精度：Nemotron 3.5 Trancis实现2.80%的AA-WER，使用约4B活性参数。
发言者在各种语文之间的归属和重叠。 S排序前扬声器提示到达带有声学特性的语言模型，因此Transis可以在生成期间标记每个转弯，而不是仅仅依靠后一步将扬声器附加到单词上。
大卫·AI-AAI-ASR-I18N主板。 在David AI的DAI-ASR-I18N综合转录和语音-分配轨道上，Transtics在被评价的开放量级模型中排名第一，总体排名第二。 基准测试的Diarization轨迹上，在超过21种语言时会说话的得分，在测试的14个系统中，它的Diarization误差率最低，略高于NVIDIA专用的Nemotron 3 Diarization模型。
图3. NVIDIA模型在David AI的多语言对位化领导板上占据了前两个斑点：Nemotron 3.5 Translax在总对位化误差率(DER)中位居最低，在21种语言中占18.26%，其次是Nemotron 3 Diarization，占18.38%. DER 测量演讲时间错误；下调更好。
(原始内容存档于2017-09-26). VoiceArena Diarization-Bench. 在VoiceArena的英语全程基准测试上，Transis张贴了14个系统中最小的充电量(cpWER)，也降低了最少的词：删除的参考词占8.32%，而中位数约为31%.
图4. NVIDIA模型在VoiceArena的14个系统比较中占据了前两个位置：Nemotron 3.5 Translacy领先32.4% cpWER，其次NVIDIA多对讲机Parakeet与Nemotron 3 Diarization领先34.2%. Diarization-Bench v1涵盖英语的完整会话，重叠的演讲和2-8个演讲人。 cpWER 在匹配扬声器标签后计数单词错误；下值更好。
更多评价。 Ocular AI的Conversion-STT评价将Transis列为顶级开放模式。 Josh Talks的印度之声领导板报导了印度各语种的成果。
从边缘运行到数据中心。 Trancis运行于单NVIDIA GPU上，从Jetson AGX Thor和DGX Spark到布莱克威尔数据中心GPU. 在一台使用NVFP4精度的NVIDIA RTX PRO 6000 Blackwell Server版GPU上，它在三分钟内处理100小时的音频，在高通量时用30秒的剪辑测量。
或用NeMo-Speech.cpp在当地运行。 转录器是为录制的音频而建的，而非实时流传。
专门的录音仍然可以要求修改域名词汇和抄录常规。 Granary v2 和 NeMo Speech 微调的地址，从Trancis的背后数据开始。
Granary v2: Translatic 后面的开放数据校正管道
一个迅速的语音模型需要训练数据，并有明确的格式化目标。 Granary v2是Hugging Face和ModelScope上的Nemotron语音集合的一部分。 由NVIDIA NeMo Curator建造，其可重复使用的管道准备了Nemotron 3.5 Trancis培训中使用的多语种演讲，制作了对齐的语录变体、任务标签和可检查的质量元数据。 公开名单涵盖48种语言的463 062小时和16种源头公司，高于Granary v1的25种语言。 这些数字说明了数据集的释放，而不是Transis的全部训练混合物或其取样权重。
Granary v2为每次录制生成3个笔录表(表1)，同样的逐字记录和干净的样式开发者可以从Tranlax中请求，并带有一个提示。 除了抄写外，该管道还制作短音频字幕，代码交换还原借词，语音定位QQA对，背景ASR偏差条款，以及自动质量分数的翻译。
表1. Granary v2 同一语句的培训目标。
管道分为四个可重复使用的阶段：
- 1. 音频准备(Metadata)：部分为原始或长式音频，在全程录音中跟踪扬声器，并标记声响事件和语言。 Granary v2 的源头已经分解，因此此版本跳过了这个阶段。
- 2. 转录(ASR)：将每种语言通向主语和恢复ASR模型，当主语不能进行幻觉检查时，返回。 人文参考笔录有助于保存信任的拼写，同时添加口语形式和不流利之处。
- 3. Task Label Generation (Text)：衍生物的笔录变体，然后检查字符率，幻觉模式，以及模型或人类参考协议。 只有被接受的记录收到背景-ASR术语和翻译。
- 4. HiFieldity(HiFi)：过滤出吵闹、剪辑或包含多种扬声器的片段，保留一套较小的高质量录音，用于ASR以外的用途。 对于VoxPopuli数据集，它将欧洲22种语言的354k小时缩短到111k小时。
图6. Granary v2数据校准程序，采用从元数据、ASR和Text阶段的原始音频到培训实例的样本。
开发者可以在自己的录音或经授权的公开公司(public corpora)上运行与雷和NVIDIA Dynamo后端的NeMo Currator管道，也可以在Granary v2公开运单的基础上发展。 运单包括tn raw,itn raw，和tn clean三种笔录表单，上面描述的标签，实体提示，衍生指令对，出处，以及每个记录的质量旗帜。 依创用CC授权使用
适应重要的词汇识别
从一个指定语言，脚本，和笔录格式的开发设置提示开始。 在下文的案例研究中，将英文和拉丁字母命名在即时剪切全基准测试 WER中从30.8%到23.8%. 对于仍然存在的域错误，NeMo Speech 微调工作流程将代表性发言与明确的书面目标对齐。
NeMo Speech Medical ASR案例研究显示，该研究从生成的医学文本和公共医学目录中构建适应数据，然后结合了大约153小时的合成语音，29小时来自Common Voice的重音英语，63小时来自MultiMed的医学录音。
图7 在Eka Care的印度医学演讲基准测试的全套测试中，一个更快速的削减 WER从30.8%到23.8%，微调带来到11.8%，比模型出箱时少约62%的抄写错误。 全套包括用于选择即时检查点和检查站的855个调试片段；以下的预留结果为清洁测试
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt
One open model for multilingual transcription, speaker attribution, translation, and audio Q&A, with top leaderboard results, plus the open Granary v2 data pipeline and fine-tuning recipes to adapt it.
A transcript can capture every spoken word and still leave users or applications wondering who said what, how to translate it, or how to format domain-specific jargon. Multilingual calls, fast-paced meetings, and medical dictation each need different output: a translation, speaker labels, or correctly spelled medical terms.
NVIDIA Nemotron 3.5 Transcribe is an open-weight, promptable model that turns recorded speech into useful text, now available on Hugging Face and ModelScope. It brings transcription, speaker attribution (diarization), translation, and audio question answering into one checkpoint: it transcribes 40 languages, translates 31 of them to or from English, and attributes speakers in 13. It handles recordings up to an hour long and labels up to eight speakers. Developers can guide output with natural language, choose where their audio is processed, and adapt recognition to domain vocabulary and accents.
Transcribe is first among open-weight models on the Hugging Face Open ASR Leaderboard for English and sits on Artificial Analysis’s accuracy–active-parameter frontier. For speaker attribution, it posts the lowest error rate on David AI’s 21-language diarization leaderboard and ranks first in VoiceArena’s 14-system comparison. It runs on a single NVIDIA GPU, from DGX Spark to Blackwell data-center systems.
Granary v2’s multilingual data (available on Hugging Face and ModelScope) and reusable curation pipeline, together with NeMo Speech fine-tuning tools, extend prompt-level control to training: developers can define the written form they need and adapt the model to the speech their applications encounter. In a medical case study, fine-tuning cut word error rate (WER) on held-out speakers by 45.8% relative to the best prompt.
Guide transcription and analysis with natural language
The same checkpoint can turn one meeting recording into different outputs, depending on the instruction:
- “Transcribe this recording and label each speaker.”
- “Translate this audio into English.”
- “What decisions were made in this meeting?”
Experience the model in action in the video below, where it tackles a classic scene from the 1940 screwball comedy His Girl Friday—a film famous for its rapid-fire, heavily overlapping dialogue.
Nemotron 3.5 Transcribe captions a scene from His Girl Friday (1940) and labels speakers, including overlapping dialogue.
For unfamiliar terms, try context before training. In a healthcare setting, for example:
Transcribe this medical discussion in the original language. Possible terms: HbA1c, metformin. Use these terms only when supported by the audio. Preserve standard medical spelling and write numbers in conventional written form.
The model card has recommended presets, including reasoning for audio questions, plus the full prompt guide and a task-by-language matrix: language support differs by task, and audio question answering is trained in English. Word-level timestamps are also supported. As with any speech model, spot-check names, speaker changes, and answers against the recording.
From live classes to lasting lessons. Agora, whose real-time voice and video platform powers live conversations in apps, used word-level timestamps in a classroom demo: Nemotron 3.5 Transcribe turns a recorded class into a transcript shown alongside the replay, so students can see what was said and when. Agora’s blog walks through how they built it.
Leaderboard wins from transcription to speaker attribution
Among open-weight models, Transcribe leads on what was said; on who said it, it tops two speaker-attribution benchmarks outright.
Hugging Face Open ASR Leaderboard. The leaderboard measures transcription accuracy as word error rate (WER) on public test sets, in three tracks:
- English: first among open-weight models at 5.33% average WER (Figure 1).
- Multilingual: 4.62% mean WER across seven languages, the lowest among evaluated open-weight models in the model card’s Open ASR comparison.
- Long-form: 10.98% mean WER, also the lowest in that comparison.
Figure 1. Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR Leaderboard (English) at 5.33% average WER, with API models excluded. Lower WER means fewer transcription errors.
Artificial Analysis Speech-to-Text Leaderboard. Artificial Analysis independently ranks speech-to-text models by AA-WER, its word error rate index. Nemotron 3.5 Transcribe ties for first among open-weight models at 2.8% AA-WER, with about 4B active parameters versus 24B for the model it ties. On AA-AgentTalk, the held-out voice-agent speech set that makes up 50% of the AA-WER index, Transcribe scores 1.8% WER, versus 2.3% for the next-best open-weight model. Transcribe pairs its speech encoder with a hybrid Mamba–Transformer mixture-of-experts language model, so only about 4B of its 30B parameters are active per token. That puts it on Artificial Analysis’s open-weight Pareto frontier of accuracy versus active parameters. On AA’s all-model chart, it ranks ahead of several widely used proprietary speech APIs.
Figure 2. Leading AA accuracy at about one-sixth the active parameters: Nemotron 3.5 Transcribe achieves 2.80% AA-WER with approximately 4B active parameters.
Speaker attribution across languages and overlap. Sortformer speaker cues reach the language model with the acoustic features, so Transcribe can tag each turn during generation instead of relying solely on a later step to attach speakers to words.
David AI DAI-ASR-I18N Leaderboard. On David AI’s DAI-ASR-I18N combined transcription-and-speaker-attribution track, Transcribe ranks first among evaluated open-weight models and second overall. On the benchmark’s diarization track, which scores who spoke when across 21 languages, it has the lowest diarization error rate of the 14 systems tested, slightly ahead of NVIDIA’s dedicated Nemotron 3 Diarization model.
Figure 3. NVIDIA models hold the top two spots on David AI’s multilingual diarization leaderboard: Nemotron 3.5 Transcribe posts the lowest overall diarization error rate (DER), 18.26% across 21 languages, followed by Nemotron 3 Diarization at 18.38%. DER measures speaker-time errors; lower is better.
VoiceArena Diarization-Bench. On VoiceArena’s English full-session benchmark, Transcribe posts the lowest concatenated minimum-permutation WER (cpWER) of 14 systems and ranks first in both in-person and online settings. It also drops the fewest words: deletions are 8.32% of reference words, versus a median of about 31%.
Figure 4. NVIDIA models hold the top two spots in VoiceArena’s 14-system comparison: Nemotron 3.5 Transcribe leads at 32.4% cpWER, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. Diarization-Bench v1 covers English full sessions, overlapping speech and 2–8 speakers. cpWER counts word errors after matching speaker labels; lower is better.
More evaluations. Ocular AI’s Converse-STT evaluation ranks Transcribe the top open model. Josh Talks’ Voice of India leaderboard reports Transcribe’s results across Indian languages.
Runs from edge to data center. Transcribe runs on a single NVIDIA GPU, from Jetson AGX Thor and DGX Spark to Blackwell data-center GPUs. On one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with NVFP4 precision, it processes 100 hours of audio in under three minutes, measured on 30-second clips at high concurrency.
Serve the checkpoint with vLLM today or run it locally with NeMo-Speech.cpp. Transcribe is built for recorded audio, not live streaming.
Specialized recordings can still call for adaptation to domain vocabulary and transcript conventions. Granary v2 and NeMo Speech fine-tuning address that, starting with the data behind Transcribe.
Granary v2: The open data curation pipeline behind Transcribe
A promptable speech model requires training data with unambiguous formatting targets. Granary v2 is part of the Nemotron speech collection on Hugging Face and ModelScope. Built with NVIDIA NeMo Curator, its reusable pipeline prepared multilingual speech used in Nemotron 3.5 Transcribe’s training, producing aligned transcript variants, task labels, and inspectable quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. These figures describe the dataset release, not Transcribe’s full training mixture or its sampling weights.
Granary v2 generates three transcript forms for each recording (Table 1), the same verbatim and clean styles developers can request from Transcribe with a prompt. Beyond transcription, the pipeline produces short audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing terms, and translations with automatic quality scores.
Table 1. Granary v2 training targets for the same utterance.
The pipeline is organized into four reusable stages:
- 1. Audio Preparation (Metadata): Segments raw or long-form audio, tracks speakers across the full recording, and tags acoustic events and language. Granary v2’s sources were already segmented, so this release skipped this stage.
- 2. Transcription (ASR): Routes each language to primary and recovery ASR models, falling back when the primary fails a hallucination check. Human reference transcripts help preserve trusted spellings while adding spoken forms and disfluencies.
- 3. Task Label Generation (Text): Derives transcript variants, then checks character rate, hallucination patterns, and model or human-reference agreement. Only accepted records receive contextual-ASR terms and translations.
- 4. High Fidelity (HiFi): Filters out segments that are noisy, clipped, or contain multiple speakers, keeping a smaller set of high-quality recordings for uses beyond ASR. For the VoxPopuli dataset, it reduced 354k hours to 111k hours across 22 European languages.
Figure 6. Granary v2 data curation pipeline, following one sample from raw audio through the Metadata, ASR, and Text stages to a training example.
Developers can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora, or build on the Granary v2 public manifests. The manifests include the three transcript forms as tn_raw, itn_raw, and itn_clean, the labels described above, entity hints, derived instruction pairs, provenance, and a per-record quality flag. Download source audio separately under each corpus’s original license, and follow the release’s published terms.
Adapt recognition to the vocabulary that matters
Start with a development-set prompt that specifies language, script, and transcript format. In the case study below, naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%. For domain errors that remain, the NeMo Speech fine-tuning workflow pairs representative speech with explicit written targets.
The NeMo Speech medical ASR case study shows this progression on Eka Care’s open benchmark of recordings from Indian doctors and medical students, which has 3,619 clips and no designated training split. The study builds adaptation data from generated medical text and public medicine catalogs, then combines approximately 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical recordings from MultiMed.
Figure 7. On the full test set of Eka Care’s Indian medical speech benchmark, a better prompt cuts WER from 30.8% to 23.8%, and fine-tuning brings it to 11.8%, about 62% fewer transcription errors than the model out of the box. The full set includes the 855 tuning clips used to choose the prompt and checkpoints; the held-out results below are the cleaner test
Auto-fetched and translated by AI Atlas. All rights belong to the original author.