自理查德·萨顿2019年的有影响力的散文以来，苦痛课的概念指导了AI中许多现代进步。 这一概念支持利用计算和学习尺度的一般方法，尽可能不出现人为设计的前科。 我们的解释是，从本质上讲，智能是从原始数据中学习的：自主地寻找规律，构建抽象，从混乱的低层次投入中发展意义，理想的是端到端，尽可能地提高数据的效率。 随着AI发展到科学，机器人，生物学等复杂领域 — — 数据稀缺，而且人类往往难以理解，更不要说工程师了 — — 将这一哲学内部化，以开发更普遍的学习算法，比以往任何时候都更重要。
语言模型或许是AI今天最大的成功故事，他们的能力主要归功于一些简单的算法的缩放，如变形器架构，下接的预测，强化学习，以及测试时间的计算。 然而，它们仍然依赖于手动的编码步骤，如代号器，这严重阻碍了它们从基础数据中学习端到端的能力。 我们认为，开发字节级语言模型是AI进步的基础 — — 不仅是改善独立的LLMs，也是一般学习方法的试金石。 如果我们甚至不能从人物和字节中很好地学习语言，那么我们在下一个领域有什么权利取得进展？
2025年，我们发展了分级网络(H-Nets)的概念，它不仅解决了语言模型中的象征性化问题，而且还朝着更加通用的架构迈出了一步，这些架构可以学习不同模式的原始数据。 我们已大幅提升H-Net技术，并讨论影响。
我们将会探索H-Net的缩放定律
- 我们为象征性的变形器制作一个标准缩放阶梯，并显示经过仔细控制的一级H-Net纯粹通过学习的象征性化提供有意义的计算效率增益(最多3-5x).
- 我们显示，在扩大更灵活的2级H-Net方面取得了经验成果，它比开源模型的效绩要好，效率得到极端提高(计算和数据减少10x)。
我们分析H-Net所学的规律，其中显示了语言模型之前所未见的丰富多样的新行为，例如内文符号化。
- 我们讨论H-Net如何超越语言建模，在生物学和愿景方面不断取得研究成果，显示作为跨模式的一般结构的前景。
我们将会更深入地探索更多模式
总之，我们的成果提供了证据，证明H-Net是一个基础结构，它真正符合苦痛的教训，直接从原始数据中学习意义，并大大提高效率和可扩展性。 这个研究议程是推进AI如何从所有数据中学习的重要机会
等级网络(H-Nets)
我们推出分级网络(纸面，博客)去年作为一个新的端到端架构，在数据中模拟多个层次的抽象，在语言建模中和之外表现出希望。 简言之：
术语的灵感来自认知心理学中现存的块化概念，这是人类智能的重要组成部分。 我们认为，对人工智能的前沿也有必要。
在语言建模方面，这个想法是构建新的学习原始，使模型可以将低层次的数据递归成更高层次的意义：字符成词，词组成词，词组成概念。
就这个职位而言，人们基本上可以认为等级网络取代了象征性的网络。 H-Net 并不是从字节到符号的预处理数据(固定的字节块)，而是直接在模型内动态地进行块化。 此外，H-Nets的签字特征是，它们按等级进行这种分割的端到端。 对比之下，标准标志性语言模型，甚至其他替代方案(有些将在《邮报二》中看到)采用多个培训阶段，数据块化/标志化的过程完全脱机。
详细情况：托肯化和BPE
语言模型并不直接读取文字，一个自然词汇是256字节值，每个UTF-8字节产生一个令牌。 然而，字节级序列对于标准语言模型的处理来说是漫长而昂贵的。 通过将字节序列压缩成经常出现的块来缓解这一问题。 最常见的方法Byte Pair Encoding(BPE)从256字节开始，在固定的训练仪上反复合并最频繁的对相邻的标志，直到词汇达到设定大小(例如Llama-3). 在我们的数据组合上，每个BPE令牌覆盖大约4.57字节，使序列比字节级要短得多。 然而，这种压缩是固定的，象征性化仍然是语言模型管道的最后一部分，它是手工设计的而不是学习出来的。
即便在取代象征化的狭义背景下，H-Net 的端到端财产也能带来真正的绩效收益。 除了示意外，H-Net作为一个基础架构还有广泛的用途：
- H-Net可以递归式地嵌入，从而可以建立多个层次的抽象。 这个员额的主要结果是比开源基线更大规模的两级分级网络。
- H-Net可以运行在任何类型的数据上，而不仅仅是字节，而不只是离散数据。 与此相反，对连续数据进行标记是一项严峻挑战。 在文章的最后部分，我们将预览一些未来应用H-Nets超越文本的研究。
图1说明了一个2级H-Net，并定义了模型的主要组成部分：编码器，主网络，以及解码器。 我们有时把编码器/解码器统称为外网，主模型称为内网。
我们计划推出H-Net，我们将会更多评论与之前的架构不同
第1部分：生物防护设备及H-Net扩容梯
与许多建筑论文一样，关于分级网络的原始论文侧重于调整特定固定大小和数据视野的语言模型(例如FineWeb-Edu的100B令牌上的~1.3B-参数模型非常常见). 尽管它们看起来很有希望，但缺少一个内容是缺乏计算-优化的缩放法，现在是一种衡量预先训练创新的标准方法。 例如，Kimi在Kimi Delta注意力(KDA)上的流行创新和注意力残留分别测得1.16x和1.25x的计算乘数。
为了验证我们改进后的H-Net架构的收获，我们首先着手建造一个现代化的LLM缩放阶梯，然后用一个精心控制的等级网络来取代其标志化组件。
- 我们建立了一个培训混合体，以在平等条件下比较字节级和BPE放大定律，仔细匹配处理，甚至再培训基线的BPE标定器。
——我们根据标准做法为基线的BPE变形器语言模型搭建了阶梯，复制文献成果。
- 然后，我们把每个运行器转换成一个字节级的H-Net作为空投替换，再使用BPE主干架构和超参数。
这种配方直接显示验证损失(1.3x)的收效较强，代码(2x)等子域的增益增加，并且改进了发行(评价2x，域外代码最多5x).
数据控制的比较
由于字节级模型和BPE模型在不同颗粒体处理相同数据，需要额外注意保持方法的控制。
数据混合：我们的培训数据集由70%的英语，20%的代码，5%的数学和5%的多语种组成，用字节数来测量。
数据处理：每个样本都是作为字节和BPE符号进行处理的，在相同的边界进行分割；因此，字节和BPE处理过的数据集都包含与对齐的文档边界完全相同的文本。
量子：我们用标准比特-每字节(bpb)度量测量下位预测损失，在不同的标点符之间实现常态化。
公司内部的BPE代号器：LLMs一般使用现成的BPE代号器；H-Net的一个优点是他们直接在训练分发上学习它们的块。 为了格外小心，我们还训练了一台BPE仪表器在BPE的缩放定律下，我们使用这个法则，最终在验证损失和下游evals上
BPE 的标致器尺寸为 128k，公司内部的代号器将数据集压缩得略多一些，将平均BPE代号长度从4.45(Llama-3代号器)提高到4.57. 培训数据集以公司内部的BPE标识器表示，由64.7%的英语、23.7%的代码、6.1%的数学和5.5%的多种语言组成。
建立强有力的业务流程保护基线
对于我们的基线来说，我们适合一个经过良好调整的缩放定律一个标准的BPE变形器。 为此，我们按照Kimi和MAI等现代的缩放配方构建了一个缩放阶梯，用L层数来充分定义一个模型家族。 我们把隐藏的维度D设为128倍，这样尺寸比D/L约为75. 对所有模型，我们使用多头注意，头数等于层数，固定头尺寸128. 对于MLP，我们使用中间尺寸为3.6D的SwiGLU层，再四舍五入到128.1的最近倍数。
然后，我们把计算-最佳的缩放定律与标准的异能-FLOP扫描法相匹配。
- 所有运行都使用Muon优化器和WSD的学习速率表，并有固定的折叠热和衰变。
- 我们调整了批量大小和特定径的学习率，并安装了超参数缩放定律，以推导到较大的定律。
我们仔细计算了所有模型中的FLOP，包括所有序列混合层和LM头。 对于软马克思的注意，我们也只计算因果的FLOP，因为由于四极复杂，过量计算非因果的FLOP会惩罚处理较长序列的字节级模型。
等效的H-Net 梯级
因为H-Nets超过字节的字节与标准的标注模型大不相同，所以直接比较它们非常非三角性。 移动的部件如此之多：块化的几个阶段，相应的压缩速率，每个阶段的架构配置(outer vs 内部网络)，以及每个组件的超参数。
我们决定研究一个更狭义的设定：H-Nets如何与BPE模型进行比较，如果仅限于被解释为纯粹取代语言模型的象征部分的话？
为了做到这一点，我们把BPE梯级的每个径向一个字节一级H-Net转换。 一个1级的H-Net可以看作是一个具有学习的符号化的标准语言模型：内网络运行于块(它扮演着符号的作用)，外网络则发挥动态符号化的作用，将字节编码成块并解码回来。 每个径都非常小心地匹配：
- H-Net的内部网络
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Since Richard Sutton’s influential essay in 2019, the concept of the bitter lesson has guided much of modern progress in AI. This concept espouses general methods that leverage computation and learn with scale, as free of human-engineered priors as possible. Our interpretation is that, fundamentally, intelligence is about learning from raw data: autonomously finding patterns, building abstractions, and developing meaning from messy low-level inputs, ideally end-to-end and as data-efficiently as possible. As AI advances into complex domains such as science, robotics, and biology – where data is scarce and often difficult for humans to understand, let alone engineer for – internalizing this philosophy to develop more general learning algorithms is more important than ever.
Language models are perhaps AI’s biggest success story today, and largely embrace the bitter lesson. Their abilities are largely attributable to scaling a few simple algorithms such as the Transformer architecture, next-token prediction, reinforcement learning, and test-time computation. However, they still rely on hand-crafted encoding steps such as tokenizers, which severely hinder their ability to learn end-to-end from the underlying data. Towards unlocking another step change of scaling, we believe that developing byte-level language models is foundational for AI progress – not just for improving standalone LLMs, but also as a litmus test for general learning methods. If we can’t even learn language well from characters and bytes, what right do we have to make progress on the next frontier?
In 2025, we developed the concept of hierarchical networks (H-Nets), which not only addressed tokenization in language models, but took a step toward even more general-purpose architectures that could learn from raw data across different modalities. We’ve since significantly improved the technology of H-Nets and are excited to share their capabilities! In this series of two posts, we’ll dig into new results and discuss the implications.
In this first post of a two-part series, we’ll explore scaling laws for H-Nets and their qualitative behaviors.
- We produce a standard scaling ladder for tokenized Transformers, and show that carefully controlled 1-stage H-Nets provide meaningful compute efficiency gains (up to 3-5x) purely through learned tokenization.
- We show empirical results from scaling up a more flexible 2-stage H-Net, where it outperforms a matched open-source model with extreme efficiency improvements (10x less compute and data).
- We analyze the patterns that H-Nets learn, where they show a rich diversity of new behaviors previously unseen in language models, such as in-context tokenization.
- We discuss how H-Nets go beyond language modeling, with ongoing research results in biology and vision, showing promise as a general architecture across modalities.
In the next post, we’ll do a deeper exploration of a larger range of models to understand better why H-Nets can be so much more efficient for language modeling.
Altogether, our results provide evidence that H-Nets are a foundational architecture that truly aligns with the bitter lesson, learning meaning directly from raw data and significantly improving efficiency and scalability. This research agenda represents a major opportunity to advance how AI can learn from all data, and we couldn’t be more excited to push the frontier!
Hierarchical Networks (H-Nets)
We introduced hierarchical networks (paper, blog) last year as a new end-to-end architecture that models multiple levels of abstraction in data, showing promise in language modeling and beyond. In a nutshell:
The terminology was inspired by the existing concept of chunking in cognitive psychology, which is an essential part of human intelligence. We believe it is also necessary for the frontier of artificial intelligence.
In the context of language modeling, the idea is to build new learning primitives that allow models to recursively chunk low-level data into higher-level meaning: characters into words, words into phrases, phrases into concepts.
In the context of this post, for the most part, one can think of hierarchical networks as replacing tokenization. Instead of pre-processing data from bytes into tokens (fixed chunks of bytes), H-Nets perform the chunking dynamically directly inside the model. Furthermore, the signature characterization of H-Nets is that they hierarchically perform this chunking end-to-end. In contrast, standard tokenized language models and even other alternatives (some of which we’ll see in Post II) employ multiple stages of training, and the process of chunking / tokenizing the data is performed entirely offline.
Details: Tokenization and BPE
Language models don’t read text directly but instead take as input a sequence of tokens from a fixed vocabulary. A natural vocabulary is the 256 byte values, producing one token per UTF-8 byte. Byte-level sequences, however, are long and expensive for standard language models to process. Tokenization aims to alleviate this problem by compressing the byte sequence into frequently occurring chunks. The most common approach, Byte Pair Encoding (BPE), starts from the 256 bytes and repeatedly merges the most frequent pair of adjacent tokens on a fixed training corpus, until the vocabulary reaches a set size (e.g. 128k for Llama-3). On our data mix, each BPE token covers about 4.57 bytes, making sequences much shorter than at the byte level. However, this compression is fixed, and tokenization remains one of the last parts of the language-modeling pipeline that is designed by hand rather than learned.
We’ll see in this post series that even in the narrower context of replacing tokenization, the end-to-end property of H-Nets provides real performance gains. Beyond tokenization, H-Nets have broad utility as a foundational architecture:
- H-Nets can be nested recursively, allowing for multiple levels of abstraction to be built. The main result of this post is a larger two-stage hierarchical network matched against an open-source baseline.
- H-Nets can operate over any type of data, not just bytes, and not just discrete data. In contrast, tokenizing continuous data is a serious challenge. In the final part of this post, we’ll preview some future research applying H-Nets beyond text.
Figure 1 illustrates a 2-stage H-Net and defines the main components of the model: the encoder, main network, and decoder. We sometimes call the encoder/decoder together the outer networks and the main model the inner network.
Throughout this post, we show results with a much improved version of H-Nets, which we plan to release. We’ll comment more on differences from the earlier architecture later in this post.
Part 1: BPE and H-Net Scaling Ladders
As with many architecture papers, the original paper on hierarchical networks focused on tuning language models of particular fixed sizes and data horizons (e.g. ~1.3B-parameter models on 100B tokens from FineWeb-Edu have been very common). Although they looked very promising, a missing piece was the lack of compute-optimal scaling laws, now a standard way to measure pretraining innovations. For example, Kimi’s popular innovations on Kimi Delta Attention (KDA) and attention residuals measured compute multipliers of 1.16x and 1.25x, respectively.
In order to validate the gains of our improved H-Net architecture, we first set out to build a modern LLM scaling ladder, and then replaced its tokenization components with a carefully controlled hierarchical network.
- We set up a training mixture for comparing byte-level and BPE scaling laws on equal terms, carefully matching the processing and even re-training a BPE tokenizer for the baseline.
- We set up a scaling ladder for a baseline BPE Transformer language model according to standard practices, reproducing results from the literature.
- We then converted each rung to a byte-level H-Net as a drop-in replacement, reusing the BPE backbone architecture and hyperparameters.
This recipe directly showed stronger results on validation loss (1.3x), with gains increasing on subdomains such as code (2x), as well as improving out of distribution (2x on evaluation, and up to 5x on out-of-domain code).
A data-controlled comparison
Since byte-level and BPE models process the same data at different granularities, extra care is needed to keep the methodology controlled.
Data mixture: Our training dataset consisted of 70% English, 20% code, 5% math, and 5% multilingual, measured by the number of bytes.
Data processing: Every sample is processed as both bytes and BPE tokens, splitting at the same boundaries; both byte- and BPE- processed datasets therefore contain exactly the same text with aligned document boundaries.
Metric: We use the standard bits-per-byte (bpb) metric for measuring next-token prediction loss in a way normalized between different tokenizers.
In-corpus BPE tokenizer: LLMs generally use off-the-shelf BPE tokenizers; an advantage of H-Nets is that they learn their chunking directly on the training distribution. To be extra careful, we also trained a BPE tokenizer on the corpus. We used this for the BPE scaling laws below, which ended up resulting in about a 1.1x compute gain on both validation loss and downstream evals (so the H-Net would have an even greater compute multiplier against a standard off-the-shelf tokenizer!).
The BPE tokenizer’s vocab size is 128k, matching the Llama-3 tokenizer that we were working off of. The in-corpus tokenizer compressed the dataset slightly more, increasing the average BPE token length from 4.45 (Llama-3 tokenizer) to 4.57. Measured in tokens of the in-corpus BPE tokenizer, the training dataset comprised 64.7% English, 23.7% code, 6.1% math, and 5.5% multilingual.
Creating a robust BPE baseline
For our baseline, we fit a well-tuned scaling law for a standard BPE Transformer. To do this, we constructed a scaling ladder, following modern scaling recipes such as Kimi and MAI, fully defining a family of models by the number of layers L. We set the hidden dimension D to a multiple of 128 such that the aspect ratio D/L is approximately 75. For all models, we use multi-head attention with the number of heads equal to the number of layers and a fixed head dimension of 128. For the MLPs, we use SwiGLU layers with the intermediate dimension set to 3.6D, again rounding to the nearest multiple of 128.1
We then fit compute-optimal scaling laws with the standard iso-FLOP sweep methodology.
- All runs use the Muon optimizer and WSD learning rate schedule with fixed-fraction warmup and decay.
- We tuned batch size and learning rate on selected rungs and fitted hyperparameter scaling laws to extrapolate to larger ones.
- We counted FLOPs carefully in all models, including all sequence-mixing layers and the LM head. For softmax attention, we also count only causal FLOPs, as, due to the quadratic complexity, over-counting non-causal FLOPs penalizes byte-level models which process longer sequences.
An equivalent H-Net ladder
Because H-Nets over bytes are so different from standard tokenized models, it’s highly nontrivial to compare them directly. There are so many moving parts: how many stages of chunking, the respective compression rates, the architecture configurations of each stage (outer vs. inner networks), and the hyperparameters for each component.
We decided to investigate a narrower setting: how do H-Nets compare against BPE models, if restricted to being interpreted as purely replacing the tokenization portion of the language model?
To do this, we converted each rung of the BPE ladder to a byte-level 1-stage H-Net. A 1-stage H-Net can be viewed as a standard language model with learned tokenization: the inner network operates over chunks (which play the role of tokens) and the outer networks perform the role of dynamic tokenization, encoding bytes into chunks and decoding them back. Each rung is matched very carefully:
- The H-Net’s inner network
Auto-fetched and translated by AI Atlas. All rights belong to the original author.