后培训探索Demoscenes模式
关于演示文法和未完成代码的可能性
最近，我对世界的降级和大小编码感到着迷，最终我看到HellMood杰出的MS DOS演示“记忆”。 在阅读他的写作时，我开始对作者如何达到其视觉效果感兴趣。 我试图创造任何我自己，所以我希望从中吸取一些灵感。
我注意到这些效果可以追溯到早期的实验，包括替代执行和从之前的演示中借用的一些想法。 我开始想知道，同样的基本想法可能还采取了什么其他方向。 我决定从几个已有的演示开始，最简单的演示，并按个性修改它们，观察这些改变如何影响视觉感受。
我遇到了Dwitter.net，它就像Twitter 但140个角色JavaScript演示。 他们已经支持“remix ” —— 任何疏漏，以便我们能够在它们之上更上一层楼。 顺便说一句，网站最喜欢的Dweet与来自Memories的parlax检查板效果相同！
更深入地挖掘，我注意到一些重混体共享了它们源代码的很大一部分，只有几个字符不同，而另一些则通过完全不同的执行实现了类似的视觉效果。
下面是一个例子，我从d/367547追溯到d/36555，而d/36467又由此推导出。 灰色的下划线显示它们之间的共同点。 蓝色的显示在前两对夫妇之间是常见的，绿色的显示对下一对夫妇是常见的。
在语言学中，语言是有成效的，即有限的词汇和有限的一套语法规则允许大量不同的表达。 编程语言也共享此属性。 现在让我们来考虑上述的Dweets。 它们包含重叠的底线，但结果不同。 假设你拿一个140个字符程序，并删除它的最后30个字符。 即使假设的字母表只有64个字符，也有几乎1.5×10^54可能的完成字符串可以形成。 几乎所有都会在战术上无效或产生无趣的结果。 尽管如此，可能性的数量是天文的。
人类不可能详尽列举所有这些，产生和测试所有可能的绳索是愚蠢的。 然而，部分方案的延续并不同样合理。
由于一个部分程序，某些字符和表达方式比其他字符和表达方式更可能遵循。 这几乎就是自递性语言模型的功能。 我想知道我是否可以使用这样的模型来抽查未完成的Dweets的可信续作并可以想象每个程序可能采取的不同方向
然后，我可以外科切除下弦或混合与匹配与我自己的审美品味。
我对自己在开始一次演示后“完成”一个演示的能力感到相当沮丧
大约在同一时间，我看到Mithil Vakde的微博(你可能最近在HackerNews上见过他)
这给了我另一个实验的借口 -- 我可以把自己的美学品味注入我的自动完成模型吗？这真是太棒了。
我理解，没有人能写一个功能决定一个演示是否美丽，但人们可以通过比较例子表达他们的喜好，对不对？
考虑到这一点，我决定让我的GitHub赞助商出钱租一个GPU，然后在训练后再做我自己的模特。
第一个障碍是找到合适的数据集。 我最初尝试用Claude Sonnet 5.5来生成合成的Dweets，但发现它们在美学上并不令人愉快。 幸运的是，Dwitter.net提供了一个API，我曾经建造一个25k Dweets的微小数据集，与~1.3k作为验证集一起用于培训。
我用Qwen2.5-Code-7B作为基模。 它很难完成我测试的大部分Dweets，这是可以理解的，考虑到他们的语法有多奇特。 Dweets广泛使用代码高尔夫技巧和数学短手，与代码相比，这与一般通用编码模型一般遇到的代码相当不寻常。
我决定继续尝试对血清进行预训，为了看它是否有效，我测量了培训前后个人代币的占比。 超常值仅仅是一个模型给一个符号分配的负对数概率。 信物的可能性越小，对模型来说就越令人惊讶。
在下图中，较暗的红色表示更大的外衣。 在CPT之后，很多代码变得明显不足为奇(特别是不熟悉的编程规范).
perplexity从28.63降至8.57，这意味着这个模型在预测这些微小的怪异程序的结构方面已经大有改进。
尽管产生的方案更加一致，但它们仍然缺乏某种“图”。 幸运的是，Dwitter对每个Dweet有“惊人的”分数。 我决定用它来构建课程
我以“惊人”的分数排名前5名，最初，每个杜威都有同样的可能性被采样。 超过400个训练步骤，我逐渐增加了分配给排名较高的课程的权重，直到前1000名占采样概率的约67%.
这真是一次突破！所生成的续作在视觉上变得更加有趣。
有趣的是，象征性可能性方面的改进甚至扩大到在课程培训期间从未抽样的Dweets。 Perplexity从7.42跌至5.65，虽然收益分布在惊人的队伍中相当均匀。
超过7000套课程，课程培训提高了所有优秀阶层的象征性可能性，包括培训期间从未抽样过的方案。 对于最初发现困难的方案来说，收益特别明显。
我现在有一个模型可以产生相当漂亮的续作。 我不知道我是否可以通过纳入我自己的美学偏好来进一步推动这一点。
9月中旬，我看到苏里亚关于“训练人工智能用密码绘画”的文章，他们用RL用p5js制作水彩画。 他们先用不同的奖励功能进行实验，然后用对偶的比对来对付一批手额画。 我想起了这个想法，并想到尝试与Demoscener类似的东西(反复称它为“模型”并不好)。
我选择用GRPO来做RLHF，由我自己担任奖励职能。
对于每一个未完成的Dweet，我抽样了一组8个完成，把它们放在一起，并选了一个我发现最美学满意的。 如果没人向我求情，我跳过了队伍。 我用这些偏好来更新Demoscener的政策，我手动检查了4万部动画
我开始欣赏VLM和HPSv3的存在。 可惜，为时已晚。
经过几轮训练后，我想看看我的品味究竟在德摩斯切纳身上消退了多少。 我邀请了一个相当不愿意的朋友从几个检查站(我随机挑选了4个)判断400套演示，却不知道是哪个模式产生的。
结果显示4号检查站是最有利的所以我用它来推断我的Mac
现在到了边上探索的尽头，我建立了一个小界面，在那里，我可以生成几个续集从同一个前缀，使它们并肩并肩，并且从任何我的眼睛。 然后，我可以调整代码，产生更多的续作，并不断探索。
每个分支都代表着可能的延续，我可以用手来对比，用手来对比，并用作进一步探索的起点。
能够漫游于这些可能性中，这是一件非常可爱的事。 我想我会花更多的时间这样做。
您可以在HuggingFace上找到Demoscener-7B和数据集。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Post training a model to explore Demoscenes
On the grammar of demoscenes and the possibilities of unfinished code
Lately, I have been fascinated by the world of demoscening and sizecoding and eventually I came across HellMood’s remarkable MS DOS demo “Memories”. Whilst reading through his writeup, I became interested in how the author had arrived at some of its visual effects. I was struggling to create any of my own, so I hoped to squeeze out a little bit of inspiration out of that.
I noticed that these effects could be traced back to earlier experiments consisting of alternative implementations and several ideas borrowed from previous demos. I began wondering how many other directions the same underlying ideas might have taken. I decided to start with a few existing demos, the simplest ones of all, and modify them character by character to observe how those changes can affect the visual feel.
I came across Dwitter.net, which is like Twitter but for 140 character JavaScript demos. They already supported “remix”-ing any dweet so that we can build on top of them. Co incidentally, the most liked Dweet on the site shared the same effect as the parallax checkerboard effect from Memories!
Digging a little deeper, I noticed some remixes shared substantial portions of their source code, differing by only a few characters, whilst others achieved similar visual effects through entirely different implementations.
Here is an example, I traced back d/36555 from d/367547 which in turn is derived from d/36467. The gray underlined substrings show what is common across all of them. Blue shows what’s common in between the first two, and green shows for the next couple.
In linguistics, language is productive i.e. a finite vocabulary and a finite set of grammatical rules permit an enormous number of different expressions. Programming languages share this property too. Now let’s consider the aforementioned Dweets. They contain overlapping substrings, but their rendered results differ. Suppose you take a 140 character program, and remove its final 30 characters. Even with a hypothetical alphabet of just 64 character, there are almost 1.5 x 10^54 possible completion strings that can be formed. Almost all will be syntactically invalid or produce uninteresting results. Nevertheless, the number of possibilities is astronomical.
It’s not humanly possible to enumerate all of them exhaustively and it would be foolish to to generate and test all possible strings. However, the continuations of a partial program are not equally plausible.
Given a partial program, certain characters and expressions are considerably more likely to follow than others. This is pretty much how autoregressive language models work. I wondered whether I could use such a model to sample plausible continuations of unfinished Dweets and visualize the different directions each program could have taken.
And then, I could surgically delete substrings or mix-and-match them to my own aesthetic taste.
I had also grown rather frustrated with my own ability to “finish” a demo after starting one, so having an autocomplete that understood the peculiar language of Dweeting rather looked like an appealing prospect to me.
Around the same time, I came across Mithil Vakde’s tweet (you might have seen him on HackerNews recently).
That gave me another excuse to experiment - could I inject my own aesthetic taste as a prior into my autocomplete model? That’d be really amazing.
I understand that no one can write a function that determines whether a demo is beautiful, but people can express their preferences by comparing examples, right?
With that in mind, I decided to put my GitHub Sponsors payout towards renting a GPU and post training a model of my own.
The first hurdle was finding a suitable dataset. I initially experimented with generating synthetic Dweets using Claude Sonnet 5.5 but found them not at all aesthetically pleasant. Fortunately, Dwitter.net provides an API, which I used to build a modest dataset of 25k Dweets to be used for training along with ~1.3k more as a validation set.
I used Qwen2.5-Code-7B as the base model. It struggled to complete most of the Dweets I tested it on, which was understandable given how peculiar their syntax is. Dweets make extensive use of code golfing tricks and mathematical shorthand, which are rather unusual compared with the code a general purpose coding model typically encounters.
I decided to try continued pretraining (CPT) on the corpus, hoping to shift the model’s probability distribution towards the pecular language of dweeting. To see whether it had worked, I measured the surprisal of individual tokens before and after training. Surprisal is simply the negative log probability a model assigns to a token given everything preceding it. The less likely a token is, the more surprising it is to the model.
In the figure below, darker red indicates greater surprisal. After CPT, much of the code becomes noticeably less surprising (particularly the unfamiliar programming idioms).
Across 30 held out dweets, perplexity fell from 28.63 to 8.57, which means that the model had become substantially better at predicting the structure of these tiny weird programs.
Although the generated programs were more consistent, they still lacked a certain “charm”. Fortunately, Dwitter has an “awesomeness” score for every Dweet. I decided to use it to construct a curriculum.
I ranked the top 5k dweets by their “awesomeness” scores and constructed a curriculum that gradually favoured the more appreciated programs. Initially, every dweet had an equal probability of being sampled. Over 400 training steps, I progressively increased the weight assigned to higher ranked programs, until the top 1000 accounted for roughly 67% of the sampling probability.
This turned out to be a breakthrough! The generated continuations became much more visually interesting.
Interestingly, the improvements in token likelihood extended even to Dweets that had never been sampled during curriculum training. Perplexity fell from 7.42 to 5.65, although the gains were spread fairly evenly across awesomeness ranks.
Across 7000 Dweets, curriculum training improved token likelihood across all awesomeness ranks, including programs never sampled during training. The gains were particularly pronounced for programs the original model found difficult.
I now had a model that could produce rather beautiful continuations. I wondered whether I could take this further by incorporating my own aesthetic preferences.
In mid-September, I came across Surya’s article on Training AI to Paint with Code, where they had used RL to produce watercolor paintings using p5js. They experimented with different reward functions before settling on pairwise comparsions against a collection of hand rated paintings. I remembered that idea and thought of trying something similar with Demoscener (it doesn’t feel alright to call it “the model” repeatedly).
I opted to do RLHF using GRPO, with myself acting as the reward function.
For each unfinished Dweet, I sampled a group of eight completions, rendered them side by side and selected the one I found most aesthetically pleasing. If none appealed to me, I skipped the group. I used these preferences to update Demoscener’s policy, then sampled fresh continuations and repeated the process over roughly 9–10 rounds of 500 groups each. Altogether, I manually inspected around 40,000 animations.
Somewhere around the 10,000th animation, I began to appreciate the existence of VLMs and HPSv3. Sadly, it was too late.
After several rounds of training, I wanted to see how much of my taste had actually rubbed off on Demoscener. I enlisted a rather unwilling friend to judge 400 sets of demos from a few of the checkpoints (I picked 4 randomly), without knowing which model had produced which.
The results showed that checkpoint 4 was the most favorable, so I used that to run inference on my Mac.
Now coming to the end of the side quest, I built a little interface where I could generate several continuations from the same prefix, render them side by side and branch off from whichever one caught my eye. I could then tweak the code, generate more continuations, and keep exploring.
Each branch represents a possible continuation, which I could compare visually, tweak by hand, and use as the starting point for further exploration.
There is something quite lovely about being able to wander through these possibilities. I think I'll be spending a little more time doing just that.
You can find Demoscener-7B and the dataset on HuggingFace.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.