我们——内森和弗洛里安——想在AI讨论一个繁忙的星期里所有混乱的权衡，并询问所有紧迫的问题，例如：
- 为什么Kimi K3.1 Pro与如此不可思议的基准测试有混合反应？它会获得通过吗？
- 开放量模型和网络安全的风险我们是不是错了？
- 为什么网络专家讨论最近的事件与AI专家有如此不同？
- 接下来几个月会怎样？什么会改变我们的想法？
我们将多写一些关于这个主题的报导，但希望你们能回到幕后，因为我们希望能够很快地更多地了解过去几年来网络安全产业和趋势，这些年来，这些都一直领导着我们今天。 在播客的最后，我们用一些更轻松的话题，从Muse -maxing到兴奋仍然雇用初级技术人才。 请给我们发送问题和话题，因为我们计划记录更多的这些播客前进。
听听Apple播客，Spotify，无论你在哪里得到你的播客。 其他互联播客请到这里 YouTube (英语).
章节
- 00: 00 Kimi K3.1 - 30: 00 Kimi K3.1 - 30: 00 Kimi K3.1 反应
- 8:15 Kimi的开放式网络原则
-11点23分你可以入侵所有时代
- 15:54 PHP2Shell、LiteLLM和新的互联网
- 24:00 公布API、保障措施和可信任的访问
- 31:41 Anthropic不雇用初级雇员
- 36:16 弗洛里安吸毒成瘾
- 45:30 总结
我们讨论的链接
- 我和迪恩·鲍尔的线索关于边疆实验室的“哭狼男孩”的故事
- Kimi K3.1号公告博客
- Kimi K3.1人工分析情报指数
- Kimi K3.1人工分析防御网络基准测试
- 方案组
- 开放量级的神话
- PHP“尚未造成灾难”更新
- Abliteration.ai服务于盲目型号API
- 通过API提供盲文DeepSeek
- Joshua Saxe关于AI和网络安全的Substack
- Azeem Azhar,“互联网的武汉时刻”
Anthropic,“Anthropic的就业情况如何变化”
- LiteLLM安全事件报告
语录
00:00 内森·兰伯特：我们与另一个互联播客重逢。 我们和Florian在一起，Eval的家伙，最近搬到了SF。 弗洛里安，是Kimi K3.1 Pro，名字是一个口吻，要达到基准测试的分数，比如在人工分析上，还是在哪里降落？
00:00:17 弗洛里安·布兰德：在过去，Kimi模型总是不辜负人们的期望，但这种发行与其他版本相比有些怪异。 我想这也是部分原因因为他们越来越专业，之前的模型下降有点出乎意料。 这一次也引起了一些批评，他们发布是一场更大的社交媒体运动，据称也涉及支付人们在不透露的情况下张贴模型。
内森·兰伯特(Nathan Lambert)：我认为很多公司都这样做，这是坏的形式。 任何公司都不应该这样做。 Kimi的媒体指示截图显示，称“不要说这是有报酬的晋升”。 据我所知之后没人能证实
00:01:15 弗洛里安·布兰德(Florian Brand)：是的，现在有一些这些型号的发布，包括美国公司，这些型号都非常明显地被天体摧毁了。 我们已看到发布，有些人透露这是付费广告，但总的来说，我们看到了发布模型随着时间推移的专业化，而且相当快。 一年前，每个人都放弃了模型，然后在之后补上vLLM，所以实际上没有人可以暂时运行任何开放模型。 现在，你得到了所有推论框架在第一天，你得到了所有的推论提供者在第一天。 或者，这也是我们看到的越来越多，模型创造者在释放重量之前有几个星期的独家访问。 K3.1 Pro也会出现这种情况。 在他们释放模型重量之前
00:02:38 内森·兰伯特：他们称重量将在11月4日出炉。 还有头条。.. 我不知道我是否能分享屏幕。 这台电脑上没有这个装置。 但标题是人工分析号，如果你看看Kimi K3.1的博客，他们的头条数字非常强，之后，他们也有一些数字，似乎更乱。 我想是Bench科学终端落后于GPT-6 Astra20分我想还有其他几个，比如Bench计划你知道终极奔驰科学与程序奔驰吗？人们是否认为基准测试对前沿模型很重要？或者你如何解释在人工分析中模型是最高的，什么，三，四？这是两个最新的Claude 5.5型号，Muse Wonder Full，然后是Kimi K3.1，看起来很疯狂。 它只是看起来像完全疯狂的分数，我认为每个人都仍然惊讶它。 但接下来你用这些细节来评估一个模型。 在推特上贴出了一条线索，也有许多人 “这似乎不妙，”然后有些人说，“是的，这很好。” 我不知道，相对K3来说，我认为这是相当一致的，这感觉就像一个大模型，后训练似乎有点不足。 也许这只是一点点RL炸。 我们看看降落的地方。
00:04:26 弗洛里安·布兰德(Florian Brand)：对于基准测试来说，其中一件事情是，现在人工分析公司使用的基准测试，他们可能会在发布这个播客后很快发布他们的新版本。
00:04:43 内森·兰伯特：我们不能告诉你确切的日期，但我们都知道。
00:04:48 弗洛里安·布兰德：一般来说，人工分析基准测试在时间范围上相当短，而像"程序本奇"(Programm Bench)这样的东西确实很长，也非常具有探索性。 其他的基准测试正在告诉模型，“好吧，这就是目标，我要你达到这个目标 ”，而程序Bench更像是，“好吧，这就是我想要的，但你需要探索很多东西。 ” 我认为这可能是我们发现一些分歧的原因：他们的RL训练仍然更倾向于模式训练的剥削性方面，而不是探索性方面，在你真正知道自己在做什么之前，你需要找到解决办法。
00:05:41 内森·兰伯特(Nathan Lambert)：但是如果你在使用一个三兆个参数模型，我觉得你希望这个模型在探索时相当有力。 因为如果你想要使用一个模型来完成相当已知的编码任务，比如使用DeepSeek-V4.1-Flash-1017，而后者在这些分数上没有落后那么远，那么它就会更快。 与实际符号数字相比，每项任务的成本可能并不大不相同。 所以我认为这对Kimi发行的批评相当合理：一个大模型必须。 因为会更慢，我仍然认为Fable真的很好，当我检查幻灯片时，我又回到了它，有时候，我感觉它让我有点好，在那里Opus真的，真的磨。 我认为开放量的模型在这种磨损角度上更好，因为你可以让它们更快，更便宜。
00:06:44 内森·兰伯特(Nathan Lambert)：似乎Kimi正准备用他们的命名变化来进行这个Flash式的发行。 以前的Kimi型号只是K2,K2.6,K3，现在这是K3.1 Pro. 公司在推特上的许多预测，投资方面的兴趣是，我们不是推论专家，但为模型服务似乎需要投入更多的资源，我认为你需要NVL72架
00:07:21 弗洛里安·布兰德(Florian Brand)：为了适应模型，你需要B300s，每个模型复制件一个节点，这有点尴尬，与其他模型相比，你甚至可以适应H100s，或者H200s，更方便使用。 与每个节点有一个模型复制件相比，或者利用现有的节点来适应全重相比，拆分模型总是尴尬的。 因此，在这方面，3万亿美元刚刚用完，或者在你能合理容纳的范围之内，但是与所有的Flash模型和所有较小的模型相比，它有点烦人，在那里你可以使用这么多现有的GPU，以至于人们拥有。
00:08:15 内森·兰伯特(Nathan Lambert)：你是否认为Kimi的射击本身在脚部稍稍减慢了自己的网络能力？在博客文章中，我可以读到其中的一些内容，“开放加权原则。 这些评估为我们开放量释放决定提供了框架。 首先，我们的目标是不释放开放权重，这将扩大网络能力的绝对前沿。 K3.1 Pro的进攻性网络能力，从设计上讲，低于我们的管道所能生产的能力，并且仍然落后于前沿专利模式。 他们继续说。.. 我不知道中训部分是否在此，但基本上他们操纵中训数据来降低网络能力，你觉得这能反映他们拥有的其他基准测试，以及一些这种长视距的弱点吗？或者你认为人们可以把这事分开？
00:09:08 弗洛里安·布兰德(Florian Brand)：我认为人们可以很好地把这些东西分开。 我和其他人谈过了，我觉得释放的风格真的很好。 您想要非常广泛的访问权限让每个人找到所有的bug，并补补所有的bug，这对世界来说是净阳性。 Kimi表示他们仍然为防守方释放有能力的模型，我从网络安全专业人士那里听到的是，利用和破坏整个组织是更困难的，因为你需要非常长的视野能力，你需要坚持你进入的所有阶段。 关闭的模型，特别是GPT-5.6 Sol，这个模型经常被命名于此，我认为这很重要。 我们希望我们所有的代码都安全，这应该很快发生。 但为此，你不需要很长的视野。 你不需要所有CTF式的挑战。 因此，把这些从训练组合中扔出去是一个合理的决定，并帮助他们定位对于恐惧者，曾经一个开放的模型具有神话层面的能力 e在某些方面，是发布，世界将结束。 我们至今还没有看到，因此，开放的模型比封闭的模型获得更多的审查，它们把自己定位为“我们是负责任的行为者”，这是一个明智的举动。
00:11:23 内森·兰伯特(Nathan Lambert)：好的，我们可以转换成这个部分的“你可以黑入一切”。 我和弗洛里安都不是网络专家，但值得讨论，因为我认为我读到的网络专家从生态系统的角度谈论这个问题。 ChatGPT出世前，所有测量数据，如脆弱性检测，开发时间，都在这些指数上，但当你来到我们推特一角的AI人民，人们喜欢辩论开放与封闭模型，混乱的中间是，日出计划和神话之类的东西对更广泛的社会有何影响？我觉得自己在试图通过这个过程，页：1
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like:
- Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted?
- Were we wrong about the risks of open-weight models and cybersecurity?
- Why do cyber experts discuss recent events so differently than AI experts?
- What comes next in the few months? What will change our minds?
We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward.
Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓
Chapters
- 00:00 Kimi K3.1 Reaction
- 08:15 Kimi’s open-weight cyber principles
- 11:23 You can hack everything era
- 15:54 PHP2Shell, LiteLLM, and the new internet
- 24:00 Abliterated APIs, safeguards, and trusted access
- 31:41 Anthropic not hiring juniors employees
- 36:16 Florian’s Muse addiction
- 45:30 Wrap-up
Links we discuss
- My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs
- Kimi K3.1 Pro announcement blog
- Kimi K3.1 Pro on Artificial Analysis Intelligence Index
- Kimi K3.1 on Artificial Analysis defensive cyber benchmarks
- Program Bench
- Open-weight Mythos would’ve been fine take
- PHP “hasn’t caused a disaster yet” update
- Abliteration.ai serving abliterated models API
- Morph serving abliterated DeepSeek via API
- Joshua Saxe’s Substack on AI and cybersecurity
- Azeem Azhar, “The internet’s Wuhan moment”
- Anthropic, “How employment is changing at Anthropic”
- LiteLLM security incident report
Transcript
00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land?
00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing.
00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that.
00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights.
00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land.
00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast.
00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know.
00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing.
00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017, which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper.
00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well.
00:07:21 Florian Brand: To fit the model, you need B300s, one node per model replica, which is kind of awkward, versus other models which you can even fit on H100s, or H200s rather, which are more readily available. Splitting models is always awkward compared to having one model replica per node, or using existing nodes to fit the full weights. So in that regard, three trillion is just out of, or just in the range of what you can host somewhat reasonably, but it is kind of annoying compared to all the Flash models and all the smaller models, where you can use so many of the existing GPUs that people have.
00:08:15 Nathan Lambert: Do you think Kimi’s shooting itself in the foot a little bit by saying it’s slowing its own cyber capabilities? In the blog post, I can read some of this, because it’s really good for people to say. “Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro’s offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models.” And they go on to say... I don’t know if the mid-training part is in this, but essentially they manipulated the mid-training data to down-weight cyber capabilities and see that that can last into the post-training. Do you think this would reflect upon the other benchmarks that they have, and maybe some of this long-horizon weakness? Or do you think that people could actually separate this?
00:09:08 Florian Brand: I think people can very well separate these things. I talked to a bunch of other people, and I think that style of release is really good. You want very broad access for everyone to find all the bugs and to patch all the bugs, which is net positive for the world. And Kimi saying they still release the capable model for the defensive side, but they don’t want to advance the offensive side. What I’ve heard from cybersecurity professionals is that exploiting and being able to go through or wreck a whole organization is way harder, because you need very long-horizon capabilities, and you need to persist through all the stages that you penetrate. That’s where closed models, especially GPT-5.6 Sol, a model that’s named often in this regard, are just way, way better than open models. And I think that is important. We want all our code to be secure, and that should happen pretty fast. But for this, you don’t need long horizon. You don’t need all the CTF-style challenges. So throwing these out of the training mix is a reasonable decision, and it helps them position against the fearmongering that once an open model with Mythos-level capabilities, which we arguably have in some regards, is released, the world will end. Which we haven’t seen so far, and I don’t think will happen in the near future. So open models get way more scrutiny than closed models in this regard, and them positioning themselves as, “We are the responsible actors,” is a smart move.
00:11:23 Nathan Lambert: Okay, we can transition into the “you can hack everything” part of this segment. There’s a lot going on here, and Florian and I are both not cyber experts, but it is worth discussing, because I’m kind of of the opinion that the cyber experts that I read talk about this from an ecosystem perspective. All measurements like vulnerability detection, time to exploitation, have been on these exponentials before ChatGPT came out, and there wasn’t a step change as AI models came out, and they focus on this ecosystem perspective. But when you come to the AI people, which is our corner of Twitter, people love to debate open versus closed models and Mythos and things like this. And then the messy middle is, what is the effect of the Daybreak programs and Mythos and things like this on broader society? I feel like I’m trying to muddle through this without getting too stuck into any one point of view. B
Auto-fetched and translated by AI Atlas. All rights belong to the original author.