Matrix 首页推荐
Matrix 是少数派的写作社区，我们主张分享真实的产品体验，有实用价值的经验与思考。我们会不定期挑选 Matrix 最优质的文章，展示来自用户的最真实的体验和观点。
文章代表作者个人观点，少数派仅对标题和排版略作修改。
AI 辅助创作声明（作者注）：
本文记录的固件代码由 Anthropic 的 Claude Code（Claude 模型）编写，作者负责提出需求、在开发板上安装验证和做决定。文章初稿由 Claude 根据开发过程中的会话记录和交接文档整理成文，作者逐段改写并核对了其中的事实、数据和引用来源；文中「顺手想到」和「放大看」两类段落的观点为作者本人所写。示意图由 Claude 按作者要求绘制，实拍照片和截图为作者本人拍摄、截取。作者已对全文做了核查和修改，文责自负。
如果有人问我，我花时间，花 Token 来做这么一个没有用的东西有价值吗？我的回答是肯定的，最大的意义其实就在于，证明 Vibe Coding 在硬件领域正在快速成长，为什么呢？因为数字世界已经到了一定的瓶颈了，是需要物理世界来反哺数字世界。如果没有 AI Agent 发展到今天这个程度，我是完全不敢想象自己要能做这件事要花多少时间去学习硬件、嵌入式、软件、通信等知识，要去查多少资料，这都需要时间的积累，但这些积累在数字世界基本都有了，硬件的东西像乐鑫这样子专业的公司做了就行了。
先把几个词说清楚，方便没碰过硬件的读者往下读。
开发板是芯片厂商给工程师验证芯片、写固件用的一块电路板，把芯片、供电、USB 口和常用外设（屏幕、麦克风、喇叭、传感器）焊在一起，拿到手就能通电写程序，它不是面向消费者的成品。ESP32 是乐鑫的一个芯片系列，自带 Wi-Fi 和蓝牙，便宜、省电、资料多，创客圈和 AI 硬件玩具里用得最多的就是它。固件就是烧进这块板子里的程序，决定了板子开机后干什么。
模块化开发板的意思是主板只带最基本的东西，摄像头、人体感应、灯这些功能做成一块块小板，按需插在主板两侧的扩展槽上，想要什么能力就插什么模块。ESP-Mosaico 是乐鑫 2026 年 8 月发的一块这样的板子，特别之处是它从一开始就按「让 AI 编程工具来写固件」设计：板子里固定住一套叫 Vibe Mode 的底层固件，负责装程序、恢复和看日志，用户不用碰烧录工具；上手不是看引脚图，而是把一句提示词交给 Claude Code 这类编程 Agent。
乐鑫 8月19日 发的 ESP-Mosaico 是一块专门给 Coding Agent 用的模块化开发板。其实我知道它很粗糙，看到用户反馈说还是 3D 打印的部件，但我还是想看看现在 AI 硬件领域生态做的最好的 ESP32 系列到底是怎么做的，尤其是它现在的 Vibe Code 模式。它的上手方式和以前的开发板不一样：不给你看引脚图，先给你一句提示词，让你丢给 AI 编程工具。那句话是这样的：
Download the code from this repository: https://github.com/esp-mosaico/esp-mosaico-vibe
Set up the environment and implement a Snake game.
第一个作品是贪吃蛇。我照做了，用 Claude Code 跑完这一句，板子上就能玩了。10月2日 Meta 开源了 Muse Gadgets 的设备 SDK，官方推荐的硬件也是 ESP32，于是接着往下做：当天这块板变成了一台 Muse 小设备，后面持续迭代了一下就能用中文对话、把回复念出来、按 Muse 的要求拍一张照片发回去、我走近的时候亮灯打招呼。
整个过程是我和 Claude Code 一起做的，前后开了四个会话，其中两个在同一块板子上并行干活。这篇按发生的顺序写，分五段，每段后面放一点做的时候顺手想到的，最后放大看。代码在这两个仓库：
- rowline/esp-mosaico-vibe 的 muse-gadget 分支：贪吃蛇、Muse 的包装工程、蓝牙诊断固件、交接记录
- rowline/muse-gadget-sdk 的 esp-mosaico 分支：对 Meta SDK 的 9 个提交
一、先玩起来
第一步：按乐鑫的剧本做贪吃蛇
ESP-Mosaico 的主控是 ESP32-S31，480 方屏、带喇叭麦克风、两侧能插扩展模块。它和普通开发板最大的区别是 Flash 前 2 MB 住着一套固定不动的固件，叫 Vibe Mode：装应用、恢复、看日志、截屏都走它，用 mosaico.py 一条命令就能装机，不用碰 esptool。开机按住橙色的 AI 键就进 Vibe Mode，所以应用写坏了也不会变砖。
那句提示词丢进去之后，AI 先花了一个晚上搭环境，坑都在环境这一层，不在游戏：
- ESP-IDF 要钉到一个具体的提交。 工作区要求 master 上的 7b9cc1ac，不是最新版，装好后报的版本号是 6.2.0。
- 国内组件镜像缺版本。 从这个网络访问乐鑫的组件服务会被转到国内镜像，镜像上没有 esp-gsp 1.5.1。最后从官方 API 下载、核对哈希，搭了一个本地组件库。
- GSP 场景编译器官方没发。 组件要求的 0.6.1 不存在，用了工作区 CI 同款的签名版 0.5.0。
以上都是 Claude Code 自己执行完成的，我什么也没有做。游戏本身用 Raylib Lite 写，规则和画法分两个文件，同一份 C 代码在浏览器模拟器和板子上跑。20×18 的格子，吃一个苹果长一节、加一分，速度从每秒 6 格涨到 12.6 格，撞墙或撞自己结束，填满算赢。触屏上滑动转向，点一下开始或朝那边转，点分数栏暂停。8 个回放测试，每个场景存成回放再跑一遍，最终状态的哈希必须一致。
模拟器里顺手抓到一个模板的 bug：空白游戏模板只听触摸点 0，浏览器把鼠标报成点 1，所以点击没反应。改成跟着先按下的那根手指或鼠标走。
装上板子是第二天的事。板子第一次插上 Mac 认不到，换了根能传数据的线才出现。第一次用的板子先跑 recover 写 Vibe Mode 基础固件，再用 system-update 装贪吃蛇。装完核对三件事：设备 ID 不变、启动 ID 是新的、设备报告健康。板上实测 33.3 fps，没掉帧，触摸正常。
贪吃蛇做完，环境也就踩通了。接下来的 Muse 全部建立在这上面。
顺手想到：为什么又是 ESP32
顺着这个看几个事情，自从 Anthropic 开源了 Claude Desktop Buddy、把自己的螃蟹装进 M5StickC 这块小板子里之后，多少人开始效仿，又有多少人去 Vibe Coding 自己的电子宠物，给个屏幕让它呈现出来，提供情绪价值，官方推荐的也是 ESP32 这个生态的东西，到这次 Muse 推荐的也是 ESP32 生态的，因为它足够简单，成长这么多年 AI 能在互联网上找到足够多的知识来帮助用户去实现他个人定制化的需求。
二、把板子变成 Muse
Meta 开源了 Muse
我买这块板子的时候预判到后面类似 GrokBot 这类产品肯定会接硬件设备的，方便我第一时间去玩，没想到 Muse 先干了，以开源项目的方式先干了。我之前也用手机模拟硬件设备接了 GrokBot。
Muse 是 Meta 做的个人 AI 助手。Muse Gadgets 开源的只有设备端：一套 ESP32 固件和一套树莓派用的 Linux SDK，Apache 2.0。助手本身不开源，设备要先去 gadgets.muse.ai 领一个 SDK token，再在 Muse App 里打开开发者模式配对。官方支持的板子里，带完整界面的是 Waveshare 1.75 寸圆屏 AMOLED 这一类。
初次使用，我发现 Muse 比 GrokBot 反应是要快很多的，应该是意图识别做的更好，GrokBot 说什么都要比较长的反应时间，GrokBot 本身没有做任何设备交互的工作，界面显示的内容和设备播报的内容不会完全一致，Muse 就好很多。
Vibe 价值就在于自由适配
把 Muse 的固件直接编译给这块板，有三件事过不去：
- 开发工具版本对不上。 Muse 固件只认 ESP-IDF 6.0.1，而 6.0.1 不支持 ESP32-S31。这块板的工作区把 ESP-IDF 钉在 6.2 的一个提交上。
- 官方刷机方式会冲掉 Vibe Mode。 Muse 用 idf.py flash 连引导程序、分区表一起写，前 2 MB 会被覆盖。
- 板子连不上 Meta 的服务器。 api.muse.ai 和 hatch.metaaivm.com 在国内直连全部超时。
如果是我本人没有 Claude 的情况下，我都看不出来这三个问题，现在有了，不妨碍我继续玩下去。我的决定是：Vibe Mode 不动，把 Muse 做成一个标准的 ESP-Mosaico 应用，用 Vibe Mode 装机。后面的 Flash 的分法是 Claude 做的，变成下面这样。
Muse 的配对信息、Wi-Fi 密码、设备凭证都在那块 64 KB 的 nvs 里，换固件不会清，所以后面十几次装机都没有重新配过对。这个点太重要了，如果不是做这个项目，我完全不知道还可以这样子操作。
代码分三层。Meta 的 SDK 只开了几个小口，板子相关的全部放在包装工程里，SDK 的 esp-mosaico 分支加起来 9 个提交，每一个默认都关着，其他板子的行为不变。
SDK 那边最早的两处改动是必须的：
- 启动钩子。 Muse 的 app_main 在起 Wi-Fi、蓝牙和界面之前，调一个弱符号 muse_gadget_platform_start()。包装工程给它一个强定义，在里面启动 ESP-Iris 设备管理服务、接受这次的固件。后来的摄像头、语音、问候也都从这里注册进去。
- 关掉 Muse 自带的回滚。 上游固件装好后 5 分钟内连不上 Muse 云端就自动回滚。这块板没配对之前肯定连不上，会被打回去。加了一个开关把它关掉，固件的验收交给 Vibe Mode。
先连上网
路由器是普通家用的，改不了网关。最后的做法是把 Mac Studio 当旁路由，只让板子这一台走它，家里别的设备不动。
固件里加了两个配置项，默认为空：
- 外网下一跳。 板子照常从路由器拿地址、续约，只是发往外网的包交给指定的主机。用的是 lwIP 自带的转发钩子，DHCP 那一套完全不动。
- DNS。 网关生效时改用局域网外的 DNS。路由器续约会把 DNS 改回去，所以每 30 秒检查一次，被改回就重新设上。
Mac 这边本来以为要给 Clash 开局域网连接和 DNS 端口，查了才发现不用：它的 TUN 模式已经接管了全部公网流量，dns-hijack: any:53 会把经过它的 DNS 查询一并答掉。只需要打开 IP 转发：
sudo sysctl -w net.inet.ip.forwarding
=
1
这条重启会失效。
配对卡住的两个假故障
无屏版 Muse 装上板子之后，日志里蓝牙一直在广播 MuseGadget-F46F0A，等人来配对。卡了一晚上，两个问题最后都证明和板子无关。
手机搜不到蓝牙。 iPhone 系统设置里的蓝牙列表只显示耳机、键盘这类设备，低功耗蓝牙的广播本来就不出现在那里。但 ESP-Mosaico 官方 BSP 的功能表里没有蓝牙，示例里一个用蓝牙的都没有，所以这块板的蓝牙到底能不能用，之前没人验证过。为了把 Muse 的因素排除掉，做了一个只做蓝牙收发的诊断固件 ble_diag：以 MosaicoBLE-DIAG 的名字广播，每 15 秒用板子自己的蓝牙扫 5 秒。结果每轮能听到 34 到 37 个设备，最强的信号 −37 dBm；手机装 nRF Connect 也看到了它。板子收发都正常。这个固件留在仓库里，以后怀疑蓝牙随时装上测一次。
Muse App 开了开发者模式，没有添加设备的入口。 查了套餐、地区、iOS 测试版，都没有定论。第二天它自己出现了，是 App 那边的问题。下面是正常时候的样子，右上角的加号就是入口，开发者模式的说明写得很直白：允许配对没经 Meta 验证的社区设备，这些设备对你的助手有完整权限。
配对本身很顺：App 里选设备，日志提示确认时短按 AI 键，填 Wi-Fi 密码，板子重启后连上 Muse。
屏幕
带屏幕的 Muse 界面是 LVGL 画的，上游用的板子是圆屏。移植有两个选择：照着 Muse 已支持的 Waveshare 板写一套驱动，或者用 ESP-Mosaico 官方的 BSP。选了 BSP：它认识这块板的两个硬件版本（引脚从 eFuse 读），管着屏幕依赖的电源，还会接住 Vibe Mode 的开机画面，让屏幕一直显示到 Muse 画出第一帧。代价是关掉 BSP 自带的 LVGL 层，图形库只留 Muse 那一份，免得两份打架。
Muse 自带一个桌面模拟器，给它加了一档 480 方屏的配置。待机、聆听、思考、配网、出错五个状态先在模拟器里看过，没有越界，再上板。
上板之后调了三处：
- 图标位置。 屏幕上麦克风和电源两个小图标，照板子的实际按键位置挪：麦克风挪到上边偏右正对 AI 键，电源挪到下边偏左正对 BOOT 键。
- 开机慢 8 秒。 电量计 BQ27220 第一次开机要写入配置，占了 8 秒。挪到后台任务，界面开机就出来，电量过几秒再显示。
- USB 掉线。 板子连上 Wi-Fi 之后，USB 上的 ESP-Iris 不再响应握手，固件装不进去。查到 ESP-Iris 同时开了 TCP 和 USB 两种通道，Wi-Fi 一通它就把 USB 丢了。关掉 TCP 通道，连续 4 分钟每 30 秒探一次，USB 都在。
顺手想到：ESP32 之外的 RK，以及无用之用
8月底很火的 MicroDuck 是破圈了，399 美元，几天卖了一万多台，没有 Vibe Coding 的人都在说想买一个，想买一个干嘛呢？朋友说单纯做一个玩具也行。这个用的是瑞芯微的 RK3566，有比 ESP32 更强的本地能力，不少做机器人的都用了瑞芯微的芯片，说实话，我们自己在预研探索的产品用的也是 RK 系列的芯片，不过他们的芯片和 ESP32 系列不同的是做不小：我们现在已经遇到了想做小但散热解决不了的问题，甚至才用到它 CPU 的 20% 左右就烫得不行。再看看做 MicroDuck 的 Pollen Robotics，去年 4月被 Hugging Face 收了，今年 9月2日英伟达又宣布 129 亿美元收购 Hugging Face，LeRobot 和这些机器人背后站着的是英伟达，英伟达的模型和算力已经很牛逼，但虚拟世界再怎么模拟计算，再怎么解决 SIM2REAL 的 GAP，GAP 就是会存在，不如真实世界来的好。做这些产品，做不出来什么有用的东西，但能让团队/人了解开发具身智能这套工具链，就像我做这个没用的 Muse Gadget 一样，我知道他是怎么回事了，后面真想做有用的，或者一直跟他的技术进步，真能做出来有用的东西。
9月国内的安克也做了一件事情，搞首届黑客松（9月7日报名，10月中旬在深圳决赛），把自己公司成熟硬件产品的 SDK 开出来，邀请各位来一起构建应用。参赛的协议里面写的很清楚，参赛的设计、代码等都归安克所有。安克可以说是国内最近几年风头正劲的公司，AI NATIVE 推进也很靠前，人才济济的他为什么要做这样的事情呢？我相信他们也发现现在的 AI 除了在数字世界让 Human In The Loop 产生价值外，很难找到在消费品领域让用户买单的应用场景。他们类似 Plaud 的安克豆已经是目前 Agent 的极限状态，因为做个会议纪要，最后还是要我自己看一遍的。
三、让它有用一点
中文字幕
配对成功后第一次用中文提问，屏幕上一片空白。Muse 的字幕字体是 unscii，只有 ASCII。固件会把弯引号、破折号这些转成英文符号，汉字直接没了。
做法是给板子加一套 16 px 的中文字体：从思源黑体 CN 里取 GB2312 的 6763 个汉字加中文标点，共 6953 个字形，转成 LVGL 的二进制字体，884 KB。它编进固件，界面启动时载入 PSRAM，实测用了 0.57 秒。文件叫 cjk_16.bin，不叫思源的名字，因为 OFL 规定改过的字体不能沿用保留名。
Muse 这边加了一个「宽字体回退」：主字体没有的字去宽字体里找。两个细节要处理：
- 基线。 LVGL 把每个字形放在主字体的基线上，中文字体的基线不一样，要把宽字体的字形抬上去，不然汉字沉在行底。
- 换行。 中文没有空格，按字换行，闭合标点不落行首，开引号不落行尾。
还顺手抓到一个老 bug：字幕缓冲 400 字节，一个汉字占 3 字节，一页中文只能放半页，后半页被跳过。改成 800 字节，补了一条测试，旧代码跑这条会挂。后来又把 Markdown 从字幕里去掉了：行首的井号、列表号、引用号，行内的加粗、代码、删除线，链接只留文字。Muse 写回复很爱用这些符号，在 16 px 的字幕里只会碍事。
拍照
Muse 的设备协议里，拍照是 Muse 主动叫设备拍：设备启动时向 Muse 声明自己支持 camera.capture，你问「看看我面前是什么」，Muse 判断需要看画面就调用它，设备拍一张 JPEG 传回去。官方 SDK 里只有 SenseCAP Watcher 实现了这个命令，而且写死了。改成任何板子只要注册了摄像头就会声明并响应，Watcher 的路径不动。
板子这边，摄像头子板插左槽。第一版按 BSP 文档里的 OV3640 写，让传感器直接出 JPEG。装上去一试，Muse 说拍了两次都失败：
日志里是 format=JPEG is not supported。我手上这块子板的传感器是 SC101IOT，不是 OV3640，它没有 JPEG 模式。第二版改成取原始的 UYVY 画面，在板上用 esp_new_jpeg 编码，顺手逆时针转 90°，因为传感器在子板上是横着装的。拍完的照片在屏幕上显示 4 秒，能看到 Muse 收到的是什么。
两个额外的处理：
- 只在拍的时候上电。 每次拍照才认领子板、开传感器、丢掉前 10 帧等自动曝光稳定、取一帧、再全部关掉。平时摄像头不通电、不占内存。
- 引脚冲突。 摄像头的数据线和闪光灯占了芯片自带 USB Serial/JTAG 的两个引脚，所以那个口关掉了，Muse 的串口工具改走 UART。ESP-Iris 走的是另一个高速 USB 口，不受影响。
第二版实测：
语音
Muse 只回文字，不回语音。日志里那句「reply audio」只是按字幕节奏留出的静音，官方文档的说法是想要语音就自己接一个文字转语音服务。两条路：云端语音服务音质好，但每条回复多一次网络请求，还要经过 Mac 出网；板上离线合成音质偏机械、只会中文，但不联网、免费。先选了后者，乐鑫 esp-sr 的中文 TTS，小乐音色。第二天又补了前者的一个变体：不走云端，走我自己 Mac 上的 Qwen3-TTS。两条路现在都在固件里。
语音数据 2.9 MB，单独占一块分区：应用分区从 14 MB 缩到 11 MB，腾出 3 MB 放 voice_data。改了分区表就要用 system-update 重装一次，配对信息所在的 nvs 不在包里，装完配对还在。
esp-sr 只认汉字、数字和中文标点，Muse 的回复里有 Markdown、表情、链接和零星英文，念之前要先清理一遍：
- Markdown、表情和链接去掉
- 英文缩写按中文习惯念，AI 念「诶艾」，Muse 念「缪斯」；其他英文单词跳过，字幕里照常显示
- 「60%」念「百分之 60」，「25°C」念「25 摄氏度」
- 长句在逗号处分段，一段一段地合成，每段不超过 80 个字
- 整条回复没有中文就不念，只显示
喇叭开着时，字幕跟着语音走；喇叭关着时按阅读节奏翻页，和上游一样。
装上去第一次说话，板子复位了。从板上留存的 core dump 读到断言 s_task_stack_is_sane_when_cache_frozen。原因是我在 Muse 的对话任务里映射 voice_data 分区，映射会改 flash 的 MMU，期间 flash 缓存被冻结，而这个任务的栈在 PSRAM，缓存冻结时不允许访问 PSRAM 上的栈。改成开机时在主任务里映射，主任务的栈在片内 RAM。这条规则后来写进了接口的注释：栈在 PSRAM 的任务不能做 mmap、不能写 NVS、不能读写 flash。
修好之后的数据：
英文也要念：接 Mac 上的 Qwen3-TTS
小乐只会中文，Muse 的回复经常中英混着。我家里那台 Mac Studio 上本来就跑着一个 Qwen3-TTS 服务，OpenAI 风格的接口，中英都念。于是固件里加了第二条路：回复先交给 Mac 念，Mac 连不上时中文退回板上的小乐。
这条路第一版就把板子搞崩了。服务端按合成速度往板子灌 PCM，板子只开了 1 秒的下载缓冲，读不完的语音堆在网络协议栈里，占住 Wi-Fi 的接收缓冲，片内 DMA 内存被吃到 1 KB 以下。改成两头都收着：服务端先发 1 秒，之后不超过实时的 1.5 倍；板子把整条语音先存进 PSRAM 再播。试过把 Wi-Fi 的缓冲挪去 PSRAM，这颗芯片上进不去，反而多出一批常驻的发送缓冲，撤回了。
对话中随机复位：代码不要从 PSRAM 执行
接着是一个更难的：对话中随机崩，空闲时也崩过一次。板上留下的 5 份 core dump 全是读内存出错，地址都是合法的 PSRAM 数据，而且全部落在 PSRAM 里只读区和可读写区的分界线之后 46 KB 以内。读了两个核的内存保护寄存器，都是对的。能稳定复现：同时跑一个内存锤子测试和边下载边播放，4 分钟左右崩一次。
查不到根因，但找到了绕法：关掉「代码从 PSRAM 执行」（CONFIG_SPIRAM_XIP_FROM_PSRAM），代码和常量改在 flash 里跑，PSRAM 只放数据，那条分界线就不存在了。同样的锤子测试跑 10 分钟零崩溃，两个核各锤了约 580 万轮；再跑 15 分钟，我同时用中英文对话，有语音、没崩。副作用是 ESP-Iris 那几项「放 PSRAM」的配置依赖这个功能，跟着失效，它的缓冲落回片内 RAM；好在开机后片内空闲 64.5 KB，和之前持平，PSRAM 反而多出约 4 MB。根因大概率在 ESP32-S31 加 ESP-IDF 主干这一层，8月中旬刚改过 S31 的 PSRAM 分区保护，这些证据可以拿去给乐鑫提 issue。
有人来了打个招呼
这是另一个会话做的。右槽插 ESP-Mosaico 的交互模块，上面有一个 PIR 人体感应和 6 颗 WS2812 灯。想要的效果是我走到桌前，它亮灯、亮屏、打个招呼。
先用模拟器把画面画出来确认：开心动画、一声「叮」、按时段的一句字幕，显示 6 秒。问候语分早上、中午、下午、晚上、深夜五档，深夜是「夜深了，Rollin，早点休息」。板子原本不对时，Wi-Fi 连上后自己去 ntp.aliyun.com 对时，对上之前说「你好，Rollin！」。
状态机只有三个时间常数，都写在源文件开头：
第一版上板，灯亮了不灭。日志显示来人和问候都在开机 90 秒时正常触发，之后我一直坐在板子前，人体感应持续报有动静，而原规则是连续 60 秒没动静才灭灯。改成灯和问候字幕一起亮、6 秒后一起灭，人在的时候再有动静只让屏幕不休眠，不再亮灯。灭灯没有日志可查，iris screenshot 在这个应用上也截不了图（Muse 自己驱动 LVGL，没接 Iris 的屏幕镜像），最后靠目测验收。
PIR 感应的是动作，不是存在。坐着一分钟不动再动一下，会被当成新来人，灯再亮 6 秒。
顺便算一笔内存账
ESP32-S31 的片内 RAM 是紧的，Wi-Fi 和 TLS 都要从里面拿 DMA 内存。加上字体、语音、问候之后，Muse 就绪时的空闲片内内存从 74.7 KB 掉到 49.0 KB，DMA 内存最低到过 12 KB。两个会话各自把能挪的挪到 PSRAM：语音的文本缓冲和字幕页、问候任务的 4 KB 栈和状态。前提是查过它们的调用链不碰 flash。
剩下的大头在 BSP 里：模块管理器以前拍照时才启动，现在开机就起，带两个各 4 KB 的片内栈，大小写死在子模块里。
顺手想到：感知这件事，语音比图像先落地
语音转文字的信息现在是最容易被从真实世界收集上来的。
雷鸟 8月发的 iO 眼镜，全天候记录对话、只存转写的文字不存音频，雷鸟这个眼镜最牛逼的取舍就是把摄像头和扬声器都去掉了，34 克，有人戴这个眼镜如果不是我知道他是什么产品，我根本不会在意，但要是有个摄像头就会特别引人注意了，哪怕我说摄像头没工作也没人信的。我戴 Meta 的眼镜出门和在公司就遇到了这样的问题，总有人问我你摄像头没工作吧，语音信息反而没啥人在意。苹果在九月的发布会上也预告了 Apple Watch Series 12 的 Siri Recap：全天候听你的对话，只留摘要不存音频，年底 beta，世界上最牛逼的公司之一现在在消费品也只能做到这个地步了，全天候收集语音信息。不是说不能 24 小时收集图像信息，但那个代价太大了，不是消费品能做的，甚至现在智能眼镜的续航根本还做不到，其他类型的设备也还没出现，哪天消费品能出现一个 24 小时记录数据的，估计也只能是特定工作场景的具身智能机器人了，除了工厂，又有哪个场景需要机器人 24 小时工作呢？
四、反过来，让 Muse 使唤板子
做到这里，板子是在「被动服务」：Muse 要看就拍，回复来了就念。我问 Muse「你能用 gadget 说话吗」，它答「没有扬声器播放的指令」。这句话点明了 Muse 和板子之间的关系：不是 MCP，是 Meta 自己的 Home Link 协议。板子连上时报一份命令清单，每条带描述和参数，Muse 看着清单决定什么时候调。上游的清单里只有五条：健康、发现、画一张网址图、放动画、拍照。
于是把板子的能力都报上去。SDK 里加了四条任何 Muse 板子都能用的：voice.say 念一段话、voice.configure 调音量、display.show_text 显示一页字、display.configure 调亮度和休眠。包装工程里加了三条交互模块专属的：presence.read 有没有人、lights.set 六颗灯变色、ir.send_nec 发一个红外码。注册的 JSON 从 1.9 KB 涨到 5.5 KB，Muse 服务端回了 200。
装上去试了一轮。第一次让它「用 gadget 说一句你好」，Muse 答「小设备没有扬声器，说不了话」，把「你好」画到了屏幕上，它记着的还是前一天清单里没有 voice.say 时的结论；新开一个对话再说一遍，板子就用 Mac 的声音说了「你好」，0.6 秒说完。「把灯打开」它先去找 Philips Hue，提醒它「我的 gadget 有灯呀」之后，六颗灯亮成暖白；再说「让灯多彩闪烁起来」，它把红橙黄绿青蓝紫轮了一遍停在紫色上，lights.set 被它连着调了好几次。
presence.read 和 ir.send_nec 还没让它调过。
顺手想到：Home Agent 这条赛道
非常让我兴奋的是，Anthropic 9月预览了 Model Hardware Standard（MHS），打通硬件设备和 Agent 的沟通渠道，其实也不是什么高端的东西，大家都能想到，我兴奋是因为我的判断是对的，也正在推动团队做类似的事情，先把硬件做稳定，把硬件的能力通过 MCP 告诉 Agent，在软件层面做各种编排定制通过硬件来实现不同的感知-执行的工作。现在 Muse Gadget 也是类似的产品，你看 Meta 在 9月的 Connect 上发了个掌机 Muse Charm，那玩意儿不就是让大家先玩起来吗？现在这个 Muse Gadget 也不过是 Home Link 里面的一部分，Meta 明显是想从 Personal Agent 到 Home Agent，现在的智能家居也会被 Muse 这样的大脑管起来。
有传言 OpenAI 和 Jony Ive 的第一款硬件是个无屏幕、带摄像头和麦克风的桌面设备，年底前亮相、2027年上市，在这条路上前面已经有很多先驱了，在现在的 Agent 和模型能力下，用云端的应该是可以做到更好的家居
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Matrix Home Page Recommendations
Matrix is a minority writing community that advocates the sharing of real product experiences with practical experience and reflection. We will occasionally select the best articles of the Martrix to show the most true experiences and views from users.
The article represents the author ' s personal point of view, with minority groups making only minor changes to the title and layout.
AI Auxiliary Creative Statement (author ' s note):
The firmware code recorded here was prepared by Claude Code (Claude model) of Anthropic, and the author was responsible for requesting, installing validation and making decisions on the board. The first draft of the article was compiled by Claude on the basis of the conversational logs and handover documents in the development process, where the author rewrites and cross-references the facts, data and sources of reference; the views of the authors themselves were written on the " thought-in-hand " and " zoom in " paragraphs. The diagram was drawn by Claude at the author ' s request and taken and intercepted by the author himself. The full text has been verified and modified by the author and the author is personally responsible.
If someone asks me, do I spend time, spend token doing something so useless? My answer is yes, and the most important point is to prove that Vibe Coding is growing fast in the hardware field. Because the digital world has reached a certain bottleneck, and the physical world is needed to reverse the digital world. Without AI Agent to this extent, I can't imagine how much time it takes to learn about hardware, embedding, software, communication, etc., and how much information it takes to find, but it takes time to accumulate, but it's almost all available in the digital world, and hardware stuff is done by companies that specialize in music.
First, a few words are made clear, so that readers who have not touched hardware can read them down.
The development panel is a circuit board used by chip manufacturers to validate chips, write solids for engineers, weld chips, electricity, USB mouths and commonly used external features (screens, microphones, horns, sensors) so that they can pass through the electronic writing program, which is not a consumer-oriented product. ESP32 is a chip series of music, with wi-fi and bluetooth, cheap, power-efficient, informative, open-air and AI hardware toys. The solid is the program to burn into the plate and decide what to do when it's on.
Modularized development boards mean that the main plate is only the most basic, with cameras, human sensors, lights, which are made into a small sheet, and are inserted on the extended slots on both sides of the main plate, as required, with whatever capacity is available. ESP-Mosaico is a board of this type issued in August 2026, especially since it was designed from the beginning by "Let's allow the AI programming tool to write firmware": the panel contained a set of bottom solids called Vibe Mode, which were used to load programs, restore and read logs, and the user did not touch the burn tool; instead of looking at the drawings, it turned over a prompt to Claude Code, a program such as Agent.
On August 19, the ESP-Mosaico was a modular board for Coding Age. Actually, I know it's rough, and I see the user feedback that it's still a 3D printout, but I'd like to see how the best ESP32 series in the AI hardware field is doing, especially in its Vibe Code model. It works differently than the old board: instead of showing you a drawing, it gives you a prompt and you throw it at the AI programming tool. Here's the sentence:
Download the code from this repository: https://github.com/esp-mosaico/esp-mosaico-vibe
Set up the environment and implement a Snake game.
The first one is snake-eating. I did it, with Claude Code, and I can play on the board. On October 2nd, Meta opened up the equipment of Muse Gadgets, SDK, and the officially recommended hardware was ESP32, so it went down: on that day, the board became a Muse small device, and after that it was able to talk in Chinese, read it out, take a picture back as requested by Muse, and light the lights as I approached.
I did the whole thing with Claude Code, with four sessions back and forth, two of them working side by side on the same board. The article is written in the order in which it occurred, in five sections, and it is thought of at the end of each paragraph with a little bit of it done. Codes in two warehouses:
- muse-gadget branch of rowline/esp-mosaico-vibe: Viper, Muse packaging works, Bluetooth diagnostic solids, handover logs
- extension/muse-gadget-sdk branch: 9 submissions to Meta SDK
One, play first.
Step one: making snakes according to the music script.
ESP-Mosaico is dominated by ESP32-S31,480 square screens, horned microphones, extension modules with both sides. The biggest difference between it and the normal development board is that the front 2 MB lives in a fixed set of solids called Vibe Mode: install applications, restore, read logs, block them, install them with a mosaico.py command, without touching esptool. Turns on the orange AI key and enters Vibe Mode, so the application is broken and it doesn't change bricks.
After prompt was thrown in, it took AI one night to set up the environment, and the pits were on the floor, not the game:
- ESP-IDF to nail to a specific submission. The workspace requires 7b9cc1ac on Master, which is not the latest version, with the finished version number 6.2.0.
- Inadequate version of the domestic components. The component service from this network for access to music is transferred to the domestic mirror, which does not have an esp-gsp 1.5.1. Finally, the official API downloaded, checked Hashi and built a local assembly library.
- GSP scene compiler officially not sent. The component required 0.6.1 does not exist, and a signed version of the workspace CI paragraph 0.5.0 is used.
That's all Claude Code did, I didn't do anything. The game itself is written by Raylib Lite, rules and drawings in two files, and the same C code runs on the browser simulator and board. The 20 x 18 grid, with an apple length section and an extra point, increases from 6 to 12.6 per second, crashes a wall or ends itself and fills it up to win. Slide on the touch screen, start or turn there, and pause the fraction bar. 8 Playback tests, each set to play back and run again, and the final state of Hashi must be consistent.
The simulator grabs the bug of a template: the blank game template listens only to touch point 0, and the browser reports the mouse as point 1, so the click does not react. Replace by following the finger or mouse pressed first.
It was the next day. For the first time, the board was plugged into the Mac, but it didn't come up with a line that passed the data. First-time board runs recover for Vibe Mode base solid, then system-update for snake-eating. Three things are checked: the device ID is unchanged, the start ID is new, the device reports health. 33.3 fps, no frame off, touch normal.
When snakes are finished, the environment breaks. The next Muse all built on this.
And then it came to mind: Why ESP32 again?
Following on to a few things, how many people have started to follow the example of Claude Desktop Buddy, how many people have gone to Vibe Coding's own electronic pets, give it to the screen, give it emotional value, and what is officially recommended for the ESP32, which is also an ESP32 ecological object, which was recommended by Muse this time because it's simple enough to grow over the years, and AI can find enough knowledge on the Internet to help users to achieve his own customized needs.
ii. Turn the board into a mus
Meta Open Source
When I bought this board, I predicted that something like GrokBot would go back to the hardware, so I could go to play first, and I didn't think Muse would do it first, in the open source. I used my cell phone to simulate hardware.
Muse is the personal AI assistant that Meta does. Muse Gadgets is open only to the end of the device: Linux SDK, Apache 2.0 for a set of ESP32 solids and a set of berry pies. The assistant does not open its own source, and the device first goes to gadgets.muse.ai to collect a SDK token and then opens the developer's mode pair in Muse App. The officially supported panel with the full interface is the Waveshare 1.75 inch screen AMOLED category.
When I first used it, I found that Muse was a lot faster than GrokBot's reaction, and it was better to try to identify it, that GrokBot said it would take longer to react, that GrokBot himself did nothing to interact with the device, that the content of the interface and the content of the device would not be exactly the same, and that Muse would be much better.
Vibe's value is freedom to fit.
Three things can't be done:
- Development of a version of the tool. Muse solidware only recognizes ESP-IDF 6.0.1 and 6.0.1 does not support ESP32-S31. The working area of this board nailed ESP-IDF to a submission in 6.2.
- The official brush will flush Vibe Mode. Muse writes with the idf.py flash chain guide, partition table, the first 2 MB will be overwritten.
- The board doesn't connect to Meta's server. api.muse.ai and catch.metaaivm.com have all timeout in the country.
If I didn't have Claude myself, I wouldn't be able to see these three problems, and now I have them, and I don't want to keep playing. My decision is to leave Vibe Mode, make Muse a standard ESP-Mosaico application, install it with Vibe Mode. The division of the back Flash was made by Claude and turned down like this.
Muse's matching messages, Wi-Fi passwords, and device vouchers are all in the 64 KB nvs, and the resets are not clear, so they have not been rematched for the last 10 times. That's too important. If it wasn't for this project, I didn't know that it was possible.
The code is in three layers. The SDK of Meta opened only a few small mouths, all the boards were in the packaging project, and the ESp-mosaico branch of SDK added up to nine submissions, each of which was closed by default, and the other boards remained intact.
The first two changes to SDK are necessary:
- Start the hook. Muse's app main prefixes Wi-Fi, Bluetooth and interface with a weak sign muse gadget platform start(). The packaging project gave it a strong definition in which ESP-Iris equipment management services were activated and the solids were accepted. Later, cameras, voices and greetings were registered from here.
- Turn off Muse's bringin' the roller. Upstream solids are loaded and not connected to Muse clouds in five minutes, and they roll back automatically. It's not going to work until it's paired. It was turned off with a switch and the solid acceptance check was handed over to Vibe Mode.
We're going online.
The routers are common for home use and can't change the gateway. The last thing is to use Mac Studio as a sidewalk, just let the board walk it, and the rest of the equipment stays.
Two configurations have been added to the solid; the default is empty:
- Extranet next jump. The board takes the address from the router, renews the contract, and only delivers the package to the designated host. It's with IwIP's own relay hooks, DHCP that set completely intact.
- DNS. Switch to DNS outside the LAN when the gateway is effective. The router continues to reset DNS, so every 30 seconds, reset.
Mac here thought it was necessary to open the Clash local area network connection and the DNS port to find out: Its TUN mode has taken over all public traffic, dns-hijack: any:53 will answer the DNS queries through it. Just open the IP forward:
sudo sysctl -w net.inet.ip.forwarding
=
1
This restart will fail.
Two fake faults stuck in pairs.
Bluetooth in the log has been broadcasting MuseGaget-F46F0A, and others have paired it with the unscreened Muse. It's been stuck all night, and both questions prove to be irrelevant.
Cell phones can't find Bluetooth. The bluetooth list in the iPhone system settings shows only devices such as headphones, keyboards, where low-capacity bluetooth broadcasts do not appear. But there's no bluetooth on the functional chart of the ESP-Mosaico official BSP, and there's no bluetooth in the example, so the plate's bluetooth is working, and nobody has ever verified it before. To remove the muse factor, a diagnostic solid for bluetooth only was made: ble diag: broadcast under the name MosaicoBLE-DIAG, with its own bluetooth broom every 15 seconds. The result is 34 to 37 devices per round, the strongest signal -37 dBm; it is also seen in a mobile phone. The board and the mail are normal. The solid was left in the warehouse and later suspected that Bluetooth was ready for testing.
Muse App opened the developer mode without adding the device to the portal. Checked the food set, the area, the iOS test version, and it's not conclusive. The next day it showed up itself, was App's problem over there. Here's what it looks like in normal times, with the upper right corner plus the entrance, and the developer's model is very clear: it allows pairing of community equipment that is not certified by Meta, which has complete permission for your assistant.
Matching per se is very smooth: App select device, short-click AI at logtip confirmation, fill in the Wi-Fi password, reboot the plate and connect to Muse.
Screen
The Muse interface with the screen is LVGL painted, and the board used upstream is a screen. There are two options for transplantation: to write a set of drivers on the Muse-supported Waveshare board, or to use the ESP-Mosaico official BSP. BSP: It knows two hardware versions of this plate (read from eFuse), runs the power on which the screen depends, and also picks up Vibe Mode's live screen, allowing the screen to be shown until Muse draws the first frame. The price is to turn off the LVGL floor of the BSP, leaving only Muse that one before two fights.
Muse brought a desktop emulator with a set of 480 screens. The standby, listening, thinking, networking, and error of five states were first seen in the simulator, not crossed the border and then put on the table.
Three times after the board:
- Icon position. The two small icons on the screen of the microphone and the power source move by the actual button position of the plate: the microphone moves to the top right-right positive-AI key and the power moves to the lower left-right-to-BOT key.
- Go slow eight seconds. BQ27220 has eight seconds to write to the configuration for the first time. Move to the backstage and the interface will be open and the power will be displayed in seconds.
- USB offline. After the board was attached to Wi-Fi, the ESP-Iris on USB no longer responded to handshakes, the solids could not fit in. When we found out that ESP-Iris had both TCP and USB access, Wi-Fi lost USB. Turning off the TCP channel for four consecutive minutes every 30 seconds, USB is in.
RK outside ESP 32, and useless.
At the end of August, MicroDuck was a piece of shit, $399, sold more than 10,000 in a few days, and everyone who didn't have Vibe Coding was talking about buying one. Friends say it's just a toy. It's a rickshaw with RK 3566, a much stronger local capability than ETS32, and a lot of robots are using a rickshaw chip. To be honest, the RK series is also used for our own pre-explored product, but unlike the ESP32 series, it's not small: we've got problems that we can't solve, we can't even use 20% of its CPU. And look at Pollen Robotics, who is a MicoDuck, who was taken over by Hugo Face last April, and on September 2nd this year, when Britain announced a further $12.9 billion acquisition of Hugging Face, LeRobot, and the robots behind them are already very good models and calculations, but how the virtual world simulates and solves SIM2REAL's GAP, GAP is not as good as the real world. Making these products, doing nothing useful, but getting teams/people to know how to develop a smart tool chain, like I did this useless Muse Gadget, I know what's going on with him, and then I really want to do something useful, or I can do something with his technological advances.
In September, Anke in the country also did something to get the first hackers out of his company's mature hardware product, SDK, and invite you to build applications together. The agreement for the contest is very clear, and the design, code, etc. for the contest belongs to Ank. Anke can be said to be a strong company in the country in recent years, and AI NATIVE has advanced very well. And I'm sure they've also found it hard to find applications in consumer goods that can be bought by users, in addition to making human In The Loop valuable in the digital world. They're like Plaud's Anker Beans, which is now the limit of Agent's, because I'm going to have to take a look at it myself.
Three, make it work.
Chinese subtitle
For the first time, a pair was successfully asked in Chinese and there was a blank on the screen. Muse's subtitle font is unscii, only ASCII. The solids transpose the curved quotes and the dashes into the English symbols, and the Chinese are gone.
This will be done by adding a set of 16 px Chinese fonts to the board: 6763 HDRs of GB2312 from ThinkBold CNs, 69553 fonts, converted into LVGL binary fonts, 884 KB. It builds solids, it loads PRRAM when the interface starts, and it is measured in 0.57 seconds. The file, cjk 16.bin, is not the name of the resource, as the changed font specified by the OFL does not have the reserved name.
Muse added a "broad font rollback" to this side: the main font does not have words to look in the wide font. Two details need to be addressed:
- Baseline. LVGL places each font on the baseline of the main font, which is different from the Chinese font, and lifts the font of the wide font, otherwise the Chinese word is sunk to the bottom of the line.
- Line break. There are no spaces in Chinese, line-to-word, closed markers do not start and quotation marks do not end.
An old bug: subtitle buffer 400 byte, a Chinese word of 3 byte, a page in Chinese only half a page, and the last half of the page was jumped. Changed to 800 byte and added a test that the old code would hang. Markdown was then removed from the subtitle: the well number, list number, reference number, thick, code, strike-line in the line, only text. Muse writes very much about these symbols, which are in the way of subtitle at 16 px.
Take pictures.
In the Muse device protocol, the photo was taken by the Muse who volunteered to shoot: when the device was activated, he declared to Muse that he supported the camera. Capture, and you asked, "Look at what's in front of me," and Muse called it as soon as he saw what was needed, and the device took a JPEG. Only SenseCAP Watcher in the official SDK fulfilled this order and wrote it dead. Change to any tablet to register the camera will announce and respond, and the WATcher path will remain intact.
This side of the board, camera panel in the left slot. The first version is written by OV3640 in the BSP document, which allows the sensor to come directly out of JPEG. Put it on, Muse said it failed twice:
Format=JPEG is not supported. The sensor I have on this plate is SC101IOT, not OV3640, which does not have JPEG mode. The second edition was changed to take the original UYVY image, encoded on the plate with esp new jpeg and rotated 90° in reverse, because the sensor was over-dressed on the subboard. The photos are on the screen for four seconds and can see what Muse received.
Two additional treatments:
- Only when you shoot. Each time you take a picture, you identify a bearing plate, an open sensor, an automatic exposure of 10 frames, etc., before they are dropped, and take a frame and turn it all off. The camera is not powered and does not account for memory.
- Caught. The camera's data lines and flashlights took two steps of the chip with USB Serial/JTAG, so the mouth was turned off and the Muse serial tool was removed. ESP-Iris is going to another high-speed USB mouth, not affected.
Release two:
Voice
Muse only returns text, not voice. The phrase "reply audio" in the log is just a silence left at the subtitle pace, and the official document says that you want a voice to switch to a voice service. Two roads: cloud voice service is sound, but each response to an additional request for a network goes through Mac; offline synthetic sound machine on board, Chinese only, but is not connected, free of charge. First, the latter, the edible esp-sr's in Chinese, TTS, voice. And the next day, one of the variants of the former was filled: no clouds, no Qwen3-TTS on my own Mac. Both roads are now in solids.
Voice data 2.9 MB, separate partition: the application partition has been reduced from 14 MB to 11 MB, free of 3 MB with voice data. Changing the partition table requires reloading with a system-update. The nvs in which the information is found are not in the bag and the pair is still complete.
esp-sr, which only recognizes Hong Kong characters, numbers and Chinese markers, Muse's reply contains Markdown, emoticons, links and sporadic English, and needs to be cleaned up before reading:
- Markdown, face and link. Drop it.
- English abbreviations in Chinese, AI, "Aye", Muse, "Muse"; other English words skip, as usual, in subtitle
- 60% "60%" and 25°C "25°C"
- Long sentences in commas, one paragraph combined, each not exceeding 80 words
- If you don't read it in Chinese, just show it.
When the horn was on, subtitle followed the voice; when the horn was closed, the page was turned by the reading rhythm, as was the case upstream.
Put it on for the first time and the board returns. The core dump left on the board read the assertion s task stack is sane wen cache frozen. The reason for this is that in my Muse dialogue mission, which maps the voice data division, the map changes the MMU of flash, during which the flash cache is frozen, and the kiosk of the mission is not allowed to visit the psalm on PSRAM, cache when it is frozen. Change to map the main mission when it's on, and the main mission is inside RAM. This rule was later included in the interface 's notes: missions in PRRAM cannot be map, NVS cannot be written, flash cannot be read and written.
Data after repair:
In English too: pick up Qwen3-TTS on Mac
Lok's only Chinese, Muse's reply is often mixed with Chinese. My home, Mac Studio, runs an Qwen3-TTS service, Open AI-style interface, Chinese and British. So the solids add a second path: the reply goes first to Mac, who can't connect to the Chinese back board.
The first edition of this road destroyed the board. The service side fills the plate with PCM at synthesis speed, with a one-second download buffer on the plate, with unreadable voice piles in the web protocol stack, holding the Wi-Fi receiving buffer, and a piece of DMA memory being eaten below 1 KB. Replace it with both: 1 second first for the service and 1.5 times later in real time; and the board saves the entire voice in PSRAM. An attempt was made to move the Wi-Fi buffer away from PSRAM, which was unable to get up on the chip, and instead an additional set of permanent sending buffers was withdrawn.
Random repetition of position in dialogue: code not executed from PRRAM
And then it's harder: there's a random fall in dialogue, and there's a fall in free time. The five core dumps left on the board are all reading memory errors, the addresses are valid PSRAM data and are all located within 46 KB after the line between read-only and read-writing areas in PSRAM. Read two nuclear memory protection repositories, all right. Stable recurrence: running a memory hammer test and downloading while playing, one crash in about four minutes.
No root cause could be found, but a deviation was found: the "code executed from CONFIG SPIRAM XIP FROM PSRAM" was turned off, the code and constant ran in the flash, and the PSRAM only left data, and the dividing line did not exist. The same hammer test runs 10 minutes and zero, two hammers each about 5.8 million rounds; 15 more minutes, I talk in Chinese and English at the same time, with voice and no fall. The side effect was that the ESP-Iris configurations of the "PSRAM" were dependent on this function, and it failed, and its buffer fell back inside RAM; it was better to leave 64.5 KB in the back of the engine than before, and PSRAM was about 4 MB. The evidence can be taken to Legitimate Zunue because the root is probably on the ESP32-S31 plus the ESP-IDF backbone, which has just been modified in mid-August to the S31 PSRAM partition protection.
Someone came to say hello.
It's another session. The right slot inserts the interactive module ESP-Mosaico with a PIR human sensor and 6 WS2812 lamps. The effect is that I walk to the table and it lights, lights, says hello.
The simulator first draws the image to confirm: Happy Animation, One Ding, subtitle in Time, 6 seconds. In the morning, noon, afternoon, evening and late evening, the greeting was "Deep night, Rollin, early rest". When the board was wrong, Wi-Fi joined up and went to ntp.aliyun.com and said, "Hello, Rollin!"
Status machines have only three time constants, all written at the beginning of the source file:
First edition on board, the lights are on. The log shows that both the people and the greetings were activated in 90 seconds, and then I sat in front of the board, and the body senses continued to show a movement, whereas the rule was 60 seconds in a row without moving. Change to light with greeting subtitle and then extinguish it with six seconds later, and when people move, they just keep the screen awake and no more lights on. There's no log to check the lights, and iris screenshot can't stop the picture on this application (Muse self-propelled LVGL, no Iris screen mirror), and lastly by visual acceptance check.
PIR senses action, not existence. One minute without moving, you'll be treated as a newcomer and the light will be on for six seconds.
And a memory check.
The ESP32-S31 film RAM is tight, and Wi-Fi and TLS are required to take the DMA memory from it. With fonts, voice, greeting, the spare fragments memory of Muse ready fell from 74.7 KB to 49.0 KB, DMA minimum to 12 KB. Each session moves the moveable to PRRAM: Voice text buffer and subtitle page, greeting mission 4 KB stack and status. If they check their call chain without touching flash.
The remaining big head is in the BSP: the module manager only started when he had taken a photo, and now it's on, with two KBs, each, and the size dies in the submodule.
By the way, it's a matter of perception that the voice is ahead of the image. Land
Word-to-text information is now the easiest to collect from the real world.
Thunderbird's iO glasses, round-the-clock conversation, only transcribed text without sound. Thunderbird's best trade-off is to take off both the camera and the speaker. 34 grams, I wouldn't care if someone wore these glasses if I didn't know what he was making, but having a camera would be particularly interesting, even if I said the camera didn't work. I went out with Meta's glasses and I had this problem at the company, and someone always asked me if your camera was not working and that voice messages weren't important. Apple also announced at the September publish meeting that Apple Watch Series 12, Siri Recap: Listening to your conversation all day, leaving only summary without audio, and at the end of the year Beta, one of the world's best companies is now only able to do that in consumer goods, collecting voice information all day. It's not that you can't collect image information 24 hours, but it's too expensive, it's too expensive for consumer goods, it's not even possible for smart glasses to continue, it's not available for other types of equipment, it's not possible for consumer goods to have 24 hours to record data, and it's estimated that they can only be smart robots for specific work scenarios, except for factories, which ones need robots 24 hours to work.
♪ In turn, let Muse make the boards ♪
And here, the board is "passive service": Muse will watch and read. I asked Muse, "Can you talk with gadget," and it answered, "No speaker." This explains the relationship between Muse and the board: not MCP, but Meta's own Home Link protocol. The boards are attached to the Times list of commands, each with a description and parameters, and Muse looks at when the list will be adjusted. There are only five entries on the upstream list: health, discovery, drawing a map of the site, animating and taking photographs.
So I reported on the board. The SDK adds four items of any Muse board that can be used: voice.say read a message, voice.configure tuning, display.show text displays a page word, display.configure tuning and hibernation. Three interactive modules have been added to the packaging project: presence.read. Registered JSON went from 1.9 KB to 5.5 KB, Muse service back 200.
Put it on and try a round. For the first time, it was allowed to say "hello" with gadget, and Muse said "hello without speaker" and "hello" to the screen, remembering the conclusion that there was no voice.say on the list the previous day; it said "hello" with Mac's voice again, 0.6 seconds after. It went to Philips Hue and reminded him that after "my gadget has a light", six lights were lit warm and white; and, "Let the light shine up," it stopped the yellow yellow and green wheel on purple, lights.set, and it was pulled several times.
I haven't made it.
And by the way: Home Agent this track
And what I'm very excited about is that in September, Anthropic previewed Model Hardware Standard (MHS) to access hardware equipment and access to Agent, which is not really a high-end thing, and that I'm excited because I'm right and I'm pushing the team to do something like this, to stabilize the hardware, to tell Agent through the MCP, to make various customizations at the software level to achieve different senses-implementations through hardware. Now Muse Gadget is a similar product, and you see, Meta put a hand in September's Conect. Now this Muse Gaget is just a part of Home Link, and Meta obviously wants to go from personal Agent to Home Agent, and now the smart home will be managed by a brain like Muse.
There's a rumor that Open AI and Johnny Ive's first hardware is a desktop device with no screens, cameras and microphones and that there's a lot of pioneers ahead of the road before the end of the year, and with the current Agent and model capabilities, it's better with clouds. Home
Auto-fetched and translated by AI Atlas. All rights belong to the original author.