对一个GPU进行批量分类和决策推论。
spinf-serve读过一段内容，并在同一个传中回答了很多关于它的问题。 每个答案都是你给出的选项的概率，而不是生成的文本。 它运行于一个24GB NVIDIA GPU上，服务于OpenAI兼容的API，与主机spinf服务相同的API. 阿帕奇2.0 (英语).
我们为自己的作品建造了这个引擎：每日的市场指数从数百万篇新闻文章中读取，每篇文章都问了几十个狭义，校准的问题。 为聊天而建造的服务引擎重读每个问题的内容，并一次生成一个符号，这使得这项工作缓慢而昂贵。 spinf-serve将每个问题都作为阅读内容的分支列出，并将许多内容装入一个前传。 批量分类是它的起源地和最强的地方。
同样的机制是现在所谓的决定模型：一个模型，它返回了对一组固定答案的概率(是或否，标签，级别)而不是一个段落。 任何开放的指令模型都以这种方式在spinf-serve中工作，专用的决策模型运行到现在。 主机spinf API是用这个引擎产生的；我们正在出版它，因为人们要求用自己的硬件运行它。
这不是聊天服务器。 除非您要求，否则任何事物都无法产生，而且它是为吞吐量而不是为单一请求的暂时性而建造的。
Linux,NVIDIA GPU配有24GB和fp8的拉力核(Ada,Hopper或Blackwell:RTX 4090 / 5090,L4,L40S,RTX 6000 Ada,H100)，是支持CUDA 13,Python 3.12的驱动程序。 Ampere牌以英寸8重量代替(配置)运行。
pip 安装“ spinf- serve [server] ”
spinf-serve服务 spinf/gemma-4-12b-fp8 # 首开时下载检查点；服务于港口 8000
或图像，安装了 NVIDIA 集装箱工具包(容量保存模型和编译缓存);
图像名称后面的参数是spinf-serve serve serve:
docker 运行 -gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve
便问。 两种分类和一种从内容的单一读取：
校对：Soup
-H"Content-Type：应用程序/json"\?
-d '{
"型号":"spinf/gemma-4-12b-fp8".
"灵枢"曰：[
{"作用":"用户"
"内容":"客户支持信息：\n I被收取了两次9月的费用，请退还其中一项费用。"}
],
"评分":{
"泻"曰：[
{"id":"论语"
"定档":"n\nTopic(开票，技术，帐户，航运，或取消):{?}"
"选项":["计费","技术","账户","航运","注销"]}
{"id":"退钱"
"腾讯":"客户是否要求退款？??.
"选择":["是","否"]
{"id":"月"("月")
"通鉴"曰："臣曰：何月被罚两次？
]
}
}'
对于每个得分的问题，答案给出每个选项的概率，对于空内容(来自12B模式的真实数字)的概率相同：
{"id":"退款","合并":[{"值":{},"选项":
{"文本":"是","日志":-0.90,"p":0.72,"地板 p":0.60}
{"文本":"否","日志": -1.86,"p":0.28,"地板 p":0.40}}
"余文":0.44 }}]] }.
p 是您选项中的概率。 地板 p是相同的问题，没有内容，所以p - 地板 p是什么
添加的内容(这里的题目是“计费”从0.47移到0.91)。 剩余值是用到文字的概率
在你的选择之外。 每月的答复列出了最可能的答案及其概率("九月第一").
同一机身带有API密钥，运行于 api.spinf.com;OpenAI SDKs工作通过/v1/chat/完成
附加体中的积分块(快速启动)。
- 一个读，许多问题。 内容读一次；每个问题都是同一个通行证的分支，只花费自己的代币。 占位符将一个模板变成一个问题网格。
- 分数可以扣分每个选项的概率及其下限和日志概率；决定是您在自己的数据(校准)上选择的阈值。
- 从同一案文中提取的数值改为：文本、 {n} 数字和{$} 数字按其概率解码
或自由文本生成到槽中。
- 同样的分数无论在批量。 有了fp8检查站，一个内容的分数是比特的
独自一人或与数以千计的其他人打包跨越电话、天数和服务器二时的系统指印称
答复不具有可比性。
- 内容中的图像、音频和视频框架，用于读取它们的模型。
- 即时包。 一项工作(内容节制、分类、产品上市、评论、品牌、娱乐)的准备、校准问题集，预先装入并在请求中命名(以下)。
- 主办或自主办。 客户端在此服务器和 api.spinf.com 之间切换，可以更改基址和
钥匙互联互通；
agent技能教授一个编码agent的API，这个服务器和包格式。
在1个RTX 4090上测量，fp8中的Gemma 4 12B:24 Wikinews文章约470个令牌，56个双选题
每一个，总共1 344个问题(固定装置/设置的公共工作)，通过HTTP发送到spinf-serve和vLLM 0.26
前缀缓存。 剧本和全部结果以基准测试/表示。
vLLM已经通过前缀缓存在56个问题中重复使用每个文章。 它不能做的是在一个前传中回答他们：它运行着1,344个单托请求，而spinf-serve则将56个问题作为一个读的分支来铺设，并将24篇文章装入2个16k个单托的通行证。 差距随着每个内容的提问次数增加而增加。
在实践中，每4090人使用12B型：自旋/调试包(每条电文106个问题，22个类别)
每小时得分约2万条消息；一篇新闻文章问了74个问题，每秒读近700个问题。
提取槽增加了解码工作(搜索槽费用高达200至600个记分符，生成)
大约2 000)，但读数相同。
一个工作很少是一个问题：内容节制是106个，每一个校准与正常流量
进入类别分数和决定。 即时包是一个文件，其中包含问题，内容是如何呈现的
对模型，答案是如何变成分数的，以及分数是如何变成决定的。 服务器预装包和
请求名称 1 ("packs": [{'id": "spinf/ modelation" }])，获取包的类别，并返回决定。
从同一字开始
官方的包装在spinf-promts中，默认加载。 服务器
从任何 GitHub 寄存器或本地文件夹中装入包，带有 -- packs; PACKS. md 列表
包我们知道，我们的和别人的，一个问题
新增你的。。。 格式，请求字段和来源：docs/prompt-packs.md.
每个都是一个检查站，我们量化和校准，发布在Hugging Face枢纽上
其配方(定量/)；引擎为它经过对等测试的修订版配音。 任何其他中央检查站
支持的建筑家族按其路径运行，在负载时间进行量化。
docs/：快速启动，实例，评分API，配置，写题，校准，即时包，FAQ；指南(决定模型，您可以采取行动的概率，几发分类，情绪，勘测编码，节制，意图路由)；以及关于引擎如何工作的工程文件(architecture，批量变异，精度，Qwen堆，提取).
0.1版 (中文(简体) ). 引擎(评分，提取，生成，介质，批量变异，两个型号堆栈)是完整的，并参照记录的引用进行对等测试；服务器会讲主机服务的公共API；四个检查站是公开的。 文件/README.md中列出了仍然缺失的内容。
问题、想法和错误都涉及这个仓库的问题。 没有论坛或聊天，没有什么太小。
spinf serve/持有引擎(protocol，编译器，打包机，计分器，提取器，服务)，服务型号'.
列表，包模块(纯Python，客户端和主机API也使用)，每个架构一个模型堆栈
型号/ 中的家族，型号/ 中的量化方案，服务器/ 中的 HTTP 服务器和 spinf- serve 命令。
量化/ 每个公布的检查点有一个文件夹(检索、校准输出、型号卡)、固定/公众
基准测试/通过量比较。 代码的地图是
文献/档案
uv 同步 -- 外部服务器
uv运行pytest #CPU测试(官方包的金色固定装置需要。/spinf-prompts或SPINF PROMPTS DIR).
uv运行 ruff 检查。 uv运行 ruff 格式 - 检查。
CUDA VISIBLE DEVICES=0 uv 运行自旋-服务器对等-固定装置/wikinews 20261024.json.gz \
--参考定着/wikinews 20261024. spinf-gemma-4-12b-fp8.json.gz - passes 2
触及一个内核，一个编译边界或一个具有GPU等值的量化计划舰船的每个变化运行：每个已公布的检查站(fixtures/)记录的参考文献上的位同日志概率。 依赖性是固定的；它们只随着对等运行而改变。 见贡献.md。
Apache 2.0 (英语：LICENSE, Notice). 我们运行和重新量化的模型，我们开发的项目，我们的技术
数据记入ACKNOWLEDGEMENS.md。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Bulk classification and decision inference on one GPU.
spinf-serve reads a piece of content once and answers many questions about it in the same pass. Each answer is a probability over the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the same API as the hosted spinf service. Apache 2.0.
We built this engine for our own work: a daily market index read from millions of news articles, each article asked dozens of narrow, calibrated questions. A serving engine built for chat re-reads the content for every question and generates one token at a time, which made that job slow and expensive. spinf-serve lays every question out as a branch of one read of the content and packs many contents into one forward pass. Bulk classification is where it started and where it is strongest.
The same mechanism is what is now called a decision model: a model that returns a probability over a fixed set of answers (yes or no, a label, a level) instead of a paragraph. Any open instruction model works this way in spinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are publishing it because people asked to run it on their own hardware.
It is not a chat server. Nothing is generated unless you ask for it, and it is built for throughput rather than for the latency of a single request.
Linux, an NVIDIA GPU with 24 GB and fp8 tensor cores (Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000 Ada, H100), a driver supporting CUDA 13, Python 3.12. Ampere cards run with int8 weights instead (configuration).
pip install "spinf-serve[server]"
spinf-serve serve spinf/gemma-4-12b-fp8 # downloads the checkpoint on first start; serves on port 8000
Or the image, with the NVIDIA Container Toolkit installed (the volume keeps the model and the compile caches;
arguments after the image name are those of spinf-serve serve):
docker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve
Then ask. Two classifications and one extraction from a single read of the content:
curl http://localhost:8000/v1/score \
-H "Content-Type: application/json" \
-d '{
"model": "spinf/gemma-4-12b-fp8",
"messages": [
{ "role": "user",
"content": "Customer support message:\nI was charged twice for September, please refund one of the charges." }
],
"scoring": {
"queries": [
{ "id": "topic",
"template": "\n\nTopic (billing, technical, account, shipping, or cancellation):{?}",
"options": [" billing", " technical", " account", " shipping", " cancellation"] },
{ "id": "refund",
"template": "\n\nIs the customer asking for a refund?\nAnswer:{?}",
"options": [" yes", " no"] },
{ "id": "month",
"template": "\n\nQ: Which month was charged twice?\nA: \"{*}\"" }
]
}
}'
For each scored question the response gives every option's probability, and the same probability on empty content (real numbers from the 12B model):
{ "id": "refund", "combinations": [ { "values": {}, "options": [
{ "text": " yes", "logprob": -0.90, "p": 0.72, "floor_p": 0.60 },
{ "text": " no", "logprob": -1.86, "p": 0.28, "floor_p": 0.40 } ],
"residual": 0.44 } ] }
p is the probability among your options. floor_p is the same question with no content, so p - floor_p is what
the content added (here topic moves " billing" from 0.47 to 0.91). residual is the probability that went to words
outside your options. For month the response lists the most likely answers with their probabilities ("September" first).
The same body, with an API key, runs against api.spinf.com; the OpenAI SDKs work through /v1/chat/completions with
the scoring block in extra_body (quickstart).
- One read, many questions. The content is read once; every question is a branch of the same pass and costs its own tokens only. Placeholders turn one template into a grid of questions.
- Scores you can threshold. A probability per option with its floor and log-probability; a decision is a threshold you choose on your own data (calibration).
- Extraction from the same read: {*} text, {n} numbers and {$} amounts decoded with their probabilities,
or free text generated into a slot.
- The same score whatever else is in the batch. With the fp8 checkpoints, a content's scores are bit-identical
alone or packed with thousands of others, across calls, days and servers. The system_fingerprint says when two
responses are not comparable.
- Images, audio and video frames in the content, for the models that read them.
- Prompt packs. Ready-made, calibrated question sets for a job (content moderation, triage, product listings, reviews, brand, entertainment), preloaded and named in a request (below).
- Hosted or self-hosted. A client switches between this server and api.spinf.com by changing the base URL and
the key. spinf-client talks to both; the
agent skills teach a coding agent the API, this server and the pack format.
Measured on one RTX 4090 with Gemma 4 12B in fp8: 24 Wikinews articles of about 470 tokens, 56 two-option questions
each, 1,344 questions in all (the public job set in fixtures/), sent over HTTP to spinf-serve and to vLLM 0.26 with
prefix caching. The script and the full results are in benchmarks/.
vLLM already reuses each article across its 56 questions through the prefix cache. What it cannot do is answer them in one forward pass: it runs 1,344 single-token requests, while spinf-serve lays the 56 questions out as branches of one read and packs the 24 articles into two 16k-token passes. The gap grows with the number of questions per content.
In practice, per 4090 with the 12B model: the spinf/moderation pack (106 questions per message, 22 categories)
scores about 20,000 messages per hour; a news article asked 74 questions is read at nearly 700 questions per second.
Extraction slots add decoding work (a searched slot costs about as much as 200 to 600 tokens of scoring, a generated
one up to about 2,000) but share the read.
A job is rarely one question: content moderation is 106 of them, each calibrated against normal traffic and combined
into category scores and a decision. A prompt pack is one file holding the questions, how the content is presented
to the model, how the answers turn into scores and how the scores turn into a decision. The server preloads packs and
a request names one ("packs": [{"id": "spinf/moderation"}]), getting the pack's categories and decision back next
to its own questions, from the same read.
The official packs are in spinf-prompts, loaded by default. The server
loads packs from any GitHub repository or local folder given with --packs; PACKS.md lists every
pack we know of, ours and others', and an issue
adds yours. Format, request fields and sources: docs/prompt-packs.md.
Each is a checkpoint we quantized and calibrated, published on the Hugging Face Hub
with its recipe (quantizations/); the engine pins the revision it was parity-tested on. Any other Hub checkpoint of
a supported architecture family runs by its path, quantized at load time.
docs/: quickstart, examples, scoring API, configuration, writing questions, calibration, prompt packs, FAQ; the guides (decision models, probabilities you can act on, few-shot classification, sentiment, survey coding, moderation, intent routing); and the engineering documents on how the engine works (architecture, batch invariance, precision, the Qwen stack, extraction).
Version 0.1. The engine (scoring, extraction, generation, media, batch invariance, the two model stacks) is complete and parity-tested against recorded references; the server speaks the public API of the hosted service; the four checkpoints are public. What is still missing is listed in docs/README.md.
Questions, ideas and bugs go to the issues of this repository. There is no forum or chat, and nothing is too small.
spinf_serve/ holds the engine (protocol, compiler, packer, scorer, extract, service), the served models'
manifests, the packs module (pure Python, also used by the client and the hosted API), one model stack per architecture
family in models/, the quantization schemes in quant/, the HTTP server in server/ and the spinf-serve command.
quantizations/ has one folder per published checkpoint (recipe, calibration output, model card), fixtures/ the public
parity fixture and references, benchmarks/ the throughput comparison. The map of the code is
docs/ARCHITECTURE.md.
uv sync --extra server
uv run pytest # CPU tests (the official packs' golden fixtures need ../spinf-prompts or SPINF_PROMPTS_DIR)
uv run ruff check . && uv run ruff format --check .
CUDA_VISIBLE_DEVICES=0 uv run spinf-serve parity --fixture fixtures/wikinews_20261024.json.gz \
--reference fixtures/wikinews_20261024.spinf-gemma-4-12b-fp8.json.gz --passes 2
Every change that touches a kernel, a compile boundary or a quantization scheme ships with the GPU parity run: bit-identical log-probabilities on the recorded reference of each published checkpoint (fixtures/). Dependencies are pinned exactly; they change only together with a parity run. See CONTRIBUTING.md.
Apache 2.0 (LICENSE, NOTICE). The models we run and re-quantize, the projects we build on, the techniques we
implement and the data we checked on are credited in ACKNOWLEDGEMENTS.md.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.