去年冬天一个晚上，我们的写作助手挂了 62 秒。
不是崩了，是"一直转圈"。用户点了发送，屏幕上一个光标闪了一分钟，然后弹出一句"生成失败，请重试"。
那晚服务端 5xx 高峰持续了 11 分钟。我们发了 4,200 次请求，其中 2,100 次以超时结束，平均每个用户等了 62 秒。事后看日志，前 300 次超时的时候我们就该停下来——但我们没有停，因为代码里写的是 max_retries=3，每一次都老老实实等满 60 秒超时。
重试本身不是问题，"在已知对方挂了的时候还重试"才是问题。
这篇文章讲我加的那个熔断器：三个阈值、60 行代码、上线半年拦住两次事故。
一、为什么大模型调用特别需要熔断
传统 HTTP 服务加熔断，通常是为了防雪崩。大模型调用有三个不一样的地方：
第一，单次请求特别慢。 首字延迟正常 800ms，但一旦服务端排队，能拖到 40 秒以上。超时设 60 秒是常规做法，这意味着一次失败要占用一个连接 60 秒。
第二，重试特别贵。 一次调用可能几千 token，重试三次就是三倍的钱。服务端挂的时候，你不但没拿到结果，还把钱花出去了。
第三，失败是成片出现的。 服务端抖动不是随机的，是一段一段的。你要是能识别出"这段是坏的"，跳过这一段，收益极大。
那次事故的数字很直观：
多出来的 2,060 次，全是在服务端已经明确故障之后发出去的。
二、三个阈值，缺一不可
我的实现只有一个类，三个参数：
class Breaker:
def __init__(self, window=20, fail_ratio=0.5, cooldown=30.0):
self.window = window # 只看最近 20 次的结果
self.fail_ratio = fail_ratio # 失败率超过 50% 就打开
self.cooldown = cooldown # 打开后 30 秒进半开
阈值一：窗口 = 20 次。 不建议用"连续失败 N 次"，因为大模型调用里混着用户取消、内容审核拦截这类非服务端失败，连续计数容易被个别请求带偏。用滑动窗口看比例，抗噪。
窗口太小（比如 5）会误判——一次批量任务是并发发的，5 个里有 3 个超时很正常；窗口太大（比如 200）反应太慢，200 次 × 60 秒的损失已经发生了。20 是我试出来的平衡点。
阈值二：失败率 = 50%。 这个不能设太高。日常我们的失败率在 0.3%–0.8%，50% 已经是明确异常。设 30% 会在早晚高峰误触发，设 80% 等于没设。
阈值三：冷却 = 30 秒。 打开后不再放行，30 秒后进"半开"，放 3 个探测请求过去。3 个里 2 个成功就关闭，否则重新打开。30 秒的依据是：服务端抖动通常以分钟为单位的块状出现，探测太频繁只是给已经挂掉的服务端添乱。
三、状态机就三个状态
import time
from collections import deque
class Breaker:
def __init__(self, window=20, fail_ratio=0.5, cooldown=30.0, probes=3):
self.window, self.fail_ratio = window, fail_ratio
self.cooldown, self.probes = cooldown, probes
self.results = deque(maxlen=window)
self.state = "closed"
self.opened_at = 0.0
self.probe_ok = 0
self.probe_total = 0
def allow(self) -> bool:
if self.state == "closed":
return True
if self.state == "open":
if time.monotonic() - self.opened_at >= self.cooldown:
self.state, self.probe_ok, self.probe_total = "half", 0, 0
return True
return False
# half-open：只放行有限个探测
return self.probe_total < self.probes
def record(self, ok: bool):
if self.state == "half":
self.probe_total += 1
self.probe_ok += int(ok)
if self.probe_total >= self.probes:
if self.probe_ok >= 2:
self.state, self.results = "closed", deque(maxlen=self.window)
else:
self.state, self.opened_at = "open", time.monotonic()
return
self.results.append(ok)
if len(self.results) == self.window:
ratio = 1 - sum(self.results) / len(self.results)
if ratio >= self.fail_ratio:
self.state, self.opened_at = "open", time.monotonic()
调用处的改动很小：
from openai import OpenAI
import os
client = OpenAI(
api_key=os.environ["API_KEY"],
base_url="https://api.tokenfreeai.com/v1", # OpenAI 兼容
timeout=60, max_retries=0, # 重试交给熔断器判断
)
breaker = Breaker()
def ask(prompt: str) -> str | None:
if not breaker.allow():
return None # 快速失败，别让用户干等
try:
r = client.chat.completions.create(
model="glm-5",
messages=[{"role": "user", "content": prompt}],
)
except Exception:
breaker.record(False)
raise
breaker.record(True)
return r.choices[0].message.content
注意两件事：max_retries=0 必须写，否则 SDK 自带的退避重试会绕过熔断器；返回 None 的时候上层要能接住，这一步比熔断逻辑本身更容易漏。
四、返回 None 之后干什么
这是很多人漏掉的一半。熔断的价值不在"拦"，在"拦了之后给什么"。
我们的三级降级：
- 换模型。 主力模型熔断后，切到同厂的轻量版，输出质量掉一档但能用。实测切换后用户投诉量没有明显上升，因为大部分请求是润色、起标题这类任务。
- 排队。 非实时任务（夜间批处理、报告生成）进延迟队列，5 分钟后重试，用户端显示"处理中，完成后通知"。
- 明说。 实时对话类如果两样都不行，直接告诉用户"当前服务繁忙"，而不是让他等一分钟再看失败。
第三条最难做，但最重要。 那个 62 秒的晚上，用户真正愤怒的不是失败，是"你不知道它会不会成功"。
五、上线半年的两次拦截
第一次是四月份，服务端区域性故障，熔断在 38 秒后打开，拦掉了 1,800 多次请求。用户平均等待从预估的 50 秒降到 1.1 秒。
第二次更隐蔽：我们自己的一个脚本死循环在重试，一小时打了 9,000 次调用。熔断器本身没拦（因为服务端是好的），但同一套状态机加了个"每小时调用量超阈值就打开"的旁路，把它挡住了——那次省了大约 380 元。
六、一个坑：别把熔断器做成分区无关的
我们刚开始只有一个全局熔断器。后来发现一个问题：主力模型熔断后，所有请求都被挡在外面，包括本来可以走备用模型的那部分。
后来改成按"模型 + 区域"分桶。每个桶一个熔断器，桶之间互不影响。这个改动只多了三行（用 dict 键存实例），但可用性提升比熔断器本身还大。
如果你的调用是走 OpenAI 兼容协议的，这件事做起来很省——同一份客户端代码，换个 model 字符串就是一个新桶，不用为每个厂商写一套适配。我们这边用的是词元云，一个密钥管着几个模型，横评和切桶都在同一个客户端里；不过你直接用各家的官方接口也完全能跑，熔断逻辑是一样的。
七、要不要上现成的
有现成的库，pybreaker、tenacity 都行。我最后自己写，原因是三个阈值的语义要按大模型调用的特点调（窗口用次数不用时间、失败判据要能排除内容审核拦截），现成库把这层封得比较死。
如果你只是想要个能跑的，用现成的完全不丢人。这 60 行真正值钱的不是代码，是那三个数字背后的判断。
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
One night last winter, our writing assistant hung up 62 seconds.
It's not a fall. It's a circle. The user called for a dispatch, a cursor on the screen flashed for a minute, and then a line "Generating failed, please retry".
The service peak of 5xx that night lasted 11 minutes. We sent 4,200 requests, 2,100 of which ended with timeout, and averaged 62 seconds per user. Then look at the log, 300 times before timeout, we should stop -- but we didn't stop because the code says max retries = 3, each time it's 60 seconds timeout.
retry is not a problem in itself, but it's "retry when the other party is known to be dead".
This article says that I added the smelter: three thresholds, 60 line codes, two accidents over six months.
I. Why LLM calls especially need to melt
Traditional HTTP services are melted, usually to prevent avalanches. LLM calls in three different places:
First, single requests are particularly slow. The initial word is delayed to a normal 800 ms, but can be delayed to more than 40 seconds once the service end is queued. timeout sets 60 seconds, which means a failure takes 60 seconds of a connection.
Second, retry is particularly expensive. One call could be thousands of tokens, and retry three times the money. When the server hung up, you didn't get the results, you spent the money.
Thirdly, failure comes in pieces. Service vibrating is not random, it is a period. If you can identify "this part is bad" and skip this part, it's good.
The number of accidents was intuitive:
The additional 2,060 times were sent after the service had clearly failed.
II. Three thresholds, not one
I have achieved only one category with three parameters:
class Breaker:
def __init__(self, window=20, fail_ratio=0.5, cooldown=30.0):
# Just look at the latest 20 times #
Self. fail ratio = fail ratio
# When it's open, 30 seconds and a half Open
Threshold I: Window = 20 times. The term "continuous failure N" is not recommended, since the LLM calls are mixed with user cancellations, content audits fail to intercept such non-services, and consecutive numbers are vulnerable to bias by individual requests. Use the slide window to see scale, anti-noise.
The window is too small (e.g. 5) to be miscalculated - a one-volume assignment is issued, and three of the five timeouts are normal; the window is too slow (e.g. 200) to react, and 200 times x 60 seconds has been lost. Twenty is the balance I tried out.
Threshold II: failure rate = 50%. This can't be too high. Our daily failure rate is 0.3-0.8 per cent, 50 per cent is clearly abnormal. 30% will be triggered by early and late peaks, 80% equals nothing.
Threshold three: cooling = 30 seconds. No more release after opening, 30 seconds after half-open, three detection requests past. Two out of three are closed or reopened. 30 seconds are based on the fact that the service vibrating usually takes place in a block form in minutes and that detection is too frequent only to cause disruption to the already hung service.
Three, three.
import time
from collections import deque
class Breaker:
def __init__(self, window=20, fail_ratio=0.5, cooldown=30.0, probes=3):
self.window, self.fail_ratio = window, fail_ratio
self.cooldown, self.probes = cooldown, probes
self.results = deque(maxlen=window)
self.state = "closed"
self.opened_at = 0.0
self.probe_ok = 0
self.probe_total = 0
def allow(self) -> bool:
if self.state == "closed":
return True
if self.state == "open":
if time.monotonic() - self.opened_at >= self.cooldown:
self.state, self.probe_ok, self.probe_total = "half", 0, 0
return True
return False
# half-open: only limited detections released
return self.probe_total < self.probes
def record(self, ok: bool):
if self.state == "half":
self.probe_total += 1
self.probe_ok += int(ok)
if self.probe_total >= self.probes:
if self.probe_ok >= 2:
self.state, self.results = "closed", deque(maxlen=self.window)
else:
self.state, self.opened_at = "open", time.monotonic()
return
self.results.append(ok)
if len(self.results) == self.window:
ratio = 1 - sum(self.results) / len(self.results)
if ratio >= self.fail_ratio:
self.state, self.opened_at = "open", time.monotonic()
Very few changes have been made:
from openai import OpenAI
import os
client = OpenAI(
api_key=os.environ["API_KEY"],
Base url= "https://api.tokenfreeai.com/v1", #Open AI compatible
Timeout = 60, max retries = 0, retry to be judged by the breaker
)
breaker = Breaker()
def ask(prompt: str) -> str | None:
if not breaker.allow():
Quick failure, don't keep the user waiting.
try:
r = client.chat.completions.create(
model="glm-5",
messages=[{"role": "user", "content": prompt}],
)
except Exception:
breaker.record(False)
raise
breaker.record(True)
return r.choices[0].message.content
Note two things: max retries=0 must be written, otherwise SDK's own retreat from retry will bypass the smelter; the upper layer will be able to catch it when you return to None, a step that is easier to escape than the smelting logic itself.
IV. BACK TO NONE
That's half what many people missed. The value of melting is not "stop," but "what after stop."
Our three-tier downgrade:
- Change the model. When the main force model melts, it cuts to a light version of the same plant, and the output quality falls off the shelf but it works. The number of user complaints did not increase significantly as the majority of requests were for tasks such as decorating and heading.
- Line up. Non-real-time tasks (night batching, report generation) in delayed queue, 5 minutes after retry, the user shows "processing, notification after completion".
- Exactly. The real-time conversation class, if it doesn't work on either, just tell the user that "current service is busy" instead of letting him wait a minute to see the failure.
The third is the hardest, but the most important. That 62-second night, the user's real anger wasn't failure, it was "You don't know if it's gonna work."
Five, two intercepts in six months.
The first, in April, was a service-end regional malfunction, which opened in 38 seconds, blocking more than 1,800 requests. The average user waits down from an estimated 50 seconds to 1.1 seconds.
For the second time, one of our own scripts is dead in retry, 9,000 calls an hour. The smelter itself was unstoppable (because the service end was good), but the same set-of-state set added a sideway "opens the ultra-threshold per hour" to block it -- saved about $380.
VI. A pit: Do not make the melting machine a component area.
We've just started with only one global melting machine. A problem was later identified: when the main power model melted, all requests were blocked outside, including the part of the alternative model that could have been taken.
It was then replaced by a "Model + Area" sub-barrel. One smelt per barrel, which does not affect each other. This change is only three rows longer (example with dict keys), but the usability is greater than the smelter itself.
If your call is on the Open AI compatibility protocol, it's a lot easier to do -- the same client code, a model string is a new barrel, not a fit for each manufacturer. We use a word cloud, a key that runs several models, cross-views and cut buckets in the same client; but you can run directly with the official interfaces of each home, with the same melting logic.
Do you want something ready?
There's a ready-made library, pybreaker, tenacity. I ended up writing on my own because the syntax of the three thresholds had to be adapted to the LLM (windows are not time-consuming, failure criteria exclude content-checking interception), and the ready-to-assembly library covered the floor.
If you just want someone who can run, it's no shame to use the ready. It's not the code that's really worth the 60 lines, it's the judgment behind the three numbers.
Auto-fetched and translated by AI Atlas. All rights belong to the original author.