故障切换是一把演练过的开关
备用节点只有在你既知道怎么切换、又确认切换有效时才有用。故障切换是一个决定、一套流程、一次演练。交给一把你从未动过的开关,是让一次优雅交接演变成宕机的原因。 提前设计好决定 在一切正常的时候,先定清楚:谁有权按开关、可接受的丢数据范围是多大、故障转移是自动还是需要人工拍板。然后趁冷静把 runbook 写下来:精确到某条命令或某个控制台步骤,确认新主库追上进度的检查,以及该向谁呼叫的通道。自动切换提速快,但必须把「脑裂」场景想透——两个都以为自己是主库的写者,损坏的状态比一个降级的单主库原地出错多得多。目标是不让中间的操作员在故障时现场发明决策,只需要执行决定已经定好的那套动作。 开关必须被测试 要在故障替你动手之前,主动按一次开关。每季度主动把流量从主库真实切走再按正确流程切回来,远比一叠架构图有用,因为它一次性验证了 DNS、证书、配置和回放追赶路径。每次演练先立好基准、确认备份、备好回滚,再开始。演练时的失败是礼物而不是挫折——它在我们选好的时间点暴露了缺的权限和过时的脚本。把耗时记下来、留存结果,让「切换能多快」有季度级的数据可对比,而不是被一句「差不多」盖过去。 不规划的代价 没有彩排的故障切换,所有代价都落在最坏的时间。第一步会先在记不全的步骤里烧掉几分钟,还要开小会去争论现在由谁拍板。而一个没练过的主库在关键时刻倒下,很可能把一次小抖动抬成丢失写入事故。纪律还会悄悄腐蚀:配置在变,角落里那套从没运行过的步骤,恰恰是最后时刻最先出问题的那一个。而切换只有在真实成功过一次之后才算数,所以关于「准备好了」的唯一诚实报告,就是最近一次演练成功的日期。 坏掉的样子 没有彩排的切换会在五个地方摔跟头。最危险的是脑裂双写：旧主库还在接受写入，备库却自行提升为主，两个主库各自向前，分叉后的数据谁都不全，事后几乎无法合并，唯一的解法是先隔离旧主再提升新主，并断言全程只有一个主库。副本滞后切换更安静：备库落后几分钟，切换显示成功，丢掉的写入要到第二天早上才以客诉的形式浮现，所以手册必须写明滞后阈值，不满足阈值就拒绝切换，而不是凭感觉放行。自动切换会抖动：健康检查在临界值上来回跳，今天升甲明天升乙，来回几次两边副本都不可信，客户端看到前后不一致的版本，没有冷却与人工确认的自动切换等于把开关交给抖动。域名切换会粘滞：解析器和客户端按旧生存时间缓存，记录改了之后仍有一部分流量继续打到已经死亡的地址，预案要提前调小生存时间，并按旧值估算残留流量。最后是回切冲突：修好的旧主带着更新的时间戳回来，不加保护直接回切就会覆盖故障期间已接受的写入，回切前必须先只读校验差异，确认追平再动手。五种失败指向同一组护栏：先隔离、再查滞后、全程只允许一个写者。 快速验证 把下面代码存为 failover_drill.py，在临时目录运行 python3 failover_drill.py。它只在内存里建一主一备的一对节点，写入真正的健康检查与滞后阈值，不改任何生产状态。四次拒绝都是反例对照，不是推荐操作，用来证明护栏真的会拦。 """Failover drill: single-primary switch with health and lag guards (stdlib only).""" from dataclasses import dataclass MAX_LAG = 128 class FailoverRefused(Exception): pass @dataclass class Node: name: str healthy: bool = True lag: int = 0 is_primary: bool = False class Cluster: def __init__(self): self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")} self.fenced: set[str] = set() @property def primary(self) -> str: primaries = [n for n in self.nodes.values() if n.is_primary] assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}" return primaries[0].name def health_check(self, name: str) -> bool: node = self.nodes[name] return node.healthy and node.lag <= MAX_LAG def fence(self, name: str) -> None: self.fenced.add(name) def failover(self, target: str, *, fence_old: bool) -> str: old = self.primary if target == old: raise FailoverRefused("target is already primary") if target not in self.nodes: raise FailoverRefused(f"unknown target {target}") candidate = self.nodes[target] if not fence_old or old not in self.fenced: raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain") if not candidate.healthy: raise FailoverRefused(f"refuse: standby {target} is unhealthy") if candidate.lag > MAX_LAG: raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})") self.nodes[old].is_primary = False candidate.is_primary = True assert self.primary == target return target def expect_refused(fn, label: str) -> None: try: fn() except FailoverRefused as exc: print(f"refused as expected ({label}): {exc}") return raise AssertionError(f"should have refused: {label}") def main() -> None: # 1. Clean drill: healthy standby, caught-up replica, old primary fenced. cluster = Cluster() assert cluster.primary == "a" assert cluster.health_check("b") cluster.fence("a") assert cluster.failover("b", fence_old=True) == "b" print("clean switch: a -> b") # 2. Stale replica must not promote: switching would drop writes. cluster = Cluster() cluster.nodes["b"].lag = 900 assert not cluster.health_check("b") cluster.fence("a") expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica") assert cluster.primary == "a" # 3. Unhealthy standby must not promote. cluster.nodes["b"].lag = 0 cluster.nodes["b"].healthy = False expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby") assert cluster.primary == "a" # 4. Unfenced old primary must block promotion (split-brain guard). cluster.nodes["b"].healthy = True unfenced = Cluster() # nobody fenced yet expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain") expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record") assert unfenced.primary == "a" # 5. Failback needs the same guards: fence the failed node before returning. cluster = Cluster() cluster.fence("a") cluster.failover("b", fence_old=True) cluster.nodes["a"].healthy = True cluster.nodes["a"].lag = 0 cluster.fence("b") assert cluster.failover("a", fence_old=True) == "a" print("failback: b -> a") print("PASS: single primary held; 4 refusals + clean switch + failback behaved as expected") if __name__ == "__main__": main() 演练共有五步：一次干净切换、三类拒绝、一次受保护的回切。干净切换只在旧主已隔离且备库健康追平时才提升，并用单主断言守住不变量；滞后副本与不健康备库的请求都会被拒绝，主库保持不动，避免丢写入；未隔离旧主的提升会被直接拒绝，这正是防脑裂的关键；回切沿反方向重复同一规则，先隔离再切换。预期输出以 PASS 结尾，并同时打印干净切换与回切的行，以及四行拒绝说明，看到这几行就说明护栏全部生效。 ...
