mux WebSocket 的心跳判死是静默的：漏 2 次 Pong 即 terminate()（无日志、无指标、无 Close 帧），而判死判据与它要检测的拥塞共用同一条连接

① 容忍窗口只有 6.0 秒：漏 2 次 Pong，第 3 跳判死

默认 websocketHeartbeatIntervalMs = 2000；MAX_MISSED_HEARTBEATS = 2。多轮独立实测（同一个零依赖原生 socket 探针，另有一轮 ws 客户端交叉复现，判词格式不同未收录原文）都落在 6000–6005 ms（验证轮 6002 / 6001 / 6001 / 6002 ms），即「最后一次被回答的 Pong → socket 被拆」为三跳 2 秒；一轮负载抖动实测 6279 ms（+4.6%）——窗口是「≥ 三跳」，tick 迟到会等比拉长。

② 判死是静默的：不发 Close 帧、不写日志、不打指标

判死分支里只有 socket.terminate()（直接销毁 TCP）。实测 close_code=none：连 Close 帧都没有，客户端只能看到 1006 空 reason，无从区分「Host 心跳判死」与「网线掉了」。Host 侧同样没有任何一行日志或指标——这直接解释了为什么用户在看「自动重连中」时，我们的看板是全绿的。

③ 判据与被判对象共用同一条连接（机制清楚，现场未证）

RemoteStreamMuxConnection.send() 把所有逻辑流的每一帧串到同一条 this.writes 链上，只等 write 回调，从不读 bufferedAmount；而 socket.ping() 写的是同一个 socket。TCP 有序：Ping 会排在已排队的数据帧之后。另一侧，浏览器在背压下会暂停读 socket——连 Ping 都读不到，自然不回 Pong。也就是说：会话越多 → 事件越多 → 发送缓冲越积压 → 越容易吃满 6 秒窗口 → 被判死 → 重连后又整批重拉。这一条我们只有代码级证据与机制推理，没有抓到「生产实例拥塞期间被杀」的现场（缺 Host 侧计数）——正好是建议 (a) 要补的观测。

原始输出（零依赖探针，两次独立运行）

     1 upgraded (HTTP/1.1 101 Switching Protocols)
   272 <- Ping #1 -> Pong
  2271 <- Ping #2 (gap 1999ms) -> Pong
  4271 <- Ping #3 (gap 2000ms) -> Pong
  6271 <- Ping #4 (gap 2000ms) -> Pong
  8004 --- muting Pong (socket keeps reading) ---
  8275 <- Ping #5 (gap 2004ms) [muted: no Pong sent]
 10275 <- Ping #6 (gap 2000ms) [muted: no Pong sent]
 12277 socket closed (no Close frame)
HEARTBEAT_TOLERANCE pings=6 median_gap_ms=2000 last_pong_to_close_ms=6005 since_mute_ms=4273 close_code=none text_frames=0 result=KILLED

     1 upgraded (HTTP/1.1 101 Switching Protocols)
  1084 <- Ping #1 -> Pong
  3084 <- Ping #2 (gap 2000ms) -> Pong
  5002 --- muting Pong (socket keeps reading) ---
  5084 <- Ping #3 (gap 2000ms) [muted: no Pong sent]
  7084 <- Ping #4 (gap 2000ms) [muted: no Pong sent]
  9085 socket closed (no Close frame)
HEARTBEAT_TOLERANCE pings=4 median_gap_ms=2000 last_pong_to_close_ms=6000 since_mute_ms=4083 close_code=none text_frames=0 result=KILLED

第二次的静音点恰好落在 Ping 前 82 ms（5002 vs 5084），仍得到 6000 ms——说明窗口锚在最后一次真正的 Pong，不是我们的动作时刻。

# 本机某一户：读 $DSH_HOME/web-url.txt 的启动令牌自动换 Cookie
DSH_HOME=<home>/.dsh node ops/dsh/upstream-issue-ws-heartbeat/heartbeat-tolerance-probe.mjs --port <dsh-web-port>

# 判据行（期望 KILLED，窗口 4–6.5 s，close_code=none）
# HEARTBEAT_TOLERANCE pings=6 median_gap_ms=2000 last_pong_to_close_ms=6005 since_mute_ms=4273 close_code=none text_frames=0 result=KILLED

部署侧现有绕法（不是修复，是止痛）：把 websocketHeartbeatIntervalMs 调大。该键在 base bundle 的 typert-gateway 行（@deepseek-ai/dsh-api-gateway）的 config 上，写进该户 $DSH_HOME/cordis.patch.yml 后需重启该户服务（会打断在跑会话）。调大 = 容忍窗口变长，代价是真死连接发现更晚——这也是建议 (b) 比 (c) 更好的原因。

Title: WebSocket heartbeat kills a healthy socket silently: 2 s Ping, two
       missed Pongs, terminate() with no log / metric / Close frame

Summary
-------
The Web mux carrier (/api/remote.mux) pings every `websocketHeartbeatIntervalMs`
(default 2000 ms) and destroys the socket once two consecutive Pings went
unanswered. Measured tolerance is about 6.0 s from the last answered Pong
(6000-6005 ms across four runs; one loaded round stretched to 6279 ms, since
latent ticks lengthen the window), and the
termination is completely silent: `socket.terminate()` sends no Close frame and
emits no log line or metric, so the browser only sees 1006 with an empty reason.
The client then invalidates its connection generation, shows the "Reconnecting"
indicator, and refetches everything on recovery (`connection/reset`).

The same socket carries every multiplexed session stream. The mux write path
serialises all streams onto one promise chain and never inspects
`socket.bufferedAmount`, while `socket.ping()` writes to that same TCP
connection. TCP ordering means a Ping queues behind whatever data is already
buffered, and a browser under backpressure stops reading the socket altogether —
so congestion can delay the very Pong that the liveness check is waiting for.

Environment
-----------
@deepseek-ai/dsh 0.1.5-rc.1, Node v24.16.0, Linux. Reproduced on
ws://127.0.0.1:4006/api/remote.mux (loopback, no reverse proxy in the path) with
a valid browser-session cookie, on 2026-09-13 ~22:00 +0800.

Steps to reproduce
------------------
  1. Open the mux WebSocket (any authenticated client) and answer Pings with
     Pongs. Confirm Ping interval is 2000 ms and the socket stays open.
  2. Keep reading the socket but stop answering Pings.
  3. Observe: after two missed Pings the Host destroys the TCP connection.

Reproduction is dependency-free (raw WebSocket over node:net) in
`heartbeat-tolerance-probe.mjs`:

  $ node heartbeat-tolerance-probe.mjs --port <dsh-web-port>
  HEARTBEAT_TOLERANCE pings=6 median_gap_ms=2000 last_pong_to_close_ms=6005
    since_mute_ms=4273 close_code=none text_frames=0 result=KILLED

Four independent runs: last_pong_to_close_ms = 6005 / 6000 / 6002 / 6001 ms
(three 2 s ticks; a loaded round measured 6279 ms). `close_code=none` means no
Close frame was sent.

Expected behaviour
------------------
  * A liveness check should distinguish "the peer is gone" from "we are
    congested". While the socket is demonstrably making progress (write
    callbacks completing, `bufferedAmount` decreasing), a late Pong should not
    be treated as death.
  * A Host-initiated close should be observable: one log line plus a metric
    carrying missed count, interval, `bufferedAmount` and time since the last
    Pong.
  * A Host-initiated close should carry a reason, e.g.
    `close(1011, 'heartbeat timeout')`, so client and operators can tell it
    apart from a dropped connection. Note the same file already closes with a
    reason on the error path (`close(1011, 'Remote stream failure could not be
    delivered')`).
  * 2 s should not be the default for a connection that carries every session
    stream; and the knob should be documented in the config catalog instead of
    only in the package README's "known limitations".

Actual behaviour
----------------
  * Two missed Pongs (third tick) = terminate, measured 6.0 s after the last
    answered Pong.
  * `socket.terminate()`: no log, no metric, no Close frame. The client sees
    1006 with an empty reason and cannot distinguish heartbeat-kill from network
    loss. Our dashboards stayed green while users watched "Reconnecting" flaps.
  * Client-side consequences: `connection.state = connecting` (indicator
    `connection.connecting` = "自动重连中" / "Reconnecting"), console warning
    `[connection] connection lost, retry #N`, then backoff 500 ms … 10 s; on
    success `emit("connection/reset")` triggers a full refetch (session list,
    per-session page/follow catch-up, models/settings/tools/skills/workspace).
    On a deployment where `session/list` already costs 9.7-11.9 s for ~2 957
    rows, each heartbeat kill is expensive, and more concurrent sessions make
    the next kill more likely — a self-reinforcing loop.

Ruled out (all tested)
----------------------
  * Network middleboxes / proxy timeouts: reproduced on loopback with no proxy.
  * Browser behaviour: reproduced with a raw socket client.
  * Host CPU / blocked event loop: the mux heartbeat timer stayed punctual
    during the kill (2000.0 ms idle; 2085 ms under 16 concurrent session/list
    calls). One socket dies, not the loop.
  * Auth / Host-Origin trust: a valid cookie was used; failures there look like
    a failed handshake, not this.
  * Client-initiated close: the probe only read; it never sent Close.
  * The `disconnected` state is a different path (browser `offline` event).

Not yet proven
--------------
What share of real-world reconnects on a busy multi-session deployment are
heartbeat kills rather than network loss. The mechanism and the window are
measured; the production rate needs the log/metric from the first suggested
fix.

Suggested fixes
---------------
  1. Observability (smallest, highest value): log + metric on heartbeat
     termination (socketId, missed, intervalMs, bufferedAmount, lastPongAgoMs),
     and `close(1011, 'heartbeat timeout')` instead of a bare `terminate()`.
  2. Predicate: treat a socket that is still flushing as alive — skip the kill
     and reset the counter while writes keep completing / `bufferedAmount`
     decreases; or at minimum require an elapsed-time window (≥3×interval)
     rather than "two missed".
  3. Defaults/docs: raise the default to 10-15 s for this carrier and register
     `websocketHeartbeatIntervalMs` in the config catalog.
  4. Longer term: separate the control plane (heartbeat, cancels, small RPC
     frames) from the bulk data plane, or at least prioritise control frames
     and bound the write queue (`bufferedAmount` cap + coalescing), so the
     liveness signal is not polluted by the congestion it guards.

Code references (0.1.5-rc.1, shipped lib/)
------------------------------------------
  * dsh-api-gateway/lib/types/stream-server.js:4   MAX_MISSED_HEARTBEATS = 2
  * dsh-api-gateway/lib/types/stream-server.js:61-77  heartbeat loop; missed
    >= 2 => setImmediate(() => socket.terminate()); :75 socket.ping()
  * dsh-api-gateway/lib/types/stream-server.js:152  a reasoned close already
exists on the last-resort path (close(1011, ...) when even the error frame
cannot be delivered) — precedent for doing the same on a heartbeat timeout
    (ordinary stream errors use the {type:'error'} frame at :147)
  * dsh-api-gateway/lib/types/stream-server.js:157-179  single serialised write
    chain, no bufferedAmount check
  * dsh-api-gateway/lib/index.js:398, :433, :458  default 2000 ms, Config
    schema, wiring into RemoteStreamMuxServer
  * dsh-api-gateway/lib/types/index.js:76  wiring into RemoteStreamMuxServer
  * dsh-client-connection/lib/client.js:943, :954, :802-804, :907-909
    connecting state, warning, backoff defaults
  * dsh-client-ui-settings-general/lib/client.js:491  the "自动重连中" label key
  * dsh-api-gateway/lib/client.js:1433  onConnected => emit("connection/reset")