会话列表接口 session/list 永不返回：一元 RPC 上零字节挂起 ≤300 秒，而同一实例的兄弟接口 100 毫秒就回

三条硬事实

用大白话说：DSH 网页左边那栏"历史会话列表"，是靠一个叫 session/list 的接口填的。接口不返回，列表就永远是空的 —— 界面看起来"登录成功了但里面什么都没有"，因为前端连"我有几个会话"都不知道。这和你的 token、跟会话文件有没有坏，都没关系。

影响面与优先级

• P0 用户可感知且不可绕过 会话列表是网页入口面；此接口不返回，整个 Web 界面对人不可用，而进程活着、鉴权正常、其它接口全通，监控看不出任何异常。
• 失败是静默且彻底的 没有错误体、没有 4xx/5xx、没有超时、没有部分结果。客户端既拿不到错误也没有东西可重试，且它分不清「这个工作区本来是空的」和「服务端挂了」。（这也是我们上一轮把它误判成"前端没选工作区"的原因，见第 6 节的诚实交代。）
• 成本对监控不可见 这条请求自身归因到的 CPU ≈ 0，read_bytes ≈ 0（全打 page cache），写入 socket 0 字节。任何按 CPU / IO 告警的看板都会报「健康」。
• 不会自愈、且随数据量单调恶化 跨浏览器重启、低负载窗口、300 秒长预算都一样。而这条路径没有内存缓存也没有 TTL，每次首屏/每次切工作区都要重新走一遍全部会话目录；上游也明说没有任何东西会删会话文件（dsh-session-persistence-jsonl/README.md 已知限制原文 "Nothing deletes session files"）。这台机器 7 天就长到约 2308 个会话 / 1006 MiB（约 526 个/天）。

1. 方法：不重启现场，用 SIGUSR1 进 inspector 取 profile

node ops/dsh/profile-dsh-node.mjs --pid <PID> --seconds 30 --out <file>.cpuprofile
node ops/dsh/profile-dsh-node.mjs --analyze <file>.cpuprofile --top 6

# 尺子先自证（阴性对照：拿一个真烧 CPU 的子进程，必须看到它的函数名）
node ops/dsh/profile-dsh-node.mjs --selftest
  → PROFILE_SELFTEST verdict=PASS samples=5248 top=runScriptInThisContext:97%,(garbage collector):3%

3. 第二个独立缺陷：每次起子进程都阻塞地探一次 systemd scope

// 调用链（profile 实测，自底向上）
spawn  ← spawnSync  ← probeLinuxScope@dsh-subprocess-local/lib/index.js:73
       ← selectContainmentMode@index.js:1384
         ← apply@index.js:118  ← spawn@index.js:1356  ← executeArgv@index.js:157

function probeLinuxScope(internals = {}) {                 // :73
  const unitBase = unitStem("dsh-subprocess-probe");
  const result = (internals.spawnSync ?? spawnSync)(internals.systemdRun ?? "systemd-run", [
    "--user", "--scope", "--quiet", "--collect", "--expand-environment=no",
    `--unit=${unitBase}`,
    "--", internals.systemctl ?? "systemctl", "--user",
    "show", `${unitBase}.scope`, "--property=ActiveState", "--value"
  ], { env: quietSystemdEnvironment(), stdio: "ignore",
       timeout: SYSTEMCTL_TIMEOUT_MS });                   // ← 同步等一轮 systemd 往返
  ...
}

selectContainmentMode(kind) {                              // :1384
  if (platform === "linux") {
    const available = this.linuxDeepProbePassed ? probeLinuxManager() : probeLinuxNative();
    if (available) this.linuxDeepProbePassed = true;       // ← 有缓存位……
    ...
}

标题

`session/list` never responds: zero bytes and no response headers for up to 300 s on a plain unary RPC, while sibling RPCs on the same instance answer in ~100 ms

Environment

Host: [host], 256 logical cores, Linux. loadavg 1/5/15 = 21.80 / 20.11 / 24.26 at 2026-09-08 08:42:21 +0800
      (several unrelated Python pytest/parity jobs at 100-510% CPU; the box is ~73% idle).

Server: node [app-dir]/app/node_modules/.bin/dsh --profile infra --no-open           --host 127.0.0.1 --port 4002 --trusted-host <host>
        PID 29582, started 2026-09-07 16:21:55 +0800, uptime ~16.3 h, systemd unit dsh-user-d,
        NRestarts=0, 12 threads, cumulative %CPU 74.3 (~0.74 cores lifetime),
        utime 2,034,878 / stime 2,351,730 jiffies (sys > user).
        RSS = VmHWM = 4.43 GiB, Swap 0, majflt = 1;
        no --max-old-space-size / --max-heap flag is set.

Packages: [app-dir]/app/node_modules/@deepseek-ai/*   (node v24.16.0)
DSH_HOME: /home/user-d/.dsh

Corpus at first measurement: ~2,306-2,311 session directories, ~1.0 GiB on disk
        (session.jsonl.zstd), 2,240 projection-cache records, 3 project directories.
        storages/workspace.json registers 1 workspace ([repo], title infra)
        with 82 sessionIds and 7 archivedSessionIds.

Auth:   launch token rotates on each restart, read from [实例目录]/web-url.txt;
        GET /?token=<tok> -> HTTP 200 in 0.488 s, sets HttpOnly cookie dsh-auth-<...> (len 171).
        All RPCs below reuse that cookie. There is no Authorization header.

Flags:  coldBlankProbeMaxBytes left at the default 1024 (dsh-base does not override it);
        projection-cache write-behind: writeEveryEvents 200, writeIntervalMs 5000.

Steps to reproduce

# 1. Get a session cookie (token rotates on restart)
TOK=$(sudo -n -u user-k sudo -n grep -o "token=[^&]*" [实例目录]/web-url.txt | cut -d= -f2)
J=/tmp/jar_$$.txt
curl -sS -m 8 -c $J -o /dev/null -w "%{http_code}\n" "http://127.0.0.1:4002/?token=$TOK"    # -> 303

# 2. Send the byte-exact envelope the shipped client sends (131 B, captured from a real
#    browser navigation - no New Session click, i.e. a plain page load)
curl -sS -m 60 -b $J -H 'Content-Type: application/json' \
  --data '{"type":"client-request","rpcId":"32f2434f-e6bc-49f2-be3b-8c649ba43d74",
           "method":"session/list","payload":{"args":{"_request":{}}}}' \
  -w '\ncode=%{http_code} total=%{time_total} first=%{time_starttransfer} size=%{size_download}\n' \
  http://127.0.0.1:4002/api/session/list
# -> code=000 total=60.01 first=0.000000 size=0     (curl exit 28)

# 3. Controls, identical cookie and transport (all 200, transfer-encoding chunked,
#    content-encoding gzip, body fully read 115-197 ms after the first byte):
curl ... --data '{"type":"client-request","rpcId":"c1","method":"session/modelCatalog",
                  "payload":{"args":{}}}' ... /api/session/modelCatalog
#    -> 200   0.033-0.238 s   1728 B   (4/4, plus 2/2 in a low-load window)
#    settings/describe            -> 200  0.049 / 0.108  26961 B
#    session/canOpenWorkspacePath -> 200  0.040 / 0.045    283 B
#    agentPresets/list (also walks the filesystem) -> 200  4.434-5.951 s  1288 B

# 4. THE KEY MEASUREMENT: issue step 2 again, and while it is in flight keep hitting a
#    sibling endpoint. The event loop is demonstrably alive.
/home/user-l/seat5/seat5_critpath.py
# list                      -> TimeoutError, total 30.0302 s, first_byte None, bytes 0
# modelCatalog while stalled -> 8/8 HTTP 200, first-byte 0.090 / 0.098 / 0.111 / 0.119 /
#                              0.143 / 0.147 / 0.166 / 1.122 s (median 0.131), 1733-1734 B
# idle baseline (x3)         -> 0.097 / 0.117 / 0.099 s
# MainThread CPU, whole 30 s window -> 25.17 s = 0.838 cores
#   of which the 12.37 s sub-window with the request in flight -> 9.99 s = 0.807 cores
# => request-attributable CPU ~ 0

# 5. Load ablation already performed: identical zero-byte behaviour at loadavg(1m)
#    10.89, 13.94, 14.89, 15.11, 16.07, 16.40, 18.50, 20.39, 34.50  (9 readings),
#    plus a 300 s-budget run (300.312 s, 0 bytes). No config change, no process restart,
#    ever on the same PID 29582.

# 6. Corpus census used above (read-only):
#    files_total 2308, session.jsonl.zstd, dir 1006.5 MiB, uncompressed 2.77 GB,
#    projcache 2240 records at 08:44, workspace.json 1 workspace / 82 sessionIds / 7 archived.

Expected behaviour

session/list is declared as a plain unary RPC: the generated descriptor (dsh-api-session-controller/lib/typert.host.js:873-905, source packages/api/session-controller/src/index.ts:214:9) has invocation: { kind: "direct" } and no mode: "stream"; its single wire parameter is _request (z.object({ cursor: z.string().optional() })); dsh-api-gateway/lib/index.js:538-548 therefore does await Reflect.apply(...) and serialises one JSON response.

Given (a) the documented design intent that listing is a light path — dsh-session-persistence-jsonl/README.md:93 "Listing reads and validates only the header frame." and dsh-session-query/README.md:145 "listSessions stays lightweight"; (b) that the projection columns can be served from memory with zero I/O (dsh-session-projection-cache/README.md:61); and (c) that an independent out-of-process replay of listArtifacts over this exact corpus (3 project dirs / 2,306 session dirs, 4,612 open + 2,306 8 KiB reads + 2,306 zstd header frames) measures 0.92-1.00 s wall / 1.16-1.23 s CPU, and an independent header-only scan measures 0.171 s serial / 0.092 s at 8-way over 2,308 files, the endpoint should return a single { items: SessionSummary[] } within a few seconds even with ~2,300 accumulated sessions, and should emit its response headers as soon as the handler resolves. If something is genuinely slow or broken, the expected contract is a bounded failure (SESSION_QUERY_PERSISTENCE_FAILED) or a partial result — not silence.

Actual behaviour

The endpoint stalls before the Response object exists, so not even response headers are emitted; the socket carries zero bytes for the whole budget. Measured 10/10 times between 08:26 and 08:52 (+0800): seven runs at -m 60 (60.011-60.018 s, first byte 0.000000, 0 B, curl exit 28), one at -m 300 (300.312 s, 0 B), one at -m 30 (30.030 s, 0 B) and two follow-up runs at -m 90 (90.0011 s / 90.0016 s, 0 B). Load-independence is total (see step 5). In the browser, the real client-issued session/list receives no response headers inside a 20 s window and the session list never renders; only the ordinary composer placeholder "Choose a workspace to start" (i18n placeholder.workspace) is shown, which is also why this failure is so easy to misread as "empty workspace".

The stall is not accompanied by request-attributable CPU (numbers in step 4): MainThread burns 0.807 cores with the request in flight versus 0.838 cores over the same window without it, statistically identical, and 1.10-1.48 cores across every sampled window (407.7 CPU-s over 360 s = 1.13 cores). Thread-level sampling shows MainThread at ~99% of one core in state R while all four libuv workers (0.07-0.09 s / 20 s) and the V8 workers (0.03-0.04 s / 20 s) idle, so it is not blocked on async I/O — yet it never settles the request.

Two side observations that may or may not be related (we are not claiming causation): while no request of ours is in flight (08:43:19/22/25/28, 3 s apart) the process holds 16 / 16 / 17 / 23 concurrently open session.jsonl.zstd descriptors under [repo-slug]/<id>/, the set changing on every sample, with /proc/<pid>/fdinfo pos mostly 0 or 8192 (the 8 KiB pre-read of readFirstZstdLine); and the projection cache is being rewritten continuously (2,240 records; 10 file mtimes changed per 10 s, 83 within 60 s, newest at 08:43:44). Every storage-domain write is serialised on one chain (dsh-storage-domain/lib/index.js:97 "every write queues on the chain, awaits backend durability", enqueue at :221), and session/list's projection column reads the same domain.

Impact

• The session list is the app's entry surface. On this instance the browser never renders a workspace session list, so the Web UI is effectively unusable for the human operator even though the process is alive, authenticated, and answering every other RPC.
• The failure is silent and total: no error body, no 4xx/5xx, no timeout, no partial result. The client has nothing to retry against and cannot distinguish "this workspace is empty" from "the server hung".
• The cost is invisible to the usual dashboards: ~0 request CPU, read_bytes ~ 0 (page cache), zero bytes written to the socket. CPU/IO-based alerting reports the instance as healthy.
• It does not self-heal: identical across browser restarts, low-load windows and a 300 s budget. It is a whole-endpoint outage, not a slow path.
• Blast radius grows monotonically with corpus size: nothing in the shipped seam deletes session files (dsh-session-persistence-jsonl/README.md known limits: "Nothing deletes session files"), the projection cache has no eviction ("No eviction or retention surface"), and the listing path has no memory cache and no TTL — every first paint and every workspace switch re-walks the entire corpus. This instance reached ~1.0 GiB / ~2,300 sessions in under 7 days (~526 sessions/day).
• Secondary, independently observable operator pain in the same process: ~0.74-0.84 cores burned continuously for 16.3 h (43,739 CPU-s); 2.6 billion read syscalls (44,220/s mean) and 4.34 TB rchar over the process lifetime; RSS pinned at its lifetime high-water mark (4.43 GiB) with no heap cap configured and 2,240 projection-cache records that nothing evicts.

What we already ruled out (all tested on the live instance)

1. Authentication / token rotation. GET /?token=... returns 200 in 0.488 s and sets the HttpOnly cookie; every subsequent RPC with it succeeds (step 3). A stale or missing cookie yields an immediate 401, never zero bytes for 300 s.
2. A malformed envelope. The 131-byte body the shipped client actually sends (captured from a real navigation) is byte-identical to the hand-written one and was replayed verbatim. Argument-shape violations are rejected synchronously by assertExactArguments (dsh-api-gateway/lib/index.js:1045, call site :746) with gateway/arguments-invalid - an error, not a hang. _request is the correct and only required key (source: json, strict codec, no acceptsUndefined; the payload must contain exactly one plain-object args field, :934).
3. Streaming / subscription semantics. The descriptor has invocation.kind: "direct" and no mode: "stream"; _request is a "reserved empty list request" (dsh-api-session-controller/lib/index.js:2690) whose only legal field, cursor, is ignored. The real streams in this gateway are $events (:13, openRemoteEvents :579-593, which requires an empty args object) plus SessionEventStream / SessionControlStream - none of them is session/list.
4. Host CPU contention. The host has 256 cores and is ~73% idle; the unrelated pytest/parity jobs (100-510% each) were never touched. The identical zero-byte result reproduces at nine loadavg(1m) readings from 10.89 to 34.50.
5. Whole-event-loop starvation / a synchronous main-thread monopoly. During a confirmed 30 s stall, 8/8 concurrent session/modelCatalog requests returned 200 with first byte at 0.090-1.122 s (median 0.131), versus an idle baseline of 0.097-0.117 s. The event loop schedules within ~100 ms granularity while session/list is stuck.
6. Data corruption / missing files / fail-closed abort. A read-only census of all ~2,311 session directories found 0 bad zstd magics, 0 empty logs and 0 missing logs; an independent parse of all 2,306 header frames succeeded for every one, with no duplicate ids and no legacy flat layout. This matters because the seam is fail-closed (dsh-session-query/lib/index.js:247-253 wraps enumeration errors into SESSION_QUERY_PERSISTENCE_FAILED): a corrupt artifact produces a fast error, the opposite of what we observe.
7. A missing or unmounted data directory. All logs are present and readable; the out-of-process listArtifacts replay over the same tree completed ALL_OK and found the >10 MB (6), 1-10 MB (99) and 100 KB-1 MB (1,841) files exactly where expected, all <7 days old.
8. Disk I/O latency or swapping. read_bytes ~ 0 during the stall (page cache), the process reports Swap 0.0 and majflt = 1 over its whole 16.3 h life, and an equivalent enumeration of the same corpus finishes in 0.92-1.00 s wall.
9. Algorithmic complexity. Listing reads only the first (header) frame per session and never event bodies, so the cost is O(sessions), not O(sessions x events). readSession / readSurface / filterEvents / observeSession are not on this path.
10. The probeSmallCold full-replay branch. It is gated by stat(path).size > coldBlankProbeMaxBytes (default 1024 B here) and is reached only for cold sessions that are uncached or blank === true; with 2,240 cached records against ~2,306 sessions that is <= ~112 stats, each returning immediately for logs larger than 1 KiB.
11. Per-endpoint serialisation by an earlier stuck request. dsh-client-connection/lib/index.js has no endpoint-level mutex or queue, only socket backpressure (res.off/once("drain"), :73/:77), and the concurrent-control experiment proves the endpoint is not saturated.

Still open / what we could not prove

We would rather hand you a precise hole than a wrong root cause. These remain candidates only:

1. Contention between the projection-cache write-behind and the listing's projection column over the single storage-domain chain (lib/index.js:97 / enqueue :221). We have existence evidence for the chain and for continuous writes, but no direct evidence that the listing is queued behind it.
2. V8 heap pressure near the default cap (RSS = VmHWM = 4.43 GiB, no --max-old-space-size). The idle V8Worker time (0.03-0.04 s per 20 s) argues against it, so the two pieces of evidence conflict.
3. The always-on header-level scan running in idle time (16-23 rolling session-log descriptors) contending for the same corpus as the listing.
4. A client-side state machine defect we reproduced separately: dsh-client-ui-workspace watchNavigation performs auto workspace selection exactly once and, if it finds no target, latches initial = "done" and never retries; startSession() with no target is a silent sessions.clear(); return;. We list this as secondary because fixing the server-side listing would reduce it to a slightly late first paint rather than an empty UI.

Suggested fixes (upstream)

1. Instrument the endpoint, then look. Add stage timings or a debug log with per-phase durations inside sessionController.list / ApiSessionList.list / listArtifacts so this class of stall stops being invisible. Without this, no amount of operator-side measurement can attribute it.
2. Stream or at least flush early. Because the handler awaits the entire corpus walk before a Response exists, clients see zero bytes and cannot distinguish a hang from a slow answer. Emitting headers (or a first partial page) before the walk would turn an outage into visible latency.
3. Put a memory cache / short TTL on the listing. The corpus walk is repeated on every first paint and every workspace switch, has no cache and no TTL, and nothing ever deletes session files. A header index (persisted or in-memory with mtime invalidation) would bound the cost instead of letting it grow with the corpus.
4. Bound the work per call. listArtifacts is a fully serial loop of await; adding bounded concurrency (the pattern SessionCorpus.projectMany already uses per dsh-session-query/README.md:104) and honoring the existing (currently ignored) _request.cursor for pagination would give the endpoint a ceiling.
5. Consider workspace scoping. session/list returns the entire corpus with no workspace argument, while the UI only needs the sessions of one workspace (82 of ~2,300 in our case) — the client joins against workspace.json by itself.
6. Warn about the coldBlankProbeMaxBytes cliff. Raising it makes probeSmallCold do a full log replay + fold for cold sessions; nothing in the config documents that this turns listing into O(events).
7. Client side. Let watchNavigation retry auto-selection when the workspace/session stores become ready, and give startSession()'s no-target path a visible outcome instead of a silent no-op.