DeepSeek Harness · Upstream · Issue #22 草案
实测对象是一台 256 核、约 4,900 个进程的共享宿主上的 DSH 0.2.2-rc.1 实例。主线程连续约 125 分钟贴在 0.99 核;同期 loopback 的 401 请求从 2 ms 涨到 0.61–0.76 s,会话列表 RPC 最长 240 s。卡着的时候抓 perf:62% 周期在 syscall,proc_pid_readdir 占 21.8%,syscr 约 6.6 万次/秒。
/proc/<dsh-pid>/io 的 syscr 两次求差 —— 若达数万次/秒,同时 perf record -p <pid> 出现 proc_pid_readdir / getdents64 / do_task_stat,就是本篇说的形态,不必再猜网络、浏览器或某条慢请求。全部读数取自同一个仍在卡着的进程(30 秒窗口内无重启、无改配置);证据分级沿用本站约定。
平台期 300 s 差分 min 0.937 / median 0.994 / max 1.018 核(n=20);5 分钟 rate 为 0.87–1.11。单线程物理上限 1.0,>1 的部分是估算器在阶梯计数器上的窗口外推,不能读成“两个核”。
空闲时 loopback GET /(返回 401)1.7–2.3 ms;钉住时 0.61–0.76 s;5 路并发串行累积到 1.87 s。会话列表 RPC 在同一窗口最长 240 s,重启后回到 0.7–1.0 s,约 12 分钟后再次劣化到 51 s。
perf record -F 99 -g -p <pid> -- sleep 6:62% 周期在 syscall;proc_pid_readdir 21.8%,调用图里 seq_read / do_task_stat / getdents64 成批出现;第二次抓取 __GI___getdents64 13.1% / proc_pid_readdir 13.0%。
/proc/<pid>/io 的 syscr 采样:660,061 / 10 s、329,661 / 5 s(≈ 6.6–6.9 万次/秒)。按每 pid 2 次 syscall 的下界估,只能给出上界 ≤ 7.3 次/秒,不是实测次数;这个数字后来被上游指出不能当计数用,我们已在 下一节 撤回。
命令都在宿主上现跑;有原始输出支撑的标“已实测”,由前两者推导的标“推断”。
| 项 | 读数 | 强度 |
|---|---|---|
| 主线程 CPU(300 s 差分) | 0.937 / 0.994 / 1.018 核,n=20 | 已实测 |
| loopback 401 延迟 | 空闲 1.7–2.3 ms → 钉住 0.61–0.76 s | 已实测 |
| 会话列表 RPC | 最长 240 s;重启后 0.7–1.0 s,12 分钟后 51 s | 已实测 |
| perf(卡着时) | 62% syscall;proc_pid_readdir 21.8% | 已实测 |
syscr 速率 | 66,006 /s(10 s 窗)与 65,932 /s(5 s 窗) | 已实测 |
| 宿主进程数 | 4,879 pid / 约 2 万线程;C 的 ps -e 一次 0.27 s | 已实测 |
| 扫描次数上界 | ≤ 7.33 次/秒(syscr ÷ (2 × pid)),不是实测次数 | 上界,非计数 |
DSH 开发团队读完材料后给了五条意见;本页据此收窄结论,并把“下一步怎么定案”写清楚。
/proc 确实是现有实现;syscr 不能直接换算成准确的扫描次数;waitForMembers()”撤回——本窗口能证明的是“主线程被 /proc 枚举占满”,不能证明是哪条调用路径贡献的,waitForMembers() 只是候选之一。ls -l /proc/<pid>/fd:0 个 /dev/pts(fd 总数 57)。但这不能排除终端巡检——已退出但仍被保留的终端句柄,仍可能在 inspectActivity() 里取快照(上游指出,我们接受)。syscr 344,192 / 5 s(≈ 68.8k/s)、perf 仍以 proc_pid_readdir / getdents64 为主 ⇒ 枚举在持续发生,来源未定。dsh-bash-local 的 DEFAULT_GRACE_MS = 3e3;按 25 ms 节拍,单次“后代不退出”的 dispose 最坏 = 120 次快照 ≈ 1.13 M syscall(4,696 pid × 2)。观测 68.8k/s ⇒ 每分钟约 4 次满宽限 dispose 即可解释全部 churn;若后代秒退(每次 1 次快照),则需要约 7.3 次/秒 的子进程生命周期(68.8k ÷ 9.4k)。两者都能对上观测值,所以速率反推定不了案。
snapshot() 前后计时计数,label 区分 construct / waitForMembers / stopDescendants / inspectActivity / groupLiveness,每 60 s 打一行 dsh_inspector{label=…} calls=… ms=…。落点:index.js:963、:1015、:1032、:1123-1132,runner-launch-*.js:600-615(snapshot 本体)。linuxProcessGroupHasLiveMembers()(runner-launch-*.js:405):它直接读 /proc、不走 snapshot(),只统计 snapshot() 会漏掉 #7600 这条已知路径——这是上游点名的关键项。两类枚举分别计 calls 与 ms,再比较各自耗时。waitForMembers 占大头 ⇒ 25 ms 改指数退避 + 快照复用 + cgroup 限定;construct 占大头 ⇒ 进程表缓存/增量刷新;inspectActivity 占大头 ⇒ 终端保留与快照解耦。开发团队给出的机制解释,与本页读数一致;同时明确它仍未被本窗口的读数证明。
linuxProcessGroupHasLiveMembers();该函数读 /proc 目录并逐 pid 读 stat,找到存活成员即提前返回(不保证读全量)——与本页观测到的枚举形态一致。0.2.2-rc.1 打 tag,因此本版本没有、也还没有包含它的发布版。dsh-subprocess-local/lib/runner-launch-*.js:405(函数本体)、:715(sleepTick() = setTimeout(15))、:1042(fallbackOwner 使用它)。syscr 68.8k/s、CPU 95.4%)正好对上——不需要任何“只跑了 11% 时间”的假设:只要进程组一直有存活成员,这个循环就是近似连续在跑。上游要求“两类枚举都计入再比较各自耗时”。补丁已写好并做过锚点/语法自检,但尚未装到任何实例。
construct / inspectForeground / inspectActivity / waitForMembers / forceStopDescendants / stopDescendants —— 覆盖 snapshot() 的各个调用点;groupLiveness —— 15 ms fallback 的 linuxProcessGroupHasLiveMembers()(直接读 /proc,不走 snapshot();只计 snapshot 会漏掉 #7600 这条路)。开关:DSH_INSPECTOR_COUNTERS=1;关闭时包装函数直接调用原函数,零计时零输出。每 60 s 每个 label 一行:
dsh_inspector label=groupLiveness calls=… ms=… pids=… at=2026-11-10T…Z
ops/dsh/inspector-counters/instrument-dsh-inspector.py:--check 锚点+文件哈希、--selftest(node --check 阴阳)、--stage 生成打了补丁的副本 + unified diff + manifest,不碰生产目录;DSH_INSPECTOR_INSTRUMENT verdict=PASS mode=check anchors=10/10;DSH_INSPECTOR_SELFTEST anchors=PASS node_check=PASS edits=10;/data/tmp/dsh-counters-rc1/(instrumented/ 两文件 + 各自的 .patch + manifest.json,含源/打补丁后的 sha256)。dsh-subprocess-local/lib/{index.js,runner-launch-B2zsQ1Dz.js};instrumented/ 覆盖这两个文件(sha 对 manifest.json);Environment=DSH_INSPECTOR_COUNTERS=1,daemon-reload 后重启;journalctl -u <unit> | grep dsh_inspector;需要回传的数据:上面 7 个 label 各自的 calls/ms(30 分钟累计)与当刻 pids;我们会按同一口径更新本页与上游结论。
决定性验证(2026-11-10 16:05–16:10,已执行;结果否定该候选):
syscr 312,590 / 5 s(62.5k/s)、主线程 1.01 核、该 pid 100% CPU;kill -TERM 2872646(python3 -m http.server 18991,存活 8,815 s)→ 3 s 后进程已退出;syscr 608,122 / 10 s(60.8k/s)、1.10 核 —— 没有下降;systemctl restart dsh-alpha-mch0 → 16:10 复测:syscr 1 / 10 s、5 s 内 tick 增量 1(空闲)、loopback 401 19 ms。怎么读这个结果:杀掉那个残留进程(至少单独)不是触发条件——CPU 没动;而 restart 立刻清零。最可能的解释是:被等待的进程组不是它(另有存活成员),或 kill 之后该 await 并没有重新评估。所以“杀遗留进程看 CPU 是否下降”这条捷径在本次实例上不成立;要定案仍需计数器(或先定位被等待的那个进程组)。
副作用:该 18991 静态服务器已被终止且不会自动恢复;实例已按要求重启(ActiveEnterTimestamp=2026-11-10 16:09:43,新 node pid 2342485)。
先界定本页能证明什么:目前核验过的是方案覆盖范围(7 个 label 落在哪些调用点)与锚点/语法(哈希、10 处锚点、node --check);计数语义是否真按 label 分开、是否零开销要靠 canary 实测,本页不声称已核验实现。
下面是从 0.2.2-rc.1 实际文件生成的两个 unified diff(未写入任何安装目录),可直接对照审阅。
index.js.patch(99 行)--- a/index.js
+++ b/index.js
@@ -1,3 +1,30 @@
+/* dsh-inspector-counters: env-gated (DSH_INSPECTOR_COUNTERS=1) */
+import __dshFs from "node:fs";
+const __dshCnt = (globalThis.__dshInspectorCounters ??= (() => {
+ const on = process.env.DSH_INSPECTOR_COUNTERS === "1";
+ const rows = new Map();
+ const t = (label, fn) => {
+ if (!on) return fn();
+ const s = process.hrtime.bigint();
+ try { return fn(); } finally {
+ const r = rows.get(label) ?? { c: 0, ms: 0 };
+ r.c += 1; r.ms += Number(process.hrtime.bigint() - s) / 1e6; rows.set(label, r);
+ }
+ };
+ if (on) {
+ const timer = setInterval(() => {
+ let pids = -1;
+ try { pids = __dshFs.readdirSync("/proc").length; } catch (_e) { pids = -1; }
+ for (const [k, v] of rows) {
+ process.stderr.write("dsh_inspector label=" + k + " calls=" + v.c + " ms=" + v.ms.toFixed(1) +
+ " pids=" + pids + " at=" + new Date().toISOString() + "\n");
+ }
+ }, 60000);
+ timer.unref?.();
+ }
+ return { t };
+})());
+
import { C as bindManagedProcess, D as createProcessInspector, E as validateSubprocessSpec, O as controlPipe, S as loadLinuxExecve, T as spawnSubprocess, _ as parseWindowsRunnerResult, c as runnerStdio, d as cleanupLinuxLaunchFiles, l as spawnRunnerInvocation, m as deserializeRunnerError, n as WINDOWS_RUNNER_SELECTION, o as runnerEnvironment, p as createLinuxLaunchFiles, s as runnerInvocationAvailable, u as targetEnvironment, w as childEnv, y as readLinuxStartupError } from "./runner-launch-B2zsQ1Dz.js";
import { logSpillFailure, prepareManagedProcessBinding } from "./output.js";
import { closeSync, constants, existsSync, mkdtempSync, openSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
@@ -960,7 +987,7 @@
this.observeShellExit = observeShellExit;
this.pid = terminal.pid;
try {
- this.rootIdentity = inspector.snapshot().tree(this.pid).find((member) => member.pid === this.pid);
+ this.rootIdentity = __dshCnt.t("construct", () => inspector.snapshot()).tree(this.pid).find((member) => member.pid === this.pid);
} catch (_rootIdentityUnavailable) {
this.rootIdentity = void 0;
}
@@ -1012,7 +1039,7 @@
this.terminal.resize(cols, rows);
}
async inspectForeground() {
- this.descendants(this.inspector.snapshot());
+ this.descendants(__dshCnt.t("inspectForeground", () => this.inspector.snapshot()));
const processGroupId = this.inspector.foregroundPgid(this.pid);
if (processGroupId === void 0) return void 0;
return {
@@ -1029,7 +1056,7 @@
revision: 0
};
revision = shell.revision;
- const observed = this.inspector.snapshot();
+ const observed = __dshCnt.t("inspectActivity", () => this.inspector.snapshot());
const descendants = this.descendants(observed);
const root = observed.tree(this.pid).find((member) => member.pid === this.pid);
if (this.exited && this.managedRangeEmpty) state = "idle";
@@ -1123,10 +1150,10 @@
async waitForMembers(members) {
if (members.length === 0) return [];
const until = Date.now() + this.graceMs;
- let survivors = this.survivors(members, this.inspector.snapshot());
+ let survivors = this.survivors(members, __dshCnt.t("waitForMembers", () => this.inspector.snapshot()));
while (survivors.length > 0 && Date.now() < until) {
await delay(Math.min(25, Math.max(1, until - Date.now())));
- survivors = this.survivors(members, this.inspector.snapshot());
+ survivors = this.survivors(members, __dshCnt.t("waitForMembers", () => this.inspector.snapshot()));
}
return survivors;
}
@@ -1138,7 +1165,7 @@
forceStopDescendants() {
let members = this.trackedDescendants;
try {
- members = this.descendants(this.inspector.snapshot());
+ members = this.descendants(__dshCnt.t("forceStopDescendants", () => this.inspector.snapshot()));
} catch (_processTableUnavailableDuringHostExit) {}
this.signalMembers(members, "SIGKILL");
}
@@ -1154,13 +1181,13 @@
return members;
}
async stopDescendants() {
- const captured = this.descendants(this.inspector.snapshot());
+ const captured = this.descendants(__dshCnt.t("stopDescendants", () => this.inspector.snapshot()));
this.signalMembers(captured, "SIGTERM");
const capturedSurvivors = await this.waitForMembers(captured);
- const members = this.unionMembers(capturedSurvivors, this.descendants(this.inspector.snapshot()));
+ const members = this.unionMembers(capturedSurvivors, this.descendants(__dshCnt.t("stopDescendants", () => this.inspector.snapshot())));
this.signalMembers(members, "SIGKILL");
const survivors = await this.waitForMembers(members);
- const observed = this.inspector.snapshot();
+ const observed = __dshCnt.t("stopDescendants", () => this.inspector.snapshot());
return this.survivors(this.unionMembers(survivors, this.descendants(observed)), observed);
}
async stopShell() {runner-launch-B2zsQ1Dz.js.patch(42 行)--- a/runner-launch-B2zsQ1Dz.js
+++ b/runner-launch-B2zsQ1Dz.js
@@ -1,3 +1,30 @@
+/* dsh-inspector-counters: env-gated (DSH_INSPECTOR_COUNTERS=1) */
+import __dshFs from "node:fs";
+const __dshCnt = (globalThis.__dshInspectorCounters ??= (() => {
+ const on = process.env.DSH_INSPECTOR_COUNTERS === "1";
+ const rows = new Map();
+ const t = (label, fn) => {
+ if (!on) return fn();
+ const s = process.hrtime.bigint();
+ try { return fn(); } finally {
+ const r = rows.get(label) ?? { c: 0, ms: 0 };
+ r.c += 1; r.ms += Number(process.hrtime.bigint() - s) / 1e6; rows.set(label, r);
+ }
+ };
+ if (on) {
+ const timer = setInterval(() => {
+ let pids = -1;
+ try { pids = __dshFs.readdirSync("/proc").length; } catch (_e) { pids = -1; }
+ for (const [k, v] of rows) {
+ process.stderr.write("dsh_inspector label=" + k + " calls=" + v.c + " ms=" + v.ms.toFixed(1) +
+ " pids=" + pids + " at=" + new Date().toISOString() + "\n");
+ }
+ }, 60000);
+ timer.unref?.();
+ }
+ return { t };
+})());
+
import { OutputCollector, prepareManagedProcessBinding } from "./output.js";
import { accessSync, chmodSync, closeSync, constants, existsSync, lstatSync, mkdtempSync, openSync, readFileSync, readSync, readdirSync, readlinkSync, rmdirSync, statSync, unlinkSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
@@ -1039,7 +1066,7 @@
});
const direct = directChildResult(child);
const pid = child.pid;
- const owner = fallbackOwner(platform, pid, child, internals.taskkill ?? taskkillProcessTree, internals.linuxProcessGroupHasLiveMembers ?? linuxProcessGroupHasLiveMembers, direct);
+ const owner = fallbackOwner(platform, pid, child, internals.taskkill ?? taskkillProcessTree, internals.linuxProcessGroupHasLiveMembers ?? ((pgid) => __dshCnt.t("groupLiveness", () => linuxProcessGroupHasLiveMembers(pgid))), direct);
return bindManagedProcess(spec, {
stdin: child.stdin,
stdout: child.stdout,只指到包内文件与行号,不含任何内网路径。
@deepseek-ai/dsh-subprocess-local/lib/runner-launch-*.js —— snapshot() 枚举 每一个 pid 并逐个 readFileSync(/proc/<pid>/stat),每次调用都建整张进程表。@deepseek-ai/dsh-subprocess-local/lib/index.js:1123-1132 —— waitForMembers() 在等待后代退出时每 25 ms 重新全量快照。@deepseek-ai/dsh-subprocess-local/lib/index.js:963 —— 每个子进程句柄在构造时各取一次快照。@deepseek-ai/dsh-subprocess-local/lib/runner-launch-*.js:405 / :715 / :1042 —— 15 ms fallback:linuxProcessGroupHasLiveMembers() 读 /proc 并逐 pid 读 stat(找到存活成员即提前返回)判断进程组是否还活着,由 sleepTick() 在每次检查后等 15 ms(已知问题 #7600)。@deepseek-ai/dsh-api-terminal-controller/lib/index.js —— 终端保留在无人持有时轮询 inspectActivity()(默认 30 s 轮询 / 2 h 未持有超时),同样走快照。/proc/<dsh-pid>/io 的 syscr,求差;数万次/秒即命中。perf record -F 99 -g -p <dsh-pid> -- sleep 5,看 proc_pid_readdir / getdents64 / do_task_stat。curl loopback 的 web 端口;主线程被钉住时应看到 0.1–1 s 的 401。/sys/fs/cgroup/<unit>/cgroup.procs 或只跟踪自己拉起的子进程,不要为无关进程付费。snapshot() 前后计时并计数,运维才能分清成本来自句柄构造、终端保留还是 waitForMembers()。完整英文稿见仓内 ops/dsh/upstream-issue-main-thread-control-plane/GITHUB-ISSUE-20261110-public.md(52 行,已做身份/路径脱敏)。
Title: One Node thread serves HTTP + agent loop, and the process
inspector enumerates every /proc entry
- 0.99 core for ~125 min (300 s deltas 0.937/0.994/1.018), while loopback
401s went 2 ms -> 0.61-0.76 s and session/list peaked at 240 s.
- perf on the still-stalled pid: 62% cycles in syscalls; proc_pid_readdir
21.8%; getdents64 13.1% in a second capture.
- /proc/<pid>/io syscr: 660,061/10 s and 329,661/5 s (~66k syscalls/s).
With 4,696-4,879 PIDs this bounds (it does not count) the enumeration rate at <=7.3/s;
the probe may return early, so even that is an upper bound.
- Code: dsh-subprocess-local snapshot() walks all PIDs; waitForMembers()
re-snapshots every 25 ms; terminal retention polls inspectActivity().
- Fix: scope enumeration to the owning cgroup; cache/backoff instead of a
25 ms full-table cadence; instrument snapshot() call sites; long term,
separate the control plane from the agent loop.