长时高扇出 workflow 作业不可靠：run 进程重启即丢  +  单进程单主线程封顶吞吐

# 面一：run 没有持久化身份、结果没有收尾记录
dsh-tool-workflow/lib/index.js:10     "… the run; non-completed reasons become tool errors, and background collection remains deferred."
dsh-tool-workflow/lib/index.js:38     const active = /* @__PURE__ */ new Map();          # run 注册表只在内存
dsh-tool-workflow/lib/index.js:231-262  start(...) → result = await run.result → return {runId, agentsStarted, result}; finally { await run.dispose(); recorder.finish(run.id, result.stopReason) }
dsh-tool-workflow/lib/index.js:76-82    finish(runId, stopReason) → append("tool-workflow/run-end", {runId, stopReason})   # 不含终值
dsh-workflow-worker-thread/lib/index.js # grep -nE "persist|storage|writeFile|appendFile" → 0 命中（引擎不落盘 run）
dsh-session/lib/index.js:600 / :667
# 面一（续）：重启后子会话连"看历史"都取不到
dsh-api-session-controller/lib/index.js:1578   if (header.origin === "subagent") throw new RemoteError("session/agent-busy", "subagent Sessions require their durable parent address", …)
dsh-api-session-controller/lib/types/history.js:346   同上（历史读取路径）     "Recovery code for a recorded tool call whose completed outcome was not durably recorded." / 恢复文案原文

# 面二：能力已随包、装配未接上
dsh-base/cordis.patch.yml:331-339      只挂 dsh-subagent-spawn-in-process(providerName: spawn) 与 dsh-subagent-fork-in-process(providerName: fork)
dsh-subagent-spawn-in-process/lib/index.js:42 / dsh-subagent-fork-in-process/lib/index.js:58   # 全树仅有的两处 subagent registerProvider
dsh-subagent/lib/types/out-of-process.js # "Provider-side vocabulary for OUT-OF-PROCESS subagent backends … belongs to the `dsh-subprocess` seam."
dsh-workflow-worker-thread:311 / dsh-code-runtime-worker-thread:737 / dsh-session-persistence-jsonl:1491   # 随包已有的三处 new Worker()

面一 · run 耐久性（上游）

• P0 落耐久 run 记录：runId、meta、逐 item/agent 状态、终态与终值；并折进会话恢复路径——进程重启后父会话应看到「起了多少项 / 完成多少项 / 已完成项的结果」，而不是一句 unknown（恢复文案至少带 runId 与进度）。
• P0 提供按 runId 查询 / 重附着入口：读回耐久记录、导出未完成项，允许只补跑未完成项。
• P1 关停为在途 tool/call 写终态（ABORTED），并与耐久记录保持一致。
• P1 兑现自述里 deferred 的 background collection：长扇出可 detach（工具立刻返回 runId），事后收结果。

面二 · 原生多进程 / worker-thread（上游）

• P0 接上 out-of-process provider：复用已随包的 out-of-process 词汇 + dsh-subprocess 机器，与两个 in-process driver 并存、按 profile 可选（如 provider: process）。
• P1 worker_threads 变体：给不需要进程隔离的 CPU 密集 agent 用（机制上已有三处 new Worker() 先例）。
• P1 每 worker 并发配额 + 排队状态可见：现在「在跑」与「排队」在用户侧不可区分。
• P1 定义 worker 化后的跨进程契约：持久化 / 取消 / session/follow / 工具调用耐久记录仍由宿主统一收口（否则与面一叠加成更难查的半途状态）。
• 验收判据（可测）：① N 路扇出期间主线程 CPU 不贴顶（空闲基线约 1% of one core）；② 墙钟 ≈ ceil(N/workers) × 单 agent 时延，而不是 N × 排队时延；③ 同宿主无关会话 canary 延迟保持平稳。

本地（本仓，P0 缓解；不属上游）

• 看门狗别把「忙」判成「死」：gateway-health.sh 的探针超时（6 s）与失败阈值（2 轮）应放宽并观察「是否在推进」；探针优先走不经网关的直连。
• 重启粒度：网关是独立进程，真坏时只重启它，不要 systemctl restart 整个单元。
• 单元内有在跑 workflow 时把「自动重启」降级为告警 + 人工。

# 1) 面一：run 无持久化身份 + 恢复只给 unknown
grep -n "background collection remains deferred" [app-dir]/app-[version]/node_modules/@deepseek-ai/dsh-tool-workflow/lib/index.js   # :10
grep -n "durably recorded"                       [app-dir]/app-[version]/node_modules/@deepseek-ai/dsh-session/lib/index.js          # :600 / :667

# 2) 面二：能力已随包、装配未接上
grep -nE "subagent-(spawn|fork)-in-process" [app-dir]/app-[version]/node_modules/@deepseek-ai/dsh-base/cordis.patch.yml            # 只有 :331 / :336 两行
ls -d [app-dir]/app-[version]/node_modules/@deepseek-ai/dsh-subprocess* [app-dir]/app-[version]/node_modules/@deepseek-ai/dsh-acp*
sudo grep -rn "subagents.registerProvider" --include=index.js [app-dir]/app-[version]/node_modules/@deepseek-ai                     # 只有 spawn:42 / fork:58

# 3) 本地扳机（两轮同形）
sudo journalctl -u dsh-gateway-health.service --since today --no-pager | grep -E "BAD|HEAL"

Title: Long-running fan-out `workflow` jobs are unreliable: runs are lost on process
       restart, and agent throughput is capped by a single JS main thread

Background
----------
`workflow` is the documented way to "orchestrate subagents at scale". We use it for a
long, wide job: 363 items, maxConcurrentAgents=256, one subagent per item. The same run
exposed two independent problems.

(1) Runs are not recoverable across a process restart.
    * The run registry is in-memory only: `const active = new Map()` (dsh-tool-workflow/
      lib/index.js:38), and the engine persists nothing (dsh-workflow-worker-thread: no
      persist/storage/writeFile/appendFile).
    * The tool awaits `run.result` and then disposes the run; the durable end-of-run
      record carries only `{runId, stopReason}` — not the final value (:231-262, :76-82).
    * The module itself documents "background collection remains deferred" (:10).
    * Cross-process recovery can only report "outcome is unknown"
      (dsh-session/lib/index.js:600/:667); there is no way to re-attach by run id.
    Measured: the run was killed twice by an external restart (10:14:45, 10:29:51);
    the second one survived ~10 minutes and completed only 12 of 363 items. Progress
    could only be inferred from each item's on-disk side effects.

(2) All agents share one JS main thread, so concurrency queues instead of scaling.
    * The shipped host mounts only in-process subagent drivers (dsh-base/cordis.patch.yml:
      331-339 -> spawn/fork; tree-wide registrations: dsh-subagent-spawn-in-process:42,
      dsh-subagent-fork-in-process:58).
    * The out-of-process vocabulary and machinery already ship
      (dsh-subagent/lib/types/out-of-process.js, dsh-subprocess, dsh-subprocess-local,
      dsh-acp, dsh-acp-app), and `new Worker()` is already used for workflow scripts,
      run_code and generation verification — it is just not used for agents.
    Measured elsewhere on the same host: K=4 and K=8 both plateau at 1.00 rpc/s with the
    main thread at 96.7%/98.3%; an unrelated control-plane call (about 0.5 s of CPU) was
    stretched to 34 s - >=45 s; and a 256-way fan-out made the per-user gateway miss a 6 s
    watchdog probe twice, triggering a restart of the whole unit (which killed the run).

Requested
---------
Face 1 (run durability), P0:
  a. Persist a durable run record (runId, meta, per-item/agent status, terminal state and
     final value) and surface it through session recovery, so a restart reports run id,
     progress and completed results instead of "outcome is unknown".
  b. Provide a query/re-attach entry point by run id, including the list of unfinished
     items so only those need re-running.
  P1: c. On shutdown/dispose, write a terminal record for in-flight tool calls (ABORTED).
      d. Implement the deferred background collection so a long fan-out can be detached
         and collected later.

Face 2 (native multi-process / worker-thread agents), P0:
  a. Ship an out-of-process subagent provider built on the existing out-of-process
     vocabulary + dsh-subprocess seam, mounted alongside the in-process drivers and
     selectable per profile.
  P1: b. A worker_threads variant for CPU-bound agents that do not need isolation.
      c. Per-worker concurrency quotas with queue state exposed to the UI.
      d. A cross-process contract for persistence, cancellation and session/follow.

Acceptance criteria (measurable): during an N-way fan-out the host main thread must stay
well below saturation; wall time should scale as ceil(N/workers) x per-agent latency
rather than N x queueing latency; and canary latency for an unrelated session on the same
host must stay flat.

What multi-process does NOT fix (stated to avoid over-claiming): single-call upstream
latency (measured on the same upstream: concurrency 71 -> p50 ~101 s; concurrency 325 ->
p50 ~246 s), a proxy/connection-pool limit, or per-step fsync/compaction barriers.

Keywords: workflow run durability, TOOL_OUTCOME_UNKNOWN, background collection deferred,
multi-process agents, worker_threads, out-of-process subagent driver, event-loop stall.