Files
codegraph/src/bin/command-supervision.ts
T
Colby Mchenry edb9f2f14c fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)
The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).

The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).

Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 08:25:23 -05:00

97 lines
4.5 KiB
TypeScript

/**
* Process supervision for long-running CLI commands (`index` / `init --index`).
*
* Indexing a large repo can run for a while on the main thread, and #999
* surfaced two ways that goes wrong when nothing is watching it:
*
* 1. **Orphaned worker.** `index` runs in a child re-exec'd with
* `--liftoff-only` (the WASM-flag relaunch). Its parent blocks in
* `spawnSync`, so when the parent shim is killed it cannot forward the
* signal — the child keeps running, now orphaned, pinning a core. The PPID
* watchdog (#277) notices the parent/host went away and exits the child.
* 2. **Wedged indexer.** The `#850` main-thread liveness watchdog — which
* SIGKILLs a process whose event loop stops turning — was wired only into
* the MCP `serve` path, so a wedged `index`/`init` was never auto-killed.
*
* Both reuse the exact mechanisms `serve` already uses; this just makes them
* available to a one-shot command. Best-effort and self-disabling: a missing
* watchdog never blocks the command from running. Both honour the same env
* switches as `serve` (`CODEGRAPH_NO_WATCHDOG`, `CODEGRAPH_PPID_POLL_MS=0`).
*
* Unlike the daemon — whose main thread only does fast, bounded work — the
* `index`/`init` path runs reference resolution and dynamic-edge synthesis
* SYNCHRONOUSLY on this thread, and on a large repo that is legitimately many
* seconds of work. So those spans yield cooperatively to the event loop
* (`src/resolution/cooperative-yield.ts`) to keep the heartbeat alive; without
* that the watchdog would SIGKILL a valid, in-progress index (#1091). The
* distinction it must preserve — kill a TRUE wedge, spare slow-but-progressing
* work — is exactly what cooperative yielding buys: a genuinely stuck span never
* reaches its next yield, so it still trips the timeout.
*/
import { installMainThreadWatchdog, WatchdogOptions } from '../mcp/liveness-watchdog';
import { supervisionLostReason, parsePpidPollMs, parseHostPpid } from '../mcp/ppid-watchdog';
import { isProcessAlive } from '../mcp/daemon-registry';
import { EARLY_PPID } from '../mcp/early-ppid';
import { HOST_PPID_ENV } from '../extraction/wasm-runtime-flags';
export interface CommandSupervision {
/** Tear down both watchdogs. Idempotent; call when the command finishes. */
stop(): void;
}
/**
* Install the liveness + PPID watchdogs for the duration of a CLI command.
* `label` is used in the shutdown notice (e.g. `"index"`). Returns a handle
* whose `stop()` must be called when the command completes so neither watchdog
* outlives it.
*
* Pass `watchdog.progressPaths` (the project's SQLite DB + `-wal`) so the
* liveness watchdog can tell a slow-but-progressing store on degraded storage
* (files advancing) from a true wedge (they aren't) — one long synchronous
* SQLite statement on a 150-IOPS disk otherwise gets a healthy index
* SIGKILLed (#1231).
*/
export function installCommandSupervision(label: string, watchdog: WatchdogOptions = {}): CommandSupervision {
// Liveness watchdog: a separate process that SIGKILLs us if our event loop
// stops turning for too long (a wedged synchronous loop). Self-disables on
// CODEGRAPH_NO_WATCHDOG.
const liveness = installMainThreadWatchdog(watchdog);
// PPID watchdog: detect that the parent (or the host threaded past the
// relaunch shim) died and we've been orphaned, then exit instead of leaking.
// Baseline from the CLI entry's earliest-possible capture — reading
// process.ppid here would miss a launcher killed during startup (#1185).
const originalPpid = EARLY_PPID;
const hostPpid = parseHostPpid(process.env[HOST_PPID_ENV]);
const pollMs = parsePpidPollMs(process.env.CODEGRAPH_PPID_POLL_MS);
let ppidTimer: ReturnType<typeof setInterval> | null = null;
if (pollMs > 0) {
ppidTimer = setInterval(() => {
const reason = supervisionLostReason({
originalPpid,
currentPpid: process.ppid,
hostPpid,
isAlive: isProcessAlive,
});
if (reason) {
try {
process.stderr.write(`[CodeGraph ${label}] Parent process exited (${reason}); aborting.\n`);
} catch { /* stderr gone with the parent — exit anyway */ }
process.exit(1);
}
}, pollMs);
// Never let the watchdog itself keep the process alive past its real work.
ppidTimer.unref();
}
let stopped = false;
return {
stop(): void {
if (stopped) return;
stopped = true;
if (ppidTimer) clearInterval(ppidTimer);
liveness?.stop();
},
};
}