fix(windows): helper threads never end mid-start or with GC marking in flight (#2309)

On Windows a worker thread that ends - terminate(), its own process.exit(),
or the process exiting - while V8's concurrent marker is marking its heap
crashes the whole process (exit 3221225477 / 0xC0000005, no error, no dump).
Bisected on the Windows 11 ARM64 VM: --no-concurrent-marking alone stops it
(18/480 -> 0/480; no other concurrency flag does), but it costs about a third
of indexing speed, so CodeGraph avoids the dangerous moments instead:

- src/worker-teardown.ts: owners terminate a worker only once it has started
  (its first message, posted after its modules load - mid-load is by far the
  worst moment), and a worker about to exit runs a full collection first,
  which finishes any marking in flight (gc reached at runtime, never global).
- Resolver pool / store writer: the worker collects before exiting on
  'close' (the normal path - main 21/480 crashed on the VM, now 2/1200), and
  the 5 s close fallback never terminates a worker still starting.
- codegraph_status change counts: the worker posts 'loaded' after its heavy
  require and collects before answering; a deadline answers unknown at once
  but terminates only once loaded (3/720 -> 0/1440), and MCPEngine.stop()
  waits for measurements in flight.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-10-02 21:58:28 +00:00
committed by GitHub
parent 3d86bcc037
commit 70fe0ecc52
13 changed files with 376 additions and 24 deletions
+1
View File
@@ -214,6 +214,7 @@ For any Windows-specific PR, bug, or implementation, validate it on the real Win
- Guest toolchain (winget): Node LTS, Git, and the **VC++ ARM64 redistributable** (required by `@rollup/rollup-win32-arm64-msvc`, which vitest pulls in).
- Fetch a contributor PR head straight from their fork to dodge `pull/<n>/head` lag: `git fetch <fork-url> <branch>` then `git checkout -f FETCH_HEAD`.
- Windows baseline: as of #2053 the full suite passes on the Windows 11 (ARM64) VM. The only expected exception is `security.test.ts > Session marker symlink resistance > does not follow a pre-planted symlink`, which needs symlink privileges (Developer Mode) — confirm any other failure against `origin/main` before blaming your PR. The former `mcp-initialize.test.ts` / `mcp-roots.test.ts` `EPERM` teardown failures came from tests spawning `serve --mcp` without the runtime flags (its `--liftoff-only` re-exec grandchild kept the cwd / SQLite file open); spawn it with `WASM_RUNTIME_FLAGS` and await the child's exit before removing the temp dir. Windows checkouts may be CRLF — split source lines on `/\r?\n/` in tests.
- Windows worker crashes: a worker thread that ends (`terminate()`, its own `process.exit()`, or the process exiting) while V8's concurrent marker is marking its heap kills the process with exit 3221225477 (0xC0000005) and leaves no error or dump. It's worst while the worker is still loading its modules. Owners must not terminate a worker before its first message (`workerStarted` / `terminateOnceStarted`, or the pool's own handshake), and a worker that exits itself calls `collectBeforeExit()` first — both in `src/worker-teardown.ts`. `--no-concurrent-marking` also stops it, but measured about a third slower indexing on the Windows VM, so it is not used.
## Releases
+1
View File
@@ -177,6 +177,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still starting up, which took the whole process down; they now finish starting first.
- On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory.
- On Windows, the CodeGraph background server no longer sometimes crashes as it shuts down (exit code 3221225477, an access violation) when it stops shortly after starting. Shutting down could stop one of its background threads while that thread was still starting up; it now lets the thread finish starting first.
- On Windows, indexing, syncing and the CodeGraph background server no longer sometimes crash (exit code 3221225477, an access violation) when they stop one of their helper threads: the ones that resolve references, write the index, and count changes for `codegraph_status`. A thread is no longer stopped while it is still starting up, and each one finishes its memory cleanup before it exits.
## [1.6.1] - 2026-09-29
+36 -3
View File
@@ -5,7 +5,7 @@ import * as path from 'path';
import { createHash } from 'crypto';
import { Readable } from 'stream';
import { validateAnswerFiles } from '../src/mcp/answer-freshness';
import { measurePendingChanges } from '../src/mcp/index-freshness';
import { endFreshnessMeasurements, measurePendingChanges } from '../src/mcp/index-freshness';
vi.mock('fs', async importOriginal => {
const actual = await importOriginal<typeof import('fs')>();
@@ -46,16 +46,47 @@ describe('bounded freshness validation (#1959)', () => {
const pending = measurePendingChanges('/freshness-timeout');
expect(measurePendingChanges('/freshness-timeout')).toBe(pending);
expect(workers).toHaveLength(1);
workers[0].emit('message', { type: 'loaded' }); // stuck past loading, in the scan
await vi.advanceTimersByTimeAsync(8000);
expect(await pending).toBeNull();
expect(workers[0].terminate).toHaveBeenCalledOnce();
});
it('answers unknown on deadline but never terminates a worker still loading its modules', async () => {
// Terminating a worker while it loads its modules can crash the process
// on Windows (0xC0000005) — see worker-start.ts.
vi.useFakeTimers();
const pending = measurePendingChanges('/freshness-still-loading');
await vi.advanceTimersByTimeAsync(8000);
expect(await pending).toBeNull();
await vi.advanceTimersByTimeAsync(1000);
expect(workers[0].terminate).not.toHaveBeenCalled();
workers[0].emit('message', { type: 'loaded' });
await vi.advanceTimersByTimeAsync(0);
expect(workers[0].terminate).toHaveBeenCalledOnce();
});
it('a server shutting down ends measurements once they have loaded', async () => {
vi.useFakeTimers();
const pending = measurePendingChanges('/freshness-shutdown');
let ended = false;
const ending = endFreshnessMeasurements().then(() => { ended = true; });
await vi.advanceTimersByTimeAsync(1000);
expect(ended).toBe(false);
expect(workers[0].terminate).not.toHaveBeenCalled();
workers[0].emit('message', { type: 'loaded' });
await ending;
expect(workers[0].terminate).toHaveBeenCalled();
workers[0].emit('exit', 1); // what a terminated worker does
expect(await pending).toBeNull();
});
it('returns unknown on deadline even if worker termination is delayed', async () => {
vi.useFakeTimers();
const pending = measurePendingChanges('/freshness-delayed-exit');
let release!: () => void;
workers[0].terminate.mockImplementation(() => new Promise<void>(resolve => { release = resolve; }));
workers[0].emit('message', { type: 'loaded' });
await vi.advanceTimersByTimeAsync(8000);
expect(await pending).toBeNull();
release();
@@ -81,11 +112,13 @@ describe('bounded freshness validation (#1959)', () => {
expect(await measurePendingChanges('/freshness-three')).toBeNull();
expect(workers).toHaveLength(2);
workers[0].emit('error', new Error('worker failed'));
workers[1].emit('message', { added: -1, modified: 0, removed: 0 });
workers[1].emit('message', { type: 'counts', counts: { added: -1, modified: 0, removed: 0 } });
expect(await first).toBeNull();
expect(await second).toBeNull();
await new Promise((r) => setTimeout(r, 0)); // the slots free once each worker is terminated
const retry = measurePendingChanges('/freshness-three');
workers[2].emit('message', { added: 0, modified: 1, removed: 0 });
workers[2].emit('message', { type: 'loaded' });
workers[2].emit('message', { type: 'counts', counts: { added: 0, modified: 1, removed: 0 } });
expect(await retry).toEqual({ added: 0, modified: 1, removed: 0 });
});
});
+28 -1
View File
@@ -1,8 +1,9 @@
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { execFileSync } from 'child_process';
import { Worker } from 'worker_threads';
import CodeGraph from '../src/index';
import { measurePendingChanges } from '../src/mcp/index-freshness';
import { ToolHandler } from '../src/mcp/tools';
@@ -52,6 +53,32 @@ describe('MCP status freshness (#1959)', () => {
expect(await measurePendingChanges(path.join(root, 'missing'))).toBeNull();
});
it('past its deadline, never terminates the worker before it has loaded its modules', async () => {
// Terminating a worker while it loads its modules can crash the whole
// process on Windows (0xC0000005). A 1 ms deadline passes while the real
// worker is still loading: the answer is unknown at once, and the worker
// is terminated only after it has posted 'loaded'.
const posted = new WeakSet<object>();
const loadedWhenEnded: boolean[] = [];
const emit = Worker.prototype.emit;
vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) {
if (event === 'message') posted.add(this);
return emit.call(this, event, ...args);
});
const terminate = Worker.prototype.terminate;
vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) {
loadedWhenEnded.push(posted.has(this));
return terminate.call(this);
});
try {
expect(await measurePendingChanges(root, 1)).toBeNull();
await vi.waitFor(() => expect(loadedWhenEnded).toHaveLength(1), { timeout: 20_000 });
expect(loadedWhenEnded).toEqual([true]);
} finally {
vi.restoreAllMocks();
}
}, 30_000);
it('counts edits committed after the index even when the working tree is clean', async () => {
fs.writeFileSync(path.join(root, 'modify.ts'), 'export const modify = 99;\n');
execFileSync('git', ['add', 'modify.ts'], { cwd: root, stdio: 'pipe' });
+131
View File
@@ -0,0 +1,131 @@
/**
* On Windows, a worker thread that ends while V8's concurrent marker is
* marking its heap can crash the whole process (0xC0000005, no error, no
* dump) — most of all while it is still loading its modules. Owners wait for a
* worker's first message before terminating it, and workers collect garbage
* before exiting (worker-teardown.ts). These pin both halves, then end real
* resolver and store workers right after starting them and record whether
* each had started when it was terminated.
*/
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { EventEmitter } from 'events';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { Worker } from 'worker_threads';
import CodeGraph from '../src/index';
import { terminateOnceStarted, workerStarted } from '../src/worker-teardown';
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
class FakeWorker extends EventEmitter {
terminate = vi.fn(async () => 0);
}
describe('terminating a worker once it has started', () => {
it('waits for the first message', async () => {
const w = new FakeWorker();
const started = workerStarted(w as unknown as Worker);
const ending = terminateOnceStarted(w, started);
await sleep(30);
expect(w.terminate).not.toHaveBeenCalled();
w.emit('message', { type: 'ready' });
await ending;
expect(w.terminate).toHaveBeenCalledOnce();
});
it('counts an error or an exit as started', async () => {
for (const event of ['error', 'exit'] as const) {
const w = new FakeWorker();
w.on('error', () => {});
const started = workerStarted(w as unknown as Worker);
w.emit(event, event === 'error' ? new Error('boom') : 1);
await terminateOnceStarted(w, started);
expect(w.terminate).toHaveBeenCalledOnce();
}
});
it('stops waiting after the cap, and never rejects', async () => {
const w = new FakeWorker();
w.terminate.mockRejectedValue(new Error('already gone'));
const t0 = Date.now();
await terminateOnceStarted(w, new Promise<void>(() => {}), 60);
expect(w.terminate).toHaveBeenCalledOnce();
expect(Date.now() - t0).toBeGreaterThanOrEqual(55);
});
});
describe('collecting garbage before a worker exits', () => {
it('runs a full collection in a real worker, and leaves no global gc behind', async () => {
const teardown = path.resolve(__dirname, '../dist/worker-teardown.js');
const run = (code: string) => new Promise<unknown>((resolve, reject) => {
const w = new Worker(code, { eval: true });
w.once('message', resolve);
w.once('error', reject);
});
const ran = await run(`
const { collectBeforeExit } = require(${JSON.stringify(teardown)});
let junk = Array.from({ length: 200000 }, (_, i) => ({ i, s: 'x'.repeat(32) }));
junk = null;
const before = process.memoryUsage().heapUsed;
const collected = collectBeforeExit();
const after = process.memoryUsage().heapUsed;
require('worker_threads').parentPort.postMessage({ collected, freed: before - after, gc: typeof globalThis.gc });
`) as { collected: boolean; freed: number; gc: string };
expect(ran.collected).toBe(true);
expect(ran.freed).toBeGreaterThan(0);
expect(ran.gc).toBe('undefined');
// A worker created afterwards doesn't get a global gc either.
expect(await run(`require('worker_threads').parentPort.postMessage(typeof globalThis.gc)`)).toBe('undefined');
}, 30_000);
});
describe('real workers are never terminated before they have started', () => {
let root: string;
let dbPath: string;
const seen = new WeakSet<object>();
const startedWhenEnded: boolean[] = [];
beforeEach(async () => {
root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-worker-start-')));
fs.writeFileSync(path.join(root, 'app.ts'), 'export function alpha() { return beta(); }\nexport function beta() { return 1; }\n');
const cg = await CodeGraph.init(root);
try { await cg.indexAll(); } finally { cg.close(); }
dbPath = path.join(root, '.codegraph', 'codegraph.db');
startedWhenEnded.length = 0;
// A worker re-emits each message it posts; note which have, and what each
// had done by the time it was terminated.
const emit = Worker.prototype.emit;
vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) {
if (event === 'message') seen.add(this);
return emit.call(this, event, ...args);
});
const terminate = Worker.prototype.terminate;
vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) {
startedWhenEnded.push(seen.has(this));
return terminate.call(this);
});
});
afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
fs.rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 });
});
it('the resolver pool, torn down while its workers boot', async () => {
vi.stubEnv('CODEGRAPH_RESOLVE_WORKERS', '2');
const { ResolverPool } = require('../dist/resolution/resolver-pool') as typeof import('../src/resolution/resolver-pool');
const pool = ResolverPool.tryCreate(dbPath, root);
expect(pool).not.toBeNull();
await pool!.destroy(0); // the close fallback fires at once, mid-boot
expect(startedWhenEnded).toEqual([true, true]);
}, 30_000);
it('the store writer, closed while its worker boots', async () => {
const { StoreWriter } = require('../dist/extraction/store-writer') as typeof import('../src/extraction/store-writer');
const writer = new StoreWriter(path.resolve(__dirname, '../dist/extraction/store-worker.js'), dbPath, false);
await writer.close(0);
expect(startedWhenEnded.every(Boolean)).toBe(true);
}, 30_000);
});
+3
View File
@@ -32,6 +32,7 @@ import { QueryBuilder } from '../db/queries';
import { createDatabase, SqliteDatabase } from '../db/sqlite-adapter';
import { finalizeStoreBundle, type KernelStoreBundle, type StoreBundle } from './store-writer';
import { decodeExtractBuffers } from './kernel/decode';
import { collectBeforeExit } from '../worker-teardown';
if (!parentPort) {
throw new Error('store-worker must be run as a worker thread');
@@ -135,6 +136,8 @@ port.on('message', (msg: InMessage) => {
} catch {
/* already closed */
}
// No GC marking in flight when the thread ends (worker-teardown.ts).
collectBeforeExit();
process.exit(0);
break;
}
+12 -4
View File
@@ -9,6 +9,7 @@
import { Worker } from 'worker_threads';
import { ExtractionResult, Language, Node, Edge, UnresolvedReference, FileRecord } from '../types';
import { terminateOnceStarted, workerStarted } from '../worker-teardown';
/** One file's complete store payload (pre-filtered — see storeFileBundle). */
export interface StoreBundle {
@@ -66,6 +67,8 @@ export function finalizeStoreBundle(
export class StoreWriter {
private worker: Worker;
/** Settles on the worker's first message or its end — see worker-teardown.ts. */
private started: Promise<void>;
private readyPromise: Promise<void>;
private firstError: Error | null = null;
private drainWaiters = new Map<number, { resolve: () => void; reject: (e: Error) => void }>();
@@ -77,6 +80,7 @@ export class StoreWriter {
constructor(workerScriptPath: string, dbPath: string, fastInit: boolean) {
this.worker = new Worker(workerScriptPath);
this.started = workerStarted(this.worker);
let readyResolve!: () => void;
let readyReject!: (e: Error) => void;
this.readyPromise = new Promise<void>((resolve, reject) => {
@@ -175,14 +179,18 @@ export class StoreWriter {
return p;
}
/** Close the worker's DB connection and join the thread. */
async close(): Promise<void> {
/**
* Close the worker's DB connection and join the thread; the worker collects
* garbage and exits by itself (worker-teardown.ts). One that hasn't by
* `timeoutMs` is terminated — but never while it is still starting up.
*/
async close(timeoutMs = 5000): Promise<void> {
if (this.exited) return;
this.worker.postMessage({ type: 'close' });
await new Promise<void>((resolve) => {
const t = setTimeout(() => {
void this.worker.terminate().then(() => resolve());
}, 5000);
void terminateOnceStarted(this.worker, this.started).then(() => resolve());
}, timeoutMs);
this.worker.once('exit', () => {
clearTimeout(t);
resolve();
+7 -3
View File
@@ -18,6 +18,7 @@ import { ToolHandler } from './tools';
import { WslSharedIndexError } from '../db/wsl-shared-index';
import { assertNoRebuild, releaseWriterLock, tryAcquireWriterLock, writerLockHeldMessage } from './writer-lock';
import { QueryPool, resolvePoolSize } from './query-pool';
import { endFreshnessMeasurements } from './index-freshness';
import { acquireProject, ProjectLease } from './project-lifecycle';
// Lazy-load the heavy CodeGraph chain (sqlite + query/graph/context layers) OFF
@@ -271,13 +272,16 @@ export class MCPEngine {
// Detach + terminate the worker pool first so no tool call routes to a
// worker mid-teardown; outstanding pool calls resolve with graceful guidance.
// Stopping waits for the workers to end: the daemon exits right after, and
// exiting while a worker is still starting up can crash the process.
// Stopping waits for the workers to end — the pool's, and any
// `codegraph_status` change count still measuring: the daemon exits right
// after, and exiting while a worker is still starting up can crash the
// process.
this.toolHandler.setQueryPool(null);
const poolDown = this.queryPool ? this.queryPool.destroy() : Promise.resolve();
this.queryPool = null;
const measurementsDown = endFreshnessMeasurements();
const drained = this.toolHandler.closeAll();
this.stopPromise = Promise.all([drained, poolDown]).then(async () => {
this.stopPromise = Promise.all([drained, poolDown, measurementsDown]).then(async () => {
if (this.initPromise) await this.initPromise;
if (this.defaultLease) {
await this.defaultLease.release();
+7 -1
View File
@@ -1,5 +1,6 @@
/** Exact CLI-parity change count, isolated from the MCP transport event loop. */
import { parentPort, workerData } from 'worker_threads';
import { collectBeforeExit } from '../worker-teardown';
if (parentPort) {
const port = parentPort;
@@ -7,6 +8,9 @@ if (parentPort) {
let counts: { added: number; modified: number; removed: number } | null = null;
try {
const CodeGraph = (require('../index') as typeof import('../index')).default;
// Loaded: from here on the owner may terminate this worker. Ending it while
// it loads those modules can crash the process on Windows (worker-teardown.ts).
port.postMessage({ type: 'loaded' });
cg = CodeGraph.openSync((workerData as { root: string }).root);
const changes = cg.getChangedFiles();
counts = {
@@ -19,5 +23,7 @@ if (parentPort) {
} finally {
try { cg?.close(); } catch { /* the worker is exiting */ }
}
port.postMessage(counts);
// The owner terminates this worker on the answer: no GC marking in flight then.
collectBeforeExit();
port.postMessage({ type: 'counts', counts });
}
+30 -8
View File
@@ -1,6 +1,7 @@
import { existsSync } from 'fs';
import * as path from 'path';
import { Worker } from 'worker_threads';
import { terminateOnceStarted, workerStarted } from '../worker-teardown';
export interface PendingChangeCounts {
added: number;
@@ -12,20 +13,31 @@ export interface PendingChangeCounts {
const MEASURE_TIMEOUT_MS = 8_000;
let liveMeasurements = 0;
const active = new Map<string, Promise<PendingChangeCounts | null>>();
/** Every measurement worker not yet terminated, with its load — for {@link endFreshnessMeasurements}. */
const live = new Map<Worker, Promise<void>>();
export function measurePendingChanges(root: string): Promise<PendingChangeCounts | null> {
export function measurePendingChanges(root: string, timeoutMs = MEASURE_TIMEOUT_MS): Promise<PendingChangeCounts | null> {
const key = path.resolve(root);
const existing = active.get(key);
if (existing) return existing;
// Status probes for many projects must not exhaust the shared daemon.
if (liveMeasurements >= 2) return Promise.resolve(null);
const pending = runMeasurement(key).finally(() => active.delete(key));
const pending = runMeasurement(key, timeoutMs).finally(() => active.delete(key));
active.set(key, pending);
return pending;
}
function runMeasurement(root: string): Promise<PendingChangeCounts | null> {
/**
* Terminate every measurement still running, each once it has loaded its
* modules (worker-teardown.ts) — for a server that is about to exit, since
* exiting tears the workers down the same way terminating them does.
*/
export async function endFreshnessMeasurements(): Promise<void> {
await Promise.all([...live].map(([worker, loaded]) => terminateOnceStarted(worker, loaded)));
}
function runMeasurement(root: string, timeoutMs: number): Promise<PendingChangeCounts | null> {
// The compiled sibling is beside us in production; Vitest loads src/ but
// builds dist/ before the tests, so use that copy for the worker there.
const sibling = path.join(__dirname, 'index-freshness-worker.js');
@@ -41,6 +53,10 @@ function runMeasurement(root: string): Promise<PendingChangeCounts | null> {
return Promise.resolve(null);
}
// Its first message is 'loaded', posted once its modules are in. Terminating
// it before then can crash the whole process on Windows (worker-teardown.ts).
const loaded = workerStarted(worker);
live.set(worker, loaded);
liveMeasurements++;
return new Promise(resolve => {
let settled = false;
@@ -48,13 +64,19 @@ function runMeasurement(root: string): Promise<PendingChangeCounts | null> {
if (settled) return;
settled = true;
clearTimeout(timer);
// A worker inside a synchronous Git call may take time to terminate.
// Return unknown on deadline, but keep its concurrency slot until exit.
void worker.terminate().catch(() => {}).finally(() => { liveMeasurements--; });
// A worker inside a synchronous Git call may take time to terminate, and
// one still loading its modules must not be terminated yet. Return
// unknown on deadline, but keep its concurrency slot until it is gone.
void terminateOnceStarted(worker, loaded).finally(() => {
live.delete(worker);
liveMeasurements--;
});
resolve(counts);
};
const timer = setTimeout(() => finish(null), MEASURE_TIMEOUT_MS);
worker.once('message', (value: PendingChangeCounts | null) => {
const timer = setTimeout(() => finish(null), timeoutMs);
worker.on('message', (msg: { type?: string; counts?: PendingChangeCounts | null }) => {
if (msg?.type !== 'counts') return;
const value = msg.counts;
const valid = value && typeof value === 'object' && ['added', 'modified', 'removed'].every(
key => Number.isSafeInteger(value[key as keyof PendingChangeCounts]) && value[key as keyof PendingChangeCounts] >= 0,
);
+17 -4
View File
@@ -16,6 +16,7 @@ import * as os from 'os';
import type { Edge, UnresolvedReference } from '../types';
import type { ResolvedRef, UnresolvedRef } from './types';
import { memoryBudgetBytes } from './memory-budget';
import { terminateOnceStarted, workerStarted } from '../worker-teardown';
/** One synthesis pass's output: its edge list + worker-measured wall clock. */
export interface SynthPassResult {
@@ -34,9 +35,14 @@ export interface ChunkResult {
interface PoolWorker {
worker: Worker;
ready: Promise<void>;
/** Settles on the worker's first message or its end — see worker-teardown.ts. */
started: Promise<void>;
busy: number;
}
/** How long destroy() waits for a worker to exit on 'close' before terminating it. */
const CLOSE_TIMEOUT_MS = 5000;
const MIN_PARALLEL_BATCH = 1000;
const CHUNK_SIZE = 500;
@@ -166,13 +172,14 @@ export class ResolverPool {
private constructor(workerScript: string, dbPath: string, projectRoot: string, size: number) {
for (let i = 0; i < size; i++) {
const worker = new Worker(workerScript);
const started = workerStarted(worker);
let readyResolve!: () => void;
let readyReject!: (e: Error) => void;
const ready = new Promise<void>((resolve, reject) => {
readyResolve = resolve;
readyReject = reject;
});
const pw: PoolWorker = { worker, ready, busy: 0 };
const pw: PoolWorker = { worker, ready, started, busy: 0 };
worker.on('message', (msg: { type: string; id?: number; message?: string; edges?: Edge[]; ms?: number } & Partial<ChunkResult>) => {
if (msg.type === 'ready') {
readyResolve();
@@ -341,14 +348,20 @@ export class ResolverPool {
);
}
async destroy(): Promise<void> {
/**
* Ask every worker to close; each collects garbage and exits by itself (see
* worker-teardown.ts). One that hasn't by `closeTimeoutMs` is terminated —
* but never while it is still starting up: a pool torn down soon after it
* booted, on a busy machine, can have a worker still loading its modules.
*/
async destroy(closeTimeoutMs = CLOSE_TIMEOUT_MS): Promise<void> {
await Promise.all(
this.workers.map(
(pw) =>
new Promise<void>((resolve) => {
const t = setTimeout(() => {
void pw.worker.terminate().then(() => resolve());
}, 5000);
void terminateOnceStarted(pw.worker, pw.started).then(() => resolve());
}, closeTimeoutMs);
pw.worker.once('exit', () => {
clearTimeout(t);
resolve();
+3
View File
@@ -26,6 +26,7 @@ import { QueryBuilder } from '../db/queries';
import { ReferenceResolver } from './index';
import { SYNTH_PASSES } from './callback-synthesizer';
import { createYielder } from './cooperative-yield';
import { collectBeforeExit } from '../worker-teardown';
import type { UnresolvedReference } from '../types';
if (!parentPort) {
@@ -126,6 +127,8 @@ port.on('message', (msg: InMessage) => {
} catch {
/* already closed */
}
// No GC marking in flight when the thread ends (worker-teardown.ts).
collectBeforeExit();
process.exit(0);
break;
}
+100
View File
@@ -0,0 +1,100 @@
/**
* Ending a worker thread without taking the process down.
*
* On Windows, a worker thread that ends — terminated, by its own
* `process.exit()`, or with the process — while V8's concurrent marker is
* still marking its heap can crash the whole process with an access violation
* (exit 3221225477 / 0xC0000005), with no error and no dump. Bisected on a
* Windows 11 ARM64 VM (Node 24): `--no-concurrent-marking` alone stops it, and
* no other concurrency flag does — but that flag makes indexing about a third
* slower, so CodeGraph avoids the dangerous moments instead.
*
* Measured there, 480 child processes per row, four workers each:
*
* ended while still loading CodeGraph's modules 18 crashed
* loaded and allocating, terminated by the owner 4 crashed
* loaded and allocating, exiting by itself 1 crashed
* loaded and allocating, full collection, then exit 0 crashed
*
* So an owner never ends a worker that is still starting (it waits for the
* worker's first message, which is posted only after its modules load), and a
* worker that is about to exit collects garbage first — a full collection
* finishes any marking in flight.
*/
import type { Worker } from 'worker_threads';
/**
* Longest an owner waits for a worker to finish starting before terminating
* it anyway. A start takes about a second normally and a few under heavy
* load; the cap only bounds a start that is wedged.
*/
export const WORKER_START_SETTLE_MS = 15_000;
/**
* Settles once `worker` has started — its first message — or has gone away.
* Call it right after creating the worker, before it can post anything.
*/
export function workerStarted(worker: Worker): Promise<void> {
return new Promise<void>((resolve) => {
worker.once('message', () => resolve());
worker.once('error', () => resolve());
worker.once('exit', () => resolve());
});
}
/**
* Terminate `worker`, but not while it is still starting: wait for `started`
* (from {@link workerStarted}) first, up to `capMs`. Never rejects.
*/
export async function terminateOnceStarted(
worker: Pick<Worker, 'terminate'>,
started: Promise<void>,
capMs = WORKER_START_SETTLE_MS
): Promise<void> {
let cap: NodeJS.Timeout | undefined;
await Promise.race([
started,
new Promise<void>((resolve) => {
cap = setTimeout(resolve, capMs);
cap.unref?.();
}),
]);
clearTimeout(cap);
try {
await worker.terminate();
} catch {
// already gone
}
}
let collect: (() => void) | null | undefined;
/**
* In a worker that is about to exit: run a full garbage collection, so no
* marking is in flight when the thread ends. Returns whether one ran. The
* collector is reached at runtime (no `--expose-gc` on the command line) and
* is never exposed as a global.
*/
export function collectBeforeExit(): boolean {
if (collect === undefined) {
try {
// eslint-disable-next-line @typescript-eslint/no-require-imports
(require('v8') as typeof import('v8')).setFlagsFromString('--expose-gc');
// eslint-disable-next-line @typescript-eslint/no-require-imports
collect = (require('vm') as typeof import('vm')).runInNewContext('gc') as () => void;
// Only the context made above needed it; workers created later don't get a global gc.
// eslint-disable-next-line @typescript-eslint/no-require-imports
(require('v8') as typeof import('v8')).setFlagsFromString('--no-expose-gc');
} catch {
collect = null;
}
}
if (!collect) return false;
try {
collect();
return true;
} catch {
return false;
}
}