fix(mcp,extraction): exit a daemon whose install changed; never store a grammar-load failure (#2335, #2336) (#2346)

* fix(mcp,extraction): a daemon whose install changed exits; grammar-load failures are never stored (#2335, #2336)

A daemon that outlived an upgrade loaded grammars lazily from an install
that no longer existed, and stored every file its watcher re-indexed as a
zero-node row under the file's new content hash: no hash-based sync ever
revisited those rows, and `codegraph status` called the index up to date.

- The daemon checks its own package.json every 30 s
  (CODEGRAPH_DAEMON_INSTALL_CHECK_MS, 0 turns it off) and exits once the
  file is gone or carries another version, so the next session starts a
  daemon from the current install.
- A result whose grammar failed to load (`parser_error`) is never stored:
  the file keeps its previous data and the next sync or index retries it,
  and a row an older engine stored that way is re-indexed even though its
  hash matches. The tag-based CFML path reports a missing grammar with the
  same code.
- A launcher whose own install was deleted no longer crashes on the spawn
  error of the daemon it tried to start.
- `codegraph status` and `status --json` (filesNeedingReindex,
  filesWithParseErrors) report indexed files whose symbols are missing
  although their content is current (#2336), and `files --json` carries
  each file's recorded errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): a session whose own install changed never claims the writer lock (#2335)

A daemon now exits once its install is upgraded or removed, and nobody
holds the writer slot for a successor at that point. Its sessions run the
same replaced code and serve themselves in-process the moment it goes; one
that found the slot free claimed it as its writer even though its engine
could not load (a real npm upgrade: "Cannot find module
'./sqlite-adapter'"), and so kept a daemon from the current install from
starting. Such a session now falls back read-only and says to restart it.

Also adds the CHANGELOG entries for the install check and grammar-load
failures (#2335) and for the status report (#2336).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-10-05 15:08:14 +00:00
committed by GitHub
parent 511d86e94d
commit dea076fd1e
15 changed files with 706 additions and 15 deletions
+48 -1
View File
@@ -1056,6 +1056,33 @@ program
}
});
/**
* The `status` warnings for indexed files whose symbols are missing although
* their content is current (#2335, #2336): one line per group, saying what is
* wrong, what to do, and naming up to three of the files.
*/
function describeFilesMissingSymbols(health: { needsReindex: string[]; parseErrors: string[] }): string[] {
const dash = getGlyphs().dash;
const sample = (paths: string[]): string =>
paths.slice(0, 3).join(', ') + (paths.length > 3 ? ` (+${formatNumber(paths.length - 3)} more)` : '');
const lines: string[] = [];
const reindex = health.needsReindex.length;
if (reindex > 0) {
lines.push(
`${formatNumber(reindex)} ${reindex === 1 ? 'file is missing its' : 'files are missing their'} symbols ${dash} ` +
`run "codegraph sync" to re-index ${reindex === 1 ? 'it' : 'them'}: ${sample(health.needsReindex)}`
);
}
const unparsed = health.parseErrors.length;
if (unparsed > 0) {
lines.push(
`${formatNumber(unparsed)} ${unparsed === 1 ? 'file could not be parsed, so its symbols are' : 'files could not be parsed, so their symbols are'} ` +
`missing: ${sample(health.parseErrors)} ${dash} "codegraph files --json" shows the errors`
);
}
return lines;
}
/**
* codegraph status [path]
*/
@@ -1103,6 +1130,9 @@ program
// Zero on a healthy index; non-zero at rest means a resolution pass was
// interrupted, so some files' call edges are missing (#1187).
const pendingRefs = cg.getPendingReferenceCount();
// Files whose content is current but whose symbols are missing — the
// content-hash check behind "up to date" cannot see them (#2336).
const health = cg.getIndexHealth();
// JSON output mode
if (options.json) {
@@ -1143,6 +1173,12 @@ program
// interrupted resolution pass left edges missing; the next
// sync sweeps them (#1187).
pendingRefs,
// Files stored without their symbols (e.g. while their grammar
// could not load, #2335); the next sync re-indexes them.
filesNeedingReindex: health.needsReindex.length,
// Files with a recorded parse error and no symbols from it
// (#2336); unchanged until the file or the parser changes.
filesWithParseErrors: health.parseErrors.length,
},
}));
cg.destroy();
@@ -1235,7 +1271,10 @@ program
console.log(` Removed: ${changes.removed.length} files`);
}
info('Run "codegraph sync" to update the index');
} else {
}
const missingSymbols = describeFilesMissingSymbols(health);
for (const line of missingSymbols) warn(line);
if (totalChanges === 0 && missingSymbols.length === 0) {
success('Index is up to date');
}
console.log();
@@ -1751,6 +1790,14 @@ program
language: f.language,
nodeCount: f.nodeCount,
size: f.size,
// What extraction recorded for the file — a parse error, a skip
// reason — so a health check needn't read the database (#2336).
errors: (f.errors ?? []).map((e) => ({
severity: e.severity,
code: e.code,
message: e.message,
line: e.line,
})),
}));
console.log(JSON.stringify(output, null, 2));
cg.destroy();
+12
View File
@@ -3253,6 +3253,18 @@ export class QueryBuilder {
return rows.map(rowToFileRecord);
}
/**
* Files stored with no nodes or with recorded extraction errors — the only
* rows that can be missing their symbols (`CodeGraph.getIndexHealth`). A
* healthy index returns few or none, so this stays cheap on a large one.
*/
getFilesWithoutNodesOrWithErrors(): FileRecord[] {
const rows = this.db
.prepare('SELECT * FROM files WHERE node_count = 0 OR errors IS NOT NULL ORDER BY path')
.all() as FileRow[];
return rows.map(rowToFileRecord);
}
/**
* Most recent index timestamp (ms since epoch) across all tracked files, or
* null when nothing is indexed yet. One indexed aggregate, no per-row scan. (#329)
+3 -1
View File
@@ -147,10 +147,12 @@ export class CfmlExtractor {
private extractTagBased(): void {
const parser = getParser('cfml');
if (!parser) {
// Same code as TreeSitterExtractor's missing parser, so a grammar that
// failed to load is never stored as this file's index data (#2335).
this.errors.push({
message: 'cfml grammar not loaded',
severity: 'error',
code: 'unsupported_language',
code: 'parser_error',
});
return;
}
+16 -1
View File
@@ -9,7 +9,7 @@
import * as path from 'path';
import * as fsp from 'fs/promises';
import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
import { Language } from '../types';
import { ExtractionError, Language } from '../types';
export type GrammarLanguage = Exclude<Language, 'svelte' | 'vue' | 'astro' | 'liquid' | 'razor' | 'yaml' | 'twig' | 'xml' | 'properties' | 'unknown'>;
@@ -828,6 +828,21 @@ export function getUnavailableGrammarErrors(): Partial<Record<Language, string>>
return out;
}
/**
* Whether these extraction errors say the file was never parsed because its
* grammar was not available (`parser_error`, recorded by the extractor when
* `getParser` has nothing for the language).
*
* That is a fact about the running process, not about the file: a daemon whose
* install was upgraded or deleted underneath it fails every lazy grammar load
* (#2335). Such a result must never replace a file's index data, and a row an
* older engine stored that way must be re-indexed — not treated as current
* because its content hash still matches.
*/
export function hasGrammarLoadFailure(errors: readonly ExtractionError[] | undefined): boolean {
return !!errors && errors.some((e) => e.code === 'parser_error');
}
/**
* Get language display name
*/
+32 -8
View File
@@ -25,7 +25,7 @@ import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './
import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer';
import { materializeKernelResult } from './kernel';
import { detectGeneratedFile } from './generated-detection';
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES } from './grammars';
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES, hasGrammarLoadFailure } from './grammars';
import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config';
import { isCodeGraphDataDir } from '../directory';
import { logDebug, logWarn } from '../errors';
@@ -2289,11 +2289,17 @@ export class ExtractionOrchestrator {
const nodeCount = result.kernelCounts?.nodes ?? result.nodes.length;
const edgeCount = result.kernelCounts?.edges ?? result.edges.length;
// A file whose grammar failed to load was never parsed (#2335).
const grammarUnavailable = hasGrammarLoadFailure(result.errors);
// Store: on the writer thread when active (fresh DB — bundles applied
// in the same file order this chain dispatches them), else on the main
// thread (SQLite connections are per-thread).
const language = detectLanguage(filePath, content, overrides);
if (storeWriter) {
if (grammarUnavailable) {
// Store nothing: a row from an earlier run keeps its data, and with no
// row (or an older hash) the next sync or index retries the file.
} else if (storeWriter) {
if (result.kernelBuffers) {
// Buffers go to the writer as-is; the worker decodes + finalizes.
// The main thread's only per-file work stays O(1) + the content hash.
@@ -2321,7 +2327,9 @@ export class ExtractionOrchestrator {
errors.push(...result.errors);
}
if (nodeCount > 0) {
if (grammarUnavailable) {
filesErrored++;
} else if (nodeCount > 0) {
filesIndexed++;
totalNodes += nodeCount;
totalEdges += edgeCount;
@@ -2735,7 +2743,9 @@ export class ExtractionOrchestrator {
errors.push(...result.errors);
}
if (result.nodes.length > 0) {
if (hasGrammarLoadFailure(result.errors)) {
filesErrored++; // nothing was stored (#2335)
} else if (result.nodes.length > 0) {
filesIndexed++;
totalNodes += result.nodes.length;
totalEdges += result.edges.length;
@@ -2918,6 +2928,12 @@ export class ExtractionOrchestrator {
result: ExtractionResult,
onYield?: MaybeYield
): Promise<void> {
// The file was never parsed: its grammar failed to load (#2335). Storing
// this would replace the file's symbols with an empty row under the new
// content hash, which no hash-based sync revisits. Keep whatever the index
// has; the stale (or missing) row makes the next sync retry the file.
if (hasGrammarLoadFailure(result.errors)) return;
// A kernel result can arrive as an undecoded buffer transport (empty
// node/edge arrays, tables riding in kernelBuffers). Decode it before
// storing — persisting the transport as-is records the file as having no
@@ -2944,7 +2960,10 @@ export class ExtractionOrchestrator {
const existingIsMarker =
existingFile.nodeCount === 0 && (existingFile.errors?.length ?? 0) > 0;
const incomingHasContent = result.nodes.length > 0;
if (!existingIsMarker || !incomingHasContent) {
// A row an older engine stored while the grammar could not load
// (#2335) records no parse at all: any real result replaces it.
const existingNeverParsed = hasGrammarLoadFailure(existingFile.errors);
if (!existingNeverParsed && (!existingIsMarker || !incomingHasContent)) {
return; // No changes
}
}
@@ -3400,13 +3419,18 @@ export class ExtractionOrchestrator {
}
const fullPath = path.join(this.rootDir, filePath);
const tracked = trackedMap.get(filePath);
// A row an older engine stored while the file's grammar could not load
// (#2335) holds no parse of these bytes: re-index it even though its
// size, mtime and hash all match. Rows with a real parse error are
// deterministic and are not retried.
const neverParsed = tracked !== undefined && hasGrammarLoadFailure(tracked.errors);
// Cheap pre-filter: an already-indexed file whose size AND mtime both match
// the DB is unchanged — skip it without reading or hashing. (A content
// change that preserves both exactly is the blind spot every mtime-based
// incremental tool accepts; `index --force` is the escape hatch. Git bumps
// mtime on every file it writes during checkout/merge, so pulls are caught.)
if (tracked) {
if (tracked && !neverParsed) {
try {
const stat = fs.statSync(fullPath);
if (stat.size === tracked.size && Math.floor(stat.mtimeMs) === Math.floor(tracked.modifiedAt)) {
@@ -3442,7 +3466,7 @@ export class ExtractionOrchestrator {
filesToIndex.push(filePath);
changedFilePaths.push(filePath);
filesAdded++;
} else if (tracked.contentHash !== contentHash) {
} else if (tracked.contentHash !== contentHash || neverParsed) {
onFileChange?.(filePath, content);
filesToIndex.push(filePath);
changedFilePaths.push(filePath);
@@ -3477,7 +3501,7 @@ export class ExtractionOrchestrator {
const result = await this.indexFile(filePath);
if (result.errors.some(e => e.severity === 'error')) failedFilePaths.push(filePath);
nodesUpdated += result.nodes.length;
if (!hasGrammarLoadFailure(result.errors)) nodesUpdated += result.nodes.length; // else nothing was stored (#2335)
const pause = backpressure?.();
if (pause) await pause;
+30
View File
@@ -25,6 +25,7 @@ import {
BuildContextOptions,
FindRelevantContextOptions,
UnresolvedReference,
IndexHealth,
} from './types';
import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve';
@@ -43,6 +44,7 @@ import {
extractFromSource,
initGrammars,
} from './extraction';
import { hasGrammarLoadFailure, isFileLevelOnlyLanguage } from './extraction/grammars';
import {
ReferenceResolver,
createResolver,
@@ -2065,6 +2067,34 @@ export class CodeGraph {
return this.queries.countGeneratedFiles();
}
/**
* Indexed files whose symbols are missing although their content is current
* — rows a content-hash comparison calls up to date (#2336). Reported by
* `status`; `sync` repairs the `needsReindex` group.
*/
getIndexHealth(): IndexHealth {
const needsReindex: string[] = [];
const parseErrors: string[] = [];
for (const file of this.queries.getFilesWithoutNodesOrWithErrors()) {
const errors = file.errors ?? [];
if (hasGrammarLoadFailure(errors)) {
// Stored without being parsed — its grammar could not load (#2335).
needsReindex.push(file.path);
} else if (file.nodeCount === 0) {
// Every parse stores at least the file node, so zero nodes means a
// wiped row (#1541) or a recorded failure — except file-level-only
// languages and files over the size limit, which are empty on purpose.
if (isFileLevelOnlyLanguage(file.language)) continue;
if (errors.length === 0) needsReindex.push(file.path);
else if (errors.some((e) => e.severity === 'error' || e.code === 'parse_error')) parseErrors.push(file.path);
} else if (errors.some((e) => e.code === 'parse_error')) {
// Parsed, but the tree had errors and no symbols survived.
parseErrors.push(file.path);
}
}
return { needsReindex, parseErrors };
}
// ===========================================================================
// Graph Query Methods
// ===========================================================================
+71 -1
View File
@@ -54,7 +54,7 @@ import {
getDaemonSocketCandidates,
getDaemonSocketPath,
} from './daemon-paths';
import { CodeGraphPackageVersion } from './version';
import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version';
import {
releaseWriterLock,
swapWriterLock,
@@ -120,6 +120,38 @@ export function finalizeDaemonExit(
/** How often the daemon sweeps connected clients for a dead peer process (#692). */
const DEFAULT_CLIENT_SWEEP_MS = 30_000;
/** How often the daemon checks that the install it runs from is still in place (#2335). */
const DEFAULT_INSTALL_CHECK_MS = 30_000;
/**
* Why the install this daemon was started from no longer matches it, or null
* while it still does. An upgrade replaces or deletes the package directory
* under a running daemon (npm reinstalling the package, the launcher pruning
* an old bundle), and everything the daemon loads lazily from there afterwards
* fails — grammars among them, so every file its watcher re-indexed came out
* empty (#2335). Such a daemon should make way for one started from the
* current install. Any other read failure, or a package.json caught mid-write,
* answers null: the next check looks again. Exported for testing.
*/
export function installChangedReason(packageJsonPath: string, runningVersion: string): string | null {
let raw: string;
try {
raw = fs.readFileSync(packageJsonPath, 'utf8');
} catch (err) {
const code = (err as NodeJS.ErrnoException).code;
return code === 'ENOENT' || code === 'ENOTDIR' ? `Install removed (${packageJsonPath} is gone)` : null;
}
let version: unknown;
try {
version = (JSON.parse(raw) as { version?: unknown } | null)?.version;
} catch {
return null;
}
return typeof version === 'string' && version !== runningVersion
? `Install replaced by v${version} (this daemon is v${runningVersion})`
: null;
}
/** How long the daemon waits for the optional client-hello before proceeding without it. */
const CLIENT_HELLO_TIMEOUT_MS = 3_000;
@@ -185,6 +217,7 @@ export class Daemon {
private lastActivityAt = Date.now();
private maxIdleTimer: NodeJS.Timeout | null = null;
private clientSweepTimer: NodeJS.Timeout | null = null;
private installCheckTimer: NodeJS.Timeout | null = null;
private engine: MCPEngine;
private stopping = false;
private socketPath: string;
@@ -372,6 +405,10 @@ export class Daemon {
clearInterval(this.clientSweepTimer);
this.clientSweepTimer = null;
}
if (this.installCheckTimer) {
clearInterval(this.installCheckTimer);
this.installCheckTimer = null;
}
process.stderr.write(`[CodeGraph daemon] Shutting down (${reason}; clients=${this.clients.size}).\n`);
for (const session of [...this.clients]) {
try { session.stop(); } catch { /* best-effort */ }
@@ -491,6 +528,31 @@ export class Daemon {
this.clientSweepTimer = setInterval(() => this.reapDeadClients(isProcessAlive), sweepMs);
this.clientSweepTimer.unref?.();
}
// An install whose version could not be read at startup has nothing to
// compare against; leave that daemon alone.
const installMs = resolveInstallCheckMs();
if (installMs > 0 && CodeGraphPackageVersion !== '0.0.0-unknown') {
this.installCheckTimer = setInterval(() => { this.checkInstall(); }, installMs);
this.installCheckTimer.unref?.();
}
}
/**
* Exit once the install this daemon runs from has been upgraded or removed
* (#2335; see {@link installChangedReason}). Its clients' proxies fall back
* and reconnect, and the next launch starts a daemon from the current
* install. Returns whether the daemon is exiting. The check is injected for
* tests; the timer passes the real one.
*/
checkInstall(
changed: () => string | null = () => installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion),
): boolean {
if (this.stopping) return false;
const reason = changed();
if (!reason) return false;
process.stderr.write(`[CodeGraph daemon] ${reason}; exiting so the next session starts a daemon from the current install.\n`);
void this.stop('install changed');
return true;
}
/**
@@ -865,6 +927,14 @@ function resolveClientSweepMs(): number {
return Math.floor(parsed); // 0 disables the sweep
}
function resolveInstallCheckMs(): number {
const raw = process.env.CODEGRAPH_DAEMON_INSTALL_CHECK_MS;
if (raw === undefined || raw === '') return DEFAULT_INSTALL_CHECK_MS;
const parsed = Number(raw);
if (!Number.isFinite(parsed) || parsed < 0) return DEFAULT_INSTALL_CHECK_MS;
return Math.floor(parsed); // 0 disables the check
}
/**
* Parse one client-hello line. Returns the peer pids if `line` is a well-formed
* client-hello (carries the `codegraph_client` marker), or null otherwise — in
+16 -1
View File
@@ -44,12 +44,13 @@ import { MCPSession } from './session';
import {
Daemon,
clearStaleDaemonLock,
installChangedReason,
isProcessAlive,
tryAcquireDaemonLock,
} from './daemon';
import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry';
import { connectWithHello, runLocalHandshakeProxy } from './proxy';
import { CodeGraphPackageVersion } from './version';
import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version';
import {
readWriterLock,
assertNoRebuild,
@@ -150,6 +151,14 @@ function makeFallbackEngine(root: string): MCPEngine {
if (existing && isProcessAlive(existing.pid)) {
return readOnlyFallback(`live daemon PID ${existing.pid} holds the project lock`);
}
// This session's own install was upgraded or removed underneath it (#2335):
// its code no longer loads whole, so as the project's writer it would only
// keep a daemon from the current install from starting. Its daemon exits
// for the same reason, which is how it got here.
if (CodeGraphPackageVersion !== '0.0.0-unknown' &&
installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion) !== null) {
return readOnlyFallback('this session\'s CodeGraph install was upgraded or removed; restart the session to use the current one');
}
return new MCPEngine({ writerLockRoot: root, queryPool: true, queryPoolDefaultMax: DIRECT_QUERY_POOL_MAX });
}
@@ -295,6 +304,12 @@ function spawnDetachedDaemon(root: string, handover = false): void {
env,
},
);
// An upgrade can delete the install this launcher runs from — its daemon
// then exits (#2335) and a respawn finds no executable. spawn reports that
// as an asynchronous 'error' event, which would crash this process if
// nobody listened. No daemon binds either way; the caller's poll gives up
// and the session is served in-process.
child.on('error', () => { /* no daemon — see above */ });
child.unref();
} finally {
// The child holds its own dup of the log fd now; the launcher doesn't need it.
+4 -2
View File
@@ -20,10 +20,12 @@
import * as fs from 'fs';
import * as path from 'path';
/** The `package.json` of the install this process runs from. */
export const CodeGraphPackageJsonPath = path.join(__dirname, '..', '..', 'package.json');
function readPackageVersion(): string {
try {
const pkgPath = path.join(__dirname, '..', '..', 'package.json');
const raw = fs.readFileSync(pkgPath, 'utf8');
const raw = fs.readFileSync(CodeGraphPackageJsonPath, 'utf8');
const parsed = JSON.parse(raw);
if (typeof parsed?.version === 'string' && parsed.version.length > 0) {
return parsed.version;
+18
View File
@@ -265,6 +265,24 @@ export interface FileRecord {
generated?: boolean;
}
/**
* Indexed files whose symbols are missing even though their content hash is
* current — what a hash comparison alone calls "up to date" (#2336). Paths are
* project-relative and sorted.
*/
export interface IndexHealth {
/**
* Files `sync` re-indexes: stored while their grammar could not load (#2335)
* or stored with no nodes and no recorded reason (#1541).
*/
needsReindex: string[];
/**
* Files with a recorded parse error and no symbols from it. Deterministic:
* they stay this way until the file (or the parser) changes.
*/
parseErrors: string[];
}
// =============================================================================
// Extraction Types
// =============================================================================