mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-24 19:30:10 +08:00
Two coupled changes addressing the issue's underlying ask — "how does the
agent know when the index lags" — without resorting to a static wait.
Per-file staleness banner
-------------------------
FileWatcher now tracks per-path `pendingFiles` (path, firstSeenMs,
lastSeenMs, indexing) — events since the last successful sync, cleared
only after a sync whose `syncStartedMs >= lastSeenMs` commits. Chokidar
initial-scan events are gated behind a `ready` flag (with `waitUntilReady()`
exposed so tests can deterministically wait through it) so a fresh startup
doesn't falsely flag every existing file as pending.
ToolHandler now wraps every code-returning response (search, context,
callers, callees, impact, trace, explore, node, files) with
`withStalenessNotice`: intersects "files referenced in the response" with
`getPendingFiles()` and emits a hybrid signal —
* banner at the top for files referenced AND pending (with edit age +
indexing/pending-sync state, telling the agent to Read those specific
files directly; the rest of the response stays fresh and codegraph
stays authoritative for it),
* compact footer for pending files elsewhere in the project not
referenced above (capped at 5).
Cost is one boolean check + N substring matches when pending; zero
allocation when idle. `codegraph_status` surfaces the same data as a
first-class `### Pending sync:` section so the agent can ask "is the index
caught up?" in one call.
Cross-project quirk: when an agent passes `projectPath` matching the
default session's project, the staleness wrapper switches from the cached
cross-project CodeGraph (no watcher) to the default one (with watcher) so
the signal still fires. Same fix applied to `handleStatus`.
CODEGRAPH_WATCH_DEBOUNCE_MS
---------------------------
MCP `serve --mcp` now reads `CODEGRAPH_WATCH_DEBOUNCE_MS` and forwards it
to `cg.watch({ debounceMs })`. Clamped to [100ms, 60s]; out-of-range or
non-numeric values fall back to the FileWatcher default (2000ms). Active
value is logged to stderr on watcher startup so it's discoverable. The
docs in `server-instructions.ts`, `installer/instructions-template.ts`,
and `.cursor/rules/codegraph.mdc` no longer claim "~500ms"; they now
describe the banner mechanism instead — since per-file staleness replaces
the "wait N ms" guidance entirely, the docs become accurate at any
debounce value.
Validation
----------
* 847 unit/integration tests pass (added 15 new ones — pending-file
tracking, banner/footer routing, status section, env-var parsing).
* Direct MCP probe through a real `codegraph serve --mcp` process: edit a
file, query within the debounce window, banner fires naming the
edited file with edit-age.
* Real Claude TUI session via `scripts/agent-eval/itrun.sh` with
`CODEGRAPH_WATCH_DEBOUNCE_MS=10000`: agent edits `math.ts`, calls
`codegraph_explore`, reads the banner, **and discloses it unprompted in
its final reply**: "note: symbol index is mid-sync for the new `divide`,
but the source it returned is verbatim from disk."
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
4a4a37d135
commit
b48170e69f
+27
-2
@@ -46,7 +46,7 @@ import {
|
||||
import { GraphTraverser, GraphQueryManager } from './graph';
|
||||
import { ContextBuilder, createContextBuilder } from './context';
|
||||
import { Mutex, FileLock } from './utils';
|
||||
import { FileWatcher, WatchOptions } from './sync';
|
||||
import { FileWatcher, WatchOptions, PendingFile } from './sync';
|
||||
|
||||
// Re-export types for consumers
|
||||
export * from './types';
|
||||
@@ -75,7 +75,7 @@ export {
|
||||
defaultLogger,
|
||||
} from './errors';
|
||||
export { Mutex, FileLock, processInBatches, debounce, throttle, MemoryMonitor } from './utils';
|
||||
export { FileWatcher, WatchOptions } from './sync';
|
||||
export { FileWatcher, WatchOptions, PendingFile } from './sync';
|
||||
export { MCPServer } from './mcp';
|
||||
|
||||
/**
|
||||
@@ -499,6 +499,31 @@ export class CodeGraph {
|
||||
return this.watcher?.isActive() ?? false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Files seen by the file watcher since the last successful sync —
|
||||
* the per-file "stale" signal MCP tools attach to responses so an agent
|
||||
* can fall back to {@link Read} for just the affected file without
|
||||
* waiting for a debounced sync to complete (issue #403).
|
||||
*
|
||||
* Returns an empty list when the watcher isn't active, or no events have
|
||||
* arrived. Each entry includes `firstSeenMs` and `lastSeenMs` (wall-clock
|
||||
* `Date.now()` values) so callers can render "edited Nms ago", plus an
|
||||
* `indexing` flag indicating whether the in-flight sync (if any) will
|
||||
* absorb that file.
|
||||
*/
|
||||
getPendingFiles(): PendingFile[] {
|
||||
return this.watcher?.getPendingFiles() ?? [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves once the file watcher has finished its initial chokidar scan.
|
||||
* Useful for tests that need a deterministic boundary before asserting on
|
||||
* `getPendingFiles()`. Resolves immediately when no watcher is active.
|
||||
*/
|
||||
waitUntilWatcherReady(timeoutMs?: number): Promise<void> {
|
||||
return this.watcher ? this.watcher.waitUntilReady(timeoutMs) : Promise.resolve();
|
||||
}
|
||||
|
||||
/**
|
||||
* Get files that have changed since last index
|
||||
*/
|
||||
|
||||
@@ -49,7 +49,7 @@ Use codegraph for **structural** questions — what calls what, what would break
|
||||
- **Don't grep first** when looking up a symbol by name. \`codegraph_search\` is faster and returns kind + location + signature in one call.
|
||||
- **Don't chain \`codegraph_search\` + \`codegraph_node\`** when you just want context — \`codegraph_context\` is one call.
|
||||
- **Don't loop \`codegraph_node\` over many symbols** — one \`codegraph_explore\` call returns several symbols' source grouped in a single capped call, while each separate node/Read call re-reads the whole context and costs far more.
|
||||
- **Index lag**: the file watcher debounces ~500ms behind writes; don't re-query immediately after editing a file in the same turn.
|
||||
- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. \`codegraph_status\` also lists pending files under "Pending sync".
|
||||
|
||||
### If \`.codegraph/\` doesn't exist
|
||||
|
||||
|
||||
@@ -185,7 +185,18 @@ export class MCPEngine {
|
||||
return;
|
||||
}
|
||||
|
||||
// Optional override for the debounce window via env var (issue #403).
|
||||
// Useful for workspaces with bursty writes (formatter-on-save chains,
|
||||
// large generated outputs) where the 2s default fires too often. Clamped
|
||||
// to [100ms, 60s]; out-of-range / non-numeric values fall back to the
|
||||
// FileWatcher default. We log the active value so it's discoverable.
|
||||
const debounceMs = parseDebounceEnv(process.env.CODEGRAPH_WATCH_DEBOUNCE_MS);
|
||||
if (debounceMs !== undefined) {
|
||||
process.stderr.write(`[CodeGraph MCP] File watcher debounce: ${debounceMs}ms (CODEGRAPH_WATCH_DEBOUNCE_MS)\n`);
|
||||
}
|
||||
|
||||
const started = this.cg.watch({
|
||||
debounceMs,
|
||||
onSyncComplete: (result) => {
|
||||
if (result.filesChanged > 0) {
|
||||
process.stderr.write(
|
||||
@@ -230,3 +241,24 @@ export class MCPEngine {
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse and clamp the CODEGRAPH_WATCH_DEBOUNCE_MS env override.
|
||||
*
|
||||
* Issue #403: workspaces with bursty writes (formatter-on-save, multi-file
|
||||
* refactors) sometimes want a longer quiet window before sync. Returns
|
||||
* `undefined` for unset / empty / non-numeric / out-of-range values so the
|
||||
* FileWatcher default (2000ms) takes over — never throws.
|
||||
*
|
||||
* Clamp range: 100ms (faster would mean a sync per keystroke) to 60s (longer
|
||||
* and the watcher feels broken). Out-of-range values are treated as "ignore
|
||||
* this misconfiguration" rather than capped, since silently capping a 0 or
|
||||
* a typoed value would mask a real config bug.
|
||||
*/
|
||||
export function parseDebounceEnv(raw: string | undefined): number | undefined {
|
||||
if (!raw || !raw.trim()) return undefined;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || !Number.isInteger(n)) return undefined;
|
||||
if (n < 100 || n > 60000) return undefined;
|
||||
return n;
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ of calls; a grep/read exploration is dozens.
|
||||
- **Don't grep first** when looking up a symbol by name — \`codegraph_search\` is faster and returns kind + location + signature.
|
||||
- **Don't chain \`codegraph_search\` + \`codegraph_node\`** when you just want context — \`codegraph_context\` is one round-trip.
|
||||
- **Don't loop \`codegraph_node\` over many symbols** — one \`codegraph_explore\` call returns them all grouped by file, while each separate call re-reads the whole context and costs far more. Use \`codegraph_node\` for a single symbol.
|
||||
- **Don't query the index immediately after editing a file** — the watcher needs ~500ms to debounce + sync. Wait for the next turn.
|
||||
- **After editing, check the staleness banner.** When a tool response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Every file NOT in that banner is fresh, so still trust codegraph. \`codegraph_status\` also lists pending files under "Pending sync".
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
+160
-6
@@ -11,6 +11,7 @@ import {
|
||||
worktreeMismatchNotice,
|
||||
type WorktreeIndexMismatch,
|
||||
} from '../sync/worktree';
|
||||
import type { PendingFile } from '../sync';
|
||||
import type { Node, Edge, SearchResult, Subgraph, TaskContext, NodeKind } from '../types';
|
||||
import { createHash } from 'crypto';
|
||||
import {
|
||||
@@ -24,7 +25,7 @@ import {
|
||||
} from 'fs';
|
||||
import { clamp, validatePathWithinRoot, validateProjectPath } from '../utils';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { join, resolve as resolvePath } from 'path';
|
||||
|
||||
/** Maximum output length to prevent context bloat (characters) */
|
||||
const MAX_OUTPUT_LENGTH = 15000;
|
||||
@@ -264,6 +265,48 @@ function markSessionConsulted(sessionId: string): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-file staleness banner emitted at the top of a tool response when the
|
||||
* file watcher has pending events for files referenced by the response.
|
||||
* The agent uses this to fall back to Read for those specific files
|
||||
* without waiting for the debounced sync (issue #403).
|
||||
*/
|
||||
export function formatStaleBanner(stale: PendingFile[]): string {
|
||||
const now = Date.now();
|
||||
const lines = stale.map((p) => {
|
||||
const ageMs = Math.max(0, now - p.lastSeenMs);
|
||||
const label = p.indexing ? 'indexing in progress' : 'pending sync';
|
||||
return ` - ${p.path} (edited ${ageMs}ms ago, ${label})`;
|
||||
});
|
||||
return (
|
||||
'⚠️ Some files referenced below were edited since the last index sync — ' +
|
||||
'their codegraph entries may be stale:\n' +
|
||||
lines.join('\n') +
|
||||
'\nFor accurate content of those specific files, Read them directly. ' +
|
||||
'The rest of this response is fresh.'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compact footer listing pending files that are NOT referenced in this
|
||||
* response. Gives the agent a complete project-wide freshness picture
|
||||
* without bloating the main banner.
|
||||
*/
|
||||
export function formatStaleFooter(stale: PendingFile[]): string {
|
||||
const MAX = 5;
|
||||
const now = Date.now();
|
||||
const shown = stale.slice(0, MAX);
|
||||
const lines = shown.map((p) => {
|
||||
const ageMs = Math.max(0, now - p.lastSeenMs);
|
||||
return ` - ${p.path} (edited ${ageMs}ms ago)`;
|
||||
});
|
||||
const more = stale.length > MAX ? `\n - …and ${stale.length - MAX} more` : '';
|
||||
return (
|
||||
`(Note: ${stale.length} file(s) elsewhere in this project are pending index ` +
|
||||
`sync but were not referenced above:\n${lines.join('\n')}${more})`
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* MCP Tool definition
|
||||
*/
|
||||
@@ -802,6 +845,84 @@ export class ToolHandler {
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Annotate a successful read-tool result with per-file staleness — the
|
||||
* non-blocking answer to issue #403. The file watcher tracks every event
|
||||
* it sees per path; here we intersect "files referenced in this response"
|
||||
* against that pending set and prepend a compact banner so the agent can
|
||||
* fall back to Read for those *specific* files without waiting for the
|
||||
* debounced sync to fire. Other pending files in the project (not
|
||||
* referenced by this response) get a small footer so the agent has a
|
||||
* complete picture without bloating the banner.
|
||||
*
|
||||
* Cost when nothing is pending — the common case — is one boolean check.
|
||||
* No I/O, no parsing of markdown beyond a per-pending-file substring scan.
|
||||
*/
|
||||
private withStalenessNotice(result: ToolResult, projectPath?: string): ToolResult {
|
||||
if (result.isError) return result;
|
||||
|
||||
let cg: CodeGraph;
|
||||
try {
|
||||
cg = this.getCodeGraph(projectPath);
|
||||
} catch {
|
||||
return result; // no default project — leave as is
|
||||
}
|
||||
|
||||
// Cross-project `projectPath` calls open a cached CodeGraph WITHOUT a
|
||||
// watcher (watchers are only attached to the default session project).
|
||||
// When the cross-project path happens to be the same project as the
|
||||
// default cg, the cached instance is the wrong one — its pendingFiles is
|
||||
// permanently empty. Detect the equal-path case and prefer the default
|
||||
// cg so the staleness signal still fires when an agent passes the
|
||||
// explicit projectPath form of its own project.
|
||||
if (this.cg && cg !== this.cg) {
|
||||
try {
|
||||
const sameProject =
|
||||
resolvePath(this.cg.getProjectRoot()) === resolvePath(cg.getProjectRoot());
|
||||
if (sameProject) cg = this.cg;
|
||||
} catch {
|
||||
/* getProjectRoot may throw on a closed instance — leave cg as is */
|
||||
}
|
||||
}
|
||||
|
||||
// Defensive: some test fakes inject a partial CodeGraph stub without the
|
||||
// newer pending-files API. Treat missing/throwing as "no pending files."
|
||||
let pending: PendingFile[] = [];
|
||||
try {
|
||||
pending = cg.getPendingFiles?.() ?? [];
|
||||
} catch {
|
||||
return result;
|
||||
}
|
||||
if (pending.length === 0) return result;
|
||||
|
||||
const [first, ...rest] = result.content;
|
||||
if (!first || first.type !== 'text') return result;
|
||||
|
||||
const text = first.text;
|
||||
const inResponse: PendingFile[] = [];
|
||||
const elsewhere: PendingFile[] = [];
|
||||
for (const p of pending) {
|
||||
// Substring match against the project-relative POSIX path — that's
|
||||
// exactly the format both the watcher and every codegraph response
|
||||
// emit, so a plain includes() is sufficient and avoids regex pitfalls.
|
||||
if (text.includes(p.path)) inResponse.push(p);
|
||||
else elsewhere.push(p);
|
||||
}
|
||||
|
||||
let banner = '';
|
||||
if (inResponse.length > 0) {
|
||||
banner = formatStaleBanner(inResponse);
|
||||
}
|
||||
let footer = '';
|
||||
if (elsewhere.length > 0) {
|
||||
footer = formatStaleFooter(elsewhere);
|
||||
}
|
||||
if (!banner && !footer) return result;
|
||||
|
||||
const composed = [banner, text, footer].filter(Boolean).join('\n\n');
|
||||
return { ...result, content: [{ type: 'text', text: composed }, ...rest] };
|
||||
}
|
||||
|
||||
/**
|
||||
* Execute a tool by name
|
||||
*/
|
||||
@@ -831,9 +952,12 @@ export class ToolHandler {
|
||||
if (typeof check === 'object' && check !== undefined) return check;
|
||||
}
|
||||
|
||||
// Read tools resolve through a single result variable so the worktree
|
||||
// mismatch notice can be prefixed in one place (issue #155). status is
|
||||
// returned directly — it embeds its own verbose warning.
|
||||
// Read tools resolve through a single result variable so cross-cutting
|
||||
// notices — worktree-index mismatch (issue #155) and per-file
|
||||
// staleness (issue #403) — can be applied in one place. status embeds
|
||||
// its own verbose worktree warning but still flows through the
|
||||
// staleness wrapper so its pending-files section stays consistent
|
||||
// with what the read tools surface.
|
||||
let result: ToolResult;
|
||||
switch (toolName) {
|
||||
case 'codegraph_search':
|
||||
@@ -851,6 +975,9 @@ export class ToolHandler {
|
||||
case 'codegraph_node':
|
||||
result = await this.handleNode(args); break;
|
||||
case 'codegraph_status':
|
||||
// status embeds the pending-files list as a first-class section
|
||||
// (see handleStatus), so we skip the auto-banner wrapper here to
|
||||
// avoid duplicating the same info at the top of the response.
|
||||
return await this.handleStatus(args);
|
||||
case 'codegraph_files':
|
||||
result = await this.handleFiles(args); break;
|
||||
@@ -859,7 +986,8 @@ export class ToolHandler {
|
||||
default:
|
||||
return this.errorResult(`Unknown tool: ${toolName}`);
|
||||
}
|
||||
return this.withWorktreeNotice(result, args.projectPath as string | undefined);
|
||||
const withWorktree = this.withWorktreeNotice(result, args.projectPath as string | undefined);
|
||||
return this.withStalenessNotice(withWorktree, args.projectPath as string | undefined);
|
||||
} catch (err) {
|
||||
return this.errorResult(`Tool execution failed: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
@@ -2016,7 +2144,18 @@ export class ToolHandler {
|
||||
* Handle codegraph_status
|
||||
*/
|
||||
private async handleStatus(args: Record<string, unknown>): Promise<ToolResult> {
|
||||
const cg = this.getCodeGraph(args.projectPath as string | undefined);
|
||||
let cg = this.getCodeGraph(args.projectPath as string | undefined);
|
||||
// Same trick as withStalenessNotice — when an explicit projectPath
|
||||
// resolves to the same project as the default session cg, prefer the
|
||||
// default so getPendingFiles() (only populated by the default's watcher)
|
||||
// is non-empty when there are pending edits.
|
||||
if (this.cg && cg !== this.cg) {
|
||||
try {
|
||||
if (resolvePath(this.cg.getProjectRoot()) === resolvePath(cg.getProjectRoot())) {
|
||||
cg = this.cg;
|
||||
}
|
||||
} catch { /* closed instance — leave as is */ }
|
||||
}
|
||||
const stats = cg.getStats();
|
||||
|
||||
// Warn when this index actually belongs to a different git working tree
|
||||
@@ -2073,6 +2212,21 @@ export class ToolHandler {
|
||||
}
|
||||
}
|
||||
|
||||
// Per-file freshness — the inverse of the auto-prepended staleness banner
|
||||
// (issue #403). Surfacing it inside `status` gives the agent a single
|
||||
// place to ask "is the index caught up?" rather than inferring from
|
||||
// banners on other tool calls.
|
||||
const pending = cg.getPendingFiles();
|
||||
if (pending.length > 0) {
|
||||
lines.push('', '### Pending sync:');
|
||||
const now = Date.now();
|
||||
for (const p of pending) {
|
||||
const ageMs = Math.max(0, now - p.lastSeenMs);
|
||||
const label = p.indexing ? 'indexing in progress' : 'pending sync';
|
||||
lines.push(`- ${p.path} (edited ${ageMs}ms ago, ${label})`);
|
||||
}
|
||||
}
|
||||
|
||||
return this.textResult(lines.join('\n'));
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@
|
||||
* - Incremental reindexing (in extraction module)
|
||||
*/
|
||||
|
||||
export { FileWatcher, WatchOptions } from './watcher';
|
||||
export { FileWatcher, WatchOptions, PendingFile } from './watcher';
|
||||
export { watchDisabledReason, detectWsl } from './watch-policy';
|
||||
export {
|
||||
installGitSyncHook,
|
||||
|
||||
+163
-6
@@ -45,6 +45,26 @@ export interface WatchOptions {
|
||||
onSyncError?: (error: Error) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-file pending entry — tracks a source file the watcher saw an event for
|
||||
* but hasn't yet synced into the index. Exposed via {@link FileWatcher.getPendingFiles}
|
||||
* so MCP tool responses can mark stale results without forcing a wait.
|
||||
*/
|
||||
export interface PendingFile {
|
||||
/** Project-relative POSIX path (e.g. "src/foo.ts"). */
|
||||
path: string;
|
||||
/** Wall-clock ms at the first event we saw for this path since the last sync. */
|
||||
firstSeenMs: number;
|
||||
/** Wall-clock ms at the most recent event we saw for this path. */
|
||||
lastSeenMs: number;
|
||||
/**
|
||||
* True when a sync is currently in flight that began AFTER this file's most
|
||||
* recent event — i.e. the next successful sync will pick it up. False when
|
||||
* the file is still in the debounce window (no sync running yet).
|
||||
*/
|
||||
indexing: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* FileWatcher monitors a project directory for changes and triggers
|
||||
* debounced sync operations via a provided callback.
|
||||
@@ -55,13 +75,43 @@ export interface WatchOptions {
|
||||
* - Debounced to avoid thrashing on rapid saves
|
||||
* - Filters to supported source files by extension
|
||||
* - Ignores .codegraph/ and .git/ regardless of .gitignore
|
||||
* - Tracks per-file pending state so MCP tools can flag stale results
|
||||
* without blocking on a sync (issue #403)
|
||||
*/
|
||||
export class FileWatcher {
|
||||
private watcher: FSWatcher | null = null;
|
||||
private debounceTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
private hasChanges = false;
|
||||
/**
|
||||
* Files seen by the watcher since the last successful sync — populated on
|
||||
* every chokidar event, cleared at the start of a sync, and re-populated by
|
||||
* events that arrive mid-sync (or restored on sync failure). Keyed by the
|
||||
* same project-relative POSIX path the rest of the codebase uses, so a
|
||||
* caller can intersect tool-response file paths against this map cheaply.
|
||||
*/
|
||||
private pendingFiles = new Map<string, { firstSeenMs: number; lastSeenMs: number }>();
|
||||
/**
|
||||
* Wall-clock ms at which the in-flight sync began. Combined with
|
||||
* {@link pendingFiles}'s `lastSeenMs`, this distinguishes "still in the
|
||||
* debounce window" (lastSeen > syncStarted, sync hasn't started yet for
|
||||
* this edit) from "currently being indexed" (lastSeen <= syncStarted).
|
||||
*/
|
||||
private syncStartedMs = 0;
|
||||
private syncing = false;
|
||||
private stopped = false;
|
||||
/**
|
||||
* False until chokidar fires its `ready` event. Gates `pendingFiles`
|
||||
* insertion so the initial crawl's `add` events (one per pre-existing
|
||||
* source file) don't pollute the per-file staleness signal. The events
|
||||
* still flow into `scheduleSync()` to preserve the previous "initial
|
||||
* scan triggers a reconciling sync" behavior.
|
||||
*/
|
||||
private chokidarReady = false;
|
||||
/**
|
||||
* Callbacks that resolve when chokidar fires `ready`. Used by tests (and
|
||||
* any production caller that cares about a clean baseline) to deterministically
|
||||
* gate on the end of the initial scan instead of guessing at a sleep duration.
|
||||
*/
|
||||
private readyWaiters: Array<() => void> = [];
|
||||
// The shared ignore matcher (built-in defaults + project .gitignore), built
|
||||
// once at start(). Same source of truth the indexer uses, so watcher scope
|
||||
// can never diverge from index scope.
|
||||
@@ -116,6 +166,26 @@ export class FileWatcher {
|
||||
ignored: (testPath: string, stats?: Stats) => this.shouldIgnore(testPath, stats),
|
||||
});
|
||||
|
||||
// Chokidar emits `add` for every pre-existing source file during its
|
||||
// initial scan. Those events should still trigger the post-startup
|
||||
// reconciling sync (preserving prior behavior), but they must NOT land
|
||||
// in pendingFiles — otherwise every file in the project shows up as
|
||||
// "edited but not indexed" on startup, which is the opposite of the
|
||||
// signal #403 is supposed to provide. Flip the flag on chokidar's
|
||||
// `ready` event; from then on, real edits populate pendingFiles.
|
||||
//
|
||||
// We also clear `pendingFiles` here as defense-in-depth: chokidar can
|
||||
// emit late initial-scan `add` events via setImmediate AFTER the
|
||||
// `ready` callback runs (observed under test-parallelism load).
|
||||
// Clearing once at ready guarantees a clean baseline; real subsequent
|
||||
// edits repopulate the set normally.
|
||||
this.watcher.on('ready', () => {
|
||||
this.chokidarReady = true;
|
||||
this.pendingFiles.clear();
|
||||
for (const cb of this.readyWaiters) cb();
|
||||
this.readyWaiters.length = 0;
|
||||
});
|
||||
|
||||
// chokidar emits 'all' for every event type; we only sync source files.
|
||||
this.watcher.on('all', (_event: string, filePath: string) => {
|
||||
if (this.stopped) return;
|
||||
@@ -128,7 +198,17 @@ export class FileWatcher {
|
||||
if (!isSourceFile(normalized)) return;
|
||||
|
||||
logDebug('File change detected', { file: normalized });
|
||||
this.hasChanges = true;
|
||||
// Only track events from after chokidar's initial scan as pending
|
||||
// edits — pre-existing files on disk are already represented by
|
||||
// (or about to be reconciled by) the index, not a user edit.
|
||||
if (this.chokidarReady) {
|
||||
const now = Date.now();
|
||||
const existing = this.pendingFiles.get(normalized);
|
||||
this.pendingFiles.set(normalized, {
|
||||
firstSeenMs: existing?.firstSeenMs ?? now,
|
||||
lastSeenMs: now,
|
||||
});
|
||||
}
|
||||
this.scheduleSync();
|
||||
});
|
||||
|
||||
@@ -187,7 +267,8 @@ export class FileWatcher {
|
||||
this.watcher = null;
|
||||
}
|
||||
|
||||
this.hasChanges = false;
|
||||
this.pendingFiles.clear();
|
||||
this.chokidarReady = false;
|
||||
this.ignoreMatcher = null;
|
||||
logDebug('File watcher stopped');
|
||||
}
|
||||
@@ -199,6 +280,30 @@ export class FileWatcher {
|
||||
return this.watcher !== null && !this.stopped;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves once chokidar has fired its `ready` event (or immediately if
|
||||
* it has already done so). Useful for tests that need a deterministic
|
||||
* boundary before asserting on `pendingFiles` — guessing a sleep duration
|
||||
* is flaky under load because chokidar can take longer than expected to
|
||||
* finish its initial crawl on slow filesystems / parallel test runs.
|
||||
*
|
||||
* Production callers don't need this: `pendingFiles` is read continuously,
|
||||
* the staleness banner is always correct (empty or populated), and the
|
||||
* initial-scan window is a small one-time startup cost.
|
||||
*/
|
||||
waitUntilReady(timeoutMs = 10000): Promise<void> {
|
||||
if (this.chokidarReady) return Promise.resolve();
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
const idx = this.readyWaiters.indexOf(handler);
|
||||
if (idx >= 0) this.readyWaiters.splice(idx, 1);
|
||||
reject(new Error(`FileWatcher.waitUntilReady timed out after ${timeoutMs}ms`));
|
||||
}, timeoutMs);
|
||||
const handler = () => { clearTimeout(t); resolve(); };
|
||||
this.readyWaiters.push(handler);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule a debounced sync.
|
||||
*/
|
||||
@@ -214,28 +319,80 @@ export class FileWatcher {
|
||||
|
||||
/**
|
||||
* Flush pending changes by running sync.
|
||||
*
|
||||
* pendingFiles is NOT cleared at the start of sync — entries are removed
|
||||
* only after sync commits successfully, and only for entries whose
|
||||
* lastSeenMs <= syncStartedMs. That way, a query that arrives mid-sync
|
||||
* still sees the affected files marked stale (the DB hasn't been updated
|
||||
* yet), and an event that lands mid-sync persists into the follow-up.
|
||||
*
|
||||
* On sync failure pendingFiles is left untouched — every edit is still
|
||||
* unindexed, and the rescheduled sync will absorb the same set next time.
|
||||
*/
|
||||
private async flush(): Promise<void> {
|
||||
// If already syncing, the post-sync check will re-trigger
|
||||
if (this.syncing || this.stopped) return;
|
||||
|
||||
this.hasChanges = false;
|
||||
this.syncStartedMs = Date.now();
|
||||
this.syncing = true;
|
||||
|
||||
try {
|
||||
const result = await this.syncFn();
|
||||
// Remove entries whose most recent event predates this sync — those
|
||||
// edits are now in the DB. Entries with lastSeenMs > syncStartedMs
|
||||
// arrived mid-sync; whether the in-flight sync captured them depends
|
||||
// on when sync read that file, so we keep them as pending and let
|
||||
// the follow-up sync handle them. We prefer false positives ("shown
|
||||
// stale, actually fresh" → at worst one extra Read) over false
|
||||
// negatives ("shown fresh, actually stale" → misleads the agent).
|
||||
for (const [filePath, info] of this.pendingFiles) {
|
||||
if (info.lastSeenMs <= this.syncStartedMs) {
|
||||
this.pendingFiles.delete(filePath);
|
||||
}
|
||||
}
|
||||
this.onSyncComplete?.(result);
|
||||
} catch (err) {
|
||||
const error = err instanceof Error ? err : new Error(String(err));
|
||||
logWarn('Watch sync failed', { error: error.message });
|
||||
// Failure: leave pendingFiles untouched. Every edit it tracks is
|
||||
// still unindexed; the rescheduled sync sees the same set.
|
||||
this.onSyncError?.(error);
|
||||
} finally {
|
||||
this.syncing = false;
|
||||
|
||||
// If new changes arrived during sync, schedule another
|
||||
if (this.hasChanges && !this.stopped) {
|
||||
// If pending files remain (mid-sync events, or this sync failed),
|
||||
// schedule another pass.
|
||||
if (this.pendingFiles.size > 0 && !this.stopped) {
|
||||
this.scheduleSync();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot of files seen by the watcher since the last successful sync.
|
||||
*
|
||||
* Used by MCP tool responses to mark stale results without blocking on a
|
||||
* sync: a tool that returns a hit in `src/foo.ts` while `src/foo.ts` is in
|
||||
* this list tells the agent "Read this file directly, the index lags."
|
||||
*
|
||||
* `indexing` is true when a sync is currently in flight whose start time is
|
||||
* AFTER this file's most recent event — i.e. that sync will absorb the
|
||||
* edit. False means the file is still inside the debounce window and no
|
||||
* sync has started yet (a follow-up call a few hundred ms later may show
|
||||
* `indexing: true` or the file may have left the list entirely).
|
||||
*
|
||||
* Cheap: O(pendingFiles.size), no I/O, no locks.
|
||||
*/
|
||||
getPendingFiles(): PendingFile[] {
|
||||
const result: PendingFile[] = [];
|
||||
for (const [filePath, info] of this.pendingFiles) {
|
||||
result.push({
|
||||
path: filePath,
|
||||
firstSeenMs: info.firstSeenMs,
|
||||
lastSeenMs: info.lastSeenMs,
|
||||
indexing: this.syncing && this.syncStartedMs >= info.lastSeenMs,
|
||||
});
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user