Files
codegraph/__tests__/git-index-currency.test.ts
Colby Mchenry 04ec5f554d fix(extraction): never read an oversize file; an MPEG-TS video named .ts is not TypeScript (#1910) (#2082)
* fix(extraction): an MPEG-TS video named .ts is not TypeScript (#1910)

`.ts` mapped to TypeScript by extension alone, so MPEG transport-stream
fixtures (`testdata/*.ts`) were parsed by tree-sitter: ~28 s of CPU for a
900 KB clip, minutes for a directory of them, for zero symbols.

Recognise the stream from the head of the file — the 0x47 sync byte at
offsets 0, 188, 376 and 564 plus a NUL byte, which every stream carries in
its first packets and UTF-8 source never does — and drop it at discovery:
not indexed, not parsed, not counted, not tallied as an unsupported
language. The check reads under 1 KB and only for `.ts` files; the batch
reader sniffs the bytes it already read, and the single-file path (sync,
watcher) does the same head read, so a clip handed in by name is skipped
too. The skip happens before parse dispatch, so no kernel mirror is needed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(extraction): a video .ts is never pending on the git path or a scoped sync

Review found two places that decide "is this a source file" without the
MPEG-TS check the scan applies: the git fast path's candidate loop in
getChangedFiles and the scoped-sync path filter (watcher events). An
untracked clip in a git repo was reported as `added`, skipped by sync
without a record, and reported as `added` again on every status; a tracked
.ts that became a clip stayed `modified` with its stale nodes. Both now
treat the clip as non-source: removed when tracked, ignored otherwise.

Regression tests fail without this change and pass with it, on both arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): require a binary head before calling a .ts file video (#1910)

Review on #1915: the sniff took four aligned 0x47 sync bytes plus any NUL
as proof of a transport stream, so TypeScript with a `G` at four 188-byte
strides and one raw NUL in a comment was dropped unindexed. The sniff now
needs the sync byte on sixteen consecutive packets (3 KB of head) and at
least 1/64 of the head to be control bytes. Compressed payload carries
8-20% of them (checked on ffmpeg-made H.264/AAC, MPEG-2 and MP2 streams);
source text carries none. A shorter clip is cheap to parse anyway.

Also drops the benchmark figures from the changelog entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): a file over the size limit is never read (#1910)

The size gate ran after `readFile`, so a committed video or blob fixture was
decoded in full — 3.4 GB of RSS for a 400 MB file, `Invalid string length`
past ~512 MB — only to be stored as skipped. It was then read again by every
pass that scans files by name: the framework detectors' `readFile` and the
resolver's cached reader, three full passes on the reporter's fixtures.

Stat first, everywhere a source file is read by path: the batch reader and
`extractFile` store an oversize file with a size stamp in place of its
content, change detection hashes the same stamp (so a same-size rewrite of a
file nothing is indexed from is not a change, while crossing the limit in
either direction is), and both by-name readers return null above the limit.
The limit moves to `src/file-limits.ts` so the three sites share one number.

strace on a sparse 400 MB `.ts`: 153 603 reads before, 0 after; indexing it
next to a real file costs no measurable RSS. Stacked on #1914, which shares
the batch-reader hunk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(extraction): one-purpose size-stamp reader, comments in operation order

`readSourceOrStamp` took a `stats` argument no caller passed and returned
stats no caller read; it now returns the text or stamp change detection
hashes. The batch reader's two comments are back in the order its code runs:
the size gate, then the MPEG-TS head check. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): bound the read itself, not only the stat before it (#1910)

A file can grow between the stat that gates the size limit and the read
that follows (a log, a download, a build output being written). Every
source read now goes through readBoundedSource/readBoundedSourceSync: the
open descriptor is re-checked, and the read stops one byte past the limit,
so an oversize file is still stamped rather than decoded. The extraction
batch reader, single-file extraction, framework detectors and the resolver's
file cache all use it. Ported from the maintainer's hardening of this fix
in #1919.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): bound the parse-retry reads too (#1910)

The retry passes for files whose worker crashed or timed out re-read the
source with an unbounded fsp.readFile. The file can have grown since the
first, bounded read; an oversize file is now skipped by the retry instead
of being decoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): check an oversize file against its size stamp everywhere (#1910)

Review on #1915:

- The viewer (readFileShape, hasDriftedOnDisk, the source endpoint) and
  MCP's drift gate hashed a file's real bytes, but the index stores the
  size stamp for a file over the limit, so every unchanged 1-8 MB file read
  as "changed on disk". oversizeStamp moves to file-limits.ts with an
  indexedHashInput helper, and all four checks hash what the index stored.
  An oversize file is now compared without being read.
- git-index-currency's "keeps a committed path pending when sync cannot
  read it" injected its failure only into readFileSync, which the bounded
  reader no longer calls, so it failed. The failure now reaches openSync
  too.
- The changelog entry drops its benchmark figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): check an oversize answer file against its size stamp (#1910)

#2043's answer-freshness check hashes each contributing file whole and
compares it with the stored contentHash. For a file over the index's size
limit the stored hash is the size stamp, so an unchanged oversize file
would read as stale. Compare it by its stamp, without reading it, like
the viewer and the MCP drift gate do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(extraction): judge a .ts clip from the bytes it is read for, not at discovery (#1910)

Discovery sniffed the head of every `.ts` file on every full scan — an
open + 3 KB read + close per file, before anything asked for its content.
On a 10k-file TypeScript tree that took a warm-cache scan from ~105 ms to
~243 ms; on a slow disk or a network share it is a random read per file
per scan, the access pattern #1231 and the SMB reports (#1014, #448) are
about.

The bytes are now judged where they are read anyway, so the check costs no
I/O and runs only for a file that is new or changed: the batch reader (as
before), `indexFile`, the sync reconcile (an untracked clip is skipped, a
tracked file that became one is removed through the same removal path as
a deletion), and both `getChangedFiles` paths. The scan lists `.ts` files
by name again; a test pins that discovery opens none of them on the walk
or the git path. Scan time is back to main's (~98 ms), and a full index is
unchanged within noise.

Also collapses the four #1910 changelog bullets (#1914 and #1915 each added
both) to two, without the benchmark figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: danusha2345 <ewidusoc498@gmail.com>
2026-09-29 03:01:46 +00:00

155 lines
7.7 KiB
TypeScript

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import * as cp from 'child_process';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { CodeGraph } from '../src';
// Wrap only I/O entry points for deterministic failure injection; all other
// calls, files, parser work and SQLite remain real.
vi.mock('child_process', async importOriginal => {
const actual = await importOriginal<typeof import('child_process')>();
return { ...actual, execFileSync: vi.fn(actual.execFileSync) };
});
vi.mock('fs', async importOriginal => {
const actual = await importOriginal<typeof import('fs')>();
return { ...actual, readFileSync: vi.fn(actual.readFileSync) };
});
describe('git index currency across commits and restores (#1829)', () => {
let root: string;
let cg: CodeGraph;
const git = (...args: string[]) => cp.execFileSync('git', args, { cwd: root, encoding: 'utf8', stdio: ['pipe', 'pipe', 'pipe'] });
const write = (name: string, symbol: string) => fs.writeFileSync(path.join(root, name), `export function ${symbol}() { return 1; }\n`);
const commit = () => { git('add', '-A'); git('commit', '-m', 'change'); return git('rev-parse', 'HEAD').trim(); };
const metadata = () => (cg as any).queries;
const symbols = (name: string) => cg.searchNodes(name).map(r => r.node.name);
const clean = () => expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] });
beforeEach(async () => {
root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-git-currency-'));
git('init'); git('config', 'user.email', 'test@example.invalid'); git('config', 'user.name', 'Test');
fs.writeFileSync(path.join(root, '.gitignore'), '.codegraph/\n');
write('source.ts', 'original'); commit();
cg = CodeGraph.initSync(root);
expect((await cg.indexAll()).success).toBe(true);
});
afterEach(() => { vi.restoreAllMocks(); cg?.close(); fs.rmSync(root, { recursive: true, force: true }); });
it.each(['index', 'sync', 'scoped'] as const)('sees a restored indexed dirty edit after %s', async mode => {
write('source.ts', 'dirtyVersion');
if (mode === 'index') await cg.indexAll();
else await cg.sync(mode === 'scoped' ? { paths: ['source.ts'] } : {});
expect(symbols('dirtyVersion')).toContain('dirtyVersion');
git('restore', 'source.ts');
expect(git('status', '--porcelain')).toBe('');
// Reopen proves that dirty candidates survive beyond one engine instance.
cg.close(); cg = CodeGraph.openSync(root);
expect(cg.getChangedFiles().modified).toEqual(['source.ts']);
await cg.sync();
expect(symbols('original')).toContain('original');
expect(symbols('dirtyVersion')).not.toContain('dirtyVersion'); clean();
});
it.each(['index', 'sync'] as const)('does not claim a commit made during %s maintenance', async mode => {
const oldHead = git('rev-parse', 'HEAD').trim();
write('source.ts', 'firstEdit');
const db = (cg as any).db;
const maintain = db.runMaintenance.bind(db);
const spy = vi.spyOn(db, 'runMaintenance').mockImplementationOnce(async () => {
write('source.ts', 'lateCommit'); commit();
return maintain();
});
if (mode === 'index') await cg.indexAll(); else await cg.sync();
expect(spy).toHaveBeenCalledOnce();
expect(metadata().getMetadata('indexed_at_commit')).toBe(oldHead);
expect(cg.getChangedFiles().modified).toEqual(['source.ts']);
await cg.sync(); expect(symbols('lateCommit')).toContain('lateCommit'); clean();
});
it('handles non-ASCII and quoted committed paths without git text escaping', async () => {
const names = ['тест.ts', 'space name.ts'];
if (process.platform !== 'win32') names.push('quote"name.ts');
for (const file of names) write(file, 'newSymbol');
commit();
expect(cg.getChangedFiles().added.sort()).toEqual(names.sort());
await cg.sync(); clean();
fs.renameSync(path.join(root, 'тест.ts'), path.join(root, 'renamed.ts')); commit();
expect(cg.getChangedFiles()).toEqual({ added: ['renamed.ts'], modified: [], removed: ['тест.ts'] });
});
it('falls back when git diff fails instead of claiming a clean index', () => {
write('new.ts', 'newSymbol'); commit();
const real = cp.execFileSync;
let injected = 0;
vi.spyOn(cp, 'execFileSync').mockImplementation(((file: string, args: string[], options: any) => {
if (file === 'git' && args[0] === 'diff') { injected++; throw new Error('Injected git diff timeout'); }
return real(file, args, options);
}) as typeof cp.execFileSync);
expect(cg.getChangedFiles().added).toEqual(['new.ts']);
expect(injected).toBeGreaterThan(0);
});
it('does not advance the commit stamp after a scoped sync', async () => {
const oldHead = metadata().getMetadata('indexed_at_commit');
write('one.ts', 'one'); write('two.ts', 'two'); commit();
await cg.sync({ paths: ['one.ts'] });
expect(metadata().getMetadata('indexed_at_commit')).toBe(oldHead);
expect(cg.getChangedFiles().added).toEqual(['two.ts']);
await cg.sync(); clean();
});
it('retains a restored dirty path when a later scoped sync touches another file', async () => {
write('source.ts', 'dirtyVersion'); await cg.sync();
git('restore', 'source.ts'); write('other.ts', 'other');
await cg.sync({ paths: ['other.ts'] });
expect(cg.getChangedFiles()).toEqual({ added: [], modified: ['source.ts'], removed: [] });
await cg.sync(); expect(symbols('original')).toContain('original'); clean();
});
it('does not call a recreated committed deletion removed when the current bytes match the DB', async () => {
fs.unlinkSync(path.join(root, 'source.ts')); commit();
write('source.ts', 'original');
clean();
write('source.ts', 'replacement');
expect(cg.getChangedFiles()).toEqual({ added: [], modified: ['source.ts'], removed: [] });
await cg.sync(); expect(symbols('replacement')).toContain('replacement'); clean();
});
it('retains deleted untracked files as candidates after they were indexed', async () => {
write('untracked.ts', 'temporary'); await cg.sync();
fs.unlinkSync(path.join(root, 'untracked.ts'));
expect(cg.getChangedFiles().removed).toEqual(['untracked.ts']);
await cg.sync(); expect(symbols('temporary')).not.toContain('temporary'); clean();
});
it('never advances freshness after a failed full index', async () => {
write('new.ts', 'newSymbol'); commit();
const controller = new AbortController(); controller.abort();
expect((await cg.indexAll({ signal: controller.signal })).success).toBe(false);
expect(cg.getChangedFiles().added).toEqual(['new.ts']);
});
it('keeps a committed path pending when sync cannot read it', async () => {
write('new.ts', 'newSymbol'); commit();
// Sync reads a source file through a bounded reader that opens a
// descriptor (#1910), so the failure is injected at openSync as well as
// readFileSync: it must reach whichever one the read goes through.
const realRead = fs.readFileSync;
const realOpen = fs.openSync;
let injected = 0;
const failNewTs = (real: (...args: any[]) => unknown) => (file: any, ...args: any[]) => {
if (String(file) === path.join(root, 'new.ts')) { injected++; throw new Error('Injected transient read error'); }
return real(file, ...args);
};
vi.spyOn(fs, 'readFileSync').mockImplementation(failNewTs(realRead as any) as typeof fs.readFileSync);
vi.spyOn(fs, 'openSync').mockImplementation(failNewTs(realOpen as any) as typeof fs.openSync);
await cg.sync();
expect(injected).toBeGreaterThan(0);
expect(symbols('newSymbol')).not.toContain('newSymbol');
vi.restoreAllMocks();
expect(cg.getChangedFiles().added).toEqual(['new.ts']);
await cg.sync(); clean();
});
});