Files
codegraph/__tests__/bounded-source.test.ts
Colby Mchenry 04ec5f554d fix(extraction): never read an oversize file; an MPEG-TS video named .ts is not TypeScript (#1910) (#2082)
* fix(extraction): an MPEG-TS video named .ts is not TypeScript (#1910)

`.ts` mapped to TypeScript by extension alone, so MPEG transport-stream
fixtures (`testdata/*.ts`) were parsed by tree-sitter: ~28 s of CPU for a
900 KB clip, minutes for a directory of them, for zero symbols.

Recognise the stream from the head of the file — the 0x47 sync byte at
offsets 0, 188, 376 and 564 plus a NUL byte, which every stream carries in
its first packets and UTF-8 source never does — and drop it at discovery:
not indexed, not parsed, not counted, not tallied as an unsupported
language. The check reads under 1 KB and only for `.ts` files; the batch
reader sniffs the bytes it already read, and the single-file path (sync,
watcher) does the same head read, so a clip handed in by name is skipped
too. The skip happens before parse dispatch, so no kernel mirror is needed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(extraction): a video .ts is never pending on the git path or a scoped sync

Review found two places that decide "is this a source file" without the
MPEG-TS check the scan applies: the git fast path's candidate loop in
getChangedFiles and the scoped-sync path filter (watcher events). An
untracked clip in a git repo was reported as `added`, skipped by sync
without a record, and reported as `added` again on every status; a tracked
.ts that became a clip stayed `modified` with its stale nodes. Both now
treat the clip as non-source: removed when tracked, ignored otherwise.

Regression tests fail without this change and pass with it, on both arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): require a binary head before calling a .ts file video (#1910)

Review on #1915: the sniff took four aligned 0x47 sync bytes plus any NUL
as proof of a transport stream, so TypeScript with a `G` at four 188-byte
strides and one raw NUL in a comment was dropped unindexed. The sniff now
needs the sync byte on sixteen consecutive packets (3 KB of head) and at
least 1/64 of the head to be control bytes. Compressed payload carries
8-20% of them (checked on ffmpeg-made H.264/AAC, MPEG-2 and MP2 streams);
source text carries none. A shorter clip is cheap to parse anyway.

Also drops the benchmark figures from the changelog entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): a file over the size limit is never read (#1910)

The size gate ran after `readFile`, so a committed video or blob fixture was
decoded in full — 3.4 GB of RSS for a 400 MB file, `Invalid string length`
past ~512 MB — only to be stored as skipped. It was then read again by every
pass that scans files by name: the framework detectors' `readFile` and the
resolver's cached reader, three full passes on the reporter's fixtures.

Stat first, everywhere a source file is read by path: the batch reader and
`extractFile` store an oversize file with a size stamp in place of its
content, change detection hashes the same stamp (so a same-size rewrite of a
file nothing is indexed from is not a change, while crossing the limit in
either direction is), and both by-name readers return null above the limit.
The limit moves to `src/file-limits.ts` so the three sites share one number.

strace on a sparse 400 MB `.ts`: 153 603 reads before, 0 after; indexing it
next to a real file costs no measurable RSS. Stacked on #1914, which shares
the batch-reader hunk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(extraction): one-purpose size-stamp reader, comments in operation order

`readSourceOrStamp` took a `stats` argument no caller passed and returned
stats no caller read; it now returns the text or stamp change detection
hashes. The batch reader's two comments are back in the order its code runs:
the size gate, then the MPEG-TS head check. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): bound the read itself, not only the stat before it (#1910)

A file can grow between the stat that gates the size limit and the read
that follows (a log, a download, a build output being written). Every
source read now goes through readBoundedSource/readBoundedSourceSync: the
open descriptor is re-checked, and the read stops one byte past the limit,
so an oversize file is still stamped rather than decoded. The extraction
batch reader, single-file extraction, framework detectors and the resolver's
file cache all use it. Ported from the maintainer's hardening of this fix
in #1919.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): bound the parse-retry reads too (#1910)

The retry passes for files whose worker crashed or timed out re-read the
source with an unbounded fsp.readFile. The file can have grown since the
first, bounded read; an oversize file is now skipped by the retry instead
of being decoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(extraction): check an oversize file against its size stamp everywhere (#1910)

Review on #1915:

- The viewer (readFileShape, hasDriftedOnDisk, the source endpoint) and
  MCP's drift gate hashed a file's real bytes, but the index stores the
  size stamp for a file over the limit, so every unchanged 1-8 MB file read
  as "changed on disk". oversizeStamp moves to file-limits.ts with an
  indexedHashInput helper, and all four checks hash what the index stored.
  An oversize file is now compared without being read.
- git-index-currency's "keeps a committed path pending when sync cannot
  read it" injected its failure only into readFileSync, which the bounded
  reader no longer calls, so it failed. The failure now reaches openSync
  too.
- The changelog entry drops its benchmark figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): check an oversize answer file against its size stamp (#1910)

#2043's answer-freshness check hashes each contributing file whole and
compares it with the stored contentHash. For a file over the index's size
limit the stored hash is the size stamp, so an unchanged oversize file
would read as stale. Compare it by its stamp, without reading it, like
the viewer and the MCP drift gate do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(extraction): judge a .ts clip from the bytes it is read for, not at discovery (#1910)

Discovery sniffed the head of every `.ts` file on every full scan — an
open + 3 KB read + close per file, before anything asked for its content.
On a 10k-file TypeScript tree that took a warm-cache scan from ~105 ms to
~243 ms; on a slow disk or a network share it is a random read per file
per scan, the access pattern #1231 and the SMB reports (#1014, #448) are
about.

The bytes are now judged where they are read anyway, so the check costs no
I/O and runs only for a file that is new or changed: the batch reader (as
before), `indexFile`, the sync reconcile (an untracked clip is skipped, a
tracked file that became one is removed through the same removal path as
a deletion), and both `getChangedFiles` paths. The scan lists `.ts` files
by name again; a test pins that discovery opens none of them on the walk
or the git path. Scan time is back to main's (~98 ms), and a full index is
unchanged within noise.

Also collapses the four #1910 changelog bullets (#1914 and #1915 each added
both) to two, without the benchmark figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: danusha2345 <ewidusoc498@gmail.com>
2026-09-29 03:01:46 +00:00

97 lines
3.8 KiB
TypeScript

/**
* Source reads are bounded by the read itself, not only by a stat taken
* before it (#1910). A file can grow between the stat and the read; the
* reader must still never hold more than the limit plus one byte.
*/
import { describe, it, expect, afterEach, vi } from 'vitest';
import * as fs from 'fs';
import * as fsp from 'fs/promises';
import * as os from 'os';
import * as path from 'path';
import { readBoundedSource, readBoundedSourceSync, MAX_SOURCE_FILE_SIZE_BYTES as LIMIT } from '../src/file-limits';
// Pass-through wrappers, so a test can inject a file growing mid-read into the
// exact calls file-limits.ts makes.
vi.mock('fs', async (importOriginal) => {
const actual = await importOriginal<typeof import('fs')>();
return { ...actual, readSync: vi.fn(actual.readSync) };
});
vi.mock('fs/promises', async (importOriginal) => {
const actual = await importOriginal<typeof import('fs/promises')>();
return { ...actual, open: vi.fn(actual.open) };
});
const dirs: string[] = [];
function sourceFile(bytes: Buffer): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-bounded-'));
dirs.push(dir);
const file = path.join(dir, 'source.ts');
fs.writeFileSync(file, bytes);
return file;
}
afterEach(() => {
for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true });
});
describe('bounded source reads (#1910)', () => {
for (const size of [0, 100, LIMIT, LIMIT + 1]) {
it(`returns the bytes up to the limit and null past it (${size} bytes)`, async () => {
const file = sourceFile(Buffer.alloc(size, 0x61));
for (const result of [readBoundedSourceSync(file), await readBoundedSource(file)]) {
expect(result.stats.size).toBe(size);
if (size > LIMIT) expect(result.bytes).toBeNull();
else expect(result.bytes?.length).toBe(size);
}
});
}
it('stops at the limit when the file grows during a synchronous read', () => {
const file = sourceFile(Buffer.from('hello'));
const readSync = vi.mocked(fs.readSync);
const original = readSync.getMockImplementation()!;
let requested = 0;
let grown = false;
readSync.mockImplementation(((fd: number, buf: Buffer, off: number, len: number, pos: number) => {
requested += len;
if (!grown) {
grown = true;
fs.writeFileSync(file, Buffer.alloc(LIMIT + 10, 0x61));
}
return original(fd, buf, off, len, pos);
}) as typeof fs.readSync);
let result;
try { result = readBoundedSourceSync(file); } finally { readSync.mockImplementation(original); }
expect(grown).toBe(true);
expect(requested).toBeLessThanOrEqual(LIMIT + 1);
expect(result.bytes).toBeNull();
});
it('rechecks the open descriptor when the file grows between stat and open', async () => {
const file = sourceFile(Buffer.from('hello'));
const open = vi.mocked(fsp.open);
const original = open.getMockImplementation()!;
let grown = false;
open.mockImplementationOnce((async (...args: Parameters<typeof fsp.open>) => {
grown = true;
fs.writeFileSync(file, Buffer.alloc(LIMIT + 1));
return original(...args);
}) as typeof fsp.open);
expect((await readBoundedSource(file)).bytes).toBeNull();
expect(grown).toBe(true);
});
it('returns multibyte source byte-for-byte', async () => {
const text = 'export const greeting = "こんにちは 🌿";';
const file = sourceFile(Buffer.from(text));
expect((await readBoundedSource(file)).bytes?.toString('utf8')).toBe(text);
expect(readBoundedSourceSync(file).bytes?.toString('utf8')).toBe(text);
});
it('refuses a path that is not a regular file', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-bounded-'));
dirs.push(dir);
expect(() => readBoundedSourceSync(dir)).toThrow(/not a regular file/);
});
});