Files
codegraph/__tests__
Colby MchenryandClaude Opus 5.5 89cd08fab8 fix(explore): a long query, like a pasted log, answers in seconds on a large index (#2804)
The prompt hook hands codegraph_explore the whole prompt, so a pasted
log, document or source file arrives as a query of hundreds of words.
findRelevantContext searched every word several times, and the reads
behind those searches scanned the index each time:

- findNodesByNameSubstring, four times per symbol word: `name LIKE
  '%word%'` with a leading wildcard reads every node of those kinds,
  about 80 ms each on vscode (562,636 nodes);
- the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name
  LIKE '%word%'`), about 120 ms per miss, and most words of a log miss;
- the fuzzy fallback after that re-read `SELECT DISTINCT name` each time
  (about 150 ms), and getAllNodeNames, documented as cached, was not;
- FTS joined nodes on the TEXT id instead of the rowid;
- scorePathRelevance re-split every query word for every scored path,
  and the deprioritize predicate stat'ed codegraph.json per candidate.

On vscode a 1.7k-character log took 80 s in explore, a 10k-character
skill document 258 s, and the hook gave up on both. A per-statement
profile put 65-80% of that in those scans.

The fix:
- A name-substring index answers the substring lookups from memory with
  exactly the rows SQLite returns, in its order: ASCII-only case folding
  and `_` as one character, like LIKE, and ties in length in the order
  the plan reads rows, the kind index's or rowid's. The plan is asked of
  SQLite once per statement shape, since a small index is scanned whole;
  a `%`, a language filter or any other plan goes to SQLite as before.
- A word of letters and digits that no name and no letters-and-digits
  run of a qualified name holds can match nothing, so the LIKE fallback
  returns empty without scanning for it.
- The fuzzy fallback reads the name list once and skips names outside
  its edit-distance length window.
- These reads are memoized per database change stamp, like the dominant
  file (PR #1864): rebuilt after any write from any connection, never
  kept inside a transaction, dropped on rebind.
- FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop
  as before.
- Path relevance prepares each query once and each path is scored once
  per search; the deprioritize patterns are read once per search.
- findRelevantContext searches at most 64 symbol names and 128 search
  terms of a query. Both lists put code-shaped words first, so a long
  paste is searched by its identifiers and its opening words. A
  16-symbol bag (75 terms) stays whole. This is the one intended change
  in output, for long queries only.

Validation (vscode ff0a86dc, median of 3 interleaved, fresh process):
- explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s
  fresh (the dominant-file aggregation is about 4 s of that) and
  0.9-2.1 s warm vs 2.7-3.9 s on main;
- explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to
  9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k
  258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s;
- the real prompt hook on the issue's prompt shapes: short 7.2 to
  7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1
  to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s;
- without the word caps, all 7 long queries gave byte-identical output
  4.3-9.0x faster;
- django 79502d08 and prometheus ed26d727: 11 of 11 queries
  byte-identical, tracebacks included, 1.0-2.9x faster;
- the substring index against the LIKE scan on vscode: 4,830 of 4,830
  lookups identical (805 words, snake_case included);
- explore-long-query.test.ts: 13 tests; 8 fail on main's source.

Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for
guidance; open PR #2260 (danusha2345) overlaps on the name-list memo
and path relevance.

Fixes #2620

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 15:48:53 +00:00
..