fix(explore): a long query, like a pasted log, answers in seconds on a large index (#2804)

The prompt hook hands codegraph_explore the whole prompt, so a pasted
log, document or source file arrives as a query of hundreds of words.
findRelevantContext searched every word several times, and the reads
behind those searches scanned the index each time:

- findNodesByNameSubstring, four times per symbol word: `name LIKE
  '%word%'` with a leading wildcard reads every node of those kinds,
  about 80 ms each on vscode (562,636 nodes);
- the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name
  LIKE '%word%'`), about 120 ms per miss, and most words of a log miss;
- the fuzzy fallback after that re-read `SELECT DISTINCT name` each time
  (about 150 ms), and getAllNodeNames, documented as cached, was not;
- FTS joined nodes on the TEXT id instead of the rowid;
- scorePathRelevance re-split every query word for every scored path,
  and the deprioritize predicate stat'ed codegraph.json per candidate.

On vscode a 1.7k-character log took 80 s in explore, a 10k-character
skill document 258 s, and the hook gave up on both. A per-statement
profile put 65-80% of that in those scans.

The fix:
- A name-substring index answers the substring lookups from memory with
  exactly the rows SQLite returns, in its order: ASCII-only case folding
  and `_` as one character, like LIKE, and ties in length in the order
  the plan reads rows, the kind index's or rowid's. The plan is asked of
  SQLite once per statement shape, since a small index is scanned whole;
  a `%`, a language filter or any other plan goes to SQLite as before.
- A word of letters and digits that no name and no letters-and-digits
  run of a qualified name holds can match nothing, so the LIKE fallback
  returns empty without scanning for it.
- The fuzzy fallback reads the name list once and skips names outside
  its edit-distance length window.
- These reads are memoized per database change stamp, like the dominant
  file (PR #1864): rebuilt after any write from any connection, never
  kept inside a transaction, dropped on rebind.
- FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop
  as before.
- Path relevance prepares each query once and each path is scored once
  per search; the deprioritize patterns are read once per search.
- findRelevantContext searches at most 64 symbol names and 128 search
  terms of a query. Both lists put code-shaped words first, so a long
  paste is searched by its identifiers and its opening words. A
  16-symbol bag (75 terms) stays whole. This is the one intended change
  in output, for long queries only.

Validation (vscode ff0a86dc, median of 3 interleaved, fresh process):
- explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s
  fresh (the dominant-file aggregation is about 4 s of that) and
  0.9-2.1 s warm vs 2.7-3.9 s on main;
- explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to
  9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k
  258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s;
- the real prompt hook on the issue's prompt shapes: short 7.2 to
  7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1
  to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s;
- without the word caps, all 7 long queries gave byte-identical output
  4.3-9.0x faster;
- django 79502d08 and prometheus ed26d727: 11 of 11 queries
  byte-identical, tracebacks included, 1.0-2.9x faster;
- the substring index against the LIKE scan on vscode: 4,830 of 4,830
  lookups identical (805 words, snake_case included);
- explore-long-query.test.ts: 13 tests; 8 fail on main's source.

Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for
guidance; open PR #2260 (danusha2345) overlaps on the name-list memo
and path relevance.

Fixes #2620

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/context/index.ts

@@ -37,6 +37,21 @@ import { describeSynthesizedHop } from '../graph/synthesized-hop';
  */
 const MAX_COPYBOOK_DECLARATION_ENTRIES = 3;
 
+/**
+ * The most symbol names, and search terms, of one query that
+ * `findRelevantContext` searches the index for. A question or a bag of symbol
+ * names stays well below both: a name like `RefreshTokenService` is one
+ * symbol but several terms (`refreshtokenservice`, `refresh`, `token`,
+ * `service`, stems). A pasted log, document or source file runs to hundreds
+ * of words, and every word costs several searches of the index, so such a
+ * query took minutes on a large project and the prompt hook gave up on it
+ * (#2620). Both lists put code-shaped words (camelCase, snake_case, dotted
+ * names) first and keep the order they appear in, so a long query is searched
+ * by its identifiers and its opening words.
+ */
+export const MAX_QUERY_SYMBOLS = 64;
+export const MAX_QUERY_TERMS = 128;
+
 /**
  * Extract likely symbol names from a natural language query
  *
@@ -220,9 +235,14 @@ export class ContextBuilder {
    * and its non-production budget cap are deliberately NOT joined: those REMOVE
    * content, and `deprioritize` is a ranking lever by definition — `exclude` is
    * the lever for taking things out of reach.
+   *
+   * Returned as a predicate with the patterns read once, for the candidate
+   * loops of one search: asking the live matcher per candidate re-checked
+   * `codegraph.json` hundreds of thousands of times on a long query (#2620).
    */
-  private isDeprioritized(filePath: string): boolean {
-    return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false;
+  private deprioritizedPaths(): (filePath: string) => boolean {
+    const matcher = this.queries.getDeprioritizedPathSnapshot?.() ?? this.queries.getDeprioritizedPathMatcher?.();
+    return (filePath) => matcher?.(filePath) ?? false;
   }
 
   /**
@@ -473,7 +493,7 @@ export class ContextBuilder {
     // === HYBRID SEARCH ===
 
     // Step 1: Extract potential symbol names from query
-    const symbolsFromQuery = extractSymbolsFromQuery(query);
+    const symbolsFromQuery = extractSymbolsFromQuery(query).slice(0, MAX_QUERY_SYMBOLS);
     logDebug('Extracted symbols from query', { query, symbols: symbolsFromQuery });
 
     // Step 2: Look up exact matches for extracted symbols
@@ -639,7 +659,8 @@ export class ContextBuilder {
       // fell through to the fuzzy fallback and returned whatever program name
       // sat within edit distance (`CVACT01Y` → `CBACT01C`).
       const searchTerms = extractSearchTerms(query)
-        .filter((term) => !copybooks.members.has(term.toUpperCase()));
+        .filter((term) => !copybooks.members.has(term.toUpperCase()))
+        .slice(0, MAX_QUERY_TERMS);
       if (searchTerms.length > 0) {
         // Search each term individually to get broader coverage,
         // then boost results that match multiple terms
@@ -759,7 +780,7 @@ export class ContextBuilder {
     // than nodes matching just one generic term. Without this, "ExecutionUtils"
     // (matches only "execution") fills budget slots meant for "ShardSearchRequest"
     // (matches "shard" + "search" + "request").
-    const queryTermsForBoost = extractSearchTerms(query);
+    const queryTermsForBoost = extractSearchTerms(query).slice(0, MAX_QUERY_TERMS);
     if (queryTermsForBoost.length >= 2) {
       // Group terms that are substrings of each other (stem variants of the same
       // root word). "indexed", "indexe", "index" should count as ONE concept match,
@@ -853,6 +874,18 @@ export class ContextBuilder {
       // SEPARATE LIKE batch so one hot single-word term can't crowd classes
       // out of the length-ordered 200-row batch.
       const camelCallableKinds: NodeKind[] = ['function', 'method', 'component'];
+      // A path's relevance to the query depends on nothing else, and the
+      // candidates of every term share their files: score each path once.
+      const isDeprioritized = this.deprioritizedPaths();
+      const pathScores = new Map<string, number>();
+      const pathScoreOf = (filePath: string): number => {
+        let score = pathScores.get(filePath);
+        if (score === undefined) {
+          score = scorePathRelevance(filePath, query, undefined, isDeprioritized(filePath));
+          pathScores.set(filePath, score);
+        }
+        return score;
+      };
       const camelSearchedTerms = new Set<string>();
       const searchIdSet = new Set(searchResults.map(r => r.node.id));
       // Track per-node term hits for multi-term boosting
@@ -902,12 +935,7 @@ export class ContextBuilder {
           if (searchIdSet.has(r.node.id)) continue;
           if (isTestFile(r.node.filePath) && !isTestQuery) continue;
 
-          const pathScore = scorePathRelevance(
-            r.node.filePath,
-            query,
-            undefined,
-            this.isDeprioritized(r.node.filePath),
-          );
+          const pathScore = pathScoreOf(r.node.filePath);
           const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4);
           termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore });
         }
@@ -994,12 +1022,7 @@ export class ContextBuilder {
         const compoundResults: SearchResult[] = [];
         for (const [, entry] of compoundTermMap) {
           if (entry.terms.size >= 2) {
-            const pathScore = scorePathRelevance(
-              entry.node.filePath,
-              query,
-              undefined,
-              this.isDeprioritized(entry.node.filePath),
-            );
+            const pathScore = pathScoreOf(entry.node.filePath);
             const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8);
             compoundResults.push({
               node: entry.node,

src/db/name-substring-index.ts

@@ -0,0 +1,222 @@
+/**
+ * An in-memory answer to the substring lookup `codegraph_explore` runs for
+ * every word of its query (#2620):
+ *
+ *   SELECT … FROM nodes WHERE name LIKE '%word%' [AND name NOT LIKE 'word%']
+ *     AND kind IN (…) ORDER BY length(name) LIMIT n
+ *
+ * A leading `%` can't use an index, so SQLite reads every node of those kinds
+ * for each call: about 80 ms on a 560k-node index, four calls a word, and a
+ * pasted log or document has hundreds of words. This index reads the names
+ * once and answers each call with one string search.
+ *
+ * It returns exactly what SQLite would. LIKE folds case for ASCII letters only,
+ * `_` matches any one character, and `length()` counts characters, not UTF-16
+ * units, so all three are done that way here. SQLite leaves rows of equal
+ * length in the order its plan read them: through `idx_nodes_kind` (kind,
+ * file_path, start_line, id) on a large index, by rowid when it scans a small
+ * one. The index keeps both orders, and the caller says which plan SQLite
+ * chose. A pattern with a `%` is left to SQLite.
+ */
+
+/**
+ * The kinds the index holds: the definitions and callables explore's
+ * camel-case and compound steps ask about, and the variables and constants its
+ * named-symbol seeding adds.
+ */
+export const NAME_SUBSTRING_KINDS: readonly string[] = [
+  'class', 'component', 'constant', 'enum', 'function', 'interface', 'method',
+  'protocol', 'struct', 'trait', 'type_alias', 'union', 'variable',
+];
+
+/** Separates the names in {@link NameSubstringIndex.hay}; no name may contain it. */
+const SEPARATOR = '\u0000';
+
+export interface NameSubstringIndex {
+  /** Kind codes: `kinds[entryKind[e]]` is entry `e`'s kind. */
+  readonly kinds: readonly string[];
+  /** Every distinct ASCII-lowercased name, each followed by {@link SEPARATOR}. */
+  readonly hay: string;
+  /** Where each distinct name starts in `hay`, plus one past the end. */
+  readonly starts: Int32Array;
+  /** The entries of distinct name `i` are `nameEntries[nameEntryStart[i] … nameEntryStart[i+1])`, ascending. */
+  readonly nameEntryStart: Int32Array;
+  readonly nameEntries: Int32Array;
+  /** Per entry, in index order. */
+  readonly entryRowid: Float64Array;
+  readonly entryKind: Uint8Array;
+  readonly entryLength: Int32Array;
+}
+
+/** Lowercase ASCII letters only, as SQLite's LIKE compares them. */
+function asciiLower(text: string): string {
+  return /[A-Z]/.test(text) ? text.replace(/[A-Z]+/g, (run) => run.toLowerCase()) : text;
+}
+
+/** `length(text)` as SQLite counts it: characters, so a surrogate pair is one. */
+function sqliteLength(text: string): number {
+  let length = text.length;
+  for (let i = 0; i < text.length; i++) {
+    const code = text.charCodeAt(i);
+    if (code >= 0xd800 && code <= 0xdbff && i + 1 < text.length) {
+      const next = text.charCodeAt(i + 1);
+      if (next >= 0xdc00 && next <= 0xdfff) { length--; i++; }
+    }
+  }
+  return length;
+}
+
+/**
+ * Build the index from rows of {@link NAME_SUBSTRING_KINDS} read in
+ * `idx_nodes_kind` order, which becomes the entries' order. Returns null when
+ * a name holds the separator, so the caller keeps asking SQLite.
+ */
+export function buildNameSubstringIndex(
+  rows: Iterable<{ rowid: number; kind: string; name: string }>,
+): NameSubstringIndex | null {
+  const kindCodes = new Map(NAME_SUBSTRING_KINDS.map((kind, i) => [kind, i]));
+  const nameIds = new Map<string, number>();
+  const names: string[] = [];
+  const rowids: number[] = [];
+  const kinds: number[] = [];
+  const lengths: number[] = [];
+  const nameOf: number[] = [];
+  for (const row of rows) {
+    const code = kindCodes.get(row.kind);
+    if (code === undefined) continue;
+    const lowered = asciiLower(row.name);
+    if (lowered.includes(SEPARATOR)) return null;
+    let id = nameIds.get(lowered);
+    if (id === undefined) {
+      id = names.length;
+      nameIds.set(lowered, id);
+      names.push(lowered);
+    }
+    rowids.push(row.rowid);
+    kinds.push(code);
+    lengths.push(sqliteLength(row.name));
+    nameOf.push(id);
+  }
+
+  const starts = new Int32Array(names.length + 1);
+  let offset = 0;
+  for (let i = 0; i < names.length; i++) {
+    starts[i] = offset;
+    offset += names[i]!.length + SEPARATOR.length;
+  }
+  starts[names.length] = offset;
+
+  // Group entries by name, keeping index order inside each group.
+  const nameEntryStart = new Int32Array(names.length + 1);
+  for (const id of nameOf) nameEntryStart[id + 1]!++;
+  for (let i = 0; i < names.length; i++) nameEntryStart[i + 1]! += nameEntryStart[i]!;
+  const fill = nameEntryStart.slice(0, names.length);
+  const nameEntries = new Int32Array(nameOf.length);
+  for (let entry = 0; entry < nameOf.length; entry++) nameEntries[fill[nameOf[entry]!]!++] = entry;
+
+  return {
+    kinds: NAME_SUBSTRING_KINDS,
+    hay: names.join(SEPARATOR) + SEPARATOR,
+    starts,
+    nameEntryStart,
+    nameEntries,
+    entryRowid: Float64Array.from(rowids),
+    entryKind: Uint8Array.from(kinds),
+    entryLength: Int32Array.from(lengths),
+  };
+}
+
+/** The distinct name whose span in `hay` holds `offset`. */
+function nameAt(starts: Int32Array, offset: number): number {
+  let lo = 0;
+  let hi = starts.length - 2;
+  while (lo < hi) {
+    const mid = (lo + hi + 1) >> 1;
+    if (starts[mid]! <= offset) lo = mid; else hi = mid - 1;
+  }
+  return lo;
+}
+
+/**
+ * A matcher for a LIKE pattern without `%`: where the next match at or after
+ * `from` starts, and whether one starts exactly at `at`. Without `_` this is a
+ * plain string search; each `_` matches any one character but the separator.
+ */
+function patternMatcher(needle: string): { next(hay: string, from: number): number; startsAt(hay: string, at: number): boolean } {
+  if (!needle.includes('_')) {
+    return {
+      next: (hay, from) => hay.indexOf(needle, from),
+      startsAt: (hay, at) => hay.startsWith(needle, at),
+    };
+  }
+  // `u`, so a wildcard takes a whole character, as SQLite's `_` does.
+  const source = needle
+    .split('_')
+    .map((part) => part.replace(/[\^$\\.*+?()[\]{}|/]/g, '\\$&'))
+    .join('[^\\u0000]');
+  const anywhere = new RegExp(source, 'gu');
+  const anchored = new RegExp(source, 'uy');
+  return {
+    next(hay, from) {
+      anywhere.lastIndex = from;
+      const match = anywhere.exec(hay);
+      return match ? match.index : -1;
+    },
+    startsAt(hay, at) {
+      anchored.lastIndex = at;
+      return anchored.test(hay);
+    },
+  };
+}
+
+/**
+ * The order SQLite's plan reads candidate rows in, which its ORDER BY keeps
+ * for rows of equal length: the kind index's, or the table's by rowid.
+ */
+export type ScanOrder = 'index' | 'rowid';
+
+/**
+ * The rowids SQLite's substring lookup returns, in its order, or null when the
+ * index can't answer exactly (a `%` in the pattern, an empty pattern, or a
+ * kind it doesn't hold). `order` is the order the plan reads rows in.
+ */
+export function searchNameSubstring(
+  index: NameSubstringIndex,
+  substring: string,
+  kinds: readonly string[],
+  excludePrefix: boolean,
+  limit: number,
+  order: ScanOrder,
+): number[] | null {
+  const needle = asciiLower(substring);
+  if (needle.length === 0 || needle.includes('%') || needle.includes(SEPARATOR)) return null;
+  const wanted = new Uint8Array(index.kinds.length);
+  for (const kind of kinds) {
+    const code = index.kinds.indexOf(kind);
+    if (code < 0) return null;
+    wanted[code] = 1;
+  }
+
+  const { hay, starts, nameEntryStart, nameEntries, entryKind, entryLength } = index;
+  const matcher = patternMatcher(needle);
+  const hits: number[] = [];
+  for (let at = matcher.next(hay, 0); at >= 0; ) {
+    const name = nameAt(starts, at);
+    if (!(excludePrefix && matcher.startsAt(hay, starts[name]!))) {
+      for (let k = nameEntryStart[name]!; k < nameEntryStart[name + 1]!; k++) {
+        const entry = nameEntries[k]!;
+        if (wanted[entryKind[entry]!]) hits.push(entry);
+      }
+    }
+    at = matcher.next(hay, starts[name + 1]!);
+  }
+  // Entry numbers follow the kind index, rowids the table: either breaks
+  // length ties the way SQLite's plan does.
+  const rowid = index.entryRowid;
+  hits.sort(order === 'index'
+    ? (a, b) => (entryLength[a]! - entryLength[b]!) || (a - b)
+    : (a, b) => (entryLength[a]! - entryLength[b]!) || (rowid[a]! - rowid[b]!));
+  // A negative LIMIT is no limit in SQLite.
+  if (limit >= 0 && hits.length > limit) hits.length = limit;
+  return hits.map((entry) => index.entryRowid[entry]!);
+}

src/db/queries.ts

@@ -23,6 +23,7 @@ import { parseQuery, boundedEditDistance } from '../search/query-parser';
 import { isGeneratedFile } from '../extraction/generated-detection';
 import { splitIdentifierSegments } from '../search/identifier-segments';
 import { isModuleKey, referenceNameTail, MEMBER_TAIL_SQL, MODULE_PARKED_SQL } from './reference-tail';
+import { NAME_SUBSTRING_KINDS, buildNameSubstringIndex, searchNameSubstring, type NameSubstringIndex, type ScanOrder } from './name-substring-index';
 
 /**
  * Files that should not be candidates for "dominant file" detection: tests,
@@ -107,6 +108,15 @@ const THROUGH_NAMES = (column: string): string => `(
  */
 export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75;
 
+/**
+ * A `deprioritize` path predicate (#982) that can also hand out a copy with
+ * the project's patterns read once, for loops over many paths (#2620).
+ */
+export interface DeprioritizedPathMatcher {
+  (filePath: string): boolean;
+  snapshot?: () => (filePath: string) => boolean;
+}
+
 /**
  * The fields that tell one node of a file from another across a re-index of
  * that file, whose node ids change with every line shift (#2276).
@@ -308,6 +318,12 @@ export class QueryBuilder {
     | { stamp: string; value: { filePath: string; edgeCount: number; nextEdgeCount: number } | null }
     | undefined;
 
+  // Query-independent reads that a search repeats for every word of its query
+  // — the name list the fuzzy fallback scans, the name-substring index — each
+  // tagged with the change stamp it was read under, like dominantFileMemo
+  // (#2620). See memoizedRead().
+  private readMemos = new Map<string, { stamp: string; value: unknown }>();
+
   // Prepared statements (lazily initialized)
   private stmts: {
     insertNode?: SqliteStatement;
@@ -448,6 +464,26 @@ export class QueryBuilder {
     // different databases report the same one — the memo goes with the old
     // connection, or a worker following a rebuilt index keeps its answer (#1864).
     this.dominantFileMemo = undefined;
+    this.readMemos.clear();
+  }
+
+  /**
+   * `compute()`'s answer, reused until the database changes (see
+   * getChangeStamp), or null inside a transaction, where a rollback could
+   * leave a stale answer under an unchanged stamp: the caller then reads the
+   * database directly, as it did before the memo.
+   */
+  private memoizedRead<T>(key: string, compute: () => T): T | null {
+    if (this.db.inTransaction !== false) {
+      this.readMemos.delete(key);
+      return null;
+    }
+    const stamp = this.getChangeStamp();
+    const memo = this.readMemos.get(key);
+    if (memo?.stamp === stamp) return memo.value as T;
+    const value = compute();
+    this.readMemos.set(key, { stamp, value });
+    return value;
   }
 
   private edgeKindStmt(sql: string): SqliteStatement {
@@ -486,6 +522,17 @@ export class QueryBuilder {
     return this.isDeprioritizedPath;
   }
 
+  /**
+   * The `deprioritize` predicate with the project's patterns read once, for a
+   * loop that asks about many paths. The plain predicate re-checks
+   * `codegraph.json` on every call, which a long query's candidate loops made
+   * hundreds of thousands of times (#2620).
+   */
+  getDeprioritizedPathSnapshot(): ((filePath: string) => boolean) | undefined {
+    const matcher = this.isDeprioritizedPath as DeprioritizedPathMatcher | undefined;
+    return matcher?.snapshot ? matcher.snapshot() : matcher;
+  }
+
   // ===========================================================================
   // Node Operations
   // ===========================================================================
@@ -1698,6 +1745,7 @@ export class QueryBuilder {
     // Apply multi-signal scoring
     if (results.length > 0 && (text || query)) {
       const scoringQuery = text || query;
+      const isDeprioritizedPath = this.getDeprioritizedPathSnapshot();
       results = results.map(r => {
         // A path the project de-prioritized is saying its symbol NAMES are not
         // the answer, so the exact-name bonus has to be damped too. The -15 path
@@ -1705,8 +1753,9 @@ export class QueryBuilder {
         // on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top
         // product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed,
         // so the tree stays findable when it genuinely is what you asked for.
-        // Evaluated once and reused: the predicate stats the config file.
-        const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false;
+        // Evaluated once per result and reused, from a snapshot that read the
+        // config once for the whole loop.
+        const deprioritized = isDeprioritizedPath?.(r.node.filePath) ?? false;
         const nameBonus = nameMatchBonus(r.node.name, scoringQuery);
         return {
           ...r,
@@ -1789,15 +1838,20 @@ export class QueryBuilder {
     const lowered = text.toLowerCase();
     const maxDist = lowered.length <= 4 ? 1 : 2;
 
-    // Pull the distinct name list once. The set is cached on QueryBuilder
-    // by getAllNodeNames(); even on a 200k-node project the distinct
-    // name set is typically O(10k) because most names repeat. The
-    // candidate-cap below bounds memory regardless.
-    const allNames = this.getAllNodeNames();
+    // The distinct name list, lowercased, read once until the database
+    // changes: this fallback runs for every query word that FTS and LIKE both
+    // miss, which is most words of a pasted log, and the DISTINCT scan alone
+    // took ~150 ms a call on a 560k-node index (#2620). Inside a transaction
+    // it is read fresh, as before.
+    const { names, lowered: loweredNames, starts } = this.fuzzyNames();
     const candidates: Array<{ name: string; dist: number }> = [];
-    for (const name of allNames) {
-      const dist = boundedEditDistance(name.toLowerCase(), lowered, maxDist);
-      if (dist <= maxDist) candidates.push({ name, dist });
+    for (let i = 0; i < names.length; i++) {
+      // An edit distance is at least the length difference, so a name outside
+      // this window could never pass the bound: skipping it changes nothing.
+      const end = starts[i + 1]! - 1;
+      if (Math.abs(end - starts[i]! - lowered.length) > maxDist) continue;
+      const dist = boundedEditDistance(loweredNames.slice(starts[i]!, end), lowered, maxDist);
+      if (dist <= maxDist) candidates.push({ name: names[i]!, dist });
     }
     candidates.sort((a, b) => a.dist - b.dist);
 
@@ -1952,10 +2006,18 @@ export class QueryBuilder {
     // nameMatchBonus) can promote results that BM25 alone undervalues.
     const ftsLimit = Math.max(limit * 5, 100);
 
+    // Joined on rowid: nodes_fts is external-content over nodes with
+    // content_rowid='rowid', so its `id` column is read back from the very row
+    // the rowid names. The integer key gives the same rows without a lookup
+    // through the TEXT id index for each match, which roughly halves this
+    // query on a large index, where explore runs it once or twice for every
+    // query word (#2620). CROSS JOIN keeps nodes_fts the outer loop, as it was:
+    // on a small index the planner would otherwise scan nodes and probe FTS
+    // once per node.
     let sql = `
       SELECT nodes.*, bm25(nodes_fts, 0, 20, 5, 1, 2) as score
       FROM nodes_fts
-      JOIN nodes ON nodes_fts.id = nodes.id
+      CROSS JOIN nodes ON nodes.rowid = nodes_fts.rowid
       WHERE nodes_fts MATCH ?
     `;
 
@@ -1993,6 +2055,21 @@ export class QueryBuilder {
   private searchNodesLike(query: string, options: SearchOptions): SearchResult[] {
     const { kinds, languages, limit = 100, offset = 0 } = options;
 
+    // This runs when FTS found nothing, and a word FTS can't find is usually
+    // in no name at all: most words of a pasted log or document. Each scan
+    // read every node, about 120 ms on a 560k-node index (#2620). A word of
+    // ASCII letters and digits can only match inside a name, or inside one
+    // run of such characters in a qualified name, and lowercasing keeps those
+    // runs whole, so when neither lowercased list holds it there is nothing
+    // to find. Any other word is scanned as before.
+    if (/^[A-Za-z0-9]+$/.test(query)) {
+      const runs = this.memoizedRead('qualifiedNameRuns', () => this.readQualifiedNameRuns());
+      if (runs !== null) {
+        const word = query.toLowerCase();
+        if (!this.fuzzyNames().lowered.includes(word) && !runs.includes(word)) return [];
+      }
+    }
+
     let sql = `
       SELECT nodes.*,
         CASE
@@ -2183,6 +2260,22 @@ export class QueryBuilder {
     sql += ' ORDER BY length(name) ASC LIMIT ?';
     params.push(limit);
 
+    // Explore asks this four times for every word of its query, and the
+    // leading `%` makes each one a scan of every node of those kinds (#2620).
+    // The in-memory index answers with the same rows in the same order: ties
+    // in length come out in the order SQLite's plan for this very statement
+    // reads rows, the kind index's or the table's. Anything it can't answer
+    // exactly (no kind filter, a language filter, a `%` in the word, another
+    // plan, a transaction) is asked of SQLite.
+    if (kinds && kinds.length > 0 && !(languages && languages.length > 0)) {
+      const index = this.memoizedRead('nameSubstringIndex', () => this.readNameSubstringIndex());
+      const order = index ? this.substringScanOrder(index, sql, params, excludePrefix ? 2 : 1) : null;
+      const rowids = order ? searchNameSubstring(index!.index, substring, kinds, excludePrefix === true, limit, order) : null;
+      if (rowids) {
+        return this.getNodeRowsByRowid(rowids).map((row) => ({ node: rowToNode(row), score: 1.0 }));
+      }
+    }
+
     const rows = this.db.prepare(sql).all(...params) as (NodeRow & { score: number })[];
     return rows.map((row) => ({
       node: rowToNode(row),
@@ -2190,6 +2283,76 @@ export class QueryBuilder {
     }));
   }
 
+  /**
+   * The order SQLite's plan for a substring lookup reads candidate rows in,
+   * which is the order its ORDER BY leaves ties of equal length in: the kind
+   * index's (kind, file_path, start_line, id), or the table's (rowid). Which
+   * one depends on the statistics — a small index is scanned whole — so it is
+   * asked of SQLite once per statement shape. Null for any other plan.
+   */
+  private substringScanOrder(
+    memo: { index: NameSubstringIndex; plans: Map<string, ScanOrder | null> },
+    sql: string,
+    params: readonly (string | number)[],
+    patternCount: number,
+  ): ScanOrder | null {
+    // The patterns (the first `patternCount` parameters) can't change the
+    // plan, since a leading `%` uses no index; the kinds and the limit can.
+    const key = `${sql}\u0000${params.slice(patternCount).join('\u0000')}`;
+    let order = memo.plans.get(key);
+    if (order === undefined) {
+      order = null;
+      try {
+        const details = (this.db.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...params) as Array<{ detail: string }>)
+          .map((row) => row.detail)
+          .filter((detail) => /^(SCAN|SEARCH) (TABLE )?nodes\b/.test(detail));
+        if (details.length === 1) {
+          if (/^SEARCH (TABLE )?nodes USING (COVERING )?INDEX idx_nodes_kind \(/.test(details[0]!)) order = 'index';
+          else if (/^SCAN (TABLE )?nodes$/.test(details[0]!)) order = 'rowid';
+        }
+      } catch {
+        order = null;
+      }
+      memo.plans.set(key, order);
+    }
+    return order;
+  }
+
+  /**
+   * Every node of the kinds the name-substring index holds, read in
+   * idx_nodes_kind order (kind, file_path, start_line, id): the order its
+   * results break length ties by.
+   */
+  private readNameSubstringIndex(): { index: NameSubstringIndex; plans: Map<string, ScanOrder | null> } | null {
+    const rows = this.db
+      .prepare(
+        `SELECT rowid AS rowid, kind, name FROM nodes WHERE kind IN (${NAME_SUBSTRING_KINDS.map(() => '?').join(',')})`
+        + ' ORDER BY kind, file_path, start_line, id'
+      )
+      .all(...NAME_SUBSTRING_KINDS) as Array<{ rowid: number; kind: string; name: string }>;
+    const index = buildNameSubstringIndex(rows);
+    return index ? { index, plans: new Map() } : null;
+  }
+
+  /** Whole node rows for `rowids`, in that order; a rowid no longer present is skipped. */
+  private getNodeRowsByRowid(rowids: readonly number[]): NodeRow[] {
+    if (rowids.length === 0) return [];
+    const byRowid = new Map<number, NodeRow>();
+    for (let i = 0; i < rowids.length; i += SQLITE_PARAM_CHUNK_SIZE) {
+      const chunk = rowids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
+      const rows = this.db
+        .prepare(`SELECT rowid AS rowid_, nodes.* FROM nodes WHERE rowid IN (${chunk.map(() => '?').join(',')})`)
+        .all(...chunk) as Array<NodeRow & { rowid_: number }>;
+      for (const row of rows) byRowid.set(row.rowid_, row);
+    }
+    const out: NodeRow[] = [];
+    for (const rowid of rowids) {
+      const row = byRowid.get(rowid);
+      if (row) out.push(row);
+    }
+    return out;
+  }
+
   // ===========================================================================
   // Edge Operations
   // ===========================================================================
@@ -4054,6 +4217,40 @@ export class QueryBuilder {
     return rows.map((r) => r.name);
   }
 
+  /**
+   * The distinct node names, in {@link getAllNodeNames} order, and the same
+   * names lowercased and newline-joined: name `i` lowercased is
+   * `lowered.slice(starts[i], starts[i + 1] - 1)`. Read once until the
+   * database changes (fresh inside a transaction). The fuzzy fallback compares
+   * against it, and the LIKE fallback asks it whether any name holds a word.
+   */
+  private fuzzyNames(): { names: string[]; lowered: string; starts: Int32Array } {
+    const read = (): { names: string[]; lowered: string; starts: Int32Array } => {
+      const names = this.getAllNodeNames();
+      const lowered = names.map((name) => name.toLowerCase());
+      const starts = new Int32Array(names.length + 1);
+      for (let i = 0; i < lowered.length; i++) starts[i + 1] = starts[i]! + lowered[i]!.length + 1;
+      return { names, lowered: lowered.join('\n') + '\n', starts };
+    };
+    return this.memoizedRead('fuzzyNames', read) ?? read();
+  }
+
+  /**
+   * The runs of ASCII letters and digits in the distinct qualified names,
+   * lowercased and newline-joined: where a word LIKE could find in a
+   * qualified name has to be.
+   */
+  private readQualifiedNameRuns(): string {
+    const runs = new Set<string>();
+    const rows = this.db.prepare('SELECT DISTINCT qualified_name AS name FROM nodes').all() as Array<{ name: string }>;
+    for (const row of rows) {
+      for (const run of row.name.toLowerCase().split(/[^a-z0-9]+/)) {
+        if (run) runs.add(run);
+      }
+    }
+    return [...runs].join('\n');
+  }
+
   /** The names of a language's nodes of the given kinds (repeats included), decoding nothing else. */
   getNodeNamesOfKinds(language: Language, kinds: readonly NodeKind[]): string[] {
     // `+language`: the kind index narrows further than the language one.

src/index.ts

@@ -30,7 +30,7 @@ import {
 } from './types';
 import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
 import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve';
-import { QueryBuilder } from './db/queries';
+import { QueryBuilder, type DeprioritizedPathMatcher } from './db/queries';
 import { importPathKeys, moduleReferenceKeys } from './db/reference-tail';
 import {
   isInitialized,
@@ -218,24 +218,47 @@ export class CodeGraph {
     // on the pattern array's identity, which the cache keeps stable.
     let cachedPatterns: string[] | undefined;
     let cachedMatcher: ReturnType<typeof ignore> | undefined;
-    this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => {
+    // The compiled matcher for the patterns `codegraph.json` holds right now,
+    // or null when there are none.
+    const currentMatcher = (): ReturnType<typeof ignore> | null => {
+      const patterns = loadDeprioritizePatterns(this.projectRoot);
+      if (patterns.length === 0) return null;
+      if (patterns !== cachedPatterns) {
+        cachedPatterns = patterns;
+        cachedMatcher = ignore().add(patterns);
+      }
+      return cachedMatcher!;
+    };
+    const matches = (matcher: ReturnType<typeof ignore> | null, filePath: string): boolean => {
+      if (!matcher) return false;
+      const rel = path.isAbsolute(filePath)
+        ? path.relative(this.projectRoot, filePath)
+        : filePath;
+      if (!rel || rel.startsWith('..')) return false;
+      return matcher.ignores(rel.split(path.sep).join('/'));
+    };
+    const deprioritized: DeprioritizedPathMatcher = (filePath: string): boolean => {
       try {
-        const patterns = loadDeprioritizePatterns(this.projectRoot);
-        if (patterns.length === 0) return false;
-        if (patterns !== cachedPatterns) {
-          cachedPatterns = patterns;
-          cachedMatcher = ignore().add(patterns);
-        }
-        const rel = path.isAbsolute(filePath)
-          ? path.relative(this.projectRoot, filePath)
-          : filePath;
-        if (!rel || rel.startsWith('..')) return false;
-        return cachedMatcher!.ignores(rel.split(path.sep).join('/'));
+        return matches(currentMatcher(), filePath);
       } catch {
         // Ranking must never take the search down with it.
         return false;
       }
-    });
+    };
+    // One `stat` for a whole loop of candidates instead of one per candidate
+    // (#2620): a search reads the config once, then asks the frozen matcher.
+    deprioritized.snapshot = () => {
+      let matcher: ReturnType<typeof ignore> | null = null;
+      try { matcher = currentMatcher(); } catch { matcher = null; }
+      return (filePath: string): boolean => {
+        try {
+          return matches(matcher, filePath);
+        } catch {
+          return false;
+        }
+      };
+    };
+    this.queries.setDeprioritizedPathMatcher(deprioritized);
 
     this.orchestrator = new ExtractionOrchestrator(this.projectRoot, this.queries);
     this.resolver = createResolver(this.projectRoot, this.queries);

src/search/query-utils.ts

@@ -215,19 +215,30 @@ export function extractSearchTerms(query: string, options?: { stems?: boolean })
 }
 
 /**
- * Score path relevance to a query
- * Higher score = more relevant path
+ * The query half of {@link scorePathRelevance}: the sub-tokens of each query
+ * word that is scored, with how many times the word occurs, and whether the
+ * query is about tests.
  */
-export function scorePathRelevance(
-  filePath: string,
-  query: string,
-  projectNameTokens?: Set<string>,
-  isDeprioritized?: boolean,
-): number {
-  const pathLower = filePath.toLowerCase();
-  const fileName = path.basename(filePath).toLowerCase();
-  const dirName = path.dirname(filePath).toLowerCase();
-  let score = 0;
+interface PathRelevanceQuery {
+  projectNameTokens: Set<string> | undefined;
+  /** No words at all: every path scores 0. */
+  empty: boolean;
+  words: Array<{ subtokens: string[]; count: number }>;
+  isTestQuery: boolean;
+}
+
+/**
+ * The few most recent queries' halves. A search scores every candidate path
+ * against one query, and re-splitting every word of it for each path was
+ * most of the scoring time on a long query: a pasted page is hundreds of
+ * words, and explore scores thousands of paths (#2620).
+ */
+const pathRelevanceQueries = new Map<string, PathRelevanceQuery>();
+const PATH_RELEVANCE_QUERY_CACHE = 8;
+
+function pathRelevanceQuery(query: string, projectNameTokens: Set<string> | undefined): PathRelevanceQuery {
+  const cached = pathRelevanceQueries.get(query);
+  if (cached && cached.projectNameTokens === projectNameTokens) return cached;
 
   // Score per original query WORD, not per sub-token. A single PascalCase word
   // splits into many sub-tokens (a project name "SuperBizAgent" →
@@ -239,30 +250,72 @@ export function scorePathRelevance(
   // camelCase/snake split per word (so `getUserName` still matches a
   // `get_user_name` path) — we just attribute each word's matches once.
   const allWords = query.split(/\s+/).filter((w) => w.length > 0);
-  if (allWords.length === 0) return 0;
 
   // A query word that just names the PROJECT (its go.mod / package.json / repo
   // name) carries no discriminative path signal — drop it so the rest of the
   // query decides the ranking, instead of every file under a `<ProjectName>…/`
   // tree winning on the project name alone (#720). Only when OTHER words remain,
   // so a bare project-name query still scores on its path.
-  const words =
+  const kept =
     projectNameTokens && projectNameTokens.size > 0
       ? allWords.filter((w) => !projectNameTokens.has(normalizeNameToken(w)))
       : allWords;
-  const scored = words.length > 0 ? words : allWords;
+  const scored = kept.length > 0 ? kept : allWords;
 
+  // A word that occurs twice adds its score twice; words with the same
+  // sub-tokens score alike, so each such group is matched once and counted.
+  const groups = new Map<string, { subtokens: string[]; count: number }>();
   for (const word of scored) {
     // Use base terms only — stem variants inflate path scores by generating
     // many near-duplicate terms that all match the same path segments.
     const subtokens = extractSearchTerms(word, { stems: false });
     if (subtokens.length === 0) continue;
+    const key = subtokens.join('\u0000');
+    const group = groups.get(key);
+    if (group) group.count++;
+    else groups.set(key, { subtokens, count: 1 });
+  }
+
+  const queryLower = query.toLowerCase();
+  const prepared: PathRelevanceQuery = {
+    projectNameTokens,
+    empty: allWords.length === 0,
+    words: [...groups.values()],
+    isTestQuery: queryLower.includes('test') || queryLower.includes('spec'),
+  };
+  pathRelevanceQueries.delete(query);
+  pathRelevanceQueries.set(query, prepared);
+  if (pathRelevanceQueries.size > PATH_RELEVANCE_QUERY_CACHE) {
+    pathRelevanceQueries.delete(pathRelevanceQueries.keys().next().value!);
+  }
+  return prepared;
+}
+
+/**
+ * Score path relevance to a query
+ * Higher score = more relevant path
+ */
+export function scorePathRelevance(
+  filePath: string,
+  query: string,
+  projectNameTokens?: Set<string>,
+  isDeprioritized?: boolean,
+): number {
+  const prepared = pathRelevanceQuery(query, projectNameTokens);
+  if (prepared.empty) return 0;
+
+  const pathLower = filePath.toLowerCase();
+  const fileName = path.basename(filePath).toLowerCase();
+  const dirName = path.dirname(filePath).toLowerCase();
+  let score = 0;
+
+  for (const { subtokens, count } of prepared.words) {
     // Exact filename match (strongest)
-    if (subtokens.some((t) => fileName.includes(t))) score += 10;
+    if (subtokens.some((t) => fileName.includes(t))) score += 10 * count;
     // Directory match
-    if (subtokens.some((t) => dirName.includes(t))) score += 5;
+    if (subtokens.some((t) => dirName.includes(t))) score += 5 * count;
     // General path match
-    else if (subtokens.some((t) => pathLower.includes(t))) score += 3;
+    else if (subtokens.some((t) => pathLower.includes(t))) score += 3 * count;
   }
 
   // Deprioritize test files unless the query is explicitly about tests, and
@@ -274,9 +327,7 @@ export function scorePathRelevance(
   //    standing statement by the project, so it is NOT waived. The name-bonus
   //    damping at the call site is what keeps such a tree findable.
   //  - a path that is both is docked ONCE, not twice.
-  const queryLower = query.toLowerCase();
-  const isTestQuery = queryLower.includes('test') || queryLower.includes('spec');
-  const offTarget = (!isTestQuery && isTestFile(filePath)) || isDeprioritized === true;
+  const offTarget = (!prepared.isTestQuery && isTestFile(filePath)) || isDeprioritized === true;
   if (offTarget) {
     score -= 15;
   }