mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-28 22:52:00 +08:00
05712869072d23ec71d23e2801bb68c1778e63d8
107
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby MchenryandClaude Opus 5.5
|
1b56cb1992 |
fix(explore): find COBOL copybooks by name, with every include site (#2342) (#2358)
`codegraph explore MYCOPYBOOK`, the `codegraph_explore` MCP tool and `codegraph context` answered "No relevant code found" for a COBOL copybook named in the query, while `query`/`node` found its include at once. Cause: the COBOL extractor records `COPY X` and `EXEC SQL INCLUDE X` as `import` nodes, and findRelevantContext filters every channel by HIGH_VALUE_NODE_KINDS, which leaves imports out on purpose (a JS/TS/Python import statement is noise in explore). With the include out of reach the text search fell through to its fuzzy fallback, so a member with a digit in it (never extracted as a symbol) came back as an unrelated program one edit away: CVACT01Y -> CBACT01C, CVCUS01Y -> CBCUS01C on CardDemo. Making COBOL imports "high value" alone would not have helped either: explore's renderer skips import nodes, and the import-to-definition step drops a COBOL include because its `imports` edge leaves the enclosing scope, not the import node. Change: - src/graph/cobol-copybooks.ts (new, shared derivation): resolves the COBOL words of a query to copybooks - every COPY / EXEC SQL INCLUDE of the member (case-insensitive) and the indexed copybook file (.cpy, or a file an include resolved to; a same-named program is not one). Gated on the project having COBOL files at all (one seek on idx_files_language), so no other language's query does extra work or changes. Plain lowercase words in a multi-word query are English, not member names. - findRelevantContext: a named copybook's top-level declarations, then its include statements, become entry points ahead of the kind-filtered channels, like an exact filename (at most half the slots when the query matched other things, include sites in those files first). They are not walked (BFS from a record layout spent the answer on whichever program MOVEs into it), the member is not re-searched as text (the fuzzy decoy), and an explicit nodeKinds filter without `import` is honored. - codegraph_explore: the copybook file is pinned like a path in the query; an include statement that is an entry point is grouped and rendered (a window around the COPY line) instead of skipped; a new "COBOL copybook" section lists every include site (path:line, COPY vs EXEC SQL INCLUDE, 25 lines then a count) or says the member has no indexed source (DB2 DCLGEN, compiler-supplied such as SQLCA/DFHAID). - QueryBuilder: three indexed lookups; CodeGraph.findNamedCopybooks exposes the derivation so explore and the context builder share one answer. No extraction change; no re-index needed. Verification: __tests__/explore-cobol-copybook.test.ts (8 tests, through ToolHandler codegraph_explore and buildContext): with src/ reverted to origin/main 6 fail (the 2 guard cases pass on both), 8/8 pass with the fix. tsc clean, npm run build OK. Issue repro fixed on the built CLI (explore + context) and through a real `serve --mcp` tools/call. Full suite on a heavily loaded box: 5716 passed, 25 failed in 15 files; 12 of those files pass alone, and the other 3 (function-ref, git-index-currency, worktree-detection) are 5 s / 60 s timeouts that fail on clean main as well in interleaved runs. Validation (both builds on one index per repo; query-side change only): - expressjs/express: 14 explore + 4 context queries byte-identical. - aws-mainframe-modernization-carddemo: 10 non-copybook queries byte-identical; 15 copybook queries all now return the copybook (when indexed) and every include site - before: 9 "No relevant code found", 4 unrelated programs, 2 without the copybook. - cicsdev/cics-genapp: 6 non-copybook queries byte-identical; 6 copybook queries fixed (LGCMAREA's 25 sites incl. multi-line EXEC SQL INCLUDE, lowercase lgpolicy, SSMAP with no indexed source). - In-process explore latency on express unchanged within noise. Thanks @popolusiak for the report. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
dea076fd1e |
fix(mcp,extraction): exit a daemon whose install changed; never store a grammar-load failure (#2335, #2336) (#2346)
* fix(mcp,extraction): a daemon whose install changed exits; grammar-load failures are never stored (#2335, #2336) A daemon that outlived an upgrade loaded grammars lazily from an install that no longer existed, and stored every file its watcher re-indexed as a zero-node row under the file's new content hash: no hash-based sync ever revisited those rows, and `codegraph status` called the index up to date. - The daemon checks its own package.json every 30 s (CODEGRAPH_DAEMON_INSTALL_CHECK_MS, 0 turns it off) and exits once the file is gone or carries another version, so the next session starts a daemon from the current install. - A result whose grammar failed to load (`parser_error`) is never stored: the file keeps its previous data and the next sync or index retries it, and a row an older engine stored that way is re-indexed even though its hash matches. The tag-based CFML path reports a missing grammar with the same code. - A launcher whose own install was deleted no longer crashes on the spawn error of the daemon it tried to start. - `codegraph status` and `status --json` (filesNeedingReindex, filesWithParseErrors) report indexed files whose symbols are missing although their content is current (#2336), and `files --json` carries each file's recorded errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(mcp): a session whose own install changed never claims the writer lock (#2335) A daemon now exits once its install is upgraded or removed, and nobody holds the writer slot for a successor at that point. Its sessions run the same replaced code and serve themselves in-process the moment it goes; one that found the slot free claimed it as its writer even though its engine could not load (a real npm upgrade: "Cannot find module './sqlite-adapter'"), and so kept a daemon from the current install from starting. Such a session now falls back read-only and says to restart it. Also adds the CHANGELOG entries for the install check and grammar-load failures (#2335) and for the status report (#2336). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
e28ac71048 |
fix(sync): same-named symbols keep their own cross-file callers across a re-index (#2276) (#2288)
* fix(sync): same-named symbols keep their own cross-file callers across a re-index (#2276) Re-storing a file re-attaches the edges other files had into it by moving each edge to the re-indexed node that replaces its target. That pairing keyed new nodes by (kind, name) with the last one winning, so in a file defining a name twice — the same method on two classes, a method's overloads — any edit, even a body-only one, moved every incoming caller of those symbols onto one of them. A body-only edit leaves the sync's definition delta empty, so nothing re-resolved them afterwards. The re-store now reads the file's old node identities alongside the edge snapshot and pairs old nodes with new ones by (kind, name), then qualified name, then signature, then position within a same-sized group of otherwise identical nodes. A target no tier can pair one-to-one is resurrected as its original unresolved reference, so resolution rebinds it instead of the store guessing. The common in-memory re-store path also now deletes the old rows, stores the new ones and re-attaches incoming edges in one transaction. Committed separately, a failure between the steps left the file looking new or already current, and other files' edges into it were lost until a full re-index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): same-named symbols keep their callers across a sync (#2276) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
455175da67 |
fix(mcp): a projectPath in a nested repo the ancestor index leaves out isn't answered from it (#2110) (#2287)
* fix(mcp): a projectPath in a nested repo the ancestor index leaves out isn't answered from it (#2110) An explicit projectPath resolves to the nearest .codegraph/ walking up, and that walk crosses git boundaries. Inside an independent repository nested under an indexed parent that excludes it (typically gitignored), every call silently answered from the parent's code. After resolving the index root, ToolHandler now asks whether the path sits in a different git repository strictly below that root (git toplevel + common dir, memoized per path and root) and, if so, whether the index holds any file under it (one primary-key range probe per call). None means the success-shaped "isn't indexed" guidance. Nested repos the parent does index (submodules, gitlinks, embedded clones), ordinary subdirectories, linked worktrees of the same repository (the when git can't run, behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): nested repositories the index leaves out (#2110) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
e12f5bbb92 |
fix(nuxt, astro): a file-routed page is bound to its component (#2267)
A Nuxt pages/x.vue or Astro src/pages/x.astro route linked to nothing, so the viewer's Steps drew a page alone and the routes list never named what serves it. Each page route now references its file's own component (never a same-named one — every index.vue is a component named index), an Astro endpoint its exported verb handlers, and the routing manifest accepts a component as what serves a route, which also lists Vue Router screens. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
0419a51636 |
fix(synthesis): link a Spring or Laravel test's request to the route it reaches (#2100)
Tests reach a controller by URL, never by a call — MockMvc's
`perform(post("/owners/new"))`, Laravel's `$this->postJson('api/me')` — so
every controller those suites exercise read as untested: the viewer's
"No test reaches this" and explore's `tests:` line both walk callers.
petclinic's processCreationForm, hit three times by OwnerControllerTests,
said no test reached it.
A new synthesis pass (tier-synthesizer section 4, registered after the
cross-tier pass, gated on java / kotlin / php) reads test files only:
- Spring: MockMvc builders (with MockMvcRequestBuilders imported),
`request(HttpMethod.X, …)`, the Kotlin `mockMvc.get(…)` DSL,
WebTestClient `.get().uri(…)`, TestRestTemplate, RestAssured.
- Laravel: `$this->get/post/…Json(…)`, `->json('POST', …)`, `->call(…)`
on a chain rooted at `$this` that only calls methods (so
`$this->app->get('config')` is not a request); Pest's helpers where the
file imports Pest\Laravel; and a project's own verb-named helpers
(koel's getAs → jsonAs → `$this->json($method, $uri)`), recognised by
their body, never by name alone.
Paths are read as built — Spring `{var}` templates, PHP / Kotlin
interpolation and concatenation become holes — and matched with the
cross-tier pass's own route matcher (a hole fills a parameter, never a
literal). One `calls` edge per (test, route), `synthesizedBy:
'test-request'`. These edges are excluded from countIncomingEdges, so a
well-tested endpoint never becomes a hub the Steps walk refuses to enter.
Routes reached by a test: petclinic 16/17, realworld 18/19, BookStack
194/340, koel 105/134 (from 0 each). A random sample of edges checked
against the test source was all correct. The pass costs 86 ms on
BookStack and 68 ms on koel.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
09b9646b7e |
perf(context): compute the dominant file once per index state (#1864) (#2086)
* perf(context): compute the dominant file once per index state (#1864) getDominantFile() backs context ranking's core-directory boost. Its answer depends only on the graph, but the whole-graph aggregation behind it (edges joined to both endpoints' nodes, grouped by file) ran on every generic codegraph_explore - seconds per call on a large index. Memoize it in QueryBuilder against a database change stamp: total_changes() (rows this connection wrote) plus PRAGMA data_version (commits by any other connection or process). Both are O(1), so no write path has to remember to invalidate, and a sync made by another process is seen on the next call. Any write forces a recompute; the heuristic's result is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(context): never keep a dominant file read inside a transaction (#1864) total_changes() counts writes that are later rolled back, so a result computed inside a transaction could survive the ROLLBACK under an unchanged stamp and describe edges that no longer exist. getDominantFile now skips the memo while a transaction is open (and on a runtime without isTransaction). Ported from the maintainer's hardening of this fix in #1919. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(db): drop the dominant-file memo when a QueryBuilder is rebound (#1864) The memo's change stamp (total_changes + data_version) is per connection, and fresh read-only connections to two different databases report the same one ("0:2"). A pool worker rebound onto a rebuilt index kept serving the old database's dominant file until something wrote through the new connection. A test rebinds onto a second project and fails without the reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): credit @danusha2345 for #1864 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
fef37765c1 |
perf(resolution): remove the indexing regressions since v1.6.0 (vscode stall, per-ref rescans) — graph byte-identical (#2072)
* perf(resolution): read JS/TS parameter lists without exponential backtracking isLocallyBoundJsName (#1759, |
||
|
Colby MchenryandClaude Opus 5.5
|
c13bd0890f |
fix(wsl): keep WSL off Windows' index on /mnt drives, and explain the I/O error (#995) (#2061)
When Windows-native and WSL CodeGraph both open the same .codegraph/codegraph.db under /mnt/<drive>/, SQLite's locking and -shm shared memory don't hold across the 9p/DrvFs bridge and WSL fails with a bare "disk I/O error". - A SQLite I/O error on the default index of a WSL /mnt/<drive> project is rewritten (in the SQLite adapter, so every open and query is covered) into the CODEGRAPH_DIR=.codegraph-wsl instruction. MCP answers it success-shaped, and a failed default-project open is reported instead of "no project loaded". - On WSL + /mnt/<drive> with CODEGRAPH_DIR unset, the data dir is resolved per root: an existing .codegraph-wsl/ wins, then an existing .codegraph/ index (kept, no silent re-index), else a fresh index goes in .codegraph-wsl/. `codegraph init` says where it went. Every other host is unchanged. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8bde4a1500 |
fix(mcp): stop pending connections and serve read-only fallbacks (#1963) (#2042)
* fix(mcp): close pending sockets and allow read-only fallback (#1963) * fix(mcp): log when a fallback serves reads without auto-sync (#1963) A read-only fallback starts no watcher and holds no writer lock, so a session that quietly stopped syncing looked the same as a healthy one in the logs. Name the holder on stderr when the fallback goes read-only. Suggested by @bompus in #1979. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(mcp): stop pending connections and serve read-only fallbacks (#1963) Pending handshakes could block daemon shutdown or register phantom sessions after disconnect. Track accepted sockets and reject late hello continuations. Use explicit read-only engines and SQLite connections for blocked fallbacks, suppressing synchronization, watching, migrations, and repairs. Preserve writer ownership and the existing project lifecycle, with regression coverage for #1963 and #1356. Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com> Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com> Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
71985f2620 |
fix(db): guard synthesis metadata JSON and keep calls ahead of imports (#2038)
* fix(db): guard synthesis metadata and migrate the expression index Bisect identifies |
||
|
Colby MchenryandClaude Opus 5.5
|
2a2f71ec41 |
fix(sync): refresh synthesized edges after incremental changes (#1988) (#2033)
* fix(sync): refresh synthesized edges after incremental changes (#1988) Scoped sync skipped synthesis and retained obsolete dispatch edges. Refresh all synthesis passes after base resolution when dependencies or source patterns change. Stage and atomically replace owned edges, preserve ordinary edges, and retry interrupted refreshes. Stabilize traversal order and migrate existing indexes for rebuild convergence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: preserve structural Go containment during synthesis refresh --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Opus 5.5
|
9ad6ee9c9a |
fix(explore): explain empty results with bounded lexical diagnostics (#1904) (#2026)
Empty explore results returned no explanation or retry guidance. Check FTS and live name segments with two bounded SQL queries to distinguish matched and unmatched words and suggest indexed names. Preserve success-shaped responses, unresolved-path notes, ranking, and non-empty output; cap added diagnostics below 1.5K characters. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e5fb5e390e |
fix(mcp): deduplicate equivalent project paths (#1057) (#2020)
* fix(mcp): share one connection across two spellings of one root (#1057) A symlinked checkout, or a case-variant of a root on a case-insensitive mount (macOS, NTFS, WSL DrvFs), resolved to two path strings and opened two connections to the same .codegraph/codegraph.db. The connection cache stays keyed by path, so a root recreated at the same path still heals in place (#925). On a miss, an open root that is the same index NOW (both data directories stat to one dev:ino, read as bigints) is reused and the new spelling filed as its alias. Comparing live identities means a deleted root never matches, even if a new directory reuses its inode. Windows has no usable inode, so it compares the case-folded realpath.native instead. statInode moves to directory.ts so the DB layer and the cache share one reading, and closeAll closes an aliased instance once. * fix(mcp): keep root aliases out of the connection cache (#1057) A second spelling of an open root was filed in projectCache under its own key, so one CodeGraph sat under two keys. Anything that drops one key and closes its connection (a bounded cache's eviction, as proposed in #1929) then left the other key holding a closed handle: every query on that spelling, and through the alias scan on the original spelling too, failed with "database is not open" until restart. Aliases now live in their own map (spelling -> projectCache key). projectCache is back to one key per instance, and an alias whose entry is gone is dropped and re-resolved instead of served. * fix(mcp): deduplicate equivalent project paths (#1057) Equivalent project paths could open duplicate cached connections. Compare current filesystem identities using bigint inodes and native Windows paths. Revalidate aliases and pin cached connection owners to symlink targets. Preserve recreation recovery and single-owner cleanup. Co-authored-by: Eric Minish <eric.minish@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Eric Minish <eric.minish@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
52b854d44a |
fix(dead-code): propagate external ancestor uncertainty to descendants (#1973) (#2007)
* fix(dead-code): treat an ancestor outside the index as opaque (#1973) An override is excluded from the dead-code report when its ancestor is in the graph, or is present but declares no members we can read. An ancestor outside the project (React.Component, stream.Transform, a NestJS interface) creates no edge at all, so a lifecycle method the framework calls looked like it overrode nothing and was listed as unreferenced. The resolver does record such an ancestor, as a failed extends / implements row. A container with one, or a project ancestor of one that has it, now counts as having an opaque ancestor, and its members are excluded under "overriding". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dead-code): propagate external ancestor uncertainty to descendants (#1973) External ancestors leave unresolved inheritance references rather than graph edges. Treat those ancestors as opaque and propagate uncertainty through every descendant root. Track visited ancestor/root pairs while preserving cycle protection and the depth bound. Cover direct and three-level inheritance, unused controls, and batched query behavior. Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5001737df3 |
fix(explore): retrieve exact Han filenames (#1372) (#2005)
* fix(explore): find files by exact Chinese filenames Co-Authored-By: GPT-5 <noreply@openai.com> * fix(explore): retrieve exact Han filenames (#1372) ASCII-only query preparation discarded Han filename queries before retrieval. Add exact basename/stem file lookup and prioritize matches before the entry-point cap while respecting node-kind filters. Preserve shared tokenization and retrieval budgets; cover Lua, Svelte, and TypeScript retrieval. Co-authored-by: 小为 <1053122090@qq.com> Co-authored-by: GPT-5 <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: 小为 <1053122090@qq.com> Co-authored-by: GPT-5 <noreply@openai.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6c414cc830 |
fix(ui): link every route to its handler, however many files they span (#1975) (#1993)
buildRoutes looked up node ids only for the first 60 distinct handler files, one getNodesInFile query each, so every later row went out with handlerId null and the viewer called the handler "not in the index". Load the nodes of every handler file in one chunked query instead (getNodesInFiles), which also drops the per-file query the cap existed to bound. Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Christopher BeaulieuandColby Mchenry
|
58c07e8745 |
fix(db): make edge rebinding atomic (#1833) (#1849)
Co-authored-by: Colby Mchenry <me@colbymchenry.com> |
||
|
|
0fd259b554 |
fix(db): graceful FTS5 fallback when Node.js build lacks FTS5 (#1532) (#1810)
* fix(db): graceful FTS5 fallback when Node.js build lacks FTS5 support (#1532)
Official Node.js binaries do not compile FTS5 by default, causing codegraph
init to fail with 'no such module: fts5'. Added runtime FTS5 detection:
- Split schema execution to try FTS5 separately, skip on failure with warning
- Added fts5Available flag to DatabaseConnection and QueryBuilder
- Bulk-load and search paths skip FTS5 operations when unavailable
- Search falls back to LIKE + fuzzy matching when FTS5 is missing
(cherry picked from commit
|
||
|
|
9181dd1ef3 |
fix(extraction): land upstream declaration initializer walks (#1511) (#1802)
Squash danusha2345's PR #1511 at |
||
|
Colby MchenryandColby McHenry
|
72c1ff13cc |
fix(resolution): restore orphan-sweep parity after interruption (#1577) (#1789)
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby MchenryandColby McHenry
|
ee83636acb |
fix(extraction): index TypeScript interface members (#1638) (#1780)
Land upstream #1686 (maxmilian + bompus kernel/CG-28 follow-ups) onto current main. tree-sitter-typescript interface members (method_signature / property_signature) were never listed in the TS extractor, so platform .d.ts APIs had no declaration nodes for call edges. Mirrors on the Rust kernel path; keeps CG-28 damping for pure-interface declaration files; filters damped files from the explore RWR seed set. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby MchenryandColby McHenry
|
9b8bb4aba0 |
fix(db): fail closed when WAL valve cannot checkpoint past caps (#1539) (#1751)
sync() armed the WAL valve but never called backpressure(), so daemon catch-up could grow the WAL without bound while query-pool readers pinned frames. Wire the writer pause into sync store + batched resolution, and abort with WalValveAbortError after parked backfills fail past the documented hard/file caps instead of disabling parking for 60s. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby McHenryandClaude Fable 5
|
b1f40c57dd |
feat(steps): cross-tier channels — a client's fetch onto its own route, queue jobs onto consumers, bus and socket events onto handlers
- resolution/tier-synthesizer.ts: http-client (literal fetch/axios/ky/got/$fetch paths, axios.create baseURL instances, template holes as :params, base-URL holes by a two-segment tail; unique match only), queue-job (BullMQ/Bull add ↔ @Process/@Processor, WorkerHost process, new Worker, queue.process), event-bus (EventEmitter2 emit ↔ @OnEvent with globs; socket emit ↔ @SubscribeMessage / socket.on both ways with tier); channel, tier, callee, registeredAt on every edge; generic transport events never pair; test and generated files never sources; registered before the emitter pass
- steps.ts: crossing() reads tier/channel before languages; an endpoint reached across a tier is a bridge box and a boundary like a screen (through=1 enters it); a channel's call is not also an effect; sites read as written; a Next 'use server' action is a crossing by its directive (when.ts directive); a function-valued constant handler (asyncHandler(...)) is a route root and borrows the file-scope calls and refs within its lines
- express.ts: app.use('/prefix', router) mounts composed onto route names in postExtract (nested, by import or require); chained router.route('/x').get(h).put(h2) extracted, across lines
- frameworks/package-deps.ts: dependencies read from workspace package.json files too (Express, React, Expo Router, NestJS detect)
- routing manifest names constant handlers; e2e/ is a test directory; explore's Flow section labels the new channels
- tests: ui-steps-cross-tier (monorepo fixture: Next client + Express/Nest API), servers test updated for the queue landing
- docs: CHANGELOG, spec §3.13 cross-tier paragraph, CLAUDE.md, callback-edge-synthesis.md, plan P3 built
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
|
||
|
Colby McHenryandClaude Opus 5
|
56dfdb0655 |
feat(ui): dead code and islands — what nothing reaches, and everything that could still reach it (CG-59)
A Dead code screen and a mark on the Map, both drawn from one derivation in src/graph/dead-code.ts so a second surface can never disagree with the first. The SQL half is four lines — no incoming edge but `contains`. It returns ~2 500 candidates on this repository and the shipped list is 20; everything in between is the feature. A candidate is dropped the moment there is any reason to believe something outside the graph reaches it: exported symbols and header declarations, test and generated files, abstract and interface members, anything carrying a `decorates` edge, overrides of an ancestor's member, names the language calls by itself, vendored directories, files nothing in the index reaches (those are islands, and the Map says so instead), names the resolver failed to resolve somewhere, and names shared with a symbol that IS referenced — the mis-resolution that leaves a used method with a self-edge and its twin with nothing. The last rule is the only one that is not a graph query: before a claim is made, the declaring file and every file that reaches it are read and the identifier counted, which is what catches the references the extractor never recorded (`this.handleMessage.bind(this)`, a call inside an object literal, a shorthand property). Every subtraction is counted and printed under the list with the scale it came from, and the caveat line above it never collapses: the claim is "no static reference in the index", not "unused". On the Map a module nothing depends on keeps its stroke and says so in its count line, and tool-generated files and modules recede to ink-4 there, in the map's file list, in search results and on the file screen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
94f4e287e6 |
feat(ui): entry points — routes, executable files and tests as flow starting points (CG-54)
`#/entry` answers "where does anything start" at full length, and turns any row that names a symbol into a flow. Server. `/api/entrypoints` gains `frameworks` (from `getDetectedFrameworks`), a `tests` list, a `routes` limit of its own, and a cache keyed on the index build — nothing here is read from disk, so unlike `/api/source` a cached answer cannot be stale about drift. `routes.items` is now a `WireList` like every other list on the payload. Routes carry where the URL is REGISTERED as well as where it is served: `getRoutingManifest` selects the route node's id, file and line, and `buildRoutes` splits the verb off the name against a fixed list (never "the first word", which would take the head off a file-routed `/blog/[slug]`). All four payroll-go routes register in one router file and three are served from another — group by the handler file and one router becomes two groups plus an orphan. `isTestFile` is split into `isTestPath` (test filename and directory conventions) + the non-production catch-all, byte-identical at every existing call site. The Tests list uses the narrow half: an example, a benchmark or a fixture is off-target for ranking but is not a test, and a heading that says "Tests" must not quietly count them. Tests rank by REACH — distinct other files touched — because Go, Rust and Java put test work inside functions where a module-level-calls ranking sees nothing. Two read-only engine queries make that affordable: `getFileReachCounts` (the mirror of `getFileDependentCounts`, driven from `nodes` by path so the cost follows the files asked about rather than the edge table) and `getFileNodes`. Viewer. `ui/src/lib/entry-model.ts` folds the four lists into file groups — pure, and `panel.rows` stays exactly the sections it draws. `EntryView` + `EntrySection` render them with the caller rail's `.filegroup` / `.row` shapes rather than a second visual language for the same idea. A row that names a callable symbol carries a `Flow ›` chip; the other end is typed or picked with `→ here` on another row. File and test rows carry none: `/api/flow` searches by name, and a file has none the path finder can look up. A project with fewer than three resolvable routes gets no Routes heading at all, not an empty one. Typing into the search box now also returns matching entry points under their own heading below the symbol matches, so a URL comes back with its handler attached; rows already in the results are dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
ecd6e1cd15 |
feat(ui): live refresh and drift banners — the viewer keeps up with the project (CG-53)
`GET /api/events` is a server-sent-event stream the viewer holds open for the life of the page. Two signals, two things the browser could not know: changed source files touched on disk, before any sync — the drift banner index the graph moved, naming what the sync re-indexed — the live refresh The server WATCHES and never syncs: the project tree through the engine's own FileWatcher with a notify-only syncFn, the index through one non-recursive fs.watch on the data directory settled at 400 ms. Both start with the first subscriber and stop with the last, so a viewer nobody has open costs no watch descriptors. Nothing polls, on either side. Drift is now parity with codegraph_node (#1474) rather than an absence. `/api/source?ondrift=current` serves a drifted file's CURRENT bytes flagged `showing: 'current'`, and the three screens that can say so switch off everything anchored to the old line numbering — gutter ports, call-site links, call arcs, the callee rail's anchoring — while keeping the source. The banner is paper-2 with a hairline rule, never amber: amber belongs to the untested badge. Also fixes a stale read this exposed. A long-lived reader holds an LRU of nodes by id that only its own writes invalidate, so `/api/node/<id>` kept answering with a symbol another process's sync had deleted while `/api/search` beside it said it was gone. GraphSession now drops the read caches when the database (or its WAL) has been written, and the Symbol view follows a symbol whose id changed because an edit above it moved its start line, carrying the trail across. Measured on a live viewer: banner 360 ms after a save, toast 440 ms after `codegraph sync` returns, 0 requests in 4 idle seconds, and the client gives up reconnecting after ~90 s with "Not live" rather than hammering a dead port. |
||
|
Colby McHenryandClaude Opus 5
|
bd99c5e99a |
feat(ui): the whole file — full source with gutter ports and intra-file call arcs (CG-52)
The File view gains a Source tab: the file itself, top to bottom, with the Symbol view's line grid, gutter ports and call-site links, a line-anchored callee rail, and — in the left margin — an arc for every call that stays inside the file, drawn from the calling line to the callee's definition line. The arcs are the point. Source order is already a layout, chosen by whoever wrote the file, so a file's internal call structure can be drawn with no algorithm placing anything. Crabviz's idea, in the one place it is legible. Everything is arithmetic, not measurement. The Symbol view queries the laid-out DOM to place a callee row beside its line; a 6 820-line file cannot afford that. Here a line is exactly 20px at `10 + (n - 1) x 20`, so ~90 line elements exist at a time and the arcs, ports, rail rows and connectors are all functions of a line number. `src/mcp/tools.ts` scrolls at a 16.6ms median frame. - `GET /api/filecode/<path>` — outline, one call group per (caller, callee) PAIR with its call-site lines, unresolved references, and the file's length. The source is NOT in it: it pages through `/api/source` 800 lines at a time with a discarded 150-line lead-in, so a page starting inside a block comment does not render prose as code, and so the ports and arcs are complete from the first frame while the text fills in behind them. - `intraFileCalls` is counted over the groups actually returned, so the header and the picture under it cannot disagree once a cap bites. - Above 40 arcs the diagram narrows to the symbol under the pointer (or the one the scroll position is inside) and the header states the total. Accent is for the pointer only, never for the filter. - Sticky outline rail at >= 1400px, following the reader down the file. - `QueryBuilder.getUnresolvedReferencesInFile` — one indexed lookup instead of one per symbol; `buildOutlineEntries` lifted out of `/api/file` so both readings of a file draw the same rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
6d0f60f32c |
feat(ui): the Map — the repository at module granularity, layered from the graph (CG-49)
`GET /api/map` rolls the whole edge table up to module granularity in one `GROUP BY`, and the Map tab draws it: one box per directory, dependencies pointing down, nothing placed by hand. Two decisions carry the screen. The vertical order rests on each link's `declared` weight — the edges resolved through an import, a qualified name, an inheritance clause or a typed receiver — not on its raw count. Bare name matching resolves `run`, `push` and `finish` across unrelated directories, and layering on raw counts put `src/db` directly under `src/bin` on this repository's own index. On declared edges the same data reproduces the pipeline CLAUDE.md describes, with a third of the mutual pairs. When too few links carry a declared edge to describe a project, the layout falls back to raw counts and the side panel says so. And the aggregation is a single scan. Grouping by the symbol names as well as the modules costs nothing extra — the join is what is expensive — so one query yields both the link weights and the tooltip's symbol pairs. Measured against this index inflated to 800k edges: 1.28s for one scan against 1.89s for two, which is the difference between meeting and missing the cold budget on a ten-thousand-file repository. Cached answers come back in ~3ms. Nothing is dropped silently: thin links are hidden until a module they touch is selected and counted in the panel, uncertain references are excluded from every number on screen and the total is printed, and mutual dependencies, module loops and file-level circular imports are listed rather than straightened away. An edge that still points up after layering is drawn dashed on selection instead of being reversed or removed. The layout — cycle-breaking, longest-path layering, barycenter ordering, ports — is a pure function of the payload in `ui/src/lib/map-model.ts`, so the tests toggle and the selection cost no round-trip and the same project always draws the same picture. Svelte Flow supplies pan, zoom and fit; never a layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
87afc50e76 |
feat(ui): the search palette, entry points and a trail that survives the URL (CG-45)
Search: `/` or ⌘K focuses the box; results arrive grouped by kind with their
glyph, signature and file:line, ↑/↓/Enter walk them, Esc dismisses. A group
appears where its best result did, so flattening the groups reproduces the
ranking the keyboard walks — the panel's flat item list IS that concatenation.
A flow question ("how does X reach Y", "X -> Y") is recognised and searches
both endpoints with a note, rather than offering a row that would land on the
phase-2 Flow view.
Entry points answer "where do I start" on the empty screen and in the resting
palette, all derived from the graph: routes, files that run something at module
level (the engine records a top-level statement as an edge out of the file node,
which is what makes src/bin/codegraph.ts the root of the CLI flow — ranked by
calls x the files they reach, so a registration table calling into itself does
not outrank the CLI), and the most depended-on symbols. Tests are excluded from
both derived lists.
Trail: hops record the direction they were walked (→ into a call, ← up to a
caller), clicking one truncates back to it, Clear keeps the place instead of
throwing it away, and the whole walk travels in the URL. A shared or reloaded
trail arrives as ids, so hops learn their names back through a new batch
endpoint and a session name cache — without it, walking back across a
truncation redrew earlier hops as raw hashes. "Read as flow" stays hidden until
there is a Flow view to send it to.
New endpoints: /api/entrypoints and /api/nodes. New engine reads:
getTopCallingFiles, getFileDependentCounts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Colby McHenryandClaude Opus 5
|
5cecaabfc2 |
feat(ui): the Symbol view — callers, gutter-ported source, line-anchored callee rail (CG-44)
The core screen of `codegraph ui`: who calls a symbol on the left, its verbatim body in the middle with a port on every line that has an outgoing edge, and what it calls on the right — each callee row placed beside the line that makes the call, with a hairline connector between them. The callee rail is the part that is not a list. A row wants to sit at the centre of its first call-site line and is pushed down only when that would collide with the row above, so the rail keeps source order; the connector still runs to the real line, so the displacement is visible rather than silent. Positions come from measuring the laid-out DOM, so they are recomputed on resize, on font load and whenever a fold opens. Honesty is carried in the drawing, not in a footnote: a filled port means the resolver matched something on that line and a hollow one means it only guessed; uncertain connectors are dashed and their targets fold away behind their count; synthesized edges are dashed differently and tagged with the mechanism that made them; references that leave the index are text with a soft underline rather than links to nowhere, and they are counted. Long bodies keep their head plus a window round every call site — windowed on graph edges only, since a function calling `console.log` two hundred times would otherwise window round every line and buy nothing. Containers over 80 lines show a members outline with per-member fan-in/fan-out instead of 700 lines of braces. Two small additions to the read-only API this needed: * `/api/node` gives every outline member its own fanIn/fanOut (two batched queries for the whole outline). A class's own fan-out is nearly always zero because its methods do the calling, so without these the outline cannot say which member carries weight. * `/api/stats` gains `blastScale` — the denominator the blast bar is drawn against, so one symbol's radius reads as wide or narrow *for this repo*. It is measured across the index's 24 most-depended-on symbols (found with a new `getTopDependedOn`, distinct dependents rather than edges), memoised against the index stamp, and reported as sampled; a symbol wider than the sample becomes the scale instead of overflowing the track. Verified against a real index in a real browser: parity with the prototype on `CodeGraph.sync` (259 lines, 27 callee rows, no overlaps), `GraphTraverser` (20-member outline), a 773-line function (26 windows, 78 connectors), light and dark, hover linking in both directions, keyboard-only navigation, and reflow on resize and on fold toggles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
951ba3678a |
feat(ui): read-only JSON API over the index for the viewer (CG-42)
Six endpoints under `/api/`, one per screen, each answering in a single
round-trip in the spirit of `codegraph_explore` — the viewer should never
have to ask a follow-up question to finish drawing a pane:
/api/stats index state, graph counts, frameworks
/api/search?q= ranked, kind-grouped symbol search
/api/node/<id> rails, members, tests, blast radius
/api/source?file=&from=&to= verbatim source + a drift verdict
/api/file/<path> outline and import rails
/api/routes URL -> handler, when there is one
It is a reader of the existing schema: no extraction or resolution changes.
It mounts on the `api` seam `startUiServer` already exposed, so it sits
behind the CG-41 loopback boundary — Host allowlist, no CORS headers,
GET/HEAD only — and every read out of the repository goes through
`resolveProjectFile`, ahead of the index lookup so a traversal is refused
as a traversal rather than reported as "not indexed".
Three properties the endpoints are built around:
- No N+1. The engine's busiest symbol has 545 incoming edges; resolving
those one `getNode` at a time is 545 queries. Every edge list is
resolved with one batched lookup, which needed four additive read-only
query methods (`getNodesByIds`/`getFanIn`/`getFanOut` on `CodeGraph`,
plus batched outgoing/incoming edge fetches and unresolved-reference
reads). `/api/node` on `LRUCache.get` answers in ~10 ms.
- Capped lists, honest totals. 545 callers cannot all be rows, so caller
groups cap at 300 — but `total` is always the real number, and the
ordering puts the useful end first (same file, then production code,
then tests). Every count in the payload is the length of a list the
same payload returns, so a badge and its rail cannot disagree.
- Nothing overclaims. Source that drifted on disk since the last index
sync is omitted rather than sliced at line ranges that may now point at
a different symbol; calls that leave the index are counted instead of
silently shortening the callee rail; imports that never resolved are
named; and a test-coverage claim reports whether its search actually
finished. `/api/routes` says a project simply is not routed, and
refuses a `limit` below three because the engine's manifest would
answer that question wrongly.
Tests: 45 against a real indexed fixture over a real loopback server,
covering every endpoint's shape, the drift verdict in all three places it
surfaces, search ranking and the filter grammar, the refusals, and the
capping/latency behaviour at 500 callers. The issue's own acceptance case
— `lru-cache.ts` `get` under 100 ms — runs against this repo's index when
one is present.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Colby Mchenry
|
41c10750e0 |
fix(erlang): give same-name different-arity functions separate arity-qualified nodes (#1610) (#1615)
Fixes #1610. Also fixes #1358 (the `<<binary>>` arity miscount in behaviour dispatch, reported separately and hit by the same code path). ## Problem Arity is part of an Erlang function's identity — `f/1` and `f/2` are unrelated top-level definitions — but the extractor merged consecutive same-name `fun_decl`s regardless of arity. Reproduced on main exactly as reported: - adjacent `f(X) -> …. f(X, Y) -> ….` → **one** node spanning both, with the first definition's signature; - interleaved `f/1, g/0, f/2` → two nodes with **identical** `qualified_name`; - `cowboy_req`'s `header(Name, Req) -> header(Name, Req, undefined).` → a **self-loop** `header → header`, with the `-spec` for `/3` swallowed by the merged span; - `-export([f/1])` marked every arity exported. ## Fix - **One node per (name, arity).** Clauses of the same name+arity still merge (that part of the old behavior was correct); a different arity starts a new node. `qualifiedName` carries the canonical spelling — `mod::f/1` — while the node **name stays bare** so search and bare-name matching are unchanged. - **`-export` and `-spec` are per-arity.** `-export([f/1])` exports exactly `f/1`; a spec sitting between two arities attaches to the arity its signature names. - **Refs carry the call-site arity** wherever it's statically known: local `f/1`, remote `mod::f/2`, `fun f/1` / `fun mod:f/1` values, `gen_server` dispatch (`handle_call/3`, `handle_cast/2`), and spawn/apply MFA lists (`spawn_link(?MODULE, work, [A, B])` → `work/2`). - **The matcher resolves only to the named arity** — same file first (a local call targets its own module) — and when no definition of that arity exists it resolves to **nothing** rather than a sibling arity: silent beats wrong. An arity-less dynamic-MFA ref resolves only when the module defines exactly one arity of that name. - **Behaviour dispatch** selects the implementer node of the site's arity, and the arity counter now skips `<<1,2,3>>` binary-literal commas per its own docstring (#1358) — `Mod:decode(<<1,2,3>>, Opts)` counts 2, not 4. - **`codegraph_explore` / `codegraph_node`** accept the written `mod:fn/3` spelling against the new arity-qualified names (the issue's measured `cowboy_stream_h:request_process/3` shape). ## Validation Minimal fixtures (all three reported shapes) now index as `gap::f/1` + `gap::f/2`, distinct `inter::f/1`/`inter::f/2`, and a real `deleg::header/2 → deleg::header/3` edge with no self-loop. Cowboy (fresh `--depth 1` clone, this build vs unmodified main build): | | main | this PR | |---|---|---| | nodes | 3,668 | 3,748 (+80 — the arity splits; no explosion) | | erlang function nodes | 2,850 | 2,930 | | behaviour dispatch edges | 38 | **44** | | `cowboy_req::header` | one node, span 420–425, /3's spec lost | `header/2` (420–421, its own spec) + `header/3` (424–425, its spec) | | delegation | self-loop | `header/2 → header/3` | `calls` edges drop 6,059 → 5,656: a sample of every removed pair shows the false-positive class the issue predicted — out-of-repo/BIF calls (`length/1`, `error/1`, `quicer:*`) that previously name-matched onto unrelated same-named in-repo functions now stay unresolved. Tests: new arity coverage in extraction + a new arity-resolution integration suite + a #1358 binary-literal behaviour test; updated existing Erlang expectations to the arity-carrying spellings. Full suite: **3,018 passed, 0 failed**. No migration: an existing Erlang index picks the new shape up on its next re-index (`codegraph sync` / re-`init`). Erlang is wasm-only (not in the native kernel), so there is no kernel-parity surface. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK |
||
|
Max HsuandClaude Opus 5
|
1d9de88ef1 |
feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting (#982) (#1463)
* feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting matchesNonProductionDir hardcodes example/sample/fixture/benchmark/demo, so a peripheral tree only the project knows about — optional-skills/, scripts/ — gets no de-prioritization. When helpers there carry generic symbol names, an exact name match hands them a large bonus and they crowd out the product code that answers the query (#982). deprioritize is the RANKING counterpart to exclude: those paths stay indexed and findable, they just stop outranking first-party code. It is deliberately distinct from the corpus-frequency discount, which keys on a name being common and is near-inert on #982's own repro where only two symbols are named usage. The -15 path penalty alone is not enough, and measuring showed why: on that repro a usage() helper sits at 74.8 against 51.2 for the top product symbol, so -15 lands at 59.8 and still leads. The path penalty is additive and the name bonus it must counter is additive and larger. A de-prioritized path is saying its symbol NAMES are not the answer, so the exact-name bonus is damped to 0.25x there as well — damped, not zeroed, so the tree still ranks when it genuinely is what you asked for. Refs #982 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky * fix(config): read deprioritize lazily and apply it in explore too Review of the first cut found two real defects. The matcher was built once in wireLayers(), which runs only from the constructor and from reopenIfReplaced(). The MCP server keeps one CodeGraph per project root alive for its whole lifetime, so editing codegraph.json appeared to do nothing until the process restarted -- exclude and include do not behave that way. The predicate now reads loadDeprioritizePatterns() per call (mtime-cached, one stat) and memoizes the compiled matcher on the pattern array's identity. A regression test writes the config after opening the project and fails on the old code. Explore passed no matcher to scorePathRelevance at either of its two call sites, so the setting only half-applied -- and #982's reproduction rows B, C and D are all codegraph explore, which made this the surface the issue actually reports on. Both sites now pass it. Explore's hard early-continue filters and its non-production budget cap are deliberately NOT joined: those REMOVE content, and deprioritize is a ranking lever by definition. README narrowed accordingly -- it previously claimed this extends the built-in list, which overstated it. Also from review: scorePathRelevance takes a boolean rather than a predicate (the caller already evaluated it, and it was being invoked twice per result), the predicate body is exception-guarded so a bad path can never take a search down, the misplaced const moved out from between imports, two vacuous test assertions tightened, and tests added for the single-penalty invariant, the deliberate isTestQuery asymmetry, and a query that genuinely targets the de-prioritized tree. Refs #982 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky * fix(search): derive the deprioritize name-bonus damping instead of picking it (#982) The 0.25 scale was a guess. On a 62k-node django index it measurably breaks the "discount, don't erase" rule the lever is built on: exact-name queries for symbols that live only in the de-prioritized tree (child, parent, method) fall behind mere prefix matches (children, all_parents, method_decorator). The prefix arm of nameMatchBonus tops out below 40, and a de-prioritized node also takes the -15 path penalty, so 80 * SCALE - 15 > 40 is the bound that keeps a damped exact match ahead of a prefix match at any corpus shape. 0.75 clears it; crowd-out removal is nearly identical to 0.5 (39 vs 40 of 88 peripheral top-10 slots cleared on django), so the deeper discount bought almost nothing and cost the invariant. Two tests pin the bound, including one that fails at the old 0.25. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Max HsuandColby McHenry
|
9219967e43 |
perf(search): seek the name index for exact-name lookups (#1542)
`nodes` carries two name indexes and neither can serve
`WHERE name = ? COLLATE NOCASE`: `idx_nodes_name` is BINARY-collated, and
`idx_nodes_lower_name` is an expression index the planner only matches against
the same expression. All three whole-name lookups in the query layer were
written that way, so each one degraded to a full table scan
(`EXPLAIN QUERY PLAN` reports `SCAN nodes`).
The LIMITs on those queries do not rescue them. SQLite can only stop early once
it has produced LIMIT rows, and the two dominant cases never get there: a query
word that names no symbol at all, and a name with only a handful of definitions.
`searchNodes` runs its supplement once per query term; `findNodesByExactName`
runs two passes per symbol extracted from the question, and extraction is
generous, so a plainly-worded question issues a dozen full scans.
Written as `lower(name) = lower(?)` the same predicate seeks
`idx_nodes_lower_name`. Measured on four indexed repositories, baseline vs fix
in one process (the only difference being how the predicate is spelled):
query "how does the retry backoff work" findNodesByExactName searchNodes
gin (2.5k nodes) 1.27ms -> 0.18ms 3.1 -> 2.6ms
Alamofire (4.5k nodes) 2.39ms -> 0.22ms 4.9 -> 4.0ms
excalidraw (11k nodes) 10.54ms -> 0.17ms 10.4 -> 5.8ms
django (62k nodes) 49.91ms -> 0.17ms 27.6 -> 4.9ms
The seek is flat across all four; the scan grows with the corpus. A one-word
query into `searchNodes` on django is unchanged (~20ms) because a single term's
scan is not what dominates it there.
Lowering the parameter in SQL rather than in JavaScript is deliberate. SQLite's
`lower()` and NOCASE both fold ASCII only, while JavaScript's `.toLowerCase()`
folds Unicode; comparing a JS-lowered parameter against `lower(name)` would
silently stop matching non-ASCII identifiers that NOCASE used to match.
`getNodesByLowerName` is spelled the same way for the same reason. It already
sought the index, but as a bare `lower(name) = ?` it took a pre-lowered
parameter on trust: any input carrying an uppercase letter returned nothing at
all. This is behaviour-neutral for its one caller — `matchFuzzy` lowers in
JavaScript before calling, and `lower()` over an already-lowered string is a
no-op, verified over the ASCII and non-ASCII cases alike. It closes the trap for
the next caller; the non-ASCII gap on the `matchFuzzy` side is a resolution
change and is deliberately not bundled here.
Result sets are unchanged, including which rows the LIMITs keep: entries under
one key in the expression index are ordered by rowid, the same order a table
scan produces. Verified over 14,400 lookups (top-400 names of the four
corpora, probed as stored / upper / lower, against all three call sites) with
zero differences, and end-to-end above with identical result ids.
Tests assert the planner's verdict rather than a wall-clock number, so they are
deterministic: they intercept the SQL each call site prepares and require an
index seek, with a guard that the lookups actually ran. Reverting any call site
turns them red.
Co-authored-by: Colby McHenry <me@colbymchenry.com>
|
||
|
|
81e1f4a92f |
fix: harden daemon and large-index recovery paths (#1562)
* fix: harden indexing recovery and daemon liveness * test: cover daemon and recovery review gaps * test: pin that a failure marker never blocks a later successful parse (#1557 retry-discard guard) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Colby McHenry <me@colbymchenry.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
d8f2eeaddf |
fix(db): loop-append dense unresolved-ref result rows; make stripped-salvage visible (#1558) (#1576)
Real-world validation of #1575 on indexes damaged by the released v1.5.0 binary surfaced both of these. getUnresolvedReferencesByFiles chunked its INPUT under SQLite's parameter limit but appended each chunk's RESULT rows with a spread — every row becomes a call argument, so a dense recovery sync (the #1541 self-heal re-indexing 919 files produced 234,440 rows) exceeded V8's argument limit and killed resolution mid-sync with "Maximum call stack size exceeded", leaving the graph 226k edges short until another sync resumed the orphans (and that sweep resolves measurably worse than the batched path — see the follow-up issue). The failed-ref retry loader had the identical pattern on unbounded result rows. Both append with a loop now (#1558). The #1575 stripped-salvage warning also never rendered: init's summary prints only index_partial warnings and counts only hard errors, so a run with salvaged files still read as fully clean — and with no hard errors the detail wasn't written to errors.log either. Salvage entries now carry code 'salvaged_stripped', the summary prints a visible warning naming the files, and errors.log is written for salvage-only runs. Validated on real corpora with full-graph dumps: healthy-path inits stay byte-identical to the pre-#1575 baseline (cpython Lib, Alamofire, with a determinism control); a realistically-damaged index (41 wiped + 5 missing files, damage generated by the released binary) heals in one plain sync to identical per-file counts and an edge set within the normal incremental residual; pathological mass damage (52% of the repo) completes without crashing. New regression test reproduces the RangeError on the old code with 200k pending refs. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
2c708caf7c | Merge branch 'main' into feature/CG-35 | ||
|
Colby McHenryandClaude Opus 5
|
9efae0f8f2 |
fix(explore): damp ambient declaration files on flow queries (CG-28)
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
03893b0ab9 |
CG-33: converge incremental sync with a full rebuild
A live, auto-synced index did not converge to a clean rebuild of the same tree — 4.3% of distinct edges wrong in both directions on this repo's own index, overwhelmingly `calls`, which is what flow queries traverse and what explore's file ranking weights. Silent: nothing warned, and the symptom read as "codegraph isn't very good" rather than "this index needs rebuilding." Two causes, and the fix needed both. Resolution binds a reference to one of the same-named definitions PROJECT-WIDE, so a definition appearing or vanishing changes the correct answer for references in files the sync never touches — and those references resolved successfully once, which deletes their unresolved_refs row, leaving nothing to revisit them with (#1240's retry only revisits refs parked as failed). Separately, when nothing disambiguated the candidates the winner came down to rowid, i.e. the order files happened to be WRITTEN, which differs between a scan-order full index and a sync that appends each file as it changes. That second one is why re-resolution alone could not converge: re-resolving against the identical graph still picked a different candidate. So getNodesByName now orders by (file_path, start_line) — a property of the code, not of the write order — and sync computes a definitionDelta and re-opens the resolution edges whose answer it may have invalidated, re-inserting each as the reference that created it for the orphan sweep to bind against the post-sync graph. The delta compares `file\0name` pairs per file rather than one name set over the batch: a commit that adds `collect` to a new file while an unrelated changed file already defines `collect` cancels out of a batch-wide set, and that miss was the largest residual class in the first measurement. Conservative where the failure modes are asymmetric — a wrong deletion is a permanent edge loss, a missed rebind is only residual drift. Edges without a refName stamp are never touched (nothing to restore them from), sources the sync already re-extracted are skipped, and a per-name ceiling declines the generic names. Edges are deleted before the sweep re-inserts, since INSERT OR IGNORE against idx_edges_identity would otherwise keep both rows when a reference rebinds elsewhere. Replaying real commits of this repo through sync, then diffing against a rebuild: 16 commits 48 -> 0; 80 commits 1,634 -> 361, with the actively misleading direction (stale edges the index keeps asserting) 671 -> 2. Index and sync wall-clock are unchanged; the ORDER BY costs 18% per uncached name lookup, which never reaches wall-clock because the resolver memoizes it. The 357-edge residual at 80 commits is one pre-existing class: refs to generic names (`push`, `join`) parked above #1240's per-name retry ceiling, which a rebuild resolves into cross-language garbage — a TS test file "calling" an R method. Converging there would mean manufacturing wrong edges, so it is left alone. And no drift metric in `codegraph status`: it cannot be computed without the rebuild it would be recommending, and a proxy would fire on that residual and train users to ignore it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby McHenryandClaude Opus 5
|
16e17495f4 |
feat(extraction): content-based generated-file detection (CG-5, #1500)
`isGeneratedFile` was path-only, but Go's own convention is a CONTENT marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one. A Go monorepo with generated CRUD in ordinarily-named files sitting beside hand-written use-cases was therefore invisible to every generated-file down-rank in the codebase — that is #1500. Measured on kubernetes/client-go (2,453 Go files): the canonical banner appears in 2,001 of them, the path check flags 0, the new content check flags exactly those 2,001 — no false positives, no misses. Design: decide at INDEX time (content is already in memory for parsing), persist on `files.generated`, read from the DB. Explore never reads file headers per request. - `hasGeneratedHeader(content)` recognizes the standard banners — Go's, protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator, FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an 8KB/60-line header window, a comment-line requirement (leader or open block comment), and markers tight enough that prose can't trip them. A generator's own source, holding the banner as a string constant in its body, is not flagged; neither is this module itself (pinned by test). - `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback. - Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no backfill: the flag derives from content the migration cannot see, so rows stay 0 until a re-index and every reader unions the flag with the path check — an un-migrated index keeps pre-#1500 behavior rather than regressing. Re-index required; noted in the CHANGELOG. - `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1) lookups. Bounded, not cached: no invalidation, so a ranking call can never serve a verdict the last sync already replaced. Wired into explore ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the context formatter, and the dominant-file/route-file hygiene filters. Cost (acceptance bar was no measurable index-time regression): a single unanchored `/generat/i` test over the header rejects ~every hand-written file before any line splitting. 4.6 µs/file on client-go (worst case — 82% generated). End-to-end `codegraph init` on client-go, n=3 alternating arms: 5.73s median with detection vs 5.76s path-only baseline; the arms cross over between runs, so the difference is inside run-to-run noise. Scope note: generated status remains a stable TIEBREAK at equal score, exactly where it was. Making it a strong negative signal is CG-10, which this unblocks by making the signal correct and available. Two pre-existing tests hard-coded schema version 8; both now track CURRENT_SCHEMA_VERSION (or the migration table) so future migrations don't require editing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
02c0e2c935 |
fix(db): stop watchdog-killed sessions from leaking the SQLite WAL without bound (#1431) (#1490)
A SIGKILL'd process (the #850 liveness watchdog, OOM, a crash) leaves its WAL on disk; the next session appends to the same file; and nothing ever truncated it — PASSIVE checkpoints fold frames but keep the file at its high-water mark, and the one shrinking path (a clean last-connection close) is exactly what a killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing until the disk filled. - journal_size_limit on every connection: resetting checkpoints now clip the WAL back to the cap instead of leaving it at its high-water mark. - healOversizedWal() fired from every DatabaseConnection.open: off-thread PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB, CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with bounded retries — concurrent passes defeat each other (each checkpoint sees the other as a busy reader). - Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the #1231 slow-disk deferral to the long-lived server so a healthy daemon mid slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs. - codegraph status shows WAL size (human + JSON) and warns when it dwarfs the DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps so kills can be placed in time. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
ce0ae30e09 |
perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)
Store-architecture arc round 2. The batched resolution loop reads unresolved_refs ONLY through the status index + the PK keyset pager; the other five ref indexes (from_node, name, file_path, from_name, failed_tail) serve sync-time paths — yet every per-batch DELETE of resolved refs maintained all of them, the biggest single main-thread stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase) and 50-81s at kernel scale. beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as refIndexLoad hooks next to the existing bulkEdgeLoad pair with the same minRefsForPool gate (small syncs never pay): drop the five for the loop, rebuild each in one scan at the end — where the table holds only the surviving FAILED refs (resolved rows are deleted by then), so the recreate is near-free. Crash inside the window heals on the next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS). Measured: - dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall ~8.5s flat — the freed main-lane time shifts into settle (the worker lane now binds the double-buffer at medium scale). - Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s, backpressure 16.8 → 7.4s — fewer index writes mean less WAL and cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the 14.8min pre-arc best; <10min-on-8c now needs ~1 more minute. Gates: dubbo/gson dumps byte-identical; linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
f6d8e8fdab |
perf(store): parse-lane index deferral — dubbo fresh init −19%, kernel-scale envelope best-ever 14.2min (§4d round 1) (#1368)
Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall 10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94% store-writer busy with B-tree maintenance as the floor (statement batching and sorted inserts already killed at ~zero). This applies the resolution phase's proven edge-index window to the whole parse lane: beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT ONLY (incremental runs delete per-file rows through the file_path indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes plus the 4 non-unique edge indexes for the parse phase's mass insert (the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it, and its source prefix keeps mid-window reads indexed), then rebuild each in one table scan before resolution, with a yield between builds (the endBulkEdgeLoad watchdog rationale). A crash inside the window heals on the next open — schema.sql re-applies CREATE INDEX IF NOT EXISTS. Measured: - dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild 665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar gap vs cbm shrinks from ~3s to ~1.1s. - Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse itself flat (linux parse is extraction-bound, not writer-bound) and the rebuild costs 21.6s — but every downstream phase dropped (resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s, maintenance 16.3s): bulk-rebuilt B-trees are densely packed where incrementally-grown ones are fragmented, so every index-mediated read for the rest of the run pays fewer pages. Gates: dubbo/gson/express/excalidraw full dumps byte-identical (dubbo's canonical 441,270 lines reproduced); linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Incremental sync paths untouched by construction (freshDb gate). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
971a5a0483 |
perf(resolution): worker connection recycling — WAL-depth writes-under-readers fix, superphase −11.4% at 8c (#1362)
The §7a.6 anomaly probed to its mechanism with five discriminating runs (§7a.7 table): main-thread B-tree writes triple under attached readers because READERS PIN WAL checkpoint progress — the deep WAL taxes every writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on identical hardware; an aggressive 64MB valve recovers the writes but overpays +129s in full-park folds; the v2 cache resurrection was falsified — long-tail name traffic is uncacheable at any capacity). Fix: workers close and reopen their read-only connections every 8 batches at the double-buffer's worker-idle boundary (ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call). Reopens are sub-millisecond, resolver caches survive (only prepared statements re-prepare), and the existing checkpoints advance instead of parking. Failed recycle downgrades to sequential, same as a failed fan-out. Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated): resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min, recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines), counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by construction (no pool → no recycling). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
7cc23668b5 |
perf(resolution): batch-loop de-quadratic — keyset reads, changes-based guard, DB-scaled valve caps + resolve profiler (#1339)
The §7a.2 per-ref profile overturned the assumption the whole arc was built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop. Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s, deletes/inserts/marks ~84s, settle 85.7s. - Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries now return summed SQLite , and zero-removals-from-claimed-work is the guard signal — the DIRECT evidence the count diff inferred (a mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real COUNT runs only on that suspicious path and arbitrates exactly as before. - Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the accumulated failed-row prefix every read; seeking past the last-seen rowid is prefix-independent and enumeration-order identical. - WAL valve caps scale with DB size (env still wins): every fold re-writes hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap); soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient WAL ≈ project size. - CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage attribution, main + workers, off by default. Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel required). Kernel-scale payoff run lands in the plan doc next. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
2adc7f60c0 |
fix(db): WAL truncate at parked barriers ONLY — the timer-path truncate loses the race it was assumed to lose (#1336)
A truncate checkpoint started against an ACTIVE writer wins the lock and then blocks that writer for its entire backfill; after the edge-index recreate's multi-GB single-transaction burst that exceeds the writer's 5s busy_timeout and fails the index with 'database is locked' (§7a.2 record run, EXIT=1 at kernel scale — the small mid-resolution truncates folded in ms and masked the hazard). Barrier truncates (backpressure/foldNow) are collision-free by construction: the writer is awaiting the valve. Dubbo gate: exit 0, peak 81MB (barrier folds carry containment), dump byte-identical. Valve + sizing suites 27/27. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
ca88d3bd15 |
fix(db,resolution): WAL file cap + cgroup cache credit + pool/parse sizing corrections from the instrumented kernel-scale runs (#1335)
Four §7a.1 instrumented-run findings, each measured: 1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still grows the FILE without bound — the writer only restarts at frame 0 when a commit finds zero reader marks, which the instrumented run showed never happens (file marched 361→721MB through two COMPLETE backfills; 22GB by phase end). backpressure() now also trips at 4× the soft cap on raw file size and TRUNCATEs at the parked barrier; the timer path truncates opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at the same 16MB valve; dumps byte-identical under aggressive folding. 2. cgroup memory credit: memory.current counts reclaimable page cache — a post-parse container read 57MB of headroom on a 6GB box and silently disabled the pool. inactive_file is credited back (the docker-stats working-set convention); the same run now reads a sane 4.4GB budget. 3. Pool at 2 cores reversed: sequential resolution measured FASTER than pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing. cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path. 4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower (493s vs 369s) — main + store-worker don't fill the second core. Floor restores the baseline (373.5s measured). Plus the observability §7a.1 burned three 25-minute cycles for: valve armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture, pool sizing decisions (incl. the disabled path), backpressure-hook presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG. Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record runs with this build follow in the migration plan §7a.1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
8c1e821495 |
fix(db): WAL valve — TRUNCATE at parked barriers, futility latch, CODEGRAPH_WAL_VALVE_DEBUG (#1334)
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with the backpressure hook DEPLOYED): 1. TRUNCATE at parked barriers: a completed passive backfill bounds the un-checkpointed backlog but the FILE only stops growing when a commit finds zero readers holding WAL marks — rare while pool workers cycle (dubbo debug baseline: file climbed monotonically through six completed pass-1 backfills). At a parked barrier the no-reader window is guaranteed, so chop the file there with wal_checkpoint(TRUNCATE) (off-thread, 2s busy_timeout — a racing reader degrades it to a no-op). 2. Futility latch: when backfill gives up (pinned reader), parking again at every over-cap boundary burns a 20-pass checkpoint attempt — each a worker thread + fresh connection against a multi-GB DB — per batch. Two consecutive give-ups now disable parking for 60s; a pinned phase degrades to pre-valve behavior instead of OOM-amplifying. 3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the caller's verbose plumbing, and give-up lines print under CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were verbose-gated. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby MchenryandClaude Fable 5
|
4efc6c70e2 |
fix(scale): kernel-scale hardening — OOM-safe pass skipping + watchdog-safe index recreate (#1323)
Two hazards found by running today's full stack against the Linux kernel (70,129 files) in the cg1212 repro container: 1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN thread. At multi-million-node scale a worker failure is usually a memory ceiling, so the retry would OOM the process and take the whole index with it. Above 1.5M nodes a failed pass is now skipped with a clear stderr message (its synthesized edges are absent; the index completes). Below that, the main-thread retry stays — small-scale worker crashes are transient and the retry keeps coverage. 2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span — measured 79s at kernel scale, past the #850 liveness watchdog's 60s stall window. A daemon-triggered re-index would have been SIGKILLed right after doing the work. Now async with an event-loop yield between builds, keeping each stall to a single index (~20s at kernel scale). Validation: full Linux kernel index to completion in the repro container — 2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min. Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate 79s), maintenance 74s. Suite green (2444). Also adds docs/design/native-extraction-kernel.md — the spike-validated design for the native extraction kernel (Rust parse+walk over dubbo's Java: 202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |