mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-18 13:53:01 +08:00
6421644ef292d0e6decd968e249c18bd648a9078
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby Mchenry
|
aeb8f95581 |
fix(kernel): find a store exported on a later line in CRLF TS/JS files (#2364)
The kernel's is_exported_later mirrors TreeSitterExtractor.isExportedLater's multiline regex, but JS's `m` flag treats `\r` as a line end while the regex crate's `(?m)` sees only `\n`. On a CRLF checkout (Windows, autocrlf) a line like `export default useStore;\r\n` matched in wasm but not in the kernel, so `const useStore = create(...)` + `export default useStore;` lost its actions (e.g. `function inc`) on the default kernel route for TS/TSX/JS/JSX. Compile it in CRLF mode, `(?mR)`: `^`/`$` then treat `\r` as a line end too, and the pattern body stays byte-identical to the TS regex. It agrees with the JS regex on CRLF, trailing whitespace and lone `\r` line ends, where an optional `\r?$` would still miss a lone `\r`. This was the only `(?m)` regex in the kernel. The other tsjs regexes are whole-string anchors, and the minified-bundle check splits lines on `\n` exactly as the TS does. Adds a kernel/wasm parity case (ts/tsx/js/jsx, LF and CRLF) that fails on the unfixed kernel for all four CRLF variants and passes with the fix. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
f40db4b9cc |
fix(js,ts): index the functions in every named object literal, not only exported ones (#2300) (#2363)
Issue #2300: a function written inside an object literal became a symbol
only when the object was an `export const`. A plain `const api = {...}`,
an object declared inside an IIFE or a function, and a namespace hung on
the page (`window.WS = {...}`, `ns.mod = {...}`) produced no member nodes,
and no node for `WS` either. Calls made inside those members were credited
to the enclosing constant or lost, and calls into them resolved to
nothing. Script-tag JavaScript, written almost entirely this way, was
mostly missing from callers and impact.
Cause: the TS/JS extractor, and the kernel's tsjs mirror of it, minted
members only for exported object-of-functions; any other literal was
walked as one opaque initializer, and an assignment to a member path was
never a declaration. Resolution had no way to reach a member through
`App.init()`, `window.App.init()` or `App.utils.pad()`.
Fix (extraction, TS and kernel byte-identical): a named object literal
owns its function members (method shorthand, `key: function`, `key: () =>`,
generators; static keys only), whether it is declared at module scope,
in a function body or IIFE, assigned to a path (`window.App = {...}`,
`App.utils = {...}`) or assigned to a plain name at module level
(`dw_page = {...}`). Members are `function` nodes qualified under the owner
(`api::load`, `window.App::init`, `App.utils::pad`), the exported case
included; a path owner is named by its last link and qualified by the
path. Calls in a member are the member's; other values are walked where
they were before. CommonJS exports, prototypes, `this.x = {...}`,
call-argument literals, a name reassigned inside a function, and
generated or minified files keep the old shape. Minified files are now
also recognised by content, in both extractors, and the kernel's
`.min.js` pattern matches the TS one.
Fix (resolution): a member is reached through its object only:
`App.init()`, `window.App.init()`, `App.utils.pad()`, a sibling's
`this.render()`, a `const { init } = App` binding; never a bare `init()`.
A same-file holder is chosen by lexical block (an IIFE's own `App` first);
otherwise a global one (`window.App = {...}`, or a classic script's
top-level `App`) when the caller neither imports nor binds the name.
Arrow members keep the `this` of the method around the literal, in the
`this.x` resolvers too. A dotted-path holder is never reached by its last
name alone, and Svelte's `$store` rule now applies only in `.svelte`
components, so the new local holders are not taken for `$n` in plain
scripts. The lookups read each file once (destructuring patterns only
when the raw text has a `} =`) and keep only the last 32 files' scans.
Contributor PR #2310 (@danusha2345): adopted its model (owner-qualified
members exported or not, path owners, IIFE/local owners, static keys,
no bare-name reach, `<script setup>` contains edges, SFC languages, test
scenarios), re-implemented on current main. Not taken: the
EXTRACTION_VERSION bump (already 28), the extraction-time binding oracle
with its ref/edge metadata and kernel ref patching (replaced by
resolution-time lookups), naming the owner `window.WS`, rewriting
`window.X.m()` ref names, flipping exported members' isExported, and
unrelated bare-call changes.
Verification: new js-object-literal-members suite (native + wasm, 18
tests); the issue's shapes fail on main and the guard test fails without
the guards. kernel-tsjs-parity passes with CODEGRAPH_KERNEL_EXPECT=1
(object-literal owners LF/CRLF, minified bundles); kernel parity sweeps
show 0 diffs on DokuWiki, vue-realworld, TodoMVC, excalidraw and this repo.
Validation (kernel loaded, before = origin/main
|
||
|
Colby Mchenry
|
8998697f1d |
fix(vue,ts): link calls in Vue templates and in top-level destructuring (#2340) (#2357)
`codegraph callers <composable>` missed most of the components using it in a Nuxt app (Issue #2340), through two extraction gaps: - A destructuring declaration at module or `<script setup>` scope (`const { a } = useFoo(1)`, `const [b] = useFoo(1)`) mints no symbol, and extractVariable skipped its declarator with `continue` before walking the initializer, so the call was never recorded (inside a function the body walker reached it). The declarator is now walked the way a function body walks it, with the enclosing scope on the stack: the file for module code, which a `<script setup>` block folds into its component. Defaults in the pattern and calls inside callbacks in the initializer count too. Mirrored in the native kernel (tsjs/extractors.rs), which extracts TS/JS in released builds. - The Vue extractor parsed only `<script>`, so calls in the template (`{{ useBar(link) }}`, `:to="useBar(link.location)"`, `v-if`, `v-for` sources, `@click="save(item)"`) made no reference. The new vue-template-calls scanner reads the root `<template>` quote-aware, collects interpolations and directive values, and names each callee the way a script call is named (`useBar`, `store.fetchUsers`). It skips `v-for` aliases and slot props in their element's subtree, arrow and function parameters, `$`-helpers, Vue's template globals, strings, regex literals, `new X()`, object-method shorthand, `v-pre` subtrees, custom blocks and non-HTML templates. Each call is a `calls` reference from the component at its own line and column; a bare call to the component's own Options API method is linked at extraction, since resolution only binds those through `this.`. Two resolution follow-ons, found validating on elk: - The vue-handler synthesizer resolved the leading identifier of every `@event` value, so `@click="save(item)"` got a second, line-1 edge, and `@click="query = ''"`, a slot prop's `hide()` or a `refresh()` destructured from useAsyncData were bound to any same-named function in another file. It now resolves only a bare handler name (`@click="save"`) and keeps its composable-destructure mapping. - The Nuxt auto-import rule answered a call to a file's own `clearError` or `navigateTo` with an edge from the caller to itself. A same-file declaration of that name now shadows the auto-import. Verification: new tests in vue-template-calls.test.ts (scanner shapes and four indexed projects, incl. the issue's repro), extraction.test.ts (module-scope destructuring in ts/tsx/js/jsx; script-setup + template refs, LF and CRLF) and kernel-tsjs-parity.test.ts (destructuring, kernel vs wasm). On main (src + kernel reverted) the 10 indexed/extraction tests fail and the 4 parity cases fail with the main kernel; all pass with the fix (CODEGRAPH_KERNEL_EXPECT=1). Issue repro on the built CLI with the kernel: callers useFoo 2 -> 5 (A.vue, B.vue, F.vue, G.ts, D.ts), callers useBar 0 -> 2 (H.vue, I.vue). kernel-parity.mjs over elk, nuxt/movies and fastify: 467/471 files byte-identical, 0 diffs, 4 deferred. Validation (kernel loaded, main vs branch, edges by natural key): elk 6282 -> 6633 (+443 / -92), vue-element-admin 2337 -> 2372 (+60 / -25), nuxt/movies 939 -> 959 (+26 / -6), fastify (plain JS control) no edge change. Every removed edge is a line-1 vue-handler edge whose pair is now linked at the handler's line (101), a name-collision guess (8), or an auto-import self-loop now resolved to the local function or route (14). Self-loops: elk 44 -> 30, others unchanged. Builds on #2341 by @drakeo338 (walk the destructuring initializer) and the template scan in #2073 by @L0garithmic. Co-authored-by: drakeo338 <paranoyouz@gmail.com> Co-authored-by: L0garithmic <L0garithmic@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
9649ce1bd6 |
fix(resolution): reject unrelated cross-language matches (#1986) (#2032)
* fix(resolution): stop expression-receiver calls from binding to unrelated methods A method called on an expression receiver reached the resolver as the bare method name, and exact-match bound it to any project method of that name: `(await list()).map(...)` in TypeScript onto a UI adapter class's `map` (0.9), and `v.iter().map(...)` in Rust onto the same TypeScript method across languages (0.5). TS/JS extraction (mirrored in the kernel): look through receiver wrappers that keep the same object — parentheses, `!`, `as`/`satisfies`, `<T>x`, `await` (including the grammar's `(a && b)!` parse of `a && b!`) — so `x!.m()` is `x.m` and `(await f()).m()` is `f().m`; any other receiver with no static type (`(a ?? b).m()`, `f().list.m()`, `arr[0].m()`) emits nothing instead of the bare name. `this`/`super` (and chains rooted at them), `window.*` chains and `new C().m()` keep their existing bare form. Resolution: a bare-named call may cross a language family only through the C ABI (a C/C++ symbol, or a free function a C/Objective-C/Swift caller reaches, e.g. a cgo `//export`). A class member of another family, or a function/class/variable outside that pair, is rejected as the candidate a strategy commits to (exact-match single/best and fuzzy), never filtered out of the pool, so no lone survivor is manufactured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resolution): reject unrelated cross-language matches (#1986) Name-based calls, constructors, and inheritance could select unrelated foreign-language symbols. Apply a shared interoperability policy to selected results without manufacturing replacement matches, including JSX/Vue synthesis. Preserve framework exports and ABI-backed calls, building on PR #1944's TS/Rust expression-receiver fixes. Validate with both repro scripts, 529 targeted tests, kernel/wasm parity, and clean TypeScript checks. Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
86708bf519 |
fix(extraction): preserve symbols with colliding same-line IDs (#1349) (#2030)
Same-kind, same-name declarations on one line shared an ID and silently overwrote each other in SQLite. Use per-extraction collision tracking to append a UTF-16 column only to later distinct declarations. Apply the same rule across wasm, Rust language walkers, Liquid, and CFML while preserving file IDs and edge endpoints. Add persistence, call-edge, identity-vector, and Unicode parity regressions. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
b65e05b98f |
fix(resolution): require receiver evidence for built-in method calls (#1987) (#2028)
* fix(resolution): resolve this.#field.method() on the field's type (#1987) A call through an ES private field fell outside the `this.<field>` shape (#1496): the extractor accepted only a property_identifier, so `this.#items.add(x)` was emitted as the bare `add` and exact-matched whichever project method shared the name, often the calling method itself. The extractor and its kernel mirror now keep `this.#items.add`, and the resolver matches that shape and reads the field's type off the class declaration. The field regexes open with a lookbehind instead of `\b`, which never matches before `#`. A builtin or external field type (`new Set()`, a `Map`, an array) now yields no edge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(resolution): require receiver evidence for built-in method calls (#1987) Unknown JavaScript receivers could match unrelated methods by name or class-name similarity. Require validated receiver evidence for built-in method names while retaining typed, imported, object-literal and class resolution. Preserve private-field receivers in TypeScript and Rust extraction and keep public/private field declarations distinct. Build on PR #1990 with regression coverage across all four JS/TS variants and kernel/WASM parity. Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <ewidusoc498@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
9fb2c25bd5 |
fix(extraction): name a function bound through a curried wrapper (#1747) (#2002)
* fix(extraction): name a function bound through a curried wrapper (#1747) reactHookBoundName already names an anonymous function after the variable_declarator that binds it. The shape is general, but the method is bounded to three React hooks, so `const run = Effect.fn("Session.run")(function* () {…})` produced no function node at all -- and its body's calls were attributed to the enclosing container, so a file gained an outgoing edge belonging to a function and the callee's caller list named the file instead. Bound the new path by a different, equally decidable test: the callee is itself a call, i.e. a factory returning the wrapper. That admits Effect.fn(...)(fn), connect(mapState)(fn) and a project's own wrap("n")(fn), and excludes the single-call forms whose argument is a computation -- useMemo(() => …, []), arr.map(…) -- which stay anonymous exactly as before. Widening to "any callee" would have given useMemo a function node and reddened react-hook-handlers.test.ts. Generators are admitted here and not in reactHookBoundName: a React handler is never a generator, while function* is the common form in the ecosystem this shape comes from, and it is what the report opens with -- broadening the callee test alone would not have fixed that repro. Mirrored in the Rust kernel; kernel-parity.mjs reports byte-parity on the new shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WgtWMywGTyu6twnk1Bu9nP (cherry picked from commit |
||
|
Colby Mchenry
|
8dcc52d3c5 | fix: retain qualified call sites in the native kernel | ||
|
Colby Mchenry
|
9181dd1ef3 |
fix(extraction): land upstream declaration initializer walks (#1511) (#1802)
Squash danusha2345's PR #1511 at |
||
|
Colby Mchenry
|
cece0720e3 |
fix(resolution): resolve this.<field>.<method>() on the fields declared type (#1496) (#1792)
Land the six-file fix from upstream PR #1691 by danusha2345
(pr-1691 at
|
||
|
Colby Mchenry
|
de5adba7ea |
fix(resolution): leave built-in Map calls unresolved (#1566) (#1790)
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby Mchenry
|
ee83636acb |
fix(extraction): index TypeScript interface members (#1638) (#1780)
Land upstream #1686 (maxmilian + bompus kernel/CG-28 follow-ups) onto current main. tree-sitter-typescript interface members (method_signature / property_signature) were never listed in the TS extractor, so platform .d.ts APIs had no declaration nodes for call edges. Mirrors on the Rust kernel path; keeps CG-28 damping for pure-interface declaration files; filters damped files from the explore RWR seed set. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby Mchenry
|
8733c2880f |
fix(extraction): index const-bound functions inside a body as symbols (#1669) (#1774)
`const handleClear = () => {…}` inside a component — every React handler
that skips useCallback — was never a symbol: the body walker only named
nested function declarations and hook-bound arrows, so the handler was
absent from callers/impact ("Symbol not found", indistinguishable from
"no callers") and its calls attributed to the component. Bind the arrow
or function expression to its declarator the way module scope already
does, in both the wasm walker and the kernel.
A navigation such a handler makes is now the handler's own edge and a hop
in the Screens `via` chain — the shape a useCallback handler already has —
so the react-router and expo-router expectations follow that convention.
Co-authored-by: danusha2345 <ewidusoc498@gmail.com>
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com>
|
||
|
Colby Mchenry
|
43271f3cd3 |
fix(extraction): index CommonJS export assignments as functions (#1675) (#1771)
`exports.getItems = async (req, res) => {…}` and `module.exports.x =
function () {…}` — the Express controller style — produced no symbol: the
arrow's parent is an assignment, not a declarator, so it stayed anonymous,
its calls attributed to the file, and `node`/`callers` answered "Symbol
not found" for a route-wired handler. Resolve the name from the export
property, mark it exported, in both the wasm walker and the kernel.
Co-authored-by: danusha2345 <ewidusoc498@gmail.com>
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com>
|
||
|
Colby Mchenry
|
a7ea5ba730 |
fix(extraction): a TS/JS call through a host-global chain emits no ref (#1707) (#1766)
`chrome.storage.local.get(key)` and `document.body.querySelector(s)` end in a platform API, but the extractor emitted the bare method name for them. That name then exact-matched whatever project symbol shared it: in a Chrome extension every `chrome.storage.local.get/set` inside a storage wrapper bound to the wrapper's own `get`/`set`, giving self-edges that are not in the source (#1707). A member chain whose root identifier is a host object the project never declares now emits nothing — a silent miss instead of a wrong edge, the same trade the literal-receiver gate makes (#1230). `window` is deliberately not a host root: `window.MyNs.doThing()` reaches a project symbol. A chain rooted at a project value keeps the bare name, so `store.getState().act()`, `ref.value .m()` and `this.<field>.m()` are untouched. The Rust kernel mirrors the same gate. Verified on Linux: fail→pass on both kernel and wasm arms for `__tests__/ts-chained-receiver.test.ts` (2 fail / 1 pass on main → 3/3 with the fix). Lands / rebases https://github.com/colbymchenry/codegraph/pull/1710 onto current main. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> Co-authored-by: Aaron Queen <bompus@users.noreply.github.com> |
||
|
Colby Mchenry
|
bb1d3093eb |
fix(extraction): never fabricate an edge from a call-result receiver (#1748)
A member call whose receiver is itself a call — `d.setdefault(k, []).append(v)`, `make().run()` — used to drop the receiver at extraction time, degrade to the bare method name, and exact-match any top-level project symbol of that name (Python and JavaScript/TypeScript). Keep the inner callee encoded as `<inner>().<method>` in the TS extractor and native kernel; the name-matcher refuses to guess for that shape (store-accessor exception only). Based on #1692, rebased onto main after #1746. Fixes #1683. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby Mchenry
|
df435d50d1 |
fix(extraction): index TS/JS generator function declarations and expressions (#1741) (#1743)
Tree-sitter kinds generator_function_declaration / generator_function were
missing from both the wasm and native kernel function-type lists, so
function* / async function* (and const g = function* () {}) produced no
nodes. Add the kinds on both paths and cover TS+JS declaration/expression
forms in extraction tests.
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com>
|
||
|
Colby McHenry
|
873f133c96 |
feat(expo-router): add Expo Router support for Screens and navigations and introduce Steps API
Introduce Expo Router integration with a new Screens view and API to surface screens and transitions, plus a new Steps API and UI to depict typed steps from anchors or symbols. Extend codegraph’s extraction and resolution to handle namespace objects (export default NAME, two-statement forms, and default bindings) and React hook bindings for handlers, improving accuracy of flows across JS ↔ native boundaries. Add Swift/React Native bridge receiver evidence (RCT_EXTERN_MODULE, RCT_EXTERN_METHOD) and related resolution logic, with tests covering namespace-object resolution, useCallback-driven handlers, and inline RN event listeners. Update UI to include a Steps tab and associated components (StepsView, StepNode, ScreenEdge) and wire navigation to expose steps-based exploration via /api/steps and UI routes. Documentation and changelog reflect the new Expo Router integration and steps surface capabilities. |
||
|
Colby Mchenry
|
838006c947 |
fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) (#1600)
Fixes #1581.
## What was wrong
`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.
Two things made "just give the worker a bigger stack" the wrong fix:
- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
several recursion points with different frame sizes, so no single stack size is a
provable bound.
Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.
## What this does
**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.
- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
`pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
`GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
`exhausted()` is one thread-local load and one compare: true once the stack pointer is
within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
found by script (every cycle in the call graph, not just direct self-calls). It returns
`Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
descending; a hook returning `false` sends its caller down the generic child walk, whose
own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
afterwards the (truncated) result is discarded and replaced by
`defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
transitively). No wire/ABI change.
Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.
## Tests
**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.
**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
— clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
`deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
exits 0 and records both files, with both functions.
**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.
**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.
**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):
| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |
Within run-to-run noise, as expected for one TLS load + compare per recursion entry.
**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:
```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node
=== reporter repro (issue #1581): 16,384-brace deep.c, codegraph init ===
│
└ Done
init exit code: 0
file: deep.c
file: deep100k.c
file: ok.c
function: add
function: bar
function: foo
=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0
=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s
=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files 2 passed (2)
Tests 18 passed (18)
```
(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)
**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:
```
head:
|
||
|
Colby McHenry
|
03d54e47a1 |
feat(kernel): R4 — Java port with Lombok synthesis, gate passed, default-on
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring the wasm extractor's Java paths bug-for-bug: package namespaces, imports, javadoc, annotations→decorates, type_list inheritance, static-final constants, enum constants, anonymous classes (including the TS side's 0-based-line quirk on the extends ref), method_invocation calls with the this.field unwrap and the Foo.getInstance().bar() chain encoding (#645/#608), static-member value reads, method-reference fn-refs (#756), value-reference edges, and the full Lombok member synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/ EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared docstring/textutil modules moved to crate level. Grammar: tree-sitter-java 0.23.5, with the wasm grammar vendored from the same tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build. Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit 341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test; full-init dump-diffs byte-identical on gson (49,766 rows), retrofit (62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. The gate caught a real cross-language bug: fn-ref dedupe and value-ref self-target checks must compare node ID STRINGS, not node-table rows — ids collide for same-(kind, name, line) nodes, which minified one-line bundles hit routinely (retrofit's website JS exposed it; latent in the TS/JS walker since R2, never released). Fixed in both walkers. Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat (parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that wall is main-thread-bound (reads + store), not worker-CPU-bound — the §6 expectation assumed otherwise. Where worker CPU binds the kernel delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s (~1.25×). The identified lever for the many-core headline is decoding kernel buffers directly into store rows (skipping per-node JS object materialization); the buffer contract already carries everything. DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
c8cca9a601 |
feat(kernel): R3 — TS/JS equivalence gate passed, kernel default-on
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Colby McHenry
|
9ad5cd7ba2 |
feat(kernel): R2 — full TypeScript/JavaScript extraction port, byte-parity with the wasm path
Replaces the R1 seed .scm emitter with a bespoke Rust walker (codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS paths function-for-function: declarations (incl. #808 field/property classification), qualified names, docstrings (#780 wrapper climbs), signatures, imports/re-exports + per-binding refs, calls with receiver-qualified callees (#1230 literal-receiver skip), instantiations, decorators, inheritance, type annotations (#381), type-alias members + tuple contracts (#359/#634), React component recognition (#841 forwardRef/memo/styled), object-of-functions / zustand-through-middleware / RTK Query endpoints + generated hooks / vuex + pinia store shapes, function-as-value capture with the flush gate (#756), and value-reference edges with the shadow prune (#895/#897). The generic query emitter is deleted — extraction parity needs logic .scm can't express; future languages get walkers too (migration plan §4a). Positions and JS string-slice semantics are emitted in UTF-16 code units natively, so kernel output is byte-identical to web-tree-sitter's — no column diff class exists. Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650 nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into __tests__/fixtures/kernel-parity/ and enforced in npm test by kernel-tsjs-parity.test.ts. The strict compare caught one real decoder bug the loose harness missed: refs must NOT carry denormalized filePath/language at the extractFromSource seam (the store fills them). Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms, identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants, agent A/B, Linux/Windows) passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |