mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-18 13:53:01 +08:00
09b9646b7e340b750a8e997bcf1ed755dd0343fb
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby Mchenry
|
5eaa6fedd5 |
fix(extraction): preserve receivers for method values (#1820) (#2034)
* fix(extraction): record callers for methods passed as values (#1820) pool.submit(obj.method), thread_pool_exec(self.store.fetch), and Go Submit(c.store.Fetch) produced no callers/impact edge because only obj.method(...) call expressions were captured. Member values are now emitted as *.method function-refs and resolve unique-or-drop (same-file first). Two methods of the same name still produce no edge. Go go c.store.Fetch(ids) remains a calls edge (it is a call expression). EXTRACTION_VERSION 27: MCP daemon catch-up rebuilds a stale index on start so the watcher does not keep serving the hole after upgrade. * fix(resolution): do not let test mocks veto #1820 method-value callers A unique production method plus a unit-test double of the same name was treated as ambiguous, so retrieve_vision_vectors still looked callback-invisible on a real repo. Prefer the non-test file when exactly one production target remains. * fix(resolution): prefer a real override over a NotImplementedError base #1820 unique-or-drop treated DocStoreConnection.delete_payload_fields (the ABC stub) and QdrantConnection.delete_payload_fields as ambiguous, so gc_stale_member_stamps stayed invisible. Skip short not-implemented stubs when exactly one real implementation remains. * fix(extraction): preserve receivers for method values (#1820) Python attributes and Go selectors used as callback values were dropped or reduced to bare method names. Preserve receiver paths in native and WASM extraction, resolve through type/import scope, and reject ambiguous or noncallable targets. Cover callback, assignment, collection, and query behavior, with extraction version 27 signaling the required reindex. Co-authored-by: Josh66 <info@aska-systems.de> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Josh66 <info@aska-systems.de> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
86708bf519 |
fix(extraction): preserve symbols with colliding same-line IDs (#1349) (#2030)
Same-kind, same-name declarations on one line shared an ID and silently overwrote each other in SQLite. Use per-extraction collision tracking to append a UTF-16 column only to later distinct declarations. Apply the same rule across wasm, Rust language walkers, Liquid, and CFML while preserving file IDs and edge endpoints. Add persistence, call-edge, identity-vector, and Unicode parity regressions. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
bd993b12fa |
fix(extraction): index Python body docstrings across backends (#1905) (#2004)
* feat(extraction): index Python docstrings, not just preceding comments (#1905) getPrecedingDocstring only walks preceding comment siblings, so a Python docstring — a bare string literal first in the body — never reached the docstring column, never entered nodes_fts, and was never shown by `codegraph node`. The identical sentence written as a leading `#` comment was both. For a Python codebase that is most of the prose there is. (#1905) Adds an optional getBodyDocstring() to LanguageExtractor, in the same shape as getSignature(), and routes every docstring call site through one docstringFor() helper that consults both sources. A node carrying a comment AND a docstring keeps both, joined: they are two things the author wrote about the same symbol and the column is free text. The Python implementation reads the grammar`s string_content rather than slicing quotes off the raw text, so r/u/b prefixes and both triple-quote forms work without a regex per case; f-strings are skipped because an interpolated string is code, not prose. Dedent follows PEP 257 (first line exempt). Scoped to Python deliberately. Julia (a string sibling before the def) and Elixir (@doc) fit the same hook and are left as follow-ups. * fix(extraction): index Python body docstrings across backends (#1905) Python extraction only consulted preceding comments, leaving body docstrings absent from search and rendered prose. Extend the contributor's hook to module and definition docstrings, and mirror it in the native kernel. Handle comments, concatenated literals, and indentation consistently while rejecting bytes, f-strings, and tuples. Verify persistence, exact-name ranking, MCP/CLI rendering, and native/WASM parity. Co-authored-by: Max Hsu <maxmilian@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Max Hsu <maxmilian@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
9181dd1ef3 |
fix(extraction): land upstream declaration initializer walks (#1511) (#1802)
Squash danusha2345's PR #1511 at |
||
|
Colby Mchenry
|
bb1d3093eb |
fix(extraction): never fabricate an edge from a call-result receiver (#1748)
A member call whose receiver is itself a call — `d.setdefault(k, []).append(v)`, `make().run()` — used to drop the receiver at extraction time, degrade to the bare method name, and exact-match any top-level project symbol of that name (Python and JavaScript/TypeScript). Keep the inner callee encoded as `<inner>().<method>` in the TS extractor and native kernel; the name-matcher refuses to guess for that shape (store-accessor exception only). Based on #1692, rebased onto main after #1746. Fixes #1683. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby Mchenry
|
838006c947 |
fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) (#1600)
Fixes #1581.
## What was wrong
`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.
Two things made "just give the worker a bigger stack" the wrong fix:
- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
several recursion points with different frame sizes, so no single stack size is a
provable bound.
Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.
## What this does
**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.
- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
`pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
`GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
`exhausted()` is one thread-local load and one compare: true once the stack pointer is
within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
found by script (every cycle in the call graph, not just direct self-calls). It returns
`Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
descending; a hook returning `false` sends its caller down the generic child walk, whose
own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
afterwards the (truncated) result is discarded and replaced by
`defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
transitively). No wire/ABI change.
Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.
## Tests
**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.
**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
— clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
`deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
exits 0 and records both files, with both functions.
**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.
**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.
**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):
| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |
Within run-to-run noise, as expected for one TLS load + compare per recursion entry.
**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:
```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node
=== reporter repro (issue #1581): 16,384-brace deep.c, codegraph init ===
│
└ Done
init exit code: 0
file: deep.c
file: deep100k.c
file: ok.c
function: add
function: bar
function: foo
=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0
=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s
=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files 2 passed (2)
Tests 18 passed (18)
```
(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)
**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:
```
head:
|
||
|
Colby Mchenry
|
38580e0b04 |
fix(python): bare class references produce references edges to classes (#1478) (#1493)
Python's class-as-value idioms (return SomeClass, x = SomeClass, registry dicts, classes passed as arguments) produced no references edges, so callers/impact on a Django/DRF serializer missed the views that consume it. Three gates dropped them: - return_statement was never dispatched by PYTHON_SPEC (kernel mirrored) - the extraction gate (definedHere) collected function/method names only - resolution accepted function/method targets only (matchFunctionRef + the function_ref import fast path) Capture return_statement for Python (single expression; tuple returns not descended), admit same-file CLASS names to the gate, and accept class targets for Python bare identifiers — scoped to Python so the TS/JS KIND FILTER contract is untouched. The docopt false-positive mechanism behind the function-only rule (lowercase locals vs same-named methods) doesn't transfer: methods stay excluded for bare ids, and the same-file/import gate + unique-or-drop rules still apply. Probed on django-rest-framework (~250 files): 559 new references→class edges, 10/10 sampled genuine (serializer_class = AuthTokenSerializer, the ModelSerializer field-mapping registry, aliases, ctor args, isinstance). EXTRACTION_VERSION 24 → 25. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Colby McHenry
|
c2503e2bee |
feat(kernel): R5 — Python and Go ports, gates passed, default-on
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the native kernel, mirroring the wasm extractors bug-for-bug. Python: decorated_definition docstrings/decorators (decorates refs only for bare-identifier decorators — the call-kind quirk), function-in-class → method, module-level assignments always extract as variable, from-import per-name binding refs, self.x fn-ref candidates as bare names. Go: receiver methods with Recv::name qualified names + contains edges to the first earlier struct of that name, type_spec struct/interface classification with embedding→extends and interface method nodes, composite-literal instantiates keeping the package qualifier, top-level var/const initializer walks attributed to the declared symbol (#693), 2-hop field chains (#1276), New().Method() re-encode (#645/#608), and the GO_SPEC fn-ref layers. Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with wasm vendored from the same tags (parser.c sha-matched) — both were 2023-era in tree-sitter-wasms. Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3 error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django (360,794), and prometheus (213,758); torture fixtures enforced in npm test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/ python/go. Full suite: 2,471 tests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |