mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-21 16:57:09 +08:00
c65f9bb7ebc7c7a38f415fd30cc071b61d6639b7
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Colby Mchenry
|
627018eb25 |
fix(cpp): an explicit specialization, like protobuf's WireFormatLite::ReadPrimitive<int32_t, WireFormatLite::TYPE_INT32>, is named after its template (#2609)
An explicit specialization of a function or member function template kept
its template arguments in its name: protobuf's `template <> bool
WireFormatLite::ReadPrimitive<int32_t, WireFormatLite::TYPE_INT32>(…)` was
named `TYPE_INT32>` and qualified `…::ReadPrimitive::TYPE_INT32>`, the
qualified id split at the `::` inside its arguments; rocksdb's `template <>
int64_t Deserialize<int64_t>(…)` was `Deserialize<int64_t>`. A call writes
the bare name, so nothing linked to them.
Both extractors (wasm and the native kernel) now name a specialization
after its template, from the template-id's own name, and take its receiver
from the text before the declarator's name. That text must end in a real
`::`: tree-sitter recovers Metal's `vertex VertexOut vertexShader(…)` with
a missing, zero-width one.
A specialization now meets its template among a name's candidates, so the
resolver chooses between them wherever it picks among C++ candidates (a
class's methods, the implicit-this walk and its bases, exact, qualified,
global, fuzzy and macro-namespace matching):
- a call that writes template arguments reaches the specialization
written with the same ones (spacing aside, or qualified further on one
side), else the template itself or an ordinary overload, else nothing:
fmt's `num_bits<unsigned short>()` stays on `num_bits<T>()` instead of
the nearer `num_bits<uint128>()`; protobuf's `Arena::Create<T>` keeps
calling `DefaultConstruct<T>`, not the tests' `DefaultConstruct<
DispatcherTestProto>` hook; a dependent `ReadPrimitive<CType,
DeclaredType>(…)` links nowhere rather than to the first specialization.
- a call that writes none, or `<>`, reaches a specialization only when
nothing else has the name.
- a class's own member template, specialized but none for these
arguments, ends the lookup there instead of falling to a base's
namesake.
The `cpp-override` pass leaves specializations out on both sides: a member
template is never virtual.
Validated on the cgstd kit against main (
|
||
|
Colby Mchenry
|
34dd3d2fb4 |
fix(cpp): a nested class defined out of line is indexed under its name, and a construction makes the class C++ finds (#2542)
A class, struct or union whose declaration writes its name in a scope —
leveldb's `struct SkipList<Key, Comparator>::Node { … }`, rocksdb's
`struct BlockBasedTable::Rep { … }` — was indexed as `<anonymous>`, so its
members belonged to no class anyone names and a call on a `Node*` or `Rep*`
was guessed by the method's name. Such a class now takes its own name,
qualified under the written scope the way an out-of-line method is, and its
members are qualified under it, in the TS extractor and the kernel alike. A
specialization (`struct fmt::formatter<Answer>`) stays unnamed: named after
its template, it stood in for the template and every other specialization.
Constructions are recorded by the type's last name, so with `DBImpl::Writer`
named, leveldb's `new log::Writer(…)` in `DBImpl` methods went to it. A C++
construction now makes the class C++ lookup finds for the type as written at
the construction (matchCppConstruction): read with its template arguments
(`new Outer<int>::Inner(…)` makes an `Outer::Inner`), at 0.9, or 0.7 when it
is only the class of that name the file can see; an enum lookup finds; through
an alias the calling function declares, the class it names; and nothing for a
template parameter, a member of a type from outside the project or a local
alias of one. Of several classes or aliases the file declares under one
qualified name (the local classes of separate gtest TEST_F bodies, the members
of unnamed specializations), the one declared around the construction. A `<`
or `>` in parentheses inside template arguments is now read as a comparison.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Colby Mchenry
|
235387ec42 |
fix(c,cpp): a struct, class, union or enum defined in a variable declaration is indexed (#2446)
`struct Foo { … } foo;`, `class Bar { … } bar;` and `static struct { … } SPT;`
outside any function or class body went to the variable extractor, which
skipped the declaration's children, so the type, its methods and its
enumerators were never indexed in either extraction engine.
Both engines now walk such a type before the variables: in C always, and in
C++ when the tree has no errors or the file is walked in brace scopes. This
converges with #2426's walk for erroring C++ files; a C++ file whose braces
don't balance is still left alone, since error recovery can run a class past
its own `}` there. The declaration's function-as-value scan skips the walked
type, which captured its own. An unnamed struct, union or enum takes the name
of the first variable its declaration declares (also inside function bodies,
where it was `<anonymous>`), as `typedef struct { … } Name;` already takes the
typedef name, and the comment above the declaration is the type's docstring.
cDeclaratorIdentifier moves to languages/c-cpp.ts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Colby Mchenry
|
9d07864e65 |
fix(cpp): a class keeps its namespaces and enclosing class past code the parser misreads (#2426)
tree-sitter recovers from a construct it can't parse by inserting the token it expected, and inside a class body that is often a `}`: an unknown macro in front of a member (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD absl::string_view name() const`) or in a class head (`struct ALIGN_AS(64U) HandleImpl`) closed the class there, and every `}` after it closed the scope one level out. Recovery also ran the other way, keeping a scope open past its own `}` once an ERROR swallowed it. The walker scoped declarations by tree nesting, so qualified names lost their namespace and class segments, or gained ones they don't belong to: protobuf's `FieldDescriptor`, `MapFieldBase` and `MutableRepeatedFieldProxyImpl`, fmt's `detail::` buffers (and `fmt::format_int` put in `detail`), rocksdb's clock_cache classes and the `Opts` structs nested in them, gtest's `PrettyUnitTestResultPrinter`. Every such file parses with errors, and the kernel defers those to wasm, so the wasm walker produced all of them; the kernel's walker has the same tree-nested stack and reaches erroring trees only under the CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT sweep hatch. For a C++ file whose tree has errors, the walker now takes each declaration's namespaces, and at declaration level its enclosing classes, from the source's braces (languages/cpp-brace-scopes.ts: comments, literals, raw strings, digit separators and preprocessor lines skipped, each #if branch read from the braces open at the #if). A class-like node ends at its body's `}`, and a class the tree glued into a namespace-level declaration's type is walked as a class. A file whose braces don't balance, and every file that parses cleanly, keeps the tree's scopes. Ported to the kernel (ccpp/brace_scopes.rs) so the hatch stays parity-true: the same erroring files diverge before and after (UTF-8/UTF-16 recovery), every other one is byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
ac1f430d3e |
fix(c,cpp): a pointer, array or function member is not a supertype (#2394)
extractInheritance carried Go's struct-embedding check (a field_declaration with no direct field_identifier names an embedded type) into every grammar with a field_declaration_list. tree-sitter-c/cpp, and Objective-C's C structs, nest a member's field_identifier inside its pointer/array/ function/reference declarator, so `SharedState* shared;`, `jv elements[];` and `virtual Status Put(...) = 0;` all became `extends` edges to the member's type: false supertypes in the type hierarchy, blast radius and codegraph_explore, and false cpp-override call edges built on them (google/leveldb: 246 of 302 extends edges). Gate the branch to Go in the TS extractor, drop the mirrored arm from the C/C++ kernel port (and the "kept for parity" quirk note), and drop the inert copy from the Rust-language port. C++ base_class_clause, Objective-C superclasses and Go embedding are unchanged. Validated on leveldb, jq, llm.c and redis: every removed extends edge is a member site, no base-clause edge is lost; gin, SDWebImage and ripgrep dumps are byte-identical; kernel/wasm parity sweeps show 0 diffs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
fa25883ab0 |
fix(cpp): resolve local constructors and macro visibility (#1839) (#2035)
* fix(cpp): macro invocations are not calls; local objects call their constructor A function-like macro invocation (`TRACE_POINT(1)`) parses as a call, and the resolver bound it to a same-named free function in another translation unit — a caller and a callee that never existed (#1838). And local object initialization (`Widget w;` / `Widget w(1);` / `Widget w{1};`) reached the class node instead of a constructor, or nothing at all for the default form (#1839). Extraction (wasm walker and the Rust kernel, in parity): - every `preproc_function_def` becomes a `constant` node whose signature is the directive, so the index knows which names are macros; - a C++ declaration emits one `calls` ref per constructed object, shaped `ns::T::T/<arity>`; pointer, reference and function declarators, `extern`, and array element braces construct nothing; the #1035 `instantiates` ref is unchanged; - constructors carry their parameter list as the signature. Resolution: - `cpp-macro-visibility.ts` walks the translation unit (root file plus in-repo includes, each file once) over per-file summaries of function-like defines, undefs and includes, evaluating what each file can decide itself (literals, its own defines, the include-guard idiom). A call whose name is definitely a macro at that point, or a name that only macros bear, resolves to nothing. A macro that exists in one build configuration only (CMSIS's `__DSB()` under the ARM compilers, a target's HAL compat header) keeps the call to the function the other configuration compiles; a wrapper macro that calls its own name is how that function gets called and hides nothing. - `cpp-constructor.ts` resolves the constructor refs: the type in the lexical namespaces of the call site, then the single overload whose parameter count admits the arguments; two admitting overloads, an initializer_list overload, or an aggregate yield no edge. - a `#define` constant is never a `calls` target and does not count toward the same-name ceiling. Measured on a betaflight tree (4,017 files, wasm arm): 153 `calls` edges lost — all fabricated (`MIN`/`MAX` bound to a CLI enum member, macros misparsed as functions, a cross-vendor `SWAPBYTE`) — and 34 constructor edges gained; indexing 34s vs 30s. Fixes #1838 Fixes #1839 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(cpp): resolve local constructors and macro visibility (#1839) Local construction lacked constructor calls, while macro invocations could bind unrelated functions. Resolve constructors using lexical types and overload signatures, preserving separately declared defaults and array element construction. Evaluate macro directives in translation-unit order, respecting reinclusion, guards and pragma once. Mirror extraction in Rust and WASM, preserve type dependencies, and distinguish them in CLI output. Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: danusha2345 <danusha2345@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
86708bf519 |
fix(extraction): preserve symbols with colliding same-line IDs (#1349) (#2030)
Same-kind, same-name declarations on one line shared an ID and silently overwrote each other in SQLite. Use per-extraction collision tracking to append a UTF-16 column only to later distinct declarations. Apply the same rule across wasm, Rust language walkers, Liquid, and CFML while preserving file IDs and edge endpoints. Add persistence, call-edge, identity-vector, and Unicode parity regressions. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
9b5c8e49dd |
fix(extraction): recover single-argument C/C++ function macros (#1373) (#2017)
Single-argument function macros were indexed under parentheses in C or the macro name in C++, breaking caller resolution. Recover both parser shapes only when a preceding local macro definition establishes the function name, and mirror the recovery in the Rust kernel. Preserve existing multi-argument recovery and ordinary declarations; add extraction, resolution, explore-flow, and parity regressions. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Colby Mchenry
|
2f1a99d34c |
fix(extraction): index C++ pure virtual methods as nodes (#1727) (#1758)
Pure-virtual declarations (`virtual int read(int key) = 0;`) parse as field_declaration, not function_definition, so they minted no method node — calls through an abstract base and cpp-override synthesis had nothing to attach to. Mirror Java interface methods: mint the node (TS + kernel), mark isAbstract, and cover with extraction/e2e/parity fixtures. Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com> |
||
|
Colby Mchenry
|
838006c947 |
fix(kernel): guard the native walkers against stack overflow and defer deep files to wasm (#1581) (#1600)
Fixes #1581.
## What was wrong
`codegraph init` / `codegraph index` died with `Segmentation fault` — the whole CLI
process, not a parse worker — on a C/C++ file with very deep brace nesting (llvm's
`clang/test/Parser/parser_overflow.c`, 16,384 nested `{`). The reporter's diagnosis is
exactly right: tree-sitter's parser is iterative, so the file parses fine, and then the
native kernel's **recursive walker** (`visit_node` → `visit_for_calls_and_structure` → …,
one frame per AST level) overflowed the thread's stack. A native overflow can't be caught
the way a wasm abort can, and a parse worker is a thread of the `codegraph` process, so
the SIGSEGV took the entire indexer down — no message, no per-file fallback, no partial
index.
Two things made "just give the worker a bigger stack" the wrong fix:
- it only moves the cliff — reproduced here: the reporter's 16,384-deep file kills a
default 4 MiB worker (rc=132 on macOS / 139 on Linux), and a 100k-deep file kills the
8 MiB **main** thread too;
- the walkers are shared by every kernel-routed language (20 of them), and each has
several recursion points with different frame sizes, so no single stack size is a
provable bound.
Meanwhile the wasm path already handles this shape gracefully: its JS walker catches its
own `RangeError` per file and stores a partial result with a `parse_error`. The kernel
just needed a way to get there instead of dying.
## What this does
**The kernel guards its own recursion against the calling thread's real stack bounds and
defers a too-deep file to wasm** — the same `defer:` routing signal it already uses for
files with parse errors, which `src/extraction/kernel/index.ts` treats as "take the wasm
path for this file", silently.
- `codegraph-kernel/src/stack.rs`: per-thread stack bounds from the OS, computed once per
thread and cached — glibc/musl `pthread_getattr_np` + `pthread_attr_getstack`, macOS
`pthread_get_stackaddr_np` + `pthread_get_stacksize_np`, Win32
`GetCurrentThreadStackLimits` (a hand-declared `kernel32` extern; no `windows-sys`).
`exhausted()` is one thread-local load and one compare: true once the stack pointer is
within a 256 KiB red zone of the limit, and it latches a flag. Where the OS can't report
bounds it falls back to a fixed 1 MiB descent budget measured from the entry stack
pointer — safe on anything from Node's 4 MiB worker default up. So the guard is exact on
the 4 MiB worker, the 8 MiB main thread, and any `resourceLimits.stackSizeMb` alike.
- `stack_guard!()` (defined in `lib.rs`) is the first statement of every recursive walker
function — all **150** self-recursive or on-cycle functions across the 15 walker modules,
found by script (every cycle in the call graph, not just direct self-calls). It returns
`Default::default()` (`()`, `false`, `None`, `""`) so an exhausted walk simply stops
descending; a hook returning `false` sends its caller down the generic child walk, whose
own guard returns at once.
- `extract_file` runs the whole walk under `stack::run_guarded`: if the flag is set
afterwards the (truncated) result is discarded and replaced by
`defer: nesting too deep for the native walker — wasm recovery handles it`.
- `parse-pool.ts`: a comment at `new Worker(scriptPath)` records why there is deliberately
no `resourceLimits.stackSizeMb` bump.
- No new crates beyond `libc` as a direct unix dependency (already in `Cargo.lock`
transitively). No wire/ABI change.
Net effect for the reporter's repo: `deep.c` goes to the wasm path, lands as
`function foo` plus a recorded parse warning, and the other 31,607 files index normally.
`CODEGRAPH_KERNEL=0` and the `exclude` workaround are no longer needed.
## Tests
**Rust unit tests** (`cargo test`, 21 passed — 7 new in `stack.rs`): the walkers for
C, C++, Rust, TypeScript and Python are driven on a **1 MiB** thread (a quarter of Node's
worker default) with 30k-deep nesting and must return `defer:` instead of crashing;
shallow files are untouched; the latch resets between runs; the OS bounds are sane on the
main thread and describe a small thread's own stack.
**`__tests__/kernel-deep-nesting.test.ts`** (new, 8 tests — skips without a staged `.node`,
fails under `CODEGRAPH_KERNEL_EXPECT=1` if the kernel is missing, like the other kernel
suites):
- every default-routed language (all 20) survives a 60k-deep expression on the main thread
— clean result or the wasm fallback's partial result, never a crash;
- the reporter's exact 16,384-brace C file is indexed (partial) on the main thread;
- 200-deep expressions in every language still take the kernel path clean (the guard never
trips on normal code);
- inside a **default-sized 4 MiB `worker_threads` Worker** through `dist/`: the reporter's
`deep.c` and a 60k-deep expression in every language come back `deferred` with exit 0,
and a normal file still extracts natively;
- end-to-end through the built CLI: `codegraph init` on a repo holding `deep.c` + `ok.c`
exits 0 and records both files, with both functions.
**Existing kernel suites**: all 15 (`kernel-*-parity`, `kernel-scaffold`,
`kernel-retry-materialize`, `kernel-grammar-parity`) pass unchanged, 147 tests — the guard
never fires on the parity fixtures.
**Reporter's probes** (`one.js` from the issue, default 4 MiB worker, this build):
`deep.c` → `deferred`, exitCode=0 (was rc=132/139); `deep100k.c` → `deferred`, exitCode=0.
Main thread: `deep.c` / `deep100k.c` → wasm partial with
`Parse error: Maximum call stack size exceeded`; a 6,000-term binary expression and a
3,000-branch `else if` chain stay on the kernel path with clean results.
**Perf** (same `dist/`, only the `.node` swapped via `CODEGRAPH_KERNEL_PATH`; interleaved
main/new ×3, `codegraph init`, macOS arm64):
| repo | main (median) | guarded (median) | nodes / edges |
|---|---|---|---|
| express (141 files) | 0.60 s (0.58–0.65) | 0.61 s (0.58–0.61) | 1,084 / identical |
| redis (786 C/H files) | 4.44 s (4.39–4.66) | 4.49 s (4.41–4.70) | 19,942 / 76,446 identical |
Within run-to-run noise, as expected for one TLS load + compare per recursion entry.
**Linux (Docker, `node:22-bookworm`, kernel built in-container, `docker run --rm --init`)** —
the reporter's platform and the glibc `pthread_getattr_np` bounds path:
```
=== platform ===
Linux efe3cc86947b 6.12.54-linuxkit #1 SMP Tue Nov 4 21:21:47 UTC 2025 aarch64 GNU/Linux
v22.22.3
-rwxr-xr-x 1 root root 35332288 Aug 22 18:02 codegraph-kernel/prebuilds/linux-arm64/codegraph-kernel.node
=== reporter repro (issue #1581): 16,384-brace deep.c, codegraph init ===
│
└ Done
init exit code: 0
file: deep.c
file: deep100k.c
file: ok.c
function: add
function: bar
function: foo
=== worker probe: kernel raw extract in a default 4 MiB worker ===
deep.c: deferred
deep.c: worker exitCode=0
deep100k.c: deferred
deep100k.c: worker exitCode=0
ok.c: kernel nodes=2
ok.c: worker exitCode=0
=== cargo test stack:: (glibc pthread_getattr_np bounds path) ===
test stack::tests::os_bounds_are_sane_on_this_platform ... ok
test stack::tests::small_stack_reports_its_own_bounds ... ok
test stack::tests::normal_files_are_untouched_by_the_guard ... ok
test stack::tests::deep_braces_c_defer_instead_of_crashing ... ok
test stack::tests::latch_resets_between_runs ... ok
test stack::tests::deep_parens_cpp_rust_ts_python_defer_instead_of_crashing ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 15 filtered out; finished in 0.23s
=== vitest: kernel-deep-nesting + kernel-scaffold ===
✓ __tests__/kernel-scaffold.test.ts (10 tests) 30ms
✓ __tests__/kernel-deep-nesting.test.ts (8 tests) 36989ms
Test Files 2 passed (2)
Tests 18 passed (18)
```
(The pre-fix crash was reproduced on macOS — rc=132 in a default worker, rc=139 on the main thread at 100k depth — not re-run inside this container; the reporter's Linux x86_64 trace is the SIGSEGV form of the same overflow.)
**Windows (Parallels ARM64 VM, MSVC 14.44, `cargo 1.97`, kernel built on the VM,
`GetCurrentThreadStackLimits` path)**:
```
head:
|
||
|
ctype_lab
|
8e3cde6994 | feat(kernel): preserve union nodes across extraction | ||
|
ctype_lab
|
85e9ac6762 |
fix(c,cpp,objc,rust): index union declarations
A `union` declaration produced no symbol at all in any of the four
languages that have one. The type never entered the graph, and neither
did anything attached to it — in Rust every `impl Trait for MyUnion`
lost its edge, and the impl's methods were left with a qualifiedName
pointing at a type the graph did not contain.
`union_specifier` / `union_item` were absent from the extraction layer
entirely: no `<x>Types` list on the TS side, no dispatch branch in
either kernel walker.
They join `structTypes` (kind `struct` — NodeKind has no `union`), which
is the extension point the table-driven extractors already provide. The
body guard in extractStruct is untouched, so a bodiless `union U;` stays
a forward declaration and is still skipped, exactly like `struct U;`.
`resolveTypeAliasKind` accepts `union_specifier` too, so
`typedef union { … } N;` takes the typedef's name the way
`typedef struct { … } N;` already did. Without it the anonymous union
body would mint a second `<anonymous>` node beside the alias.
Both walkers change together so kernel<->wasm parity holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Colby Mchenry
|
2d72891b59 |
feat(kernel): R7a C/C++ walker — dual-lang ccpp module, preParse hoist, 7 new blanks, c/cpp default-routed (#1346)
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps byte-identical on all five + linux at kernel scale (10.4M dump lines, same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x). Deferral guard corrected by measurement (C/C++ error incidence 9-42%; --max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse cost deferred files paid. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |