mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-16 20:53:57 +08:00
fix(cpp): a method a macro of its file defines, like protobuf's PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int), is indexed where the macro is invoked (#2605)
protobuf declares its descriptor accessors in their classes and defines
them below with macros, like `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor,
number, int)`; its reflection pastes `Reflection::Get##TYPENAME` in
another. The parser never sees those definitions (the preParse blanks the
invocations) and a declaration mints no node, so the methods had none and
every call to them linked to nothing.
- After either extractor (the kernel or the wasm walker), a step over the
original source expands the file's own function-like macros whose body
defines `Class::method(...) ... {`, at each invocation at namespace
scope, and mints a method node per definition: named and qualified like
an out-of-line definition, contained by the file and the same-file
class, with the expanded head as its signature. Constructors,
destructors and variadic macros are left out.
- The parse worker leaves the raw kernel fast path only for a file that
may define one (a cheap text test), so the step sees it.
- The C++ resolver reads such a method's return type and parameters from
its signature, and never takes it for a deleted overload.
A/B vs main on the kernel path, every changed site triaged: leveldb and
redis identical; fmt +13 nodes and rocksdb +11 (gtest's TestEventRepeater
overrides); protobuf +257 nodes, 6,203 calls and 2 references added, 252
calls retargeted, 11 metadata-only, none removed. Kernel and wasm dumps
are byte-identical, protobuf's one pre-existing Swift line aside.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
@@ -99,6 +99,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- In C++, a call a method makes with no receiver, or through `this->`, now links to the method C++ itself would call: one of the method's own class, else one the class inherits from a base class, else one of a class it is nested in. The same goes for a call in a lambda inside the method, and of a class's overloads the one the call's arguments fit is taken. Before, the call was matched by its name alone, so another class's method of that name could win because it was declared nearer or found first: in protocolbuffers/protobuf, every generated message's `Swap` and move assignment calling `InternalSwap(…)` reached `Any::InternalSwap`, and the `GetArena()` messages inherit from `MessageLite` reached an arena helper's `GetArena`, while googletest's `~linked_ptr() { depart(); }`, vendored in facebook/rocksdb, reached `linked_ptr_internal::depart`. Calls C++ doesn't look up in the class are matched as before: a free function's, a call to a parameter or local of that name, and a template's bare call into a base class that depends on its template parameters. Re-index C++ projects after upgrading. Thanks @danusha2345 for reporting the C++ case in #1952.
|
||||
- In C++, a call on the variable of a range-based `for` loop over `auto` elements, like `cfd->Ref()` inside facebook/rocksdb's `for (auto cfd : cfds)`, now links to the method of the type the loop goes over. That type is read from what the loop iterates: a `std::vector`, `std::set`, `absl::Span` or similar container, protocolbuffers/protobuf's `RepeatedPtrField` or RocksDB's `autovector`, whether a variable, parameter or member holds it, an `auto` variable is set to it or a function or method returns it, and a project class you loop over through its `begin()`, like RocksDB's `ColumnFamilySet`. Before, the call was matched by the variable's name, or by an earlier variable of that name: in RocksDB, `blob_file.GetBlobFileNumber()` over a list of `BlobFileGarbage` reached `BlobFileAddition::GetBlobFileNumber`, and in protobuf's Objective-C generator, calls over a list of `std::unique_ptr<MessageGenerator>` reached other generators' methods. A call on an element of a standard-library type, like `file.find(…)` over a `std::vector<std::string>`, no longer links to a project method, and `->` on an element held in a `std::unique_ptr` or `std::shared_ptr` reaches the type it holds. Re-index C++ projects after upgrading.
|
||||
- In C++, a call on a variable, parameter or member declared as one of your classes, like `iter->Valid()` on an `Iterator* iter` or `it.Valid()` on a `const Iterator& it`, now links to that method of the class C++ itself would find for the type where the call is written, or of a class it inherits from. Before, the method was picked by the type's last name, so a class of the same name nested elsewhere could win: in facebook/rocksdb, calls on an `Iterator` reached `MemTableRep::Iterator`, which declares `Valid` and `Next` itself while the `Iterator` the code means inherits them, google/leveldb's reached `SkipList::Iterator`, and in protocolbuffers/protobuf the Java generator's `context->options()` reached the hpb generator's `Context` and `text_parser.Parse(…)` on a `TextFormat::Parser` reached the `.proto` parser. When that class and the classes it derives from have no method of that name, the call no longer links to an unrelated class's method that shares it, as protobuf's `field->full_name()` on a `FieldDescriptor*` did. A type is also found through a `using` declaration the file writes, like protobuf's `using google::protobuf::Descriptor;`, before a class of the same name declared at global scope elsewhere in the repository; this also applies to base classes and to the type a smart pointer holds. A type written from the global scope, like `::upb::Arena& arena`, is looked up there, so a namespace's own `upb::Arena` is not taken for it, nor is an alias of that name the calling function declares. A type written as a member of a class template, like `SkipList<Key, Comparator>::Iterator iter` in leveldb's and RocksDB's skip-list tests, is now read whole, so its calls reach the skip list's own iterator rather than the general `Iterator`, and RocksDB's skip-list memtable iterator no longer links its calls on the skip list's iterator to its own methods; a standard container's `iterator` written this way is the library's, never a class of yours named `iterator`. Re-index C++ projects after upgrading.
|
||||
- In C++, a method defined by a macro of the same file is now indexed where the file invokes the macro, so calls to it link. protocolbuffers/protobuf's descriptor classes only declare most of their accessors and define them further down with macros like `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)`, and its reflection defines `GetInt32`, `SetInt32` and the like by pasting names together in a macro. Those methods had no node at all, so calls like `field->number()`, `descriptor->full_name()` or `reflection->GetInt32(…)` linked to nothing. Calls through what such a method returns, like `field->containing_type()->full_name()`, now link too. A macro defined in another file, and a method declared but defined nowhere in the repository, are still not indexed. Re-index C++ projects after upgrading.
|
||||
- In C++, text inside a string literal is no longer taken for a variable's declaration. That covers the C code protocolbuffers/protobuf's upb generator writes in a raw string (`R"cc( … upb_MiniTableField field = $field_init$; … )cc"`) and log messages like facebook/rocksdb's `"… in WriteBatch handler; …"`. Before, a later call like `field.ctype()` on the function's own `upb::FieldDefPtr field` parameter, or `handler->PutCF(…)` on a `WriteBatch::Handler* handler`, was matched against the type that text seemed to name, so it linked to nothing or to another class's method. Re-index C++ projects after upgrading.
|
||||
- In C++, a call on an `auto` variable now links to the method of the class C++ finds for the type its initializer gives it, as a call on a variable declared with that type does. The type is read from a cast or smart pointer written out, like facebook/rocksdb's `auto iter = std::unique_ptr<Iterator>(…)` or `auto* mutex = static_cast<InstrumentedMutex*>(arg)`, from `new` or `std::make_unique` of a class, and from what a function or method is declared to return, like the `Iterator*` of `txn->GetIterator(…)`, read to the end of an initializer that goes on to the next line. Before, such a variable took the type of an earlier variable of the same name, or none: in RocksDB, `iter->SeekToFirst()` reached `MemTableRep::Iterator` and `mutex->Lock()` reached `port::Mutex`. A chained call like `cfd->current()->storage_info()` now looks up the class the inner call is declared to return the same way, through a smart pointer too, so protocolbuffers/protobuf's Java generator no longer reaches the C# generator's `EnumGenerator`. A type from outside the project no longer links to a class of the project that shares its name: protobuf's `status.ok()` on an `absl::Status` reached `upb::Status::ok`, and fmtlib/fmt's `os.flush()` on a `std::ostream&` reached fmt's own `ostream::flush`. The same holds for a variable declared in a macro body, or typed by what a function in another file returns, and for a class the index doesn't hold whose name only another class shares: protobuf's `_status.ok()` in its `RETURN_IF_ERROR` macro reached `upb::Status::ok`, and `range.options()` on its generated `DescriptorProto::ExtensionRange` reached `Descriptor::ExtensionRange::options`. A namespace a macro stands for is the project's under either name: facebook/rocksdb declares its classes in `namespace ROCKSDB_NAMESPACE` with `#define ROCKSDB_NAMESPACE rocksdb`, so `db->Put(…)` on a `rocksdb::DB*` links to RocksDB's own `DB::Put`. Re-index C++ projects after upgrading.
|
||||
- In C++, a call on a data member of the calling method's class, like `versions_->GetColumnFamilySet()` in facebook/rocksdb's `db_impl_write.cc`, now links to the method of the member's type wherever the class is declared. Before, only the call's own file and the header of the same name were read for the member, and RocksDB declares `DBImpl` in `db_impl.h` while its methods are spread over `db_impl_write.cc`, `db_impl_compaction_flush.cc`, `db_impl_open.cc` and more, so such calls were guessed by the member's name. A member declared below an inline method that uses it, one a base class declares, and, in a gtest `TEST_F` or `TEST_P` body, one of the test's fixture, like RocksDB's `db_`, are found the same way, and a member of a class nested in the caller's class is no longer taken for the caller's own. A call on a member of a standard-library type, like `std::atomic<bool> shutting_down_`, no longer links to a project method that only shares the method's name. Re-index C++ projects after upgrading.
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
/**
|
||||
* A C++ method a function-like macro defines, like protobuf's
|
||||
* `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)` expanding to
|
||||
* `inline int FieldDescriptor::number() const { … }`, is a method node where
|
||||
* the file invokes the macro. Its class only declares it (`int number()
|
||||
* const;`), which mints no node, so it had none at all and no call to it
|
||||
* linked.
|
||||
*/
|
||||
import { describe, it, expect, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { CodeGraph } from '../src';
|
||||
import { extractFromSource } from '../src/extraction/tree-sitter';
|
||||
import type { Node } from '../src/types';
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function indexed(files: Record<string, string>): Promise<CodeGraph> {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-macro-methods-'));
|
||||
roots.push(root);
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true });
|
||||
fs.writeFileSync(path.join(root, rel), content);
|
||||
}
|
||||
return CodeGraph.init(root, { index: true });
|
||||
}
|
||||
|
||||
function callable(cg: CodeGraph, qualifiedName: string): Node {
|
||||
const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName);
|
||||
if (!node) throw new Error(`no function or method ${qualifiedName}`);
|
||||
return node;
|
||||
}
|
||||
|
||||
/** `calls` callees of a function or method, as `qualifiedName@file:line`. */
|
||||
function calls(cg: CodeGraph, caller: string): string[] {
|
||||
return cg
|
||||
.getCallees(callable(cg, caller).id)
|
||||
.filter((r) => r.edge.kind === 'calls')
|
||||
.map((r) => `${r.node.qualifiedName}@${r.node.filePath}:${r.node.startLine}`)
|
||||
.sort();
|
||||
}
|
||||
|
||||
/** The methods of a file, as `qualifiedName:line`. */
|
||||
function methodsIn(cg: CodeGraph, file: string): string[] {
|
||||
return cg.getNodesByKind('method').filter((n) => n.filePath === file).map((n) => `${n.qualifiedName}:${n.startLine}`).sort();
|
||||
}
|
||||
|
||||
const lines = (...l: string[]): string => [...l, ''].join('\n');
|
||||
|
||||
/** protobuf's descriptor.h, in small: accessors declared in the class, defined by macros below it. */
|
||||
const DESCRIPTOR_H = lines(
|
||||
'namespace google {',
|
||||
'namespace protobuf {',
|
||||
'class FieldDescriptor;',
|
||||
'class Descriptor {',
|
||||
' public:',
|
||||
' absl::string_view name() const;',
|
||||
' absl::string_view full_name() const;',
|
||||
' int field_count() const;',
|
||||
' const FieldDescriptor* field(int index) const;',
|
||||
' const Descriptor* containing_type() const;',
|
||||
' private:',
|
||||
' int field_count_;',
|
||||
' const FieldDescriptor* fields_;',
|
||||
' const Descriptor* containing_type_;',
|
||||
'};',
|
||||
'class FieldDescriptor {',
|
||||
' public:',
|
||||
' int number() const;',
|
||||
' const Descriptor* containing_type() const;',
|
||||
' private:',
|
||||
' int number_;',
|
||||
' const Descriptor* containing_type_;',
|
||||
'};',
|
||||
'',
|
||||
'#define PROTOBUF_DEFINE_ACCESSOR(CLASS, FIELD, TYPE) \\',
|
||||
' inline TYPE CLASS::FIELD() const { return FIELD##_; }',
|
||||
'',
|
||||
'// Name and full name are stored in a single array to save space.',
|
||||
'#define PROTOBUF_DEFINE_NAME_ACCESSOR(CLASS) \\',
|
||||
' inline absl::string_view CLASS::name() const { return all_names_.name(); } \\',
|
||||
' inline absl::string_view CLASS::full_name() const { \\',
|
||||
' return all_names_.full_name(); \\',
|
||||
' }',
|
||||
'',
|
||||
'#define PROTOBUF_DEFINE_ARRAY_ACCESSOR(CLASS, FIELD, TYPE) \\',
|
||||
' inline TYPE CLASS::FIELD(int index) const { \\',
|
||||
' return FIELD##s_ + index; \\',
|
||||
' }',
|
||||
'',
|
||||
'PROTOBUF_DEFINE_NAME_ACCESSOR(Descriptor)',
|
||||
'PROTOBUF_DEFINE_ACCESSOR(Descriptor, field_count, int)',
|
||||
'PROTOBUF_DEFINE_ARRAY_ACCESSOR(Descriptor, field, const FieldDescriptor*)',
|
||||
'PROTOBUF_DEFINE_ACCESSOR(Descriptor, containing_type,',
|
||||
' const Descriptor*)',
|
||||
'PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, int)',
|
||||
'PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, containing_type, const Descriptor*)',
|
||||
'',
|
||||
'#undef PROTOBUF_DEFINE_ACCESSOR',
|
||||
'} // namespace protobuf',
|
||||
'} // namespace google',
|
||||
);
|
||||
|
||||
describe('a C++ method a macro defines', () => {
|
||||
it('reproduction: protobuf descriptor accessors, and calls through what they return', async () => {
|
||||
const cg = await indexed({
|
||||
// A namesake of the class an accessor returns: the type is read as written.
|
||||
'src/legacy.h': lines(
|
||||
'namespace legacy {',
|
||||
'class Descriptor {',
|
||||
' public:',
|
||||
' absl::string_view full_name() const { return ""; }',
|
||||
'};',
|
||||
'} // namespace legacy',
|
||||
),
|
||||
'src/descriptor.h': DESCRIPTOR_H,
|
||||
'src/printer.cc': lines(
|
||||
'#include "src/legacy.h"',
|
||||
'#include "src/descriptor.h"',
|
||||
'namespace google {',
|
||||
'namespace protobuf {',
|
||||
'int Number(const FieldDescriptor* field) { return field->number(); }',
|
||||
'int Count(const Descriptor& d) { return d.field_count(); }',
|
||||
'absl::string_view Owner(const FieldDescriptor* field) { return field->containing_type()->full_name(); }',
|
||||
'int First(const Descriptor* d) { return d->field(0)->number(); }',
|
||||
'absl::string_view Parent(const FieldDescriptor* field) {',
|
||||
' auto parent = field->containing_type();',
|
||||
' return parent->full_name();',
|
||||
'}',
|
||||
'} // namespace protobuf',
|
||||
'} // namespace google',
|
||||
),
|
||||
});
|
||||
try {
|
||||
expect(methodsIn(cg, 'src/descriptor.h')).toEqual([
|
||||
'google::protobuf::Descriptor::containing_type:43',
|
||||
'google::protobuf::Descriptor::field:42',
|
||||
'google::protobuf::Descriptor::field_count:41',
|
||||
'google::protobuf::Descriptor::full_name:40',
|
||||
'google::protobuf::Descriptor::name:40',
|
||||
'google::protobuf::FieldDescriptor::containing_type:46',
|
||||
'google::protobuf::FieldDescriptor::number:45',
|
||||
]);
|
||||
expect(calls(cg, 'google::protobuf::Number')).toEqual(['google::protobuf::FieldDescriptor::number@src/descriptor.h:45']);
|
||||
expect(calls(cg, 'google::protobuf::Count')).toEqual(['google::protobuf::Descriptor::field_count@src/descriptor.h:41']);
|
||||
expect(calls(cg, 'google::protobuf::Owner')).toEqual([
|
||||
'google::protobuf::Descriptor::full_name@src/descriptor.h:40',
|
||||
'google::protobuf::FieldDescriptor::containing_type@src/descriptor.h:46',
|
||||
]);
|
||||
expect(calls(cg, 'google::protobuf::First')).toEqual([
|
||||
'google::protobuf::Descriptor::field@src/descriptor.h:42',
|
||||
'google::protobuf::FieldDescriptor::number@src/descriptor.h:45',
|
||||
]);
|
||||
expect(calls(cg, 'google::protobuf::Parent')).toEqual([
|
||||
'google::protobuf::Descriptor::full_name@src/descriptor.h:40',
|
||||
'google::protobuf::FieldDescriptor::containing_type@src/descriptor.h:46',
|
||||
]);
|
||||
const number = callable(cg, 'google::protobuf::FieldDescriptor::number');
|
||||
expect(number.signature).toBe('inline int FieldDescriptor::number() const');
|
||||
expect(number.returnType).toBeUndefined();
|
||||
expect(callable(cg, 'google::protobuf::FieldDescriptor::containing_type').returnType).toBe('Descriptor');
|
||||
const holders = cg.getIncomingEdgesTo([number.id], ['contains']);
|
||||
expect(holders.map((e) => cg.getNode(e.source)?.qualifiedName).sort()).toEqual(['google::protobuf::FieldDescriptor', 'src/descriptor.h']);
|
||||
} finally {
|
||||
cg.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('a macro of a source file that pastes the method name, for a class its header declares', async () => {
|
||||
const cg = await indexed({
|
||||
'src/reflection.h': lines(
|
||||
'class Reflection {',
|
||||
' public:',
|
||||
' int GetInt32(int index) const;',
|
||||
' void SetInt32(int index, int value) const;',
|
||||
' bool GetBool(int index) const;',
|
||||
' void SetBool(int index, bool value) const;',
|
||||
'};',
|
||||
),
|
||||
'src/reflection.cc': lines(
|
||||
'#include "src/reflection.h"',
|
||||
'#define DEFINE_PRIMITIVE_ACCESSORS(TYPENAME, TYPE) \\',
|
||||
' TYPE Reflection::Get##TYPENAME(int index) const { \\',
|
||||
' return TYPE(); \\',
|
||||
' } \\',
|
||||
' void Reflection::Set##TYPENAME(int index, TYPE value) const {}',
|
||||
'',
|
||||
'DEFINE_PRIMITIVE_ACCESSORS(Int32, int)',
|
||||
'DEFINE_PRIMITIVE_ACCESSORS(Bool, bool)',
|
||||
'#undef DEFINE_PRIMITIVE_ACCESSORS',
|
||||
'namespace {',
|
||||
'class Local {',
|
||||
' public:',
|
||||
' int Get() const;',
|
||||
'};',
|
||||
'#define DEFINE_LOCAL_GET(C) inline int C::Get() const { return 1; }',
|
||||
'DEFINE_LOCAL_GET(Local)',
|
||||
'} // namespace',
|
||||
),
|
||||
'src/use.cc': lines(
|
||||
'#include "src/reflection.h"',
|
||||
'int Read(const Reflection& r) { r.SetBool(0, true); return r.GetInt32(1); }',
|
||||
),
|
||||
});
|
||||
try {
|
||||
// An anonymous namespace qualifies nothing, as for a definition written out.
|
||||
expect(methodsIn(cg, 'src/reflection.cc')).toEqual([
|
||||
'Local::Get:17',
|
||||
'Reflection::GetBool:9',
|
||||
'Reflection::GetInt32:8',
|
||||
'Reflection::SetBool:9',
|
||||
'Reflection::SetInt32:8',
|
||||
]);
|
||||
expect(calls(cg, 'Read')).toEqual(['Reflection::GetInt32@src/reflection.cc:8', 'Reflection::SetBool@src/reflection.cc:9']);
|
||||
} finally {
|
||||
cg.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('a macro-defined overload is picked by the parameters its expansion writes, and is never a deleted one', async () => {
|
||||
const cg = await indexed({
|
||||
'src/getter.h': lines(
|
||||
'class Getter {',
|
||||
' public:',
|
||||
' int Get(int index) const;',
|
||||
' int Get(const char* key) const;',
|
||||
'};',
|
||||
'#define DEFINE_GET(PARAM) \\',
|
||||
' inline int Getter::Get(PARAM) const { return 0; }',
|
||||
'DEFINE_GET(int index)',
|
||||
'// Between two invocations, another class deletes a method of the same name.',
|
||||
'class Blocked {',
|
||||
' public:',
|
||||
' int Get(double value) = delete;',
|
||||
'};',
|
||||
'DEFINE_GET(const char* key)',
|
||||
),
|
||||
'src/use.cc': lines(
|
||||
'#include "src/getter.h"',
|
||||
'int ByKey(const Getter& g) { return g.Get("key"); }',
|
||||
'int ByIndex(const Getter& g) { return g.Get(1); }',
|
||||
),
|
||||
});
|
||||
try {
|
||||
expect(calls(cg, 'ByKey')).toEqual(['Getter::Get@src/getter.h:14']);
|
||||
expect(calls(cg, 'ByIndex')).toEqual(['Getter::Get@src/getter.h:8']);
|
||||
} finally {
|
||||
cg.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('an explicit specialization a macro defines is named without its template arguments', async () => {
|
||||
const cg = await indexed({
|
||||
'src/wire.h': lines(
|
||||
'class Wire {',
|
||||
' public:',
|
||||
' template <typename T, int Kind> static bool Read(T* value);',
|
||||
'};',
|
||||
'#define READ_PRIMITIVE(CPPTYPE, KIND) \\',
|
||||
' template <> inline bool Wire::Read<CPPTYPE, KIND>(CPPTYPE* value) { return true; }',
|
||||
'READ_PRIMITIVE(int, 1)',
|
||||
'READ_PRIMITIVE(double, 2)',
|
||||
),
|
||||
'src/use.cc': lines(
|
||||
'#include "src/wire.h"',
|
||||
'bool Use(int* value) { return Wire::Read<int, 1>(value); }',
|
||||
),
|
||||
});
|
||||
try {
|
||||
expect(methodsIn(cg, 'src/wire.h')).toEqual(['Wire::Read:7', 'Wire::Read:8']);
|
||||
expect(calls(cg, 'Use')).toHaveLength(1);
|
||||
expect(calls(cg, 'Use')[0]).toMatch(/^Wire::Read@src\/wire\.h:[78]$/);
|
||||
} finally {
|
||||
cg.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('what is not a method a macro defines', async () => {
|
||||
const cg = await indexed({
|
||||
'src/other.h': lines(
|
||||
'class Widget {',
|
||||
' public:',
|
||||
' Widget();',
|
||||
' void Draw() const;',
|
||||
' void Paint() const;',
|
||||
' int Size() const;',
|
||||
' static void Log(int x);',
|
||||
'};',
|
||||
'/*',
|
||||
'#define COMMENTED(C) inline void C::Draw() const {}',
|
||||
'*/',
|
||||
'#define DEFINE_DRAW(C) inline void C::Draw() const {}',
|
||||
'#define DEFINE_CTOR(C) inline C::C() {}',
|
||||
'#define DEFINE_FREE(NAME) inline void NAME##_fn() {}',
|
||||
'#define CALL_LOG(x) Widget::Log(x);',
|
||||
'#define DECLARE_SIZE(C) int C::Size() const;',
|
||||
'COMMENTED(Widget)',
|
||||
'DEFINE_CTOR(Widget)',
|
||||
'DEFINE_FREE(widget)',
|
||||
'DECLARE_SIZE(Widget)',
|
||||
'inline void Use() {',
|
||||
' DEFINE_DRAW(Widget)',
|
||||
' CALL_LOG(1)',
|
||||
'}',
|
||||
'struct Holder {',
|
||||
' DEFINE_DRAW(Widget)',
|
||||
'};',
|
||||
'#undef DEFINE_DRAW',
|
||||
'DEFINE_DRAW(Widget)',
|
||||
'#define DEFINE_DRAW(C) inline void C::Paint() const {}',
|
||||
'DEFINE_DRAW(Widget)',
|
||||
),
|
||||
});
|
||||
try {
|
||||
// Only the invocation of the redefined macro, at namespace scope, defines one.
|
||||
expect(methodsIn(cg, 'src/other.h')).toEqual(['Widget::Paint:31', 'Widget::Widget:3']);
|
||||
} finally {
|
||||
cg.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('is minted alike whether the kernel or the wasm extractor reads the file', () => {
|
||||
const minted = (kernel: boolean): string[] => {
|
||||
const saved = process.env.CODEGRAPH_KERNEL;
|
||||
if (kernel) delete process.env.CODEGRAPH_KERNEL;
|
||||
else process.env.CODEGRAPH_KERNEL = '0';
|
||||
try {
|
||||
const result = extractFromSource('src/descriptor.h', DESCRIPTOR_H, 'cpp');
|
||||
const byId = new Map(result.nodes.map((n) => [n.id, n]));
|
||||
return result.nodes
|
||||
.filter((n) => n.kind === 'method' && n.signature?.includes('::'))
|
||||
.map((n) => {
|
||||
const holders = result.edges.filter((e) => e.kind === 'contains' && e.target === n.id).map((e) => byId.get(e.source)?.qualifiedName);
|
||||
return `${n.id} ${n.qualifiedName} ${n.startLine}:${n.startColumn}-${n.endLine}:${n.endColumn} ${n.returnType ?? '-'} ${n.signature} <- ${holders.sort().join(',')}`;
|
||||
})
|
||||
.sort();
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.CODEGRAPH_KERNEL;
|
||||
else process.env.CODEGRAPH_KERNEL = saved;
|
||||
}
|
||||
};
|
||||
const wasm = minted(false);
|
||||
expect(wasm).toHaveLength(7);
|
||||
expect(minted(true)).toEqual(wasm);
|
||||
});
|
||||
});
|
||||
@@ -81,6 +81,11 @@ export interface CppBraceScopes {
|
||||
closeOf(open: number): number | undefined;
|
||||
/** The named namespaces whose braces hold `offset`, outermost first, as written (`a::b`). */
|
||||
namespacesAt(offset: number): string[];
|
||||
/**
|
||||
* Whether `offset` is at namespace scope: every brace around it opens a
|
||||
* namespace, named or not, and none a class, a function or a linkage block.
|
||||
*/
|
||||
atNamespaceScope(offset: number): boolean;
|
||||
/** The 1-based line and the column of `offset`. */
|
||||
positionOf(offset: number): { line: number; column: number };
|
||||
}
|
||||
@@ -204,6 +209,7 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null {
|
||||
// first branch (null until a second branch starts).
|
||||
const branches: Array<{ atIf: number[]; afterFirst: number[] | null }> = [];
|
||||
const namespaceAt = new Map<number, string>(); // `{` offset → name, for named namespaces
|
||||
const namespaceBraces = new Set<number>(); // `{` offsets of every namespace, named or not
|
||||
let pendingNamespace: { name: string; brace: number } | null = null;
|
||||
let balanced = true;
|
||||
let lineStart = true;
|
||||
@@ -322,6 +328,7 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null {
|
||||
open.push(i);
|
||||
if (pendingNamespace && pendingNamespace.brace === i) {
|
||||
if (pendingNamespace.name) namespaceAt.set(i, pendingNamespace.name);
|
||||
namespaceBraces.add(i);
|
||||
pendingNamespace = null;
|
||||
}
|
||||
} else if (c === 125) {
|
||||
@@ -339,9 +346,17 @@ export function scanCppBraceScopes(source: string): CppBraceScopes | null {
|
||||
if (close !== undefined) namespaces.add(brace, close, name);
|
||||
}
|
||||
let lineStarts: number[] | null = null;
|
||||
let scopes: NestedIntervals<boolean> | null = null;
|
||||
return {
|
||||
closeOf: (at) => closes.get(at),
|
||||
namespacesAt: (offset) => namespaces.at(offset),
|
||||
atNamespaceScope: (offset) => {
|
||||
if (!scopes) {
|
||||
scopes = new NestedIntervals<boolean>();
|
||||
for (const [brace, close] of [...closes].sort((a, b) => a[0] - b[0])) scopes.add(brace, close, namespaceBraces.has(brace));
|
||||
}
|
||||
return scopes.at(offset).every((isNamespace) => isNamespace);
|
||||
},
|
||||
positionOf: (offset) => {
|
||||
if (!lineStarts) {
|
||||
lineStarts = [0];
|
||||
|
||||
@@ -0,0 +1,571 @@
|
||||
/**
|
||||
* C++ methods a function-like macro defines, minted where the file invokes
|
||||
* the macro.
|
||||
*
|
||||
* protocolbuffers/protobuf defines most of its descriptor accessors this way.
|
||||
* `FieldDescriptor` declares `int number() const;`, and descriptor.h defines
|
||||
* it further down with `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number,
|
||||
* int)`, the macro being `inline TYPE CLASS::FIELD() const { return FIELD##_; }`.
|
||||
* The parser never sees that definition (the preParse blanks the invocation,
|
||||
* an unknown statement without a semicolon), and a declaration without a body
|
||||
* mints no node, so `FieldDescriptor::number` had no node at all and none of
|
||||
* its callers linked.
|
||||
*
|
||||
* This reads the file's own `#define`s whose body defines `Class::method(…) … {`
|
||||
* at its top level, expands each invocation at namespace scope (its
|
||||
* parameters, and `##` pasting), and mints a method node per definition, as the
|
||||
* walker mints an out-of-line definition: named and qualified like one (the
|
||||
* enclosing namespaces composed with the written receiver), and contained by
|
||||
* the file and by the same-file class of the receiver's name. The node spans
|
||||
* the invocation. Its signature is the expanded head (`inline const
|
||||
* FileDescriptor* Descriptor::file() const`), which the resolver reads where it
|
||||
* would read a definition's source: the return type a chained call goes on
|
||||
* through, and the parameters an overload is picked by. The expanded body is
|
||||
* not read, so the method has no calls of its own.
|
||||
*
|
||||
* It runs on the original source after either extractor (the kernel or the
|
||||
* wasm walker), so both mint the same nodes. Only macros the file defines
|
||||
* itself are expanded: an extraction sees one file. Constructors and
|
||||
* destructors are left out: constructor resolution reads a constructor's
|
||||
* parameter list as its signature, and a destructor is not called by name.
|
||||
*/
|
||||
import { Edge, ExtractionResult, Node } from '../../types';
|
||||
import { generateNodeId } from '../tree-sitter-helpers';
|
||||
import { normalizeCppReturnType, stripCppTemplateArgs } from './c-cpp';
|
||||
import { scanCppBraceScopes } from './cpp-brace-scopes';
|
||||
|
||||
/** A function-like macro a file defines (not a variadic one: none defines a method in practice). */
|
||||
interface CppMacro {
|
||||
name: string;
|
||||
params: string[];
|
||||
/** Its replacement text: continuations spliced, comments blanked. */
|
||||
body: string;
|
||||
/** Where its `#` is. */
|
||||
at: number;
|
||||
}
|
||||
|
||||
/** A method definition at the top level of a macro's expansion. */
|
||||
interface CppDefinitionHead {
|
||||
/** The receiver as written, template arguments dropped (`Descriptor`, `internal::ExtensionSet`). */
|
||||
receiver: string;
|
||||
name: string;
|
||||
/** The text before the receiver: specifiers and the return type (`inline const FileDescriptor*`). */
|
||||
returnText: string;
|
||||
/** Everything up to the body's `{`, whitespace collapsed. */
|
||||
head: string;
|
||||
}
|
||||
|
||||
const isWordStart = (c: number): boolean => (c >= 97 && c <= 122) || (c >= 65 && c <= 90) || c === 95;
|
||||
const isWordChar = (c: number): boolean => isWordStart(c) || (c >= 48 && c <= 57);
|
||||
|
||||
/** A function-like `#define` whose replacement, on any of its lines, writes `::` and a `{`. */
|
||||
const METHOD_MACRO_CANDIDATE = /^[ \t]*#[ \t]*define[ \t]+[A-Za-z_]\w*\([^)\n]*\)((?:[^\n]*\\\r?\n)*[^\n]*)/gm;
|
||||
|
||||
/**
|
||||
* Whether the C++ `source` may define a method with a macro of its own: a
|
||||
* cheap test of the raw text, so the bulk-index fast path, which never sees
|
||||
* this pass, is left only for a file that can't have one.
|
||||
*/
|
||||
export function mayDefineCppMacroMethods(source: string): boolean {
|
||||
if (!source.includes('define')) return false;
|
||||
METHOD_MACRO_CANDIDATE.lastIndex = 0;
|
||||
for (let m = METHOD_MACRO_CANDIDATE.exec(source); m; m = METHOD_MACRO_CANDIDATE.exec(source)) {
|
||||
if (m[1]!.includes('::') && m[1]!.includes('{')) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Add the methods the file's own macros define (see the module comment) to an extraction result. */
|
||||
export function appendCppMacroMethods(result: ExtractionResult, filePath: string, source: string, preParse: (source: string) => string): void {
|
||||
if (!mayDefineCppMacroMethods(source)) return;
|
||||
const minted = cppMacroMethods(filePath, source, preParse, result.nodes);
|
||||
result.nodes.push(...minted.nodes);
|
||||
result.edges.push(...minted.edges);
|
||||
}
|
||||
|
||||
/**
|
||||
* The method nodes, and their `contains` edges, for every definition the
|
||||
* file's own method-defining macros expand to where the file invokes them at
|
||||
* namespace scope. `nodes` are the file's extracted nodes: the class a
|
||||
* method's receiver names, and the ids already taken.
|
||||
*/
|
||||
function cppMacroMethods(
|
||||
filePath: string,
|
||||
source: string,
|
||||
preParse: (source: string) => string,
|
||||
nodes: readonly Node[],
|
||||
): { nodes: Node[]; edges: Edge[] } {
|
||||
const out = { nodes: [] as Node[], edges: [] as Edge[] };
|
||||
const { macros, undefs } = cppMacroDirectives(source);
|
||||
const defining = macros.filter((m) => cppDefinitionHeads(m.body.replace(/\s*##\s*/g, '')).length > 0);
|
||||
if (defining.length === 0) return out;
|
||||
// Scopes as the parser saw them (after preParse, offsets unchanged), like the walker's.
|
||||
const scopes = scanCppBraceScopes(preParse(source));
|
||||
if (!scopes) return out;
|
||||
|
||||
const names = new Set(defining.map((m) => m.name));
|
||||
const taken = new Set(nodes.map((n) => n.id));
|
||||
const fileId = `file:${filePath}`;
|
||||
for (const call of cppMacroInvocations(source, names)) {
|
||||
if (!scopes.atNamespaceScope(call.at)) continue;
|
||||
const macro = liveMacro(defining, undefs, call.name, call.at);
|
||||
const expansion = macro && expandCppMacro(macro, call.args);
|
||||
if (!expansion) continue;
|
||||
const start = scopes.positionOf(call.at);
|
||||
const end = scopes.positionOf(call.end);
|
||||
const prefix = scopes.namespacesAt(call.at);
|
||||
for (const def of cppDefinitionHeads(expansion)) {
|
||||
const receiverLast = def.receiver.split('::').pop()!;
|
||||
// A constructor's or destructor's.
|
||||
if (def.name === receiverLast || def.name.startsWith('~')) continue;
|
||||
let id = generateNodeId(filePath, 'method', def.name, start.line);
|
||||
if (taken.has(id)) id = `${id}:${start.column}`;
|
||||
for (let k = 1; taken.has(id); k++) id = `${generateNodeId(filePath, 'method', def.name, start.line)}:${start.column}:${k}`;
|
||||
taken.add(id);
|
||||
const returnType = normalizeCppReturnType(cppTypeText(def.returnText));
|
||||
out.nodes.push({
|
||||
id,
|
||||
kind: 'method',
|
||||
name: def.name,
|
||||
qualifiedName: composeQualifiedName(prefix, def.receiver, def.name),
|
||||
filePath,
|
||||
language: 'cpp',
|
||||
startLine: start.line,
|
||||
endLine: end.line,
|
||||
startColumn: start.column,
|
||||
endColumn: end.column,
|
||||
signature: def.head,
|
||||
...(returnType ? { returnType } : {}),
|
||||
updatedAt: Date.now(),
|
||||
});
|
||||
out.edges.push({ source: fileId, target: id, kind: 'contains' });
|
||||
// As for an out-of-line definition, the class of its receiver's name in the same file holds it.
|
||||
const owner = nodes.find((n) => n.name === def.receiver && n.filePath === filePath &&
|
||||
(n.kind === 'struct' || n.kind === 'union' || n.kind === 'class' || n.kind === 'enum' || n.kind === 'trait'));
|
||||
if (owner) out.edges.push({ source: owner.id, target: id, kind: 'contains' });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The qualified name of a method written `receiver::name` inside the
|
||||
* namespaces `prefix`, as the walker composes an out-of-line definition's
|
||||
* (TreeSitterExtractor.composeReceiverQualifiedName): the receiver is anchored
|
||||
* at the first namespace it re-spells.
|
||||
*/
|
||||
function composeQualifiedName(prefix: readonly string[], receiver: string, name: string): string {
|
||||
const base = `${receiver}::${name}`;
|
||||
if (prefix.length === 0) return base;
|
||||
const anchor = prefix.indexOf(receiver.split('::')[0]!);
|
||||
const kept = anchor === -1 ? prefix : prefix.slice(0, anchor);
|
||||
return kept.length > 0 ? `${kept.join('::')}::${base}` : base;
|
||||
}
|
||||
|
||||
/** The type a definition's leading text writes, without its specifiers, template header and attributes. */
|
||||
function cppTypeText(returnText: string): string {
|
||||
let text = returnText.trim();
|
||||
for (;;) {
|
||||
const before = text;
|
||||
if (text.startsWith('template')) {
|
||||
const open = text.indexOf('<');
|
||||
const close = open < 0 ? -1 : matching(text, open, '<', '>');
|
||||
if (close < 0) return '';
|
||||
text = text.slice(close + 1).trim();
|
||||
}
|
||||
text = text.replace(/^\[\[[^\]]*\]\]\s*/, '')
|
||||
.replace(/^(?:inline|static|virtual|constexpr|consteval|explicit|extern|friend|__forceinline)\b\s*/, '')
|
||||
// An attribute macro in front (`PROTOBUF_NOINLINE`, `ABSL_MUST_USE_RESULT`).
|
||||
.replace(/^[A-Z][A-Z0-9]*_[A-Z0-9_]*(?:\s*\([^()]*\))?\s+/, '');
|
||||
if (text === before) return text;
|
||||
}
|
||||
}
|
||||
|
||||
/** Where the `close` matching the `open` at `text[from]` is, or -1. */
|
||||
function matching(text: string, from: number, open: string, close: string): number {
|
||||
let depth = 0;
|
||||
for (let i = from; i < text.length; i++) {
|
||||
const c = text[i]!;
|
||||
if (c === '"' || c === "'") i = quotedEnd(text, i) - 1;
|
||||
else if (c === open) depth++;
|
||||
else if (c === close && --depth === 0) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
/** Past the string or character literal whose quote is at `i`. */
|
||||
function quotedEnd(text: string, i: number): number {
|
||||
const quote = text[i];
|
||||
let j = i + 1;
|
||||
while (j < text.length && text[j] !== quote && text[j] !== '\n') j += text[j] === '\\' ? 2 : 1;
|
||||
return Math.min(text.length, j + 1);
|
||||
}
|
||||
|
||||
/** Past the blanks at `i`. */
|
||||
function skipBlank(text: string, i: number): number {
|
||||
while (i < text.length && /\s/.test(text[i]!)) i++;
|
||||
return i;
|
||||
}
|
||||
|
||||
/** The end of the identifier at `i`, or -1. */
|
||||
function wordEnd(text: string, i: number): number {
|
||||
if (!isWordStart(text.charCodeAt(i))) return -1;
|
||||
let j = i + 1;
|
||||
while (j < text.length && isWordChar(text.charCodeAt(j))) j++;
|
||||
return j;
|
||||
}
|
||||
|
||||
/**
|
||||
* The qualified name starting at `i` that an argument list follows
|
||||
* (`Descriptor::file(`, `internal::ExtensionSet::Has(`, `Box<T>::get(`,
|
||||
* `Matrix::operator()(`): its segments and the offset of the `(`. Null when no
|
||||
* such name starts there, or it has no `::`.
|
||||
*/
|
||||
function qualifiedCallableAt(text: string, i: number): { segments: string[]; open: number } | null {
|
||||
const segments: string[] = [];
|
||||
let j = i;
|
||||
if (text.startsWith('::', j)) j = skipBlank(text, j + 2);
|
||||
for (;;) {
|
||||
if (text.startsWith('operator', j) && !isWordChar(text.charCodeAt(j + 8))) {
|
||||
const op = /^operator\s*(\(\s*\)|\[\s*\]|[^\s\w(]+|\w+)\s*\(/.exec(text.slice(j));
|
||||
if (!op || segments.length === 0) return null;
|
||||
segments.push(`operator${op[1]!.replace(/\s+/g, '')}`);
|
||||
return { segments, open: j + op[0].length - 1 };
|
||||
}
|
||||
const tilde = text[j] === '~';
|
||||
const end = wordEnd(text, tilde ? skipBlank(text, j + 1) : j);
|
||||
if (end < 0) return null;
|
||||
const word = text.slice(tilde ? skipBlank(text, j + 1) : j, end);
|
||||
segments.push(tilde ? `~${word}` : word);
|
||||
let k = skipBlank(text, end);
|
||||
if (text[k] === '<' && !tilde) {
|
||||
const close = matching(text, k, '<', '>');
|
||||
if (close < 0) return null;
|
||||
segments[segments.length - 1] += text.slice(k, close + 1);
|
||||
k = skipBlank(text, close + 1);
|
||||
}
|
||||
if (text.startsWith('::', k)) {
|
||||
if (tilde) return null;
|
||||
j = skipBlank(text, k + 2);
|
||||
continue;
|
||||
}
|
||||
return text[k] === '(' && segments.length > 1 ? { segments, open: k } : null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* After a definition's parameter list (from `i`): the offset of its body's
|
||||
* `{`, past the qualifiers it may write (`const`, `noexcept(…)`, `override`,
|
||||
* `&&`, an attribute, an attribute macro, a trailing return type). -1 for
|
||||
* anything else: a declaration (`;`), `= default`, a constructor's
|
||||
* initializer list.
|
||||
*/
|
||||
function bodyOpenAfter(text: string, i: number): number {
|
||||
let j = skipBlank(text, i);
|
||||
for (;;) {
|
||||
const c = text[j];
|
||||
if (c === '{') return j;
|
||||
if (c === '&') {
|
||||
j = skipBlank(text, j + (text[j + 1] === '&' ? 2 : 1));
|
||||
continue;
|
||||
}
|
||||
if (text.startsWith('[[', j)) {
|
||||
const close = text.indexOf(']]', j + 2);
|
||||
if (close < 0) return -1;
|
||||
j = skipBlank(text, close + 2);
|
||||
continue;
|
||||
}
|
||||
if (text.startsWith('->', j)) {
|
||||
// A trailing return type runs to the body.
|
||||
let depth = 0;
|
||||
for (let k = j + 2; k < text.length; k++) {
|
||||
const d = text[k];
|
||||
if (d === '(' || d === '<' || d === '[') depth++;
|
||||
else if (d === ')' || d === '>' || d === ']') depth--;
|
||||
else if (d === '{' && depth <= 0) return k;
|
||||
else if (d === ';' && depth <= 0) return -1;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
const end = wordEnd(text, j);
|
||||
if (end < 0) return -1;
|
||||
const word = text.slice(j, end);
|
||||
if (!/^(?:const|volatile|override|final|noexcept|throw|[A-Z][A-Z0-9]*_[A-Z0-9_]*)$/.test(word)) return -1;
|
||||
j = skipBlank(text, end);
|
||||
if (text[j] === '(' && word !== 'const' && word !== 'volatile' && word !== 'override' && word !== 'final') {
|
||||
const close = matching(text, j, '(', ')');
|
||||
if (close < 0) return -1;
|
||||
j = skipBlank(text, close + 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** The method definitions at the top level of a macro's expansion `text`, in order. */
|
||||
function cppDefinitionHeads(text: string): CppDefinitionHead[] {
|
||||
const heads: CppDefinitionHead[] = [];
|
||||
let segment = 0; // where the current top-level declaration began
|
||||
for (let i = 0; i < text.length; ) {
|
||||
const c = text[i]!;
|
||||
if (c === '"' || c === "'") {
|
||||
i = quotedEnd(text, i);
|
||||
continue;
|
||||
}
|
||||
if (c === ';' || c === '}') {
|
||||
segment = ++i;
|
||||
continue;
|
||||
}
|
||||
if (c === '{' || c === '(') {
|
||||
// A body or group that is no definition read here (a namespace, a class, a statement).
|
||||
const close = matching(text, i, c, c === '{' ? '}' : ')');
|
||||
if (close < 0) return heads;
|
||||
if (c === '{') segment = close + 1;
|
||||
i = close + 1;
|
||||
continue;
|
||||
}
|
||||
const callable = isWordStart(text.charCodeAt(i)) || (c === ':' && text[i + 1] === ':') ? qualifiedCallableAt(text, i) : null;
|
||||
if (!callable) {
|
||||
const end = wordEnd(text, i);
|
||||
i = end < 0 ? i + 1 : end;
|
||||
continue;
|
||||
}
|
||||
const close = matching(text, callable.open, '(', ')');
|
||||
if (close < 0) return heads;
|
||||
const open = bodyOpenAfter(text, close + 1);
|
||||
if (open < 0) {
|
||||
i = close + 1;
|
||||
continue;
|
||||
}
|
||||
// An explicit specialization's arguments (`Read<int32_t, TYPE_INT32>`) are no part of its name.
|
||||
const last = callable.segments[callable.segments.length - 1]!;
|
||||
const name = last.startsWith('operator') ? last : stripCppTemplateArgs(last);
|
||||
const receiver = stripCppTemplateArgs(callable.segments.slice(0, -1).join('::'));
|
||||
if (receiver) {
|
||||
heads.push({
|
||||
receiver,
|
||||
name,
|
||||
returnText: text.slice(segment, i).trim(),
|
||||
head: text.slice(segment, open).replace(/\s+/g, ' ').trim(),
|
||||
});
|
||||
}
|
||||
const bodyClose = matching(text, open, '{', '}');
|
||||
if (bodyClose < 0) return heads;
|
||||
segment = i = bodyClose + 1;
|
||||
}
|
||||
return heads;
|
||||
}
|
||||
|
||||
/**
|
||||
* The macro of `name` in effect at `offset`: its last `#define` before it,
|
||||
* unless an `#undef` came between.
|
||||
*/
|
||||
function liveMacro(macros: readonly CppMacro[], undefs: ReadonlyArray<{ name: string; at: number }>, name: string, offset: number): CppMacro | null {
|
||||
let found: CppMacro | null = null;
|
||||
for (const m of macros) if (m.name === name && m.at < offset && (!found || m.at > found.at)) found = m;
|
||||
if (!found) return null;
|
||||
return undefs.some((u) => u.name === name && u.at > found!.at && u.at < offset) ? null : found;
|
||||
}
|
||||
|
||||
/** A macro's expansion for the arguments `args` (as written), or null when they don't fit its parameters. */
|
||||
function expandCppMacro(macro: CppMacro, args: string[]): string | null {
|
||||
const values = macro.params.length === 0 && args.length === 1 && args[0]!.trim() === '' ? [] : args.map((a) => a.trim());
|
||||
if (values.length !== macro.params.length) return null;
|
||||
const valueOf = new Map<string, string>(macro.params.map((p, k) => [p, values[k]!]));
|
||||
return macro.body
|
||||
.replace(/[A-Za-z_]\w*/g, (word) => valueOf.get(word) ?? word)
|
||||
// `a ## b`: one token.
|
||||
.replace(/\s*##\s*/g, '');
|
||||
}
|
||||
|
||||
/**
|
||||
* The function-like `#define`s of a C++ source and its `#undef`s, in order,
|
||||
* read past comments and literals: a `#define` in a comment or a raw string
|
||||
* defines nothing.
|
||||
*/
|
||||
function cppMacroDirectives(source: string): { macros: CppMacro[]; undefs: Array<{ name: string; at: number }> } {
|
||||
const macros: CppMacro[] = [];
|
||||
const undefs: Array<{ name: string; at: number }> = [];
|
||||
for (const d of cppCodeScan(source, null).directives) {
|
||||
const text = withoutLineComment(d.text.replace(/\\\r?\n/g, ' ').replace(/\/\*[\s\S]*?\*\//g, ' '));
|
||||
const define = /^#[ \t]*define[ \t]+([A-Za-z_]\w*)\(([^)]*)\)/.exec(text);
|
||||
if (define) {
|
||||
const params = define[2]!.split(',').map((p) => p.trim()).filter((p) => p !== '');
|
||||
if (!params.some((p) => p.endsWith('...'))) macros.push({ name: define[1]!, params, body: text.slice(define[0].length), at: d.at });
|
||||
continue;
|
||||
}
|
||||
const undef = /^#[ \t]*undef[ \t]+([A-Za-z_]\w*)/.exec(text);
|
||||
if (undef) undefs.push({ name: undef[1]!, at: d.at });
|
||||
}
|
||||
return { macros, undefs };
|
||||
}
|
||||
|
||||
/** The invocations of the macros `names` in code (not in a directive, comment or literal), with their arguments. */
|
||||
function cppMacroInvocations(source: string, names: ReadonlySet<string>): Array<{ name: string; at: number; end: number; args: string[] }> {
|
||||
const calls: Array<{ name: string; at: number; end: number; args: string[] }> = [];
|
||||
for (const word of cppCodeScan(source, names).words) {
|
||||
const open = skipBlankAndComments(source, word.end);
|
||||
if (source[open] !== '(') continue;
|
||||
const args: string[] = [];
|
||||
let depth = 0;
|
||||
let from = open + 1;
|
||||
let close = -1;
|
||||
for (let i = open; i < source.length; i++) {
|
||||
const c = source[i]!;
|
||||
if (c === '"' || c === "'") i = quotedEnd(source, i) - 1;
|
||||
else if (c === '/' && source[i + 1] === '*') {
|
||||
const end = source.indexOf('*/', i + 2);
|
||||
i = end < 0 ? source.length : end + 1;
|
||||
} else if (c === '/' && source[i + 1] === '/') {
|
||||
const end = source.indexOf('\n', i);
|
||||
i = end < 0 ? source.length : end - 1;
|
||||
} else if (c === '(') depth++;
|
||||
else if (c === ')' && --depth === 0) {
|
||||
close = i;
|
||||
break;
|
||||
} else if (c === ',' && depth === 1) {
|
||||
args.push(source.slice(from, i));
|
||||
from = i + 1;
|
||||
}
|
||||
}
|
||||
if (close < 0) continue;
|
||||
args.push(source.slice(from, close));
|
||||
calls.push({ name: word.name, at: word.at, end: close + 1, args: args.map(withoutComments) });
|
||||
}
|
||||
return calls;
|
||||
}
|
||||
|
||||
function withoutComments(text: string): string {
|
||||
return text.replace(/\/\*[\s\S]*?\*\//g, ' ').replace(/\/\/[^\n]*/g, ' ');
|
||||
}
|
||||
|
||||
/** A spliced directive without the `//` comment that ends it, one in a string literal aside. */
|
||||
function withoutLineComment(text: string): string {
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
const c = text[i];
|
||||
if (c === '"' || c === "'") i = quotedEnd(text, i) - 1;
|
||||
else if (c === '/' && text[i + 1] === '/') return text.slice(0, i);
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
/** Past blanks and comments from `i`. */
|
||||
function skipBlankAndComments(source: string, i: number): number {
|
||||
for (;;) {
|
||||
i = skipBlank(source, i);
|
||||
if (source.startsWith('//', i)) {
|
||||
const end = source.indexOf('\n', i);
|
||||
i = end < 0 ? source.length : end;
|
||||
} else if (source.startsWith('/*', i)) {
|
||||
const end = source.indexOf('*/', i + 2);
|
||||
i = end < 0 ? source.length : end + 2;
|
||||
} else {
|
||||
return i;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const RAW_STRING_PREFIX = /^(?:u8|[uUL])?R$/;
|
||||
|
||||
/**
|
||||
* One pass over C/C++ source, skipping comments and string, character and raw
|
||||
* string literals: the preprocessor directives (each with its `\`
|
||||
* continuations), and, when `names` is given, the code's identifiers among
|
||||
* them.
|
||||
*/
|
||||
function cppCodeScan(source: string, names: ReadonlySet<string> | null): {
|
||||
directives: Array<{ at: number; text: string }>;
|
||||
words: Array<{ name: string; at: number; end: number }>;
|
||||
} {
|
||||
const directives: Array<{ at: number; text: string }> = [];
|
||||
const words: Array<{ name: string; at: number; end: number }> = [];
|
||||
const n = source.length;
|
||||
let lineStart = true;
|
||||
let lastWordStart = -1;
|
||||
let lastWordEnd = -1;
|
||||
for (let i = 0; i < n; ) {
|
||||
const c = source.charCodeAt(i);
|
||||
if (c === 10) {
|
||||
lineStart = true;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (c === 32 || c === 9 || c === 13 || c === 11 || c === 12) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (c === 47 && source.charCodeAt(i + 1) === 47) {
|
||||
// A `\` at the end continues a line comment.
|
||||
let end = i;
|
||||
for (;;) {
|
||||
end = source.indexOf('\n', end);
|
||||
if (end < 0) {
|
||||
end = n;
|
||||
break;
|
||||
}
|
||||
let k = end - 1;
|
||||
if (source.charCodeAt(k) === 13) k--;
|
||||
if (source.charCodeAt(k) !== 92) break;
|
||||
end++;
|
||||
}
|
||||
i = end;
|
||||
continue;
|
||||
}
|
||||
if (c === 47 && source.charCodeAt(i + 1) === 42) {
|
||||
const end = source.indexOf('*/', i + 2);
|
||||
i = end < 0 ? n : end + 2;
|
||||
continue;
|
||||
}
|
||||
if (c === 35 && lineStart) {
|
||||
let j = i + 1;
|
||||
for (; j < n; j++) {
|
||||
const d = source.charCodeAt(j);
|
||||
if (d === 47 && source.charCodeAt(j + 1) === 42) {
|
||||
const end = source.indexOf('*/', j + 2);
|
||||
j = end < 0 ? n - 1 : end + 1;
|
||||
} else if (d === 10) {
|
||||
let k = j - 1;
|
||||
if (source.charCodeAt(k) === 13) k--;
|
||||
if (source.charCodeAt(k) !== 92) break;
|
||||
}
|
||||
}
|
||||
directives.push({ at: i, text: source.slice(i, j) });
|
||||
i = j;
|
||||
continue;
|
||||
}
|
||||
lineStart = false;
|
||||
if (c === 34) {
|
||||
if (lastWordEnd === i && RAW_STRING_PREFIX.test(source.slice(lastWordStart, i))) {
|
||||
const paren = source.indexOf('(', i + 1);
|
||||
const delimiter = paren < 0 ? '' : source.slice(i + 1, paren);
|
||||
if (paren >= 0 && /^[^\s()\\"]{0,16}$/.test(delimiter)) {
|
||||
const end = source.indexOf(`)${delimiter}"`, paren + 1);
|
||||
i = end < 0 ? n : end + delimiter.length + 2;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
i = quotedEnd(source, i);
|
||||
continue;
|
||||
}
|
||||
if (c === 39) {
|
||||
// A digit separator (`1'000`) is no character literal.
|
||||
if (i > 0 && isWordChar(source.charCodeAt(i - 1)) && lastWordEnd !== i) {
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
i = quotedEnd(source, i);
|
||||
continue;
|
||||
}
|
||||
if (isWordChar(c)) {
|
||||
let j = i + 1;
|
||||
while (j < n && isWordChar(source.charCodeAt(j))) j++;
|
||||
if (isWordStart(c)) {
|
||||
lastWordStart = i;
|
||||
lastWordEnd = j;
|
||||
if (names?.has(source.slice(i, j))) words.push({ name: source.slice(i, j), at: i, end: j });
|
||||
}
|
||||
i = j;
|
||||
continue;
|
||||
}
|
||||
i++;
|
||||
}
|
||||
return { directives, words };
|
||||
}
|
||||
@@ -19,6 +19,7 @@ import { detectLanguage, loadGrammarsForLanguages, resetParser } from './grammar
|
||||
import { tryKernelExtractRaw } from './kernel';
|
||||
import { commonJsRequireRefs } from './commonjs-requires';
|
||||
import { getAllFrameworkResolvers, getApplicableFrameworks } from '../resolution/frameworks';
|
||||
import { mayDefineCppMacroMethods } from './languages/cpp-macro-methods';
|
||||
import type { Language, ExtractionResult } from '../types';
|
||||
|
||||
// Emscripten prints `Aborted()` (and a follow-up RuntimeError diag
|
||||
@@ -90,6 +91,8 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
|
||||
// buffer clone is a flat memcpy). Only when no applicable framework has
|
||||
// an extract() hook: those merge extra nodes/refs into the DECODED
|
||||
// result inside extractFromSource, so such files keep the decoded path.
|
||||
// So does a C++ file that may define methods with its own macros: they
|
||||
// are minted after the kernel, in extractFromSource.
|
||||
let result: ExtractionResult | undefined;
|
||||
const frameworksNeedDecode =
|
||||
frameworkNames && frameworkNames.length > 0
|
||||
@@ -98,7 +101,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
|
||||
language
|
||||
).some((fw) => !!fw.extract)
|
||||
: false;
|
||||
if (!frameworksNeedDecode) {
|
||||
if (!frameworksNeedDecode && !(language === 'cpp' && mayDefineCppMacroMethods(content!))) {
|
||||
const raw = tryKernelExtractRaw(filePath!, content!, language);
|
||||
if (raw) {
|
||||
result = {
|
||||
|
||||
@@ -23,6 +23,7 @@ import type { LanguageExtractor, ExtractorContext } from './tree-sitter-types';
|
||||
import { EXTRACTORS } from './languages';
|
||||
import { stripCppTemplateArgs, isCppConstructorDeclaration, cDeclaratorIdentifier, isClassLikeDefinition } from './languages/c-cpp';
|
||||
import { NestedIntervals, scanCppBraceScopes, type CppBraceScopes } from './languages/cpp-brace-scopes';
|
||||
import { appendCppMacroMethods } from './languages/cpp-macro-methods';
|
||||
import { rustImplTypeName } from './languages/rust';
|
||||
import { goAliasTypeNames, goEmbeddedTypeName } from './languages/go';
|
||||
import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeRefs } from './languages/dart';
|
||||
@@ -8371,6 +8372,12 @@ export function extractFromSource(
|
||||
);
|
||||
result = extractor.extract();
|
||||
}
|
||||
// C++ methods the file's own macros define, read from the original source
|
||||
// after either extractor so both mint them (cpp-macro-methods.ts).
|
||||
if (detectedLanguage === 'cpp') {
|
||||
const preParse = EXTRACTORS.cpp?.preParse;
|
||||
appendCppMacroMethods(result, filePath, source, (text) => (preParse ? preParse(text, filePath) : text));
|
||||
}
|
||||
}
|
||||
|
||||
// Framework-specific extraction (routes, middleware, etc.)
|
||||
|
||||
@@ -9438,17 +9438,31 @@ function cppSubscriptElement(type: CppWrittenType, at: UnresolvedRef, context: R
|
||||
return element ? { ...element, text: element.text.replace(/&+\s*$/, '').trim() } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The expanded definition a C++ method that a macro defines carries as its
|
||||
* signature (cpp-macro-methods.ts), `inline const FileDescriptor*
|
||||
* Descriptor::file() const` for protobuf's `PROTOBUF_DEFINE_ACCESSOR(Descriptor,
|
||||
* file, const FileDescriptor*)`: its source is the invocation, which spells
|
||||
* neither its return type nor its parameters. Null for any other node.
|
||||
*/
|
||||
function cppMacroMethodHead(fn: Node): string | null {
|
||||
if (fn.kind !== 'method' || !fn.signature) return null;
|
||||
const name = fn.name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
return new RegExp(`::\\s*${name}\\s*\\(`).test(fn.signature) ? fn.signature : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The return type of a C or C++ function, as its declaration writes it:
|
||||
* `std::vector<const FieldDescriptor*>` for `std::vector<const
|
||||
* FieldDescriptor*> FieldNumberOrder(…)`. (The index keeps only `vector`.)
|
||||
* Null for none that can be read: a constructor, `void`, `auto` or
|
||||
* `decltype(…)`.
|
||||
* `decltype(…)`. A method a macro defines is read from its expansion.
|
||||
*/
|
||||
function cppWrittenReturnType(fn: Node, context: ResolutionContext): string | null {
|
||||
const lines = context.getFileLines?.(fn.filePath) ?? context.readFile(fn.filePath)?.split(/\r?\n/);
|
||||
if (!lines) return null;
|
||||
const head = lines.slice(fn.startLine - 1, fn.startLine + 2).join(' ').slice(fn.startColumn);
|
||||
const macro = cppMacroMethodHead(fn);
|
||||
const lines = macro === null ? context.getFileLines?.(fn.filePath) ?? context.readFile(fn.filePath)?.split(/\r?\n/) : null;
|
||||
if (macro === null && !lines) return null;
|
||||
const head = macro ?? lines!.slice(fn.startLine - 1, fn.startLine + 2).join(' ').slice(fn.startColumn);
|
||||
const name = fn.name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&').replace(/^operator/, 'operator\\s*');
|
||||
const declared = new RegExp(String.raw`^(.*?)(?:\b[A-Za-z_]\w*\s*(?:<[^<>;{}]*>)?\s*::\s*)*\b${name}\s*\(`).exec(head);
|
||||
let written = declared?.[1];
|
||||
@@ -11169,9 +11183,11 @@ const CPP_DELETED_FUNCTIONS = new WeakMap<ResolutionContext, Map<string, boolean
|
||||
/**
|
||||
* Whether the C++ function or method `n`, named `name`, is declared deleted:
|
||||
* from its name to the declaration's `;` or body, it says `= delete`
|
||||
* (protobuf's `void reset(T* PROTOBUF_NULLABLE) = delete;`).
|
||||
* (protobuf's `void reset(T* PROTOBUF_NULLABLE) = delete;`). A method a macro
|
||||
* defines has a body.
|
||||
*/
|
||||
function isCppDeletedFunction(n: Node, name: string, context: ResolutionContext): boolean {
|
||||
if (cppMacroMethodHead(n) !== null) return false;
|
||||
let memo = CPP_DELETED_FUNCTIONS.get(context);
|
||||
if (!memo) CPP_DELETED_FUNCTIONS.set(context, (memo = new Map()));
|
||||
const hit = memo.get(n.id);
|
||||
@@ -16859,7 +16875,11 @@ function cppTemplateArgumentsEnd(text: string, at: number): number {
|
||||
/** The balanced `( … )` after `name` from `line`/`column` of `file` (up to a dozen lines), or null. */
|
||||
function cppParenListAfter(file: string, line: number, column: number, name: string, context: ResolutionContext): string | null {
|
||||
const lines = context.getFileLines?.(file) ?? context.readFile(file)?.split(/\r?\n/) ?? [];
|
||||
const text = lines.slice(line - 1, line + 11).join('\n');
|
||||
return cppParenListIn(lines.slice(line - 1, line + 11).join('\n'), column, name);
|
||||
}
|
||||
|
||||
/** The `( … )` that follows `name` in `text`, from `column` on, without its parentheses. */
|
||||
function cppParenListIn(text: string, column: number, name: string): string | null {
|
||||
const at = new RegExp(`\\b${name.replace(/[~]/g, '\\$&')}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text.slice(column));
|
||||
if (!at) return null;
|
||||
const open = column + at.index + at[0].length - 1;
|
||||
@@ -16913,10 +16933,11 @@ const CPP_OVERLOAD_PARAMETERS = new WeakMap<ResolutionContext, Map<string, strin
|
||||
* absl::optional<Semantic> semantic)` below the class, whose declaration
|
||||
* gave `semantic` its `= absl::nullopt`. Its own list when the class
|
||||
* declares the method with the same parameter types and no default, or
|
||||
* not at all.
|
||||
* not at all. A method a macro defines writes its list in its expansion.
|
||||
*/
|
||||
function cppOverloadParameters(n: Node, name: string, context: ResolutionContext): string | null {
|
||||
const own = cppParenListAfter(n.filePath, n.startLine, 0, name, context);
|
||||
const macro = cppMacroMethodHead(n);
|
||||
const own = macro !== null ? cppParenListIn(macro, 0, name) : cppParenListAfter(n.filePath, n.startLine, 0, name, context);
|
||||
if (own === null || n.kind !== 'method') return own;
|
||||
let memo = CPP_OVERLOAD_PARAMETERS.get(context);
|
||||
if (!memo) CPP_OVERLOAD_PARAMETERS.set(context, (memo = new Map()));
|
||||
|
||||
Reference in New Issue
Block a user