diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e19a3f1..89b38000 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,7 +95,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call on a variable, parameter or member declared as a standard-library or other outside type with a lowercase name, like `std::string`, `std::vector`, `std::set` or `absl::flat_hash_set`, no longer links to a project method that only shares the method's name, even one of a project class in another namespace with the type's name, like your own `util::flat_hash_set`. A variable declared outside the calling function, like another function's or a file's `std::string text`, no longer links its calls to such a class either. In google/leveldb, `saved_key_.clear()` on a `std::string saved_key_;` reached `Slice::clear`, and in protocolbuffers/protobuf, `proto.append(…)` on a `std::string` reached an unrelated `LeftoverBuffer::append`. The same goes for `->` through a raw pointer like `std::string* out`, while `->` through a smart pointer, iterator or optional still links to the type it holds. Comments also no longer pass for a variable's declaration, so more calls now link to the method of the type the variable is really declared with. Re-index C++ projects after upgrading. - In C++, a class written with attribute macros the parser can't read now keeps its members. That covers several macros between `class` and the class name, like protocolbuffers/protobuf's generated `class PROTOBUF_EXPORT PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED Any final`, a macro with arguments there, like RocksDB's `struct ALIGN_AS(64U) HandleImpl`, or one before a template specialization, and macros on a member: in front of it (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD`), between a pointer and its name (`PROTOBUF_NONNULL`), after its parameters or its name (`ABSL_ATTRIBUTE_LIFETIME_BOUND`, google/leveldb's `LOCKS_EXCLUDED(mu_)` and `GUARDED_BY(mu_)`), or alone on a line above a comment ({fmt}'s `FMT_BEGIN_EXPORT`). Before, protobuf's generated message classes were missing and their members, like `GetDescriptor`, `GetReflection` and `unknown_fields`, were indexed as functions outside any class, and leveldb's annotated methods were indexed under the annotation's name, so calls to them linked to nothing. Re-index C++ projects after upgrading. - In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop. Re-index C++ projects after upgrading. -- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, or a class that has no method of that name and neither do the classes it derives from, the call no longer links to a project method; a class derived from something that can't be followed, like a `decltype(…)`, may still have it, so that call is matched as before. The same goes for `->` on a container's iterator, like `it->data()` on a `std::deque::const_iterator it` or `loc->path_size()` on protocolbuffers/protobuf's `RepeatedPtrField::iterator loc`: the call links to that method of the container's element type, and to no project method when the element is a standard-library type like `std::string`. That includes an iterator an `auto` variable gets from a standard-library container's `begin()`, `find()` and the like, or from a standard search like `std::lower_bound`, as in facebook/rocksdb's `auto it = logs_.begin(); it->IsSyncing()`. Re-index C++ projects after upgrading. +- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, or a class that has no method of that name and neither do the classes it derives from, the call no longer links to a project method; a class derived from something that can't be followed, like a `decltype(…)`, may still have it, so that call is matched as before. The same goes for `->` on a container's iterator, like `it->data()` on a `std::deque::const_iterator it` or `loc->path_size()` on protocolbuffers/protobuf's `RepeatedPtrField::iterator loc`: the call links to that method of the container's element type, and to no project method when the element is a standard-library type like `std::string`. That includes an iterator an `auto` variable gets from a standard-library container's `begin()`, `find()` and the like, or from a standard search like `std::lower_bound`, as in facebook/rocksdb's `auto it = logs_.begin(); it->IsSyncing()`. A call on a dereferenced iterator or pointer, like `(*it)->Visit()` over a `std::vector` or `(*it).Weight()` over a `std::vector`, and on a variable a dereference sets, like `auto first = *nodes.begin()`, links the same way, where before such a call went to whichever class had a method of that name. Re-index C++ projects after upgrading. - In C++, a call a method makes with no receiver, or through `this->`, now links to the method C++ itself would call: one of the method's own class, else one the class inherits from a base class, else one of a class it is nested in. The same goes for a call in a lambda inside the method, and of a class's overloads the one the call's arguments fit is taken. Before, the call was matched by its name alone, so another class's method of that name could win because it was declared nearer or found first: in protocolbuffers/protobuf, every generated message's `Swap` and move assignment calling `InternalSwap(…)` reached `Any::InternalSwap`, and the `GetArena()` messages inherit from `MessageLite` reached an arena helper's `GetArena`, while googletest's `~linked_ptr() { depart(); }`, vendored in facebook/rocksdb, reached `linked_ptr_internal::depart`. Calls C++ doesn't look up in the class are matched as before: a free function's, a call to a parameter or local of that name, and a template's bare call into a base class that depends on its template parameters. Re-index C++ projects after upgrading. Thanks @danusha2345 for reporting the C++ case in #1952. - In C++, a call on the variable of a range-based `for` loop over `auto` elements, like `cfd->Ref()` inside facebook/rocksdb's `for (auto cfd : cfds)`, now links to the method of the type the loop goes over. That type is read from what the loop iterates: a `std::vector`, `std::set`, `absl::Span` or similar container, protocolbuffers/protobuf's `RepeatedPtrField` or RocksDB's `autovector`, whether a variable, parameter or member holds it, an `auto` variable is set to it or a function or method returns it, and a project class you loop over through its `begin()`, like RocksDB's `ColumnFamilySet`. Before, the call was matched by the variable's name, or by an earlier variable of that name: in RocksDB, `blob_file.GetBlobFileNumber()` over a list of `BlobFileGarbage` reached `BlobFileAddition::GetBlobFileNumber`, and in protobuf's Objective-C generator, calls over a list of `std::unique_ptr` reached other generators' methods. A call on an element of a standard-library type, like `file.find(…)` over a `std::vector`, no longer links to a project method, and `->` on an element held in a `std::unique_ptr` or `std::shared_ptr` reaches the type it holds. Re-index C++ projects after upgrading. - In C++, a call on a variable, parameter or member declared as one of your classes, like `iter->Valid()` on an `Iterator* iter` or `it.Valid()` on a `const Iterator& it`, now links to that method of the class C++ itself would find for the type where the call is written, or of a class it inherits from. Before, the method was picked by the type's last name, so a class of the same name nested elsewhere could win: in facebook/rocksdb, calls on an `Iterator` reached `MemTableRep::Iterator`, which declares `Valid` and `Next` itself while the `Iterator` the code means inherits them, google/leveldb's reached `SkipList::Iterator`, and in protocolbuffers/protobuf the Java generator's `context->options()` reached the hpb generator's `Context` and `text_parser.Parse(…)` on a `TextFormat::Parser` reached the `.proto` parser. When that class and the classes it derives from have no method of that name, the call no longer links to an unrelated class's method that shares it, as protobuf's `field->full_name()` on a `FieldDescriptor*` did. A type is also found through a `using` declaration the file writes, like protobuf's `using google::protobuf::Descriptor;`, before a class of the same name declared at global scope elsewhere in the repository; this also applies to base classes and to the type a smart pointer holds. A type written from the global scope, like `::upb::Arena& arena`, is looked up there, so a namespace's own `upb::Arena` is not taken for it, nor is an alias of that name the calling function declares. A type written as a member of a class template, like `SkipList::Iterator iter` in leveldb's and RocksDB's skip-list tests, is now read whole, so its calls reach the skip list's own iterator rather than the general `Iterator`, and RocksDB's skip-list memtable iterator no longer links its calls on the skip list's iterator to its own methods; a standard container's `iterator` written this way is the library's, never a class of yours named `iterator`. Re-index C++ projects after upgrading. diff --git a/__tests__/cpp-dereferenced-iterator-calls.test.ts b/__tests__/cpp-dereferenced-iterator-calls.test.ts new file mode 100644 index 00000000..d05ffb42 --- /dev/null +++ b/__tests__/cpp-dereferenced-iterator-calls.test.ts @@ -0,0 +1,176 @@ +/** + * A C++ call on a dereferenced variable, like `(*it)->Visit()` over a + * `std::vector`, or on a local an iterator's dereference initializes, + * like `auto first = *nodes.begin(); first->Visit()`, calls a member of the + * class the dereferenced value has. The extractor records the first by its + * bare name, which went to whichever class had a method of that name, and + * the second's local had no type. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-deref-iterator-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** `Other` comes first: a call matched by its bare name lands on it. */ +const NODES = { + 'src/node.h': lines( + 'class Other {', + ' public:', + ' int Visit() { return 2; }', + ' int Weight() const { return 3; }', + '};', + 'class Node {', + ' public:', + ' int Visit() { return 0; }', + ' int Weight() const { return 1; }', + ' Node* Child() { return this; }', + '};', + ), +}; + +describe('a C++ call on a dereferenced iterator or pointer', () => { + it('reproduction: `(*it)->Visit()` over a `std::vector`, and `(*it).Weight()` over a `std::vector`', async () => { + const cg = await indexed({ + ...NODES, + 'src/walk.cc': lines( + '#include ', + '#include "src/node.h"', + 'int Walk(std::vector& nodes) {', + ' int sum = 0;', + ' for (std::vector::iterator it = nodes.begin(); it != nodes.end(); ++it) sum += (*it)->Visit();', + ' return sum;', + '}', + 'int Weigh(std::vector& values) {', + ' int sum = 0;', + ' for (auto vit = values.begin(); vit != values.end(); ++vit) sum += (*vit).Weight();', + ' return sum;', + '}', + ), + }); + try { + expect(calls(cg, 'Walk')).toEqual(['Node::Visit']); + expect(calls(cg, 'Weigh')).toEqual(['Node::Weight']); + } finally { + cg.close(); + } + }); + + it('a local the dereference initializes, an element held in a smart pointer, and a raw pointer', async () => { + const cg = await indexed({ + ...NODES, + 'src/first.cc': lines( + '#include ', + '#include ', + '#include "src/node.h"', + 'int First(std::vector& nodes) {', + ' auto first = *nodes.begin();', + ' return first->Visit();', + '}', + 'int Owned(std::vector>& owned) {', + ' auto oit = owned.begin();', + ' return (*oit)->Weight();', + '}', + 'int Pointed(Node* node) {', + ' return (*node).Weight();', + '}', + ), + }); + try { + expect(calls(cg, 'First')).toEqual(['Node::Visit']); + expect(calls(cg, 'Owned')).toEqual(['Node::Weight']); + expect(calls(cg, 'Pointed')).toEqual(['Node::Weight']); + } finally { + cg.close(); + } + }); + + it('`->` through an iterator in a chain, and in what initializes a local', async () => { + const cg = await indexed({ + ...NODES, + 'src/chain.cc': lines( + '#include ', + '#include "src/node.h"', + 'int Chain(std::vector& values) {', + ' std::vector::iterator it = values.begin();', + ' return it->Child()->Visit();', + '}', + 'int Local(std::vector& values) {', + ' std::vector::iterator it = values.begin();', + ' auto child = it->Child();', + ' return child->Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'Chain')).toEqual(['Node::Child', 'Node::Visit']); + expect(calls(cg, 'Local')).toEqual(['Node::Child', 'Node::Visit']); + } finally { + cg.close(); + } + }); + + it('an element of a library type has no project method to call, nor does a free function of the name', async () => { + const cg = await indexed({ + ...NODES, + 'src/names.cc': lines( + '#include ', + '#include ', + '#include "src/node.h"', + 'class Label {', + ' public:', + ' int size() const { return 0; }', + '};', + 'int Visit() { return 9; }', + 'int Total(std::vector& names) {', + ' int n = 0;', + ' for (auto sit = names.begin(); sit != names.end(); ++sit) n += (*sit).size();', + ' return n;', + '}', + 'int Each(std::vector& nodes) {', + ' std::vector::iterator it = nodes.begin();', + ' return (*it)->Visit();', + '}', + ), + }); + try { + expect(calls(cg, 'Total')).toEqual([]); + expect(calls(cg, 'Each')).toEqual(['Node::Visit']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index f8f0b658..cb13b1a5 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -9806,13 +9806,14 @@ function cppIteratorElement(type: CppWrittenType, at: UnresolvedRef, context: Re /** * The type of the object `x.m` or `x->m` (`op`) names a member of, given - * `x`'s text: `x`'s own type for `.`, what it points at or holds for `->`. + * `x`'s text: `x`'s own type for `.`, what it points at, holds or iterates + * over for `->` (cppDereferenced). */ function cppMemberObject(object: string, op: string, at: UnresolvedRef, context: ResolutionContext, depth: number): CppWrittenType | null { const type = cppExpressionType(object, at, context, depth + 1); const followed = type && cppFollowWrittenAliases(type, at, context); if (!followed) return null; - if (op === '->') return cppPointee(followed); + if (op === '->') return cppDereferenced(followed, at, context); return isCppPointerType(followed.text) ? null : followed; } @@ -9834,7 +9835,7 @@ function cppExpressionType(expr: string, at: UnresolvedRef, context: ResolutionC if (text.startsWith('*')) { const pointer = cppExpressionType(text.slice(1), at, context, depth + 1); const followed = pointer && cppFollowWrittenAliases(pointer, at, context); - return followed && cppPointee(followed); + return followed && cppDereferenced(followed, at, context); } const caller = context.getNodeById?.(at.fromNodeId); if (!caller) return null; @@ -10317,10 +10318,29 @@ function cppArrowTarget( ): CppWrittenType | null { const held = cppHeldType(raw); if (held) return isCppLibraryHolder(raw, context) ? { ...site, text: held } : null; - const container = CPP_ITERATOR_OF.exec(cppWithoutTypeKeywords(raw))?.[1]; - const written = declared ?? site; - const element = container ? cppRangeElementOf({ ...written, text: container }, ref, context, 1) : null; - return element && (element.scope === written.scope || !isCppTemplateParameterType(element)) ? element : null; + return cppIteratorTarget({ ...(declared ?? site), text: raw }, ref, context); +} + +/** + * The element a container's iterator of the type `iterator` points at, read + * from the container's template arguments (cppArrowTarget). Null for any + * other type, and for an element a class template's own `begin()` or + * `operator*` writes with its template parameter. + */ +function cppIteratorTarget(iterator: CppWrittenType, at: UnresolvedRef, context: ResolutionContext): CppWrittenType | null { + const container = CPP_ITERATOR_OF.exec(cppWithoutTypeKeywords(iterator.text))?.[1]; + const element = container ? cppRangeElementOf({ ...iterator, text: container }, at, context, 1) : null; + return element && (element.scope === iterator.scope || !isCppTemplateParameterType(element)) ? element : null; +} + +/** + * What dereferencing a value of the type `type` gives, for `*x` and `x->`: + * what a pointer points at or a holder holds (cppPointee), or the element a + * container's iterator points at (cppIteratorTarget) — the `Node*` of + * `*nodes.begin()` over a `std::vector`. + */ +function cppDereferenced(type: CppWrittenType, at: UnresolvedRef, context: ResolutionContext): CppWrittenType | null { + return cppPointee(type) ?? cppIteratorTarget(type, at, context); } /** Whether the holder `raw` names (cppHeldType) is the library's: no type of the project's has its name. */ @@ -10750,15 +10770,79 @@ function cppImplicitReceiver(ref: UnresolvedRef, context: ResolutionContext): 'b }; const verdict = at(Math.max(0, ref.column)); if (verdict !== null || !/[^\x00-\x7f]/.test(line)) return verdict; - // The column may count UTF-8 bytes, which wider characters before the call outnumber. + const index = cppUtf8Column(line, ref.column); + return index === null ? null : at(index); +} + +/** + * Where in `line` a call's `column` is when it counts UTF-8 bytes, which + * wider characters before the call outnumber. Null when no character starts + * there, or the two agree. + */ +function cppUtf8Column(line: string, column: number): number | null { let bytes = 0; let index = 0; - while (index < line.length && bytes < ref.column) { + while (index < line.length && bytes < column) { const point = line.codePointAt(index)!; bytes += point < 0x80 ? 1 : point < 0x800 ? 2 : point < 0x10000 ? 3 : 4; index += point > 0xffff ? 2 : 1; } - return bytes === ref.column && index !== ref.column ? at(index) : null; + return bytes === column && index !== column ? index : null; +} + +/** A call on a dereferenced variable, up to the member's name: `(*it)->`, `(*node).`. */ +const CPP_DEREFERENCED_RECEIVER = /^\(\s*\*\s*([A-Za-z_]\w*)\s*\)\s*(->|\.)\s*(?:template\s+)?/; + +/** + * The variable a C++ call the extractor recorded by its bare name is made on + * through a dereference, with the operator — `it` and `->` for + * `(*it)->Visit()` — read at the call's column, where its receiver starts. + * Null for any other call, `(*this).Visit()` included (cppImplicitReceiver's). + */ +function cppDereferencedReceiver(ref: UnresolvedRef, context: ResolutionContext): { name: string; op: string } | null { + const line = (context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1]; + if (line === undefined) return null; + const at = (column: number): { name: string; op: string } | null => { + const text = line.slice(column); + const receiver = CPP_DEREFERENCED_RECEIVER.exec(text); + if (!receiver || receiver[1] === 'this') return null; + const rest = text.slice(receiver[0].length); + if (!rest.startsWith(ref.referenceName) || /[\w$]/.test(rest.charAt(ref.referenceName.length))) return null; + return cppArgumentListFollows(rest, ref.referenceName.length) ? { name: receiver[1]!, op: receiver[2]! } : null; + }; + const found = at(Math.max(0, ref.column)); + if (found || !/[^\x00-\x7f]/.test(line)) return found; + const index = cppUtf8Column(line, ref.column); + return index === null ? null : at(index); +} + +/** + * A C++ call the extractor recorded by its bare name on a dereferenced + * variable — `(*it)->Visit()` over a `std::vector`, `(*vit).Weight()` + * over a `std::vector`, `(*node).Weight()` on a `Node*` — calls a + * member of the class the dereferenced value has (cppMemberObject), looked + * up as C++ looks it up where that type is written. By its bare name the + * call went to whichever class had a method of that name. + * - The member of that class, or of a class it derives from. + * - null when no project method is called: a library type (`(*it).size()` + * over a `std::vector`), a template parameter's, or a class + * that surely lacks the method (cppLacksMethod). + * - undefined when the call is on no such receiver, or its type can't be + * read or placed: the name strategies decide, as before. + */ +function matchCppDereferencedCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + const receiver = cppDereferencedReceiver(ref, context); + if (!receiver) return undefined; + const object = cppMemberObject(`(*${receiver.name})`, receiver.op, ref, context, 0); + if (!object || isCppPointerType(object.text)) return undefined; + if (isCppLibraryType(object.text, context)) return null; + const found = cppClassWritten(object.text, object, ref, context); + if (found === undefined) return undefined; + if (found === null) return null; + if (!found.exact || !CPP_CLASS_KINDS.has(found.node.kind)) return undefined; + const method = cppMethodOf(found.node, ref.referenceName, ref, context); + if (method) return { original: ref, targetNodeId: method.id, confidence: 0.9, resolvedBy: 'instance-method' }; + return cppLacksMethod(found.node, context) ? null : undefined; } /** Whether `text` from `from` on is an argument list, after any template arguments (`(`). */ @@ -11910,7 +11994,7 @@ function cppChainInnerType(inner: string, ref: UnresolvedRef, context: Resolutio const recvType = op ? cppExpressionType(recv, ref, context, 1) : null; const followed = recvType && cppFollowWrittenAliases(recvType, ref, context); if (!followed) return null; - const object = op === '->' ? cppPointee(followed) : isCppPointerType(followed.text) ? null : followed; + const object = op === '->' ? cppDereferenced(followed, ref, context) : isCppPointerType(followed.text) ? null : followed; const cls = object && cppWrittenClass(object, ref, context); return cls ? cppReturnTypeOf(cppMethodsOf(cls, inner.slice(dot + 1), ref, context), context) : null; } @@ -16352,6 +16436,15 @@ function matchReferenceInner( if (prefixed !== undefined) return prefixed; } + // A C++ call on a dereferenced variable, which the extractor records by its + // bare name — `(*it)->Visit()` — is a call on what the variable points at, + // or on nothing of the project's: the strategies below would take any + // `Visit`, even a free function of that name. + if (ref.language === 'cpp' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName)) { + const dereferenced = nmTimedT('cppDeref', ref, () => matchCppDereferencedCall(ref, context)); + if (dereferenced !== undefined) return dereferenced; + } + // Try strategies in order of confidence let result: ResolvedRef | null;