mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-24 19:30:10 +08:00
fix(ts): a type-only re-export like export type * from './types' now parses (#2837)
TypeScript 5.0's type-only re-export of a whole module, `export type * from './x'` and `export type * as ns from './x'`, has no rule in the vendored tree-sitter-typescript grammar (v0.23.2, the revision the kernel's crate compiles) or in upstream master: `type` parses as an ERROR inside the export statement. The graph was already right, since the import resolver reads the statement from the file (PRs #1955 and #2526), but the error tree meant a barrel of re-exports got the "parse produced no symbols" warning and a parse_error row, `codegraph status` listed it among the files it could not parse, and the native kernel deferred every file with such a line to wasm. The TypeScript extractor's preParse (.ts and .tsx) now blanks that `type` to four spaces where a statement starts (a line start, or after `;`, `{`, `}` or a block comment) when `*` is followed by `from` or `as`, so the line reads as the `export *` it is to the graph. Offsets are kept. The kernel route applies the preParse before calling the kernel, so both engines parse the same bytes and no Rust change is needed. A grammar patch would mean vendoring and regenerating the grammar for both dialects, two wasm rebuilds and a kernel off the crates.io crate, for a statement `export *` already models. A literal pre-check skips other files: 16 ms over strapi's 4,659 TS files. The only node change is an import node's stored statement text, which now reads `export * from './x'`. Import nodes are filtered from search, explore and the context builder, and nothing in resolution reads it. status-index-health.test.ts used `export type * from` as its unparseable file (issue #2336); it now uses a file with merge-conflict markers. Validation: - New ts-export-type-star.test.ts, 15 tests: 13 fail with main's typescript.ts, 11 with the blank unwired, all pass with the fix (kernel and wasm, .ts and .tsx, LF and CRLF). - A/B, kernel engine, main vs fix, dumps by natural keys: masto.js, kysely, TanStack/query, graffle and strapi keep every edge and ref (162,421 and 192,248 on strapi). The only changes are 883 import signatures, one per `export type *` statement, and 76 cleared warnings (masto.js 11, kysely 1, query 4, graffle 28, strapi 32). The Go and Python controls gin and requests are byte-identical. - Kernel parity, typescript and tsx: 0 diffs on every sweep. Deferrals drop masto.js 11 to 0, graffle 203 to 13, strapi 105 to 69, query 32 to 27, kysely 14 to 12; the rest are other grammar gaps. - Index time, median of 3 interleaved: graffle 12.7 s vs 13.1 s, query 11.2 s vs 11.4 s, within noise. Fixes #2474 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
51c4124dad
commit
cf3e6bc525
@@ -282,6 +282,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704)
|
||||
- An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project).
|
||||
- A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project.
|
||||
- In TypeScript, a file that re-exports a whole module's types, like `export type * from './types'` or `export type * as Types from './types'`, now parses cleanly. Before, the parser couldn't read that line, so a barrel made of re-exports was reported as producing no symbols and `codegraph status` listed it among the files it could not parse. Re-index a project that uses it after upgrading. Thanks @gavinwilliams for the report and @danusha2345 for the pull request. (#2474)
|
||||
- Git sync hooks now refresh the project they were installed for, so a project inside a larger repository stays current after commit, pull and checkout; installing them no longer breaks an existing hook written in another language, and runs even when your own hook ends with `exit`.
|
||||
- In TypeScript and JavaScript, a call through a field the calling class inherits, like `this._store.add(…)` in a class whose base class declares `_store`, now links to the method of the field's class, with the base class found the way the subclass's file imports it, under its own name, an alias or a re-export. Before, such a call linked only when some other class with the calling class's name happened to declare the field, which could be another app's class: that last resort now needs a type the calling file imports or declares. A call on a field or variable whose type comes from a package outside your project, like `TelemetryReporter` from `@vscode/extension-telemetry`, no longer links to a project class that only shares the type's name, unless a file in your project declares that package (`declare module '…'`). Re-index TypeScript and JavaScript projects after upgrading.
|
||||
- In TypeScript and JavaScript, a call on a variable, parameter or field whose class the file imports, like `store.add(…)` with `store: DisposableStore`, now links to the method of the class that import names when other classes in the project share its name, such as a vendored copy, a test mock or another app's class. Before, it linked to whichever of them was indexed first, or to the one in the nearest folder: in vscode, calls on classes imported from vscode's own `vs/base` and `vs/platform` reached the copies the Copilot extension vendors or its test fixtures, in bitwarden the desktop app's `initService.init()` reached the browser extension's `InitService`, and in react-native `Keyboard` and `WebSocket` reached the Jest mock of `NativeEventEmitter`. A call whose first match was a test file's class, which then linked to nothing, now reaches the imported class too. The import is followed through re-exports, `import type`, namespace imports like `new async.Delayer()` and `require(…)`. Re-index TypeScript and JavaScript projects after upgrading.
|
||||
|
||||
@@ -5,9 +5,9 @@
|
||||
* "Up to date" used to mean only "every file's content hash matches the
|
||||
* index". Two kinds of file pass that check with nothing usable in the graph:
|
||||
*
|
||||
* - a file the parser could not read — `export type * from` is valid
|
||||
* TypeScript 5.0 the bundled grammar has no rule for, so the file is stored
|
||||
* with a parse error and no symbols (#2336);
|
||||
* - a file the parser could not read, like one left with merge-conflict
|
||||
* markers, which is stored with a parse error and no symbols (#2336; the
|
||||
* report's file, TypeScript 5.0's `export type * from`, parses since #2474);
|
||||
* - a row stored without its symbols — by an engine whose grammar failed to
|
||||
* load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes.
|
||||
*
|
||||
@@ -96,7 +96,12 @@ describe('status reports files indexed without their symbols (#2336, #2335)', ()
|
||||
it('a file the parser could not read keeps status from reporting "up to date"', async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'src', 'index.ts'),
|
||||
'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n'
|
||||
'export * from "./constants/index.js";\n' +
|
||||
'<<<<<<< HEAD\n' +
|
||||
'export type { Row } from "./types/database.js";\n' +
|
||||
'=======\n' +
|
||||
'export type { RowId } from "./types/database.js";\n' +
|
||||
'>>>>>>> feature\n'
|
||||
);
|
||||
await index();
|
||||
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
/**
|
||||
* TypeScript 5.0's type-only re-export of a whole module parses (#2474):
|
||||
*
|
||||
* export { FooClient, type FooClientOptions } from "./client.js";
|
||||
* export type * from "./types.js";
|
||||
* export type * as Shapes from "./shapes.js";
|
||||
*
|
||||
* The grammar has no rule for `export type *` (upstream's neither), so `type`
|
||||
* parsed as an error. A barrel of re-exports was then reported as producing no
|
||||
* symbols, `codegraph status` listed it among the files it could not parse,
|
||||
* and the native kernel handed every file holding such a line to the wasm
|
||||
* parser. The TypeScript extractor's preParse now blanks that `type` to spaces
|
||||
* where a statement starts, offsets kept, so the line reads as the `export *`
|
||||
* it is to the graph, and both engines parse the same bytes. The import
|
||||
* resolver reads the statement as written from the file, so a name imported
|
||||
* through the barrel still reaches the file that declares it.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach, afterAll } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { CodeGraph } from '../src';
|
||||
import { extractFromSource } from '../src/extraction';
|
||||
import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
|
||||
import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
|
||||
import { blankExportTypeStar } from '../src/extraction/languages/typescript';
|
||||
import type { ExtractionResult, Language } from '../src/types';
|
||||
|
||||
const KERNEL_PATH = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'codegraph-kernel',
|
||||
'prebuilds',
|
||||
`${process.platform}-${process.arch}`,
|
||||
'codegraph-kernel.node'
|
||||
);
|
||||
const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1';
|
||||
|
||||
/** The issue's barrel, plus the namespace form. */
|
||||
const BARREL = [
|
||||
'export { FooClient, FooApiError, type FooClientOptions } from "./client.js";',
|
||||
'export { DAYS_OF_WEEK } from "./types.js";',
|
||||
'export type * from "./types.js";',
|
||||
'export type * as Shapes from "./shapes.js";',
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
describe('blankExportTypeStar', () => {
|
||||
it('blanks the `type` of a type-only star re-export, keeping every offset', () => {
|
||||
const cases: Array<[string, string]> = [
|
||||
['export type * from "./types";', 'export * from "./types";'],
|
||||
['export type * as Types from "./types";', 'export * as Types from "./types";'],
|
||||
["export type * as 'my types' from './types';", "export * as 'my types' from './types';"],
|
||||
['export type *from"./types";', 'export *from"./types";'],
|
||||
['export\ttype\t*\tfrom "./types";', 'export\t \t*\tfrom "./types";'],
|
||||
// `from` on the next line.
|
||||
['export type *\n from "./types";', 'export *\n from "./types";'],
|
||||
// Indented in an ambient module, and after `{`, `;` or a block comment on the same line.
|
||||
['declare module "m" {\n export type * from "./y";\n}', 'declare module "m" {\n export * from "./y";\n}'],
|
||||
['declare module "m" { export type * from "./y"; }', 'declare module "m" { export * from "./y"; }'],
|
||||
['export * from "./a"; export type * from "./b";', 'export * from "./a"; export * from "./b";'],
|
||||
['/* types */ export type * from "./b";', '/* types */ export * from "./b";'],
|
||||
['export * from "./a";\r\nexport type * from "./b";\r\n', 'export * from "./a";\r\nexport * from "./b";\r\n'],
|
||||
];
|
||||
for (const [source, expected] of cases) {
|
||||
const blanked = blankExportTypeStar(source);
|
||||
expect(blanked).toBe(expected);
|
||||
expect(blanked).toHaveLength(source.length);
|
||||
}
|
||||
});
|
||||
|
||||
it('leaves everything else as written', () => {
|
||||
for (const source of [
|
||||
'export type Star = "*";',
|
||||
'export type { A } from "./a";',
|
||||
'export { type A } from "./a";',
|
||||
'export * from "./a";',
|
||||
'export * as ns from "./a";',
|
||||
// Not where a statement starts: a mention in a comment, a doc or a string.
|
||||
'// Re-export them with export type * from "./types".',
|
||||
'/**\n * export type * from "./types";\n */',
|
||||
'const hint = "export type * from \'./types\'";',
|
||||
]) {
|
||||
expect(blankExportTypeStar(source)).toBe(source);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
|
||||
|
||||
describe('a type-only star re-export parses in both engines', () => {
|
||||
let savedEnv: Record<string, string | undefined> = {};
|
||||
|
||||
beforeAll(async () => {
|
||||
await initGrammars();
|
||||
await loadGrammarsForLanguages(['typescript', 'tsx']);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const k of ENV_KEYS) {
|
||||
if (savedEnv[k] === undefined) delete process.env[k];
|
||||
else process.env[k] = savedEnv[k];
|
||||
}
|
||||
resetKernelForTests();
|
||||
});
|
||||
|
||||
function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult {
|
||||
if (backend === 'wasm') {
|
||||
process.env.CODEGRAPH_KERNEL = '0';
|
||||
return extractFromSource(file, source, language);
|
||||
}
|
||||
delete process.env.CODEGRAPH_KERNEL;
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
// Null when the kernel defers the file to the wasm parser, as it does every file with a parse error.
|
||||
const result = tryKernelExtract(file, source, language);
|
||||
expect(result, `kernel extraction of ${file}`).not.toBeNull();
|
||||
return result!;
|
||||
}
|
||||
|
||||
const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const);
|
||||
|
||||
for (const crlf of [false, true]) {
|
||||
const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s);
|
||||
for (const [file, language] of [
|
||||
['src/index.ts', 'typescript'],
|
||||
['src/index.tsx', 'tsx'],
|
||||
] as const) {
|
||||
it.each(backends)(`a barrel in ${file}${crlf ? ' (CRLF)' : ''}: %s`, (backend) => {
|
||||
const result = extract(backend, file, eol(BARREL), language);
|
||||
expect(result.errors).toEqual([]);
|
||||
// Each statement imports its module, as `export * from` does.
|
||||
expect(result.nodes.filter((n) => n.kind === 'import').map((n) => `${n.name}@${n.startLine}`)).toEqual([
|
||||
'./client.js@1',
|
||||
'./types.js@2',
|
||||
'./types.js@3',
|
||||
'./shapes.js@4',
|
||||
]);
|
||||
expect(result.unresolvedReferences.map((r) => `${r.referenceName}@${r.line}`).sort()).toEqual([
|
||||
'./client.js@1',
|
||||
'./shapes.js@4',
|
||||
'./types.js@2',
|
||||
'./types.js@3',
|
||||
'DAYS_OF_WEEK@2',
|
||||
'FooApiError@1',
|
||||
'FooClient@1',
|
||||
'FooClientOptions@1',
|
||||
]);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
it.each(backends)('in a declaration file\'s declare module body: %s', (backend) => {
|
||||
const source = 'declare module "sdk" {\n export type * from "./types";\n export function connect(): void;\n}\n';
|
||||
const result = extract(backend, 'typings/sdk.d.ts', source, 'typescript');
|
||||
expect(result.errors).toEqual([]);
|
||||
expect(result.nodes.filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.name}`)).toEqual([
|
||||
'import ./types',
|
||||
'function connect',
|
||||
]);
|
||||
});
|
||||
|
||||
it.runIf(kernelAvailable)('the kernel and the wasm extractor agree', () => {
|
||||
const canon = (r: ExtractionResult) => [
|
||||
...r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(),
|
||||
...r.unresolvedReferences.map((ref) => JSON.stringify(ref, Object.keys(ref).sort())).sort(),
|
||||
];
|
||||
for (const [file, language] of [
|
||||
['src/index.ts', 'typescript'],
|
||||
['src/index.tsx', 'tsx'],
|
||||
] as const) {
|
||||
expect(canon(extract('kernel', file, BARREL, language))).toEqual(canon(extract('wasm', file, BARREL, language)));
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('indexing a barrel with `export type *` (#2474)', () => {
|
||||
let root = '';
|
||||
let cg: CodeGraph;
|
||||
|
||||
beforeAll(async () => {
|
||||
root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-export-type-star-'));
|
||||
const files: Record<string, string> = {
|
||||
'packages/sdk/src/client.ts':
|
||||
'export interface FooClientOptions { baseUrl: string }\n' +
|
||||
'export class FooApiError extends Error {}\n' +
|
||||
'export class FooClient {\n' +
|
||||
' constructor(private opts: FooClientOptions) {}\n' +
|
||||
' get(path: string) { return this.opts.baseUrl + path; }\n' +
|
||||
'}\n',
|
||||
'packages/sdk/src/types.ts': "export const DAYS_OF_WEEK = ['mon', 'tue'];\nexport type Schedule = { day: string };\n",
|
||||
'packages/sdk/src/shapes.ts': 'export interface Circle { radius: number }\n',
|
||||
'packages/sdk/src/index.ts': BARREL,
|
||||
// Same names, declared where nothing re-exports them: what a name match could pick.
|
||||
'app/shapes.ts': 'export interface FooClientOptions { decoy: true }\nexport type Schedule = { decoy: true };\n',
|
||||
'app/main.ts':
|
||||
'import { FooClient, type FooClientOptions } from "../packages/sdk/src/index.js";\n' +
|
||||
'import type { Schedule } from "../packages/sdk/src/index.js";\n' +
|
||||
'export function run(o: FooClientOptions, s: Schedule) {\n' +
|
||||
' return new FooClient(o).get(s.day);\n' +
|
||||
'}\n',
|
||||
};
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true });
|
||||
fs.writeFileSync(path.join(root, rel), content);
|
||||
}
|
||||
cg = await CodeGraph.init(root, { index: true });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cg?.close();
|
||||
if (root) fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('stores the barrel without a parse error', () => {
|
||||
const barrel = cg.getFile('packages/sdk/src/index.ts');
|
||||
expect(barrel).not.toBeNull();
|
||||
expect(barrel!.errors ?? []).toEqual([]);
|
||||
});
|
||||
|
||||
it('links each name imported through the barrel to the file that declares it', () => {
|
||||
const ids = cg.getNodesInFile('app/main.ts').map((n) => n.id);
|
||||
const imported = cg
|
||||
.getOutgoingEdgesFrom(ids, ['imports'])
|
||||
.map((e) => cg.getNode(e.target)!)
|
||||
.filter((t) => t.kind !== 'file')
|
||||
.map((t) => `${t.name} -> ${t.filePath}`)
|
||||
.sort();
|
||||
expect(imported).toEqual([
|
||||
'FooClient -> packages/sdk/src/client.ts',
|
||||
'FooClientOptions -> packages/sdk/src/client.ts',
|
||||
// Reached only through `export type * from "./types.js"`.
|
||||
'Schedule -> packages/sdk/src/types.ts',
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -69,6 +69,32 @@ export function blankFlowSyntax(source: string, filePath?: string): string {
|
||||
.replace(/\.\.\.(?=\s*[},])/g, ' ');
|
||||
}
|
||||
|
||||
/** Whether the file writes `export type *` anywhere: most files don't, and skip the rest. */
|
||||
const EXPORT_TYPE_STAR_HINT = /export\s+type\s*\*/;
|
||||
/**
|
||||
* `export type *` where a statement starts (a line's first word, or after `;`,
|
||||
* `{`, `}` or the end of a block comment), followed by `from` or `as`.
|
||||
*/
|
||||
const EXPORT_TYPE_STAR = /(^|[;{}]|\*\/)([ \t]*export\s+)type(?=\s*\*\s*(?:from|as)\b)/gm;
|
||||
|
||||
/**
|
||||
* TypeScript 5.0's type-only re-export of a whole module, `export type * from
|
||||
* './types'` and `export type * as ns from './types'`, with its `type` blanked
|
||||
* to spaces (offsets survive), so it reads as the `export *` it is to the
|
||||
* graph. The grammar has no rule for it (nor does upstream's): `type` parsed
|
||||
* as an error, so a barrel of re-exports was reported as producing no symbols
|
||||
* and counted by `codegraph status` as a file it could not parse, and the
|
||||
* kernel handed every such file to the wasm parser (#2474). Only a statement
|
||||
* start is blanked, so a mention in a comment or a doc keeps its text. The
|
||||
* import resolver reads the statement as written, `type` included, from the
|
||||
* file itself (`extractReExports`). Applied before both engines parse (the
|
||||
* kernel's preParse hoist), so they agree.
|
||||
*/
|
||||
export function blankExportTypeStar(source: string): string {
|
||||
if (!EXPORT_TYPE_STAR_HINT.test(source)) return source;
|
||||
return source.replace(EXPORT_TYPE_STAR, '$1$2 ');
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether `node` is declared in a `declare module 'x' { … }` or `declare
|
||||
* global { … }` body, which exports it without an `export` keyword.
|
||||
@@ -147,7 +173,7 @@ export function isAmbientFunctionSignature(node: SyntaxNode, filePath: string):
|
||||
}
|
||||
|
||||
export const typescriptExtractor: LanguageExtractor = {
|
||||
preParse: blankFlowSyntax,
|
||||
preParse: (source, filePath) => blankExportTypeStar(blankFlowSyntax(source, filePath)),
|
||||
functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'],
|
||||
isAmbientFunction: isAmbientFunctionSignature,
|
||||
// `class` is a class expression, `register(class Widget extends Base {…})`:
|
||||
|
||||
Reference in New Issue
Block a user