From cf3e6bc525743de7a5ee771ae26448b796a6d9fa Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 09:09:05 +0000 Subject: [PATCH] fix(ts): a type-only re-export like export type * from './types' now parses (#2837) TypeScript 5.0's type-only re-export of a whole module, `export type * from './x'` and `export type * as ns from './x'`, has no rule in the vendored tree-sitter-typescript grammar (v0.23.2, the revision the kernel's crate compiles) or in upstream master: `type` parses as an ERROR inside the export statement. The graph was already right, since the import resolver reads the statement from the file (PRs #1955 and #2526), but the error tree meant a barrel of re-exports got the "parse produced no symbols" warning and a parse_error row, `codegraph status` listed it among the files it could not parse, and the native kernel deferred every file with such a line to wasm. The TypeScript extractor's preParse (.ts and .tsx) now blanks that `type` to four spaces where a statement starts (a line start, or after `;`, `{`, `}` or a block comment) when `*` is followed by `from` or `as`, so the line reads as the `export *` it is to the graph. Offsets are kept. The kernel route applies the preParse before calling the kernel, so both engines parse the same bytes and no Rust change is needed. A grammar patch would mean vendoring and regenerating the grammar for both dialects, two wasm rebuilds and a kernel off the crates.io crate, for a statement `export *` already models. A literal pre-check skips other files: 16 ms over strapi's 4,659 TS files. The only node change is an import node's stored statement text, which now reads `export * from './x'`. Import nodes are filtered from search, explore and the context builder, and nothing in resolution reads it. status-index-health.test.ts used `export type * from` as its unparseable file (issue #2336); it now uses a file with merge-conflict markers. Validation: - New ts-export-type-star.test.ts, 15 tests: 13 fail with main's typescript.ts, 11 with the blank unwired, all pass with the fix (kernel and wasm, .ts and .tsx, LF and CRLF). - A/B, kernel engine, main vs fix, dumps by natural keys: masto.js, kysely, TanStack/query, graffle and strapi keep every edge and ref (162,421 and 192,248 on strapi). The only changes are 883 import signatures, one per `export type *` statement, and 76 cleared warnings (masto.js 11, kysely 1, query 4, graffle 28, strapi 32). The Go and Python controls gin and requests are byte-identical. - Kernel parity, typescript and tsx: 0 diffs on every sweep. Deferrals drop masto.js 11 to 0, graffle 203 to 13, strapi 105 to 69, query 32 to 27, kysely 14 to 12; the rest are other grammar gaps. - Index time, median of 3 interleaved: graffle 12.7 s vs 13.1 s, query 11.2 s vs 11.4 s, within noise. Fixes #2474 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/status-index-health.test.ts | 13 +- __tests__/ts-export-type-star.test.ts | 240 +++++++++++++++++++++++++ src/extraction/languages/typescript.ts | 28 ++- 4 files changed, 277 insertions(+), 5 deletions(-) create mode 100644 __tests__/ts-export-type-star.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b879cfdd..298b336b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -282,6 +282,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704) - An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project). - A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project. +- In TypeScript, a file that re-exports a whole module's types, like `export type * from './types'` or `export type * as Types from './types'`, now parses cleanly. Before, the parser couldn't read that line, so a barrel made of re-exports was reported as producing no symbols and `codegraph status` listed it among the files it could not parse. Re-index a project that uses it after upgrading. Thanks @gavinwilliams for the report and @danusha2345 for the pull request. (#2474) - Git sync hooks now refresh the project they were installed for, so a project inside a larger repository stays current after commit, pull and checkout; installing them no longer breaks an existing hook written in another language, and runs even when your own hook ends with `exit`. - In TypeScript and JavaScript, a call through a field the calling class inherits, like `this._store.add(…)` in a class whose base class declares `_store`, now links to the method of the field's class, with the base class found the way the subclass's file imports it, under its own name, an alias or a re-export. Before, such a call linked only when some other class with the calling class's name happened to declare the field, which could be another app's class: that last resort now needs a type the calling file imports or declares. A call on a field or variable whose type comes from a package outside your project, like `TelemetryReporter` from `@vscode/extension-telemetry`, no longer links to a project class that only shares the type's name, unless a file in your project declares that package (`declare module '…'`). Re-index TypeScript and JavaScript projects after upgrading. - In TypeScript and JavaScript, a call on a variable, parameter or field whose class the file imports, like `store.add(…)` with `store: DisposableStore`, now links to the method of the class that import names when other classes in the project share its name, such as a vendored copy, a test mock or another app's class. Before, it linked to whichever of them was indexed first, or to the one in the nearest folder: in vscode, calls on classes imported from vscode's own `vs/base` and `vs/platform` reached the copies the Copilot extension vendors or its test fixtures, in bitwarden the desktop app's `initService.init()` reached the browser extension's `InitService`, and in react-native `Keyboard` and `WebSocket` reached the Jest mock of `NativeEventEmitter`. A call whose first match was a test file's class, which then linked to nothing, now reaches the imported class too. The import is followed through re-exports, `import type`, namespace imports like `new async.Delayer()` and `require(…)`. Re-index TypeScript and JavaScript projects after upgrading. diff --git a/__tests__/status-index-health.test.ts b/__tests__/status-index-health.test.ts index b94562c8..7a5571e1 100644 --- a/__tests__/status-index-health.test.ts +++ b/__tests__/status-index-health.test.ts @@ -5,9 +5,9 @@ * "Up to date" used to mean only "every file's content hash matches the * index". Two kinds of file pass that check with nothing usable in the graph: * - * - a file the parser could not read — `export type * from` is valid - * TypeScript 5.0 the bundled grammar has no rule for, so the file is stored - * with a parse error and no symbols (#2336); + * - a file the parser could not read, like one left with merge-conflict + * markers, which is stored with a parse error and no symbols (#2336; the + * report's file, TypeScript 5.0's `export type * from`, parses since #2474); * - a row stored without its symbols — by an engine whose grammar failed to * load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes. * @@ -96,7 +96,12 @@ describe('status reports files indexed without their symbols (#2336, #2335)', () it('a file the parser could not read keeps status from reporting "up to date"', async () => { fs.writeFileSync( path.join(dir, 'src', 'index.ts'), - 'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n' + 'export * from "./constants/index.js";\n' + + '<<<<<<< HEAD\n' + + 'export type { Row } from "./types/database.js";\n' + + '=======\n' + + 'export type { RowId } from "./types/database.js";\n' + + '>>>>>>> feature\n' ); await index(); diff --git a/__tests__/ts-export-type-star.test.ts b/__tests__/ts-export-type-star.test.ts new file mode 100644 index 00000000..eec3300f --- /dev/null +++ b/__tests__/ts-export-type-star.test.ts @@ -0,0 +1,240 @@ +/** + * TypeScript 5.0's type-only re-export of a whole module parses (#2474): + * + * export { FooClient, type FooClientOptions } from "./client.js"; + * export type * from "./types.js"; + * export type * as Shapes from "./shapes.js"; + * + * The grammar has no rule for `export type *` (upstream's neither), so `type` + * parsed as an error. A barrel of re-exports was then reported as producing no + * symbols, `codegraph status` listed it among the files it could not parse, + * and the native kernel handed every file holding such a line to the wasm + * parser. The TypeScript extractor's preParse now blanks that `type` to spaces + * where a statement starts, offsets kept, so the line reads as the `export *` + * it is to the graph, and both engines parse the same bytes. The import + * resolver reads the statement as written from the file, so a name imported + * through the barrel still reaches the file that declares it. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import { blankExportTypeStar } from '../src/extraction/languages/typescript'; +import type { ExtractionResult, Language } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +/** The issue's barrel, plus the namespace form. */ +const BARREL = [ + 'export { FooClient, FooApiError, type FooClientOptions } from "./client.js";', + 'export { DAYS_OF_WEEK } from "./types.js";', + 'export type * from "./types.js";', + 'export type * as Shapes from "./shapes.js";', + '', +].join('\n'); + +describe('blankExportTypeStar', () => { + it('blanks the `type` of a type-only star re-export, keeping every offset', () => { + const cases: Array<[string, string]> = [ + ['export type * from "./types";', 'export * from "./types";'], + ['export type * as Types from "./types";', 'export * as Types from "./types";'], + ["export type * as 'my types' from './types';", "export * as 'my types' from './types';"], + ['export type *from"./types";', 'export *from"./types";'], + ['export\ttype\t*\tfrom "./types";', 'export\t \t*\tfrom "./types";'], + // `from` on the next line. + ['export type *\n from "./types";', 'export *\n from "./types";'], + // Indented in an ambient module, and after `{`, `;` or a block comment on the same line. + ['declare module "m" {\n export type * from "./y";\n}', 'declare module "m" {\n export * from "./y";\n}'], + ['declare module "m" { export type * from "./y"; }', 'declare module "m" { export * from "./y"; }'], + ['export * from "./a"; export type * from "./b";', 'export * from "./a"; export * from "./b";'], + ['/* types */ export type * from "./b";', '/* types */ export * from "./b";'], + ['export * from "./a";\r\nexport type * from "./b";\r\n', 'export * from "./a";\r\nexport * from "./b";\r\n'], + ]; + for (const [source, expected] of cases) { + const blanked = blankExportTypeStar(source); + expect(blanked).toBe(expected); + expect(blanked).toHaveLength(source.length); + } + }); + + it('leaves everything else as written', () => { + for (const source of [ + 'export type Star = "*";', + 'export type { A } from "./a";', + 'export { type A } from "./a";', + 'export * from "./a";', + 'export * as ns from "./a";', + // Not where a statement starts: a mention in a comment, a doc or a string. + '// Re-export them with export type * from "./types".', + '/**\n * export type * from "./types";\n */', + 'const hint = "export type * from \'./types\'";', + ]) { + expect(blankExportTypeStar(source)).toBe(source); + } + }); +}); + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a type-only star re-export parses in both engines', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['typescript', 'tsx']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource(file, source, language); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + // Null when the kernel defers the file to the wasm parser, as it does every file with a parse error. + const result = tryKernelExtract(file, source, language); + expect(result, `kernel extraction of ${file}`).not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s); + for (const [file, language] of [ + ['src/index.ts', 'typescript'], + ['src/index.tsx', 'tsx'], + ] as const) { + it.each(backends)(`a barrel in ${file}${crlf ? ' (CRLF)' : ''}: %s`, (backend) => { + const result = extract(backend, file, eol(BARREL), language); + expect(result.errors).toEqual([]); + // Each statement imports its module, as `export * from` does. + expect(result.nodes.filter((n) => n.kind === 'import').map((n) => `${n.name}@${n.startLine}`)).toEqual([ + './client.js@1', + './types.js@2', + './types.js@3', + './shapes.js@4', + ]); + expect(result.unresolvedReferences.map((r) => `${r.referenceName}@${r.line}`).sort()).toEqual([ + './client.js@1', + './shapes.js@4', + './types.js@2', + './types.js@3', + 'DAYS_OF_WEEK@2', + 'FooApiError@1', + 'FooClient@1', + 'FooClientOptions@1', + ]); + }); + } + } + + it.each(backends)('in a declaration file\'s declare module body: %s', (backend) => { + const source = 'declare module "sdk" {\n export type * from "./types";\n export function connect(): void;\n}\n'; + const result = extract(backend, 'typings/sdk.d.ts', source, 'typescript'); + expect(result.errors).toEqual([]); + expect(result.nodes.filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.name}`)).toEqual([ + 'import ./types', + 'function connect', + ]); + }); + + it.runIf(kernelAvailable)('the kernel and the wasm extractor agree', () => { + const canon = (r: ExtractionResult) => [ + ...r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(), + ...r.unresolvedReferences.map((ref) => JSON.stringify(ref, Object.keys(ref).sort())).sort(), + ]; + for (const [file, language] of [ + ['src/index.ts', 'typescript'], + ['src/index.tsx', 'tsx'], + ] as const) { + expect(canon(extract('kernel', file, BARREL, language))).toEqual(canon(extract('wasm', file, BARREL, language))); + } + }); +}); + +describe('indexing a barrel with `export type *` (#2474)', () => { + let root = ''; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-export-type-star-')); + const files: Record = { + 'packages/sdk/src/client.ts': + 'export interface FooClientOptions { baseUrl: string }\n' + + 'export class FooApiError extends Error {}\n' + + 'export class FooClient {\n' + + ' constructor(private opts: FooClientOptions) {}\n' + + ' get(path: string) { return this.opts.baseUrl + path; }\n' + + '}\n', + 'packages/sdk/src/types.ts': "export const DAYS_OF_WEEK = ['mon', 'tue'];\nexport type Schedule = { day: string };\n", + 'packages/sdk/src/shapes.ts': 'export interface Circle { radius: number }\n', + 'packages/sdk/src/index.ts': BARREL, + // Same names, declared where nothing re-exports them: what a name match could pick. + 'app/shapes.ts': 'export interface FooClientOptions { decoy: true }\nexport type Schedule = { decoy: true };\n', + 'app/main.ts': + 'import { FooClient, type FooClientOptions } from "../packages/sdk/src/index.js";\n' + + 'import type { Schedule } from "../packages/sdk/src/index.js";\n' + + 'export function run(o: FooClientOptions, s: Schedule) {\n' + + ' return new FooClient(o).get(s.day);\n' + + '}\n', + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + }); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + it('stores the barrel without a parse error', () => { + const barrel = cg.getFile('packages/sdk/src/index.ts'); + expect(barrel).not.toBeNull(); + expect(barrel!.errors ?? []).toEqual([]); + }); + + it('links each name imported through the barrel to the file that declares it', () => { + const ids = cg.getNodesInFile('app/main.ts').map((n) => n.id); + const imported = cg + .getOutgoingEdgesFrom(ids, ['imports']) + .map((e) => cg.getNode(e.target)!) + .filter((t) => t.kind !== 'file') + .map((t) => `${t.name} -> ${t.filePath}`) + .sort(); + expect(imported).toEqual([ + 'FooClient -> packages/sdk/src/client.ts', + 'FooClientOptions -> packages/sdk/src/client.ts', + // Reached only through `export type * from "./types.js"`. + 'Schedule -> packages/sdk/src/types.ts', + ]); + }); +}); diff --git a/src/extraction/languages/typescript.ts b/src/extraction/languages/typescript.ts index ec28e1e8..5b435d91 100644 --- a/src/extraction/languages/typescript.ts +++ b/src/extraction/languages/typescript.ts @@ -69,6 +69,32 @@ export function blankFlowSyntax(source: string, filePath?: string): string { .replace(/\.\.\.(?=\s*[},])/g, ' '); } +/** Whether the file writes `export type *` anywhere: most files don't, and skip the rest. */ +const EXPORT_TYPE_STAR_HINT = /export\s+type\s*\*/; +/** + * `export type *` where a statement starts (a line's first word, or after `;`, + * `{`, `}` or the end of a block comment), followed by `from` or `as`. + */ +const EXPORT_TYPE_STAR = /(^|[;{}]|\*\/)([ \t]*export\s+)type(?=\s*\*\s*(?:from|as)\b)/gm; + +/** + * TypeScript 5.0's type-only re-export of a whole module, `export type * from + * './types'` and `export type * as ns from './types'`, with its `type` blanked + * to spaces (offsets survive), so it reads as the `export *` it is to the + * graph. The grammar has no rule for it (nor does upstream's): `type` parsed + * as an error, so a barrel of re-exports was reported as producing no symbols + * and counted by `codegraph status` as a file it could not parse, and the + * kernel handed every such file to the wasm parser (#2474). Only a statement + * start is blanked, so a mention in a comment or a doc keeps its text. The + * import resolver reads the statement as written, `type` included, from the + * file itself (`extractReExports`). Applied before both engines parse (the + * kernel's preParse hoist), so they agree. + */ +export function blankExportTypeStar(source: string): string { + if (!EXPORT_TYPE_STAR_HINT.test(source)) return source; + return source.replace(EXPORT_TYPE_STAR, '$1$2 '); +} + /** * Whether `node` is declared in a `declare module 'x' { … }` or `declare * global { … }` body, which exports it without an `export` keyword. @@ -147,7 +173,7 @@ export function isAmbientFunctionSignature(node: SyntaxNode, filePath: string): } export const typescriptExtractor: LanguageExtractor = { - preParse: blankFlowSyntax, + preParse: (source, filePath) => blankExportTypeStar(blankFlowSyntax(source, filePath)), functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'], isAmbientFunction: isAmbientFunctionSignature, // `class` is a class expression, `register(class Widget extends Base {…})`: