mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-28 22:52:00 +08:00
fix(python): resolve imports from the package roots the project has (#1921)
Python imports now resolve the way the interpreter finds modules: from the package roots the project has, not by any file whose path merely ends in the module's name. A project root is the repository, each service with its own pyproject.toml, setup.cfg or setup.py, its src/, and the package directories its build config declares; a local root answers only importers within its reach, so a test fixture's copy of a package never becomes the application's. Relative imports climb from the importer's own package. Before, `import json` or `from datetime import date` could link to a project file or function of that name, and a src-layout package like flask's was found through a test fixture named flask.py. On flask, fastapi, mealie and django, relative imports now resolve (about 2,200 links added), stdlib and third-party names stop linking to project namesakes (about 1,300 wrong links removed), and imports of a module's function land on the function rather than its file. A sync re-opens what an added or removed module, an edited __init__.py or a changed package root can move. Indexing CPU time is unchanged. Merged with main: a function-local `import a.b as X` (PR 2607) now uses the root-aware lookup, the outside-module receiver check (PR 2606) passes the importer, and plain imports are read at module level only, since the mappings apply to every function of the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
505c355be3
commit
6fe9dc1266
@@ -204,6 +204,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- In React with TypeScript, a component that renders a generic component and passes it type arguments, like outline's `<PaginatedList<Document> items={…} />` or excalidraw's `<DropdownMenuItemContentRadio<"contain" | "overlap"> …>`, now links to the component it renders. Before, a tag followed by type arguments was skipped, so the generic component looked unused by the screens that render it, and `codegraph_explore`'s call paths, callers and impact stopped short of it. Re-index React projects after upgrading.
|
||||
- In React, a component that a file declares itself as a value is now what that file's JSX renders, whatever else in the repository shares its name. A lazily loaded page, like `const RegisterPage = Loadable(lazy(() => import('./pages/Register')))`, a `dynamic(() => import(…))` component or one loaded only in development (`IS_DEV ? lazy(…) : null`), links to the component its module exports, including the one its loader picks and one an `export default observer(Login)` wraps; a wrapped component like `const Settings = observer(function Settings() {…})`, `React.memo(Rows, areEqual)` or `forwardRef(ComboboxInner) as {…}` links to the function it wraps; and an alias like `const DialogPortal = DialogPrimitive.Portal` links to its own declaration. Before, `<RegisterPage />` was linked to any lone component of that name elsewhere, often in another app of the same repository, so `codegraph_explore`, callers and impact followed it into the wrong app while the page it really renders had no caller. React Router layouts declared this way follow the same rule, and a TSX type named like a value the file declares now stays in that file. Re-index React projects after upgrading.
|
||||
- In React, a JSX tag imported from one of your own modules now links to the component that module exports under that name, read the way a lazily loaded page's module already is. That includes going through an `index` file or barrel that passes the component on, like outline's `Sidebar/index.ts` with `import Sidebar from './App'; export default Sidebar` or its `Avatar/index.ts` with `const Avatar = AvatarWithHoverCard`, and past a wrapper such as `export default observer(AppSidebar)`, `React.memo(StaticCanvas, areEqual)` or mastodon's `const MemoizedBlurhash = memo(Blurhash)`. Before, a tag was matched by its name and the import only broke ties, so outline's `<Sidebar>` linked to an unrelated styled `Sidebar` in a sibling file, a default import under another name linked to whatever other component its module exported, and an import through a barrel could land on another app's component of the same name. A tag whose module exports an unnamed component, like `export default () => …`, now links to nothing instead of to a namesake elsewhere. A lazily loaded React Router page exported as such a value or wrapper links to the component it holds too, the one with the markup. `codegraph sync` and the file watcher redraw these links when only the file that passes the component on changes. Re-index React projects after upgrading.
|
||||
- Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704)
|
||||
- An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project).
|
||||
- A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project.
|
||||
- Git sync hooks now refresh the project they were installed for, so a project inside a larger repository stays current after commit, pull and checkout; installing them no longer breaks an existing hook written in another language, and runs even when your own hook ends with `exit`.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -4490,6 +4490,211 @@ export class QueryBuilder {
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Every resolution edge leaving a node of `language` — the re-resolution set
|
||||
* when a project-wide input to that language's import resolution changes
|
||||
* (Python package roots). Same exclusion as
|
||||
* {@link getResolutionEdgesByTargetName}: synthesized edges carry no refName
|
||||
* stamp and are left alone.
|
||||
*/
|
||||
getResolutionEdgesBySourceLanguage(
|
||||
language: Language
|
||||
): Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT e.*, src.file_path AS source_file_path, src.language AS source_language
|
||||
FROM edges e
|
||||
JOIN nodes src ON src.id = e.source
|
||||
WHERE src.language = ?
|
||||
AND e.kind != 'contains'
|
||||
AND (e.provenance IS NULL OR e.provenance != 'heuristic')`
|
||||
)
|
||||
.all(language) as Array<EdgeRow & { source_file_path: string; source_language: Language }>;
|
||||
return rows.map((row) => ({
|
||||
...rowToEdge(row),
|
||||
edgeId: row.id,
|
||||
sourceFilePath: row.source_file_path,
|
||||
sourceLanguage: row.source_language,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolution edges leaving a `language` node whose TARGET is in one of
|
||||
* `targetFiles`, or whose SOURCE is under one of `sourceDirs` — the narrow
|
||||
* re-resolution set when modules or packages appear or vanish below an
|
||||
* unchanged set of Python roots.
|
||||
*/
|
||||
getResolutionEdgesTouchingFiles(
|
||||
language: Language,
|
||||
targetFiles: string[],
|
||||
sourceDirs: string[],
|
||||
importNodeSegments: string[] = []
|
||||
): Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
// Collect candidate edge ids through narrow indexed lookups, then load
|
||||
// the full rows edge-first (CROSS JOIN pins the join order; left to
|
||||
// itself the planner drove the query from the language index and one
|
||||
// rename took 18 s).
|
||||
const candidates = new Set<number>();
|
||||
const idsWhere = (column: 'source' | 'target', nodeIds: string[]): void => {
|
||||
for (let i = 0; i < nodeIds.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = nodeIds.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const rows = this.db
|
||||
.prepare(`SELECT id FROM edges WHERE ${column} IN (${chunk.map(() => '?').join(',')})`)
|
||||
.all(...chunk) as Array<{ id: number }>;
|
||||
for (const r of rows) candidates.add(r.id);
|
||||
}
|
||||
};
|
||||
// Edges into the affected module files.
|
||||
for (let i = 0; i < targetFiles.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = targetFiles.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const nodes = this.db
|
||||
.prepare(`SELECT id FROM nodes WHERE file_path IN (${chunk.map(() => '?').join(',')})`)
|
||||
.all(...chunk) as Array<{ id: string }>;
|
||||
idsWhere('target', nodes.map((n) => n.id));
|
||||
}
|
||||
// Edges out of a changed package: its nodes by an index range on the
|
||||
// path (`dir/` up to `dir0`, the character after '/').
|
||||
for (const dir of sourceDirs) {
|
||||
const nodes = (dir === ''
|
||||
? this.db.prepare('SELECT id FROM nodes WHERE language = ?').all(language)
|
||||
: this.db
|
||||
.prepare('SELECT id FROM nodes WHERE file_path >= ? AND file_path < ?')
|
||||
.all(`${dir}/`, `${dir}0`)) as Array<{ id: string }>;
|
||||
idsWhere('source', nodes.map((n) => n.id));
|
||||
}
|
||||
// Imports the resolver couldn't place and bound to their own `import`
|
||||
// node rather than parking as a failure, matched by the module name they
|
||||
// spell: ONE read of the language's import nodes, filtered here. (A LIKE
|
||||
// scan of the edges per name measured ~225 ms each on django's 213k.)
|
||||
if (importNodeSegments.length > 0) {
|
||||
const wanted = new Set(importNodeSegments);
|
||||
const nodes = this.db
|
||||
.prepare("SELECT id, name FROM nodes WHERE kind = 'import' AND language = ?")
|
||||
.all(language) as Array<{ id: string; name: string }>;
|
||||
idsWhere('target', nodes.filter((n) => n.name.split('.').some((seg) => wanted.has(seg))).map((n) => n.id));
|
||||
}
|
||||
return this.loadResolutionEdges(language, [...candidates]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolution edges leaving any node of `files` — the candidates when an
|
||||
* importer's bindings may have moved (a Python package's `__init__.py`
|
||||
* edited). The caller narrows them by the names each file binds.
|
||||
*/
|
||||
getResolutionEdgesFromFiles(
|
||||
language: Language,
|
||||
files: string[]
|
||||
): Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
const ids: number[] = [];
|
||||
for (let i = 0; i < files.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = files.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT e.id AS id FROM nodes n CROSS JOIN edges e ON e.source = n.id
|
||||
WHERE n.file_path IN (${chunk.map(() => '?').join(',')})`
|
||||
)
|
||||
.all(...chunk) as Array<{ id: number }>;
|
||||
for (const r of rows) ids.push(r.id);
|
||||
}
|
||||
return this.loadResolutionEdges(language, ids);
|
||||
}
|
||||
|
||||
/**
|
||||
* Full rows for resolution edges by id, loaded edge-first (CROSS JOIN pins
|
||||
* the join order; left to itself the planner drove the query from the
|
||||
* language index and one rename took 18 s).
|
||||
*/
|
||||
private loadResolutionEdges(
|
||||
language: Language,
|
||||
ids: number[]
|
||||
): Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
type Row = EdgeRow & { source_file_path: string; source_language: Language };
|
||||
const out: Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> = [];
|
||||
for (let i = 0; i < ids.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = ids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT e.*, src.file_path AS source_file_path, src.language AS source_language
|
||||
FROM edges e
|
||||
CROSS JOIN nodes src ON src.id = e.source
|
||||
CROSS JOIN nodes tgt ON tgt.id = e.target
|
||||
WHERE e.id IN (${chunk.map(() => '?').join(',')})
|
||||
AND src.language = ?
|
||||
AND e.kind != 'contains'
|
||||
AND (e.provenance IS NULL OR e.provenance != 'heuristic')`
|
||||
)
|
||||
.all(...chunk, language) as Row[];
|
||||
for (const row of rows) {
|
||||
out.push({ ...rowToEdge(row), edgeId: row.id, sourceFilePath: row.source_file_path, sourceLanguage: row.source_language });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-queue a language's parked failures in `names`' files whose reference
|
||||
* starts with one of the names bound there (`N`, `N.run`).
|
||||
*/
|
||||
requeueFailedReferencesBinding(language: Language, names: Map<string, Set<string>>): number {
|
||||
const files = [...names.keys()];
|
||||
const ids: number[] = [];
|
||||
for (let i = 0; i < files.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = files.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT id, file_path, reference_name FROM unresolved_refs
|
||||
WHERE status = 'failed' AND language = ? AND file_path IN (${chunk.map(() => '?').join(',')})`
|
||||
)
|
||||
.all(language, ...chunk) as Array<{ id: number; file_path: string; reference_name: string }>;
|
||||
for (const r of rows) {
|
||||
if (names.get(r.file_path)?.has(r.reference_name.split('.')[0]!)) ids.push(r.id);
|
||||
}
|
||||
}
|
||||
let changed = 0;
|
||||
this.db.transaction(() => {
|
||||
for (let i = 0; i < ids.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = ids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
changed += this.db
|
||||
.prepare(`UPDATE unresolved_refs SET status = 'pending' WHERE id IN (${chunk.map(() => '?').join(',')})`)
|
||||
.run(...chunk).changes;
|
||||
}
|
||||
})();
|
||||
return changed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return a language's parked (status='failed') refs to the pending set —
|
||||
* all of them, or only those naming one of `segments` as a dotted part
|
||||
* (`x`, `a.x`, `x.b`, `.x`).
|
||||
*/
|
||||
requeueFailedReferencesByLanguage(language: Language, segments?: string[], files?: Set<string>): number {
|
||||
const sql = "UPDATE unresolved_refs SET status = 'pending' WHERE status = 'failed' AND language = ?";
|
||||
if (!segments) return this.db.prepare(sql).run(language).changes;
|
||||
if (segments.length === 0) return 0;
|
||||
// One read of the parked rows, matched here, then an update by id.
|
||||
const wanted = new Set(segments);
|
||||
const ids = (
|
||||
this.db
|
||||
.prepare("SELECT id, reference_name, file_path FROM unresolved_refs WHERE status = 'failed' AND language = ?")
|
||||
.all(language) as Array<{ id: number; reference_name: string; file_path: string }>
|
||||
)
|
||||
.filter((r) => r.reference_name.split('.').some((seg) => wanted.has(seg)))
|
||||
.map((r) => {
|
||||
files?.add(r.file_path);
|
||||
return r.id;
|
||||
});
|
||||
let changed = 0;
|
||||
this.db.transaction(() => {
|
||||
for (let i = 0; i < ids.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = ids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
changed += this.db
|
||||
.prepare(`UPDATE unresolved_refs SET status = 'pending' WHERE id IN (${chunk.map(() => '?').join(',')})`)
|
||||
.run(...chunk).changes;
|
||||
}
|
||||
})();
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** Delete edges by primary key — the rebind pass's half of a re-resolution. */
|
||||
deleteEdgesByIds(edgeIds: number[]): number {
|
||||
if (edgeIds.length === 0) return 0;
|
||||
|
||||
@@ -3337,6 +3337,97 @@ export class ExtractionOrchestrator {
|
||||
return refs.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-open every resolution a language made, for the sync's sweep to redo
|
||||
* against the current graph — the answer when an input to that language's
|
||||
* resolution changed project-wide rather than per definition. Python
|
||||
* package roots are that kind of input: an added `__init__.py` or an edited
|
||||
* `pyproject.toml` moves where modules start for importers in files the
|
||||
* sync never touched, and CG-33's per-name rebind can't see it. Parked
|
||||
* failures are re-queued too, since some of them now resolve. Same
|
||||
* conservatism as {@link resurrectStaleResolutionEdges}: an edge without a
|
||||
* refName stamp is never deleted.
|
||||
*/
|
||||
resurrectResolutionEdgesForLanguage(language: Language): number {
|
||||
return (
|
||||
this.reopenEdges(this.queries.getResolutionEdgesBySourceLanguage(language)) +
|
||||
this.queries.requeueFailedReferencesByLanguage(language)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The narrow form, for modules or packages that appeared or vanished below
|
||||
* an unchanged set of roots: re-open the edges whose target lies in
|
||||
* `targetFiles` or whose source lies under `sourceDirs`, and re-queue the
|
||||
* parked failures naming one of `moduleLeaves`. Returns the files whose
|
||||
* references were re-opened.
|
||||
*/
|
||||
resurrectResolutionEdgesTouching(
|
||||
language: Language,
|
||||
scope: { targetFiles: string[]; sourceDirs: string[]; moduleLeaves: string[] }
|
||||
): string[] {
|
||||
const files = new Set<string>();
|
||||
const edges = this.queries.getResolutionEdgesTouchingFiles(language, scope.targetFiles, scope.sourceDirs, scope.moduleLeaves);
|
||||
for (const e of edges) files.add(e.sourceFilePath);
|
||||
this.reopenEdges(edges);
|
||||
this.queries.requeueFailedReferencesByLanguage(language, scope.moduleLeaves, files);
|
||||
return [...files];
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-open what importers bound from a Python package whose `__init__.py`
|
||||
* was edited: in each importing file, the edges and parked failures whose
|
||||
* reference starts with a name bound from that package (`N`, `N.run`,
|
||||
* `pkg.N`). A def or re-export added, removed or repointed there moves
|
||||
* exactly those answers — `from pkg import N` prefers what `__init__.py`
|
||||
* binds over a submodule `pkg/N` — and nothing in the importer's other
|
||||
* references. A name whose import edge already lands where the package
|
||||
* binds it now is left alone, so an edit that moves nothing re-resolves
|
||||
* nothing. Returns the files whose references were re-opened.
|
||||
*/
|
||||
resurrectPythonPackageImporters(bindings: Map<string, Map<string, string | null>>): string[] {
|
||||
if (bindings.size === 0) return [];
|
||||
const edges = this.queries.getResolutionEdgesFromFiles('python', [...bindings.keys()]);
|
||||
const refHead = (e: { metadata?: Record<string, unknown> }): string | null => {
|
||||
const refName = e.metadata?.refName;
|
||||
return typeof refName === 'string' ? refName.split('.')[0]! : null;
|
||||
};
|
||||
// Where each bound name's own import edge lands today.
|
||||
const bindsNow = new Map<string, string>();
|
||||
for (const e of edges) {
|
||||
const refName = e.metadata?.refName;
|
||||
if (e.kind !== 'imports' || typeof refName !== 'string' || refName.includes('.')) continue;
|
||||
const target = this.queries.getNodeById(e.target);
|
||||
if (target) bindsNow.set(`${e.sourceFilePath}\0${refName}`, `${target.filePath}\0${target.kind === 'file' ? '' : target.name}`);
|
||||
}
|
||||
const moved = new Map<string, Set<string>>();
|
||||
for (const [file, names] of bindings) {
|
||||
for (const [name, binding] of names) {
|
||||
if (binding !== null && bindsNow.get(`${file}\0${name}`) === binding) continue;
|
||||
let set = moved.get(file);
|
||||
if (!set) moved.set(file, (set = new Set()));
|
||||
set.add(name);
|
||||
}
|
||||
}
|
||||
if (moved.size === 0) return [];
|
||||
this.reopenEdges(edges.filter((e) => moved.get(e.sourceFilePath)?.has(refHead(e) ?? '') === true));
|
||||
this.queries.requeueFailedReferencesBinding('python', moved);
|
||||
return [...moved.keys()];
|
||||
}
|
||||
|
||||
private reopenEdges(edges: Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }>): number {
|
||||
const edgeIds: number[] = [];
|
||||
const refs: UnresolvedReference[] = [];
|
||||
for (const e of edges) {
|
||||
const ref = resurrectRefFromDroppedEdge(e);
|
||||
if (!ref) continue;
|
||||
edgeIds.push(e.edgeId);
|
||||
refs.push(ref);
|
||||
}
|
||||
if (refs.length > 0) this.queries.replaceResolutionEdgesWithUnresolvedRefs(edgeIds, refs);
|
||||
return refs.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync the index with the current file state.
|
||||
*
|
||||
|
||||
@@ -148,6 +148,9 @@ export interface IndexOptions {
|
||||
*
|
||||
* Provides the primary interface for interacting with the code knowledge graph.
|
||||
*/
|
||||
/** project_metadata key for the Python package-root digest (see sync). */
|
||||
const PYTHON_ROOTS_METADATA_KEY = 'python_root_fingerprint';
|
||||
|
||||
export class CodeGraph {
|
||||
private db: DatabaseConnection;
|
||||
private queries: QueryBuilder;
|
||||
@@ -737,6 +740,8 @@ export class CodeGraph {
|
||||
// engine produces richer extraction than the one on disk. Only on a
|
||||
// real full index — a sync touches a subset, so it must NOT advance the
|
||||
// extraction stamp (the bulk would still be stale). See extraction-version.ts.
|
||||
if (result.success) this.stampPythonRootFingerprint();
|
||||
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
try {
|
||||
this.queries.setMetadata('indexed_with_version', CodeGraphPackageVersion);
|
||||
@@ -834,6 +839,18 @@ export class CodeGraph {
|
||||
*
|
||||
* Uses a mutex to prevent concurrent indexing operations.
|
||||
*/
|
||||
/** Indexed Python file paths — the before/after sets a sync diffs. */
|
||||
private pythonFilePaths(): Set<string> {
|
||||
return new Set(this.queries.getAllFilePaths().filter((f) => f.endsWith('.py')));
|
||||
}
|
||||
|
||||
/** Record the Python package-root digest a later sync compares against. */
|
||||
private stampPythonRootFingerprint(fingerprint = this.resolver.getPythonRootFingerprint()): void {
|
||||
try {
|
||||
this.queries.setMetadata(PYTHON_ROOTS_METADATA_KEY, fingerprint);
|
||||
} catch { /* metadata is advisory */ }
|
||||
}
|
||||
|
||||
async sync(options: IndexOptions = {}): Promise<SyncResult> {
|
||||
return this.indexMutex.withLock(async () => {
|
||||
const startedAt = Date.now();
|
||||
@@ -920,6 +937,7 @@ export class CodeGraph {
|
||||
const fullReconcile = !options.paths || options.paths.length === 0;
|
||||
const gitState = this.orchestrator.beginGitIndexState(fullReconcile);
|
||||
|
||||
const pythonFilesBefore = this.pythonFilePaths();
|
||||
// An interrupted index may have absorbed its changed files before
|
||||
// resolution/synthesis. Detect those orphans BEFORE this sync adds refs.
|
||||
let refreshSynthesis = this.queries.getMetadata('synthesis_pending') === '1' ||
|
||||
@@ -1227,6 +1245,59 @@ export class CodeGraph {
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) {
|
||||
console.error(`[phase-timing] sync-route-modules: ${Date.now() - tModules}ms (${reopened} refs re-opened)`);
|
||||
}
|
||||
}
|
||||
|
||||
// Python module answers this sync may have moved in files it never
|
||||
// touched. If the root set changed (a top-level package appeared, a
|
||||
// build config was edited) any import anywhere may bind differently:
|
||||
// re-open them all. Otherwise only modules or packages appeared or
|
||||
// vanished below the roots, and only the edges into those module
|
||||
// names, out of a changed package, and the failures naming them can
|
||||
// move. The orphan sweep below re-resolves whatever is re-opened
|
||||
// exactly as a full index would. An index stamped by an older engine
|
||||
// has no fingerprint — record one rather than re-resolving on upgrade.
|
||||
{
|
||||
const before = this.queries.getMetadata(PYTHON_ROOTS_METADATA_KEY);
|
||||
const now = this.resolver.getPythonRootFingerprint();
|
||||
if (before !== null && before !== now) {
|
||||
this.resolver.clearCaches();
|
||||
this.orchestrator.resurrectResolutionEdgesForLanguage('python');
|
||||
// Any Python edge may move, so any synthesis built on one may too.
|
||||
refreshSynthesis = true;
|
||||
this.queries.setMetadata('synthesis_pending', '1');
|
||||
} else if (before !== null) {
|
||||
const after = this.pythonFilePaths();
|
||||
const moved = [...after].filter((f) => !pythonFilesBefore.has(f));
|
||||
for (const f of pythonFilesBefore) if (!after.has(f)) moved.push(f);
|
||||
const reopened = new Set<string>();
|
||||
if (moved.length > 0) {
|
||||
this.resolver.clearCaches();
|
||||
for (const f of this.orchestrator.resurrectResolutionEdgesTouching('python', this.resolver.getPythonReopenScope(moved))) reopened.add(f);
|
||||
}
|
||||
// An edited `__init__.py` can add, drop or repoint what its
|
||||
// package binds, which moves what its importers' names resolve to.
|
||||
const editedInits = (result.changedFilePaths ?? []).filter(
|
||||
(f) => pythonFilesBefore.has(f) && after.has(f) && /(?:^|[\\/])__init__\.py$/.test(f)
|
||||
);
|
||||
if (editedInits.length > 0) {
|
||||
this.resolver.clearCaches();
|
||||
for (const f of this.orchestrator.resurrectPythonPackageImporters(this.resolver.getPythonPackageImporters(editedInits))) reopened.add(f);
|
||||
}
|
||||
// Resolve the narrow set now, the way the changed files' own
|
||||
// references were resolved above, rather than leaving a handful of
|
||||
// refs to the orphan sweep and its whole-graph synthesis pass.
|
||||
if (reopened.size > 0) {
|
||||
this.resolver.resolveAndPersist(this.queries.getUnresolvedReferencesByFiles([...reopened]));
|
||||
// Same rule as the changed files above: a moved edge in a file
|
||||
// synthesis read or wrote means synthesis must be rebuilt.
|
||||
if (!refreshSynthesis && [...reopened].some((f) =>
|
||||
this.queries.hasSynthesizedEdgesTouchingFile(f) || this.queries.wasSynthesisInput(f))) {
|
||||
refreshSynthesis = true;
|
||||
this.queries.setMetadata('synthesis_pending', '1');
|
||||
}
|
||||
}
|
||||
}
|
||||
if (before !== now) this.stampPythonRootFingerprint(now);
|
||||
}
|
||||
|
||||
// Orphan sweep (#1187). A resolution pass that dies mid-run — the #850
|
||||
|
||||
+695
-198
File diff suppressed because it is too large
Load Diff
+45
-8
@@ -34,7 +34,7 @@ import { clearCppTypeAliasMemos } from './cpp-type-aliases';
|
||||
import { clearCppIncluderMemos } from './cpp-includers';
|
||||
import { clearCFeatureVariantMemos } from './c-feature-variants';
|
||||
import { matchShopifyThemeFile } from './shopify-themes';
|
||||
import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile } from './import-resolver';
|
||||
import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile, pythonRootFingerprint, pythonReopenScope, pythonPackageImporters } from './import-resolver';
|
||||
import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool';
|
||||
import { resolveAliasBinding } from './alias-binding';
|
||||
import { detectFrameworks, getResolvingFrameworks } from './frameworks';
|
||||
@@ -692,6 +692,8 @@ export class ReferenceResolver {
|
||||
if (resolved !== null && this.isIndexedFile(resolved)) return false;
|
||||
return this.isDeclaredOutsidePackage(source, fromFile);
|
||||
},
|
||||
resolveModuleFile: (source, fromFile, language) =>
|
||||
resolveImportPath(source, fromFile, language, this.context),
|
||||
isDeclaredPackage: (source, fromFile) => this.isDeclaredOutsidePackage(source, fromFile),
|
||||
isIndexedFile: (filePath) => this.isIndexedFile(filePath),
|
||||
getNodesInFile: (filePath: string) => {
|
||||
@@ -1022,15 +1024,16 @@ export class ReferenceResolver {
|
||||
this.reExportCache.set(filePath, []);
|
||||
return [];
|
||||
}
|
||||
// Re-exports are a JS/TS-only construct, and what matters is the
|
||||
// BARREL file's own language — not the consuming reference's. A
|
||||
// `.svelte`/`.vue` consumer threads its own language down the
|
||||
// re-export chase, which would make extractReExports() bail on a
|
||||
// `.ts` index barrel and silently break the chain (#629). Re-key
|
||||
// What matters is the BARREL file's own language — not the consuming
|
||||
// reference's. A `.svelte`/`.vue` consumer threads its own language
|
||||
// down the re-export chase, which would make extractReExports() bail
|
||||
// on a `.ts` index barrel and silently break the chain (#629). Re-key
|
||||
// the parse on the barrel's extension so the chase works no matter
|
||||
// what kind of file imports through it.
|
||||
// what kind of file imports through it. Python re-exports through its
|
||||
// imports (a package `__init__.py`), so a `.py` barrel parses as Python.
|
||||
const isJsFamily = /\.(?:d\.ts|[cm]?tsx?|[cm]?jsx?|ets)$/i.test(filePath);
|
||||
const reExports = extractReExports(content, isJsFamily ? 'typescript' : language);
|
||||
const barrelLanguage = isJsFamily ? 'typescript' : /\.py$/i.test(filePath) ? 'python' : language;
|
||||
const reExports = extractReExports(content, barrelLanguage);
|
||||
this.reExportCache.set(filePath, reExports);
|
||||
return reExports;
|
||||
},
|
||||
@@ -2650,6 +2653,40 @@ export class ReferenceResolver {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Digest of the package roots Python import resolution rests on (see
|
||||
* `pythonRootFingerprint`). Build configs are read from disk rather than
|
||||
* the file cache, so a sync after an edit sees the edit. Empty for a
|
||||
* project with no Python.
|
||||
*/
|
||||
getPythonRootFingerprint(): string {
|
||||
return pythonRootFingerprint(this.diskContext());
|
||||
}
|
||||
|
||||
/** What a sync must re-open after these Python files were added or removed (see `pythonReopenScope`). */
|
||||
getPythonReopenScope(changedFiles: string[]): ReturnType<typeof pythonReopenScope> {
|
||||
return pythonReopenScope(this.diskContext(), changedFiles);
|
||||
}
|
||||
|
||||
/** Importers of Python packages whose `__init__.py` was edited, with the names they bind (see `pythonPackageImporters`). */
|
||||
getPythonPackageImporters(initFiles: string[]): Map<string, Map<string, string | null>> {
|
||||
return pythonPackageImporters(this.context, initFiles);
|
||||
}
|
||||
|
||||
/** The resolution context, reading build configs from disk rather than the file cache. */
|
||||
private diskContext(): ResolutionContext {
|
||||
return {
|
||||
...this.context,
|
||||
readFile: (filePath: string) => {
|
||||
try {
|
||||
return fs.readFileSync(path.join(this.projectRoot, filePath), 'utf-8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get detected frameworks
|
||||
*/
|
||||
|
||||
@@ -3889,22 +3889,31 @@ function pythonFromImports(filePath: string, context: ResolutionContext): Map<st
|
||||
function isPythonNameImportedFromOutside(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean {
|
||||
const module = pythonFromImports(ref.filePath, context).get(name);
|
||||
if (!module || module.startsWith('.')) return false;
|
||||
return !isPythonProjectModule(module, context);
|
||||
return !isPythonProjectModule(module, ref.filePath, context);
|
||||
}
|
||||
|
||||
/** Whether some project file is the module `a.b` (`a/b.py` or `a/b/__init__.py`). */
|
||||
function isPythonProjectModule(module: string, context: ResolutionContext): boolean {
|
||||
/** Whether some project file is the module `a.b` as `fromFile` imports it (`a/b.py` or `a/b/__init__.py`). */
|
||||
function isPythonProjectModule(module: string, fromFile: string, context: ResolutionContext): boolean {
|
||||
let memo = PY_MODULE_LOCAL.get(context);
|
||||
if (!memo) {
|
||||
memo = new Map();
|
||||
PY_MODULE_LOCAL.set(context, memo);
|
||||
}
|
||||
let local = memo.get(module);
|
||||
// The import resolver decides where a module lives (package roots), and the
|
||||
// answer can depend on the importer (a monorepo service's own root), so key
|
||||
// by both. A file that merely ENDS in `requests/__init__.py` — a copy nested
|
||||
// in a test fixture — is not the `requests` an import names.
|
||||
const key = `${fromFile}\0${module}`;
|
||||
let local = memo.get(key);
|
||||
if (local === undefined) {
|
||||
const rel = module.replace(/\./g, '/');
|
||||
local = context.getAllFiles().some((f) =>
|
||||
f === `${rel}.py` || f.endsWith(`/${rel}.py`) || f === `${rel}/__init__.py` || f.endsWith(`/${rel}/__init__.py`));
|
||||
memo.set(module, local);
|
||||
if (context.resolveModuleFile) {
|
||||
local = context.resolveModuleFile(module, fromFile, 'python') !== null;
|
||||
} else {
|
||||
const rel = module.replace(/\./g, '/');
|
||||
local = context.getAllFiles().some((f) =>
|
||||
f === `${rel}.py` || f.endsWith(`/${rel}.py`) || f === `${rel}/__init__.py` || f.endsWith(`/${rel}/__init__.py`));
|
||||
}
|
||||
memo.set(key, local);
|
||||
}
|
||||
return local;
|
||||
}
|
||||
@@ -3928,7 +3937,7 @@ export function isPythonOutsideModuleReceiver(name: string, ref: UnresolvedRef,
|
||||
// the module `a` tells that `x` comes from outside.
|
||||
return [...keys].every((key) => {
|
||||
const module = key.slice(0, key.indexOf(':'));
|
||||
return !module.startsWith('.') && !isPythonProjectModuleAnyRoot(module, context);
|
||||
return !module.startsWith('.') && !isPythonProjectModuleAnyRoot(module, ref.filePath, context);
|
||||
});
|
||||
});
|
||||
return outside && !pythonBindsLocally(name, ref, context, false);
|
||||
@@ -3939,9 +3948,9 @@ export function isPythonOutsideModuleReceiver(name: string, ref: UnresolvedRef,
|
||||
* with the package itself indexed as the root — `b/c.py` or `c.py`. Erring
|
||||
* toward "project" only leaves a call to the name strategies, as before.
|
||||
*/
|
||||
function isPythonProjectModuleAnyRoot(module: string, context: ResolutionContext): boolean {
|
||||
function isPythonProjectModuleAnyRoot(module: string, fromFile: string, context: ResolutionContext): boolean {
|
||||
const parts = module.split('.');
|
||||
return parts.some((_, i) => isPythonProjectModule(parts.slice(i).join('.'), context));
|
||||
return parts.some((_, i) => isPythonProjectModule(parts.slice(i).join('.'), fromFile, context));
|
||||
}
|
||||
|
||||
const JAVA_TYPE_KINDS: ReadonlySet<string> = new Set(['class', 'interface', 'enum', 'struct', 'record', 'trait']);
|
||||
@@ -20446,6 +20455,14 @@ function matchReferenceInner(
|
||||
return matchFunctionRef(ref, context);
|
||||
}
|
||||
|
||||
// A python name imported from a module the project does not provide — `from
|
||||
// requests import get` with no `requests` at a package root — names nothing
|
||||
// in the project. Name matching would bind the import to any same-named
|
||||
// symbol, such as a copy of the package nested in a test fixture.
|
||||
if (ref.language === 'python' && ref.referenceKind === 'imports' && isPythonNameImportedFromOutside(ref.referenceName, ref, context)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// ArkTS chained UI attributes — emitted with a leading dot (`.titleStyle`,
|
||||
// `.width`) by the extractor — resolve ONLY to decorator-marked attribute
|
||||
// helpers: `@Extend`/`@Styles`/`@AnimatableExtend` functions (and global
|
||||
|
||||
@@ -184,6 +184,12 @@ export interface ResolutionContext {
|
||||
* to no project file. Supplied by the coordinator, like `resolveImport`.
|
||||
*/
|
||||
isOutOfRepoImport?(source: string, fromFile: string, language: Language): boolean;
|
||||
/**
|
||||
* The project file a module specifier, written in `fromFile`, names — the
|
||||
* import resolver's own answer (for Python: from the package roots), or null
|
||||
* when no project file provides it. Supplied by the coordinator.
|
||||
*/
|
||||
resolveModuleFile?(source: string, fromFile: string, language: Language): string | null;
|
||||
/**
|
||||
* Whether `source`, imported from `fromFile`, is a package: one the file's
|
||||
* package.json or an enclosing one declares, a Node built-in, or a
|
||||
|
||||
Reference in New Issue
Block a user