fix(explore): find COBOL copybooks by name, with every include site (#2342) (#2358)

`codegraph explore MYCOPYBOOK`, the `codegraph_explore` MCP tool and
`codegraph context` answered "No relevant code found" for a COBOL copybook
named in the query, while `query`/`node` found its include at once.

Cause: the COBOL extractor records `COPY X` and `EXEC SQL INCLUDE X` as
`import` nodes, and findRelevantContext filters every channel by
HIGH_VALUE_NODE_KINDS, which leaves imports out on purpose (a JS/TS/Python
import statement is noise in explore). With the include out of reach the
text search fell through to its fuzzy fallback, so a member with a digit in
it (never extracted as a symbol) came back as an unrelated program one edit
away: CVACT01Y -> CBACT01C, CVCUS01Y -> CBCUS01C on CardDemo. Making COBOL
imports "high value" alone would not have helped either: explore's renderer
skips import nodes, and the import-to-definition step drops a COBOL include
because its `imports` edge leaves the enclosing scope, not the import node.

Change:
- src/graph/cobol-copybooks.ts (new, shared derivation): resolves the COBOL
  words of a query to copybooks - every COPY / EXEC SQL INCLUDE of the member
  (case-insensitive) and the indexed copybook file (.cpy, or a file an
  include resolved to; a same-named program is not one). Gated on the
  project having COBOL files at all (one seek on idx_files_language), so no
  other language's query does extra work or changes. Plain lowercase words
  in a multi-word query are English, not member names.
- findRelevantContext: a named copybook's top-level declarations, then its
  include statements, become entry points ahead of the kind-filtered
  channels, like an exact filename (at most half the slots when the query
  matched other things, include sites in those files first). They are not
  walked (BFS from a record layout spent the answer on whichever program
  MOVEs into it), the member is not re-searched as text (the fuzzy decoy),
  and an explicit nodeKinds filter without `import` is honored.
- codegraph_explore: the copybook file is pinned like a path in the query;
  an include statement that is an entry point is grouped and rendered (a
  window around the COPY line) instead of skipped; a new "COBOL copybook"
  section lists every include site (path:line, COPY vs EXEC SQL INCLUDE,
  25 lines then a count) or says the member has no indexed source (DB2
  DCLGEN, compiler-supplied such as SQLCA/DFHAID).
- QueryBuilder: three indexed lookups; CodeGraph.findNamedCopybooks exposes
  the derivation so explore and the context builder share one answer.
No extraction change; no re-index needed.

Verification: __tests__/explore-cobol-copybook.test.ts (8 tests, through
ToolHandler codegraph_explore and buildContext): with src/ reverted to
origin/main 6 fail (the 2 guard cases pass on both), 8/8 pass with the fix.
tsc clean, npm run build OK. Issue repro fixed on the built CLI (explore +
context) and through a real `serve --mcp` tools/call. Full suite on a heavily
loaded box: 5716 passed, 25 failed in 15 files; 12 of those files pass alone,
and the other 3 (function-ref, git-index-currency, worktree-detection) are
5 s / 60 s timeouts that fail on clean main as well in interleaved runs.

Validation (both builds on one index per repo; query-side change only):
- expressjs/express: 14 explore + 4 context queries byte-identical.
- aws-mainframe-modernization-carddemo: 10 non-copybook queries
  byte-identical; 15 copybook queries all now return the copybook (when
  indexed) and every include site - before: 9 "No relevant code found",
  4 unrelated programs, 2 without the copybook.
- cicsdev/cics-genapp: 6 non-copybook queries byte-identical; 6 copybook
  queries fixed (LGCMAREA's 25 sites incl. multi-line EXEC SQL INCLUDE,
  lowercase lgpolicy, SSMAP with no indexed source).
- In-process explore latency on express unchanged within noise.

Thanks @popolusiak for the report.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-10-05 20:39:03 +00:00
committed by GitHub
parent 8998697f1d
commit 1b56cb1992
7 changed files with 540 additions and 9 deletions
+1
View File
@@ -30,6 +30,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332)
- In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305)
- In Vue and Nuxt components, calls written in the template (`{{ formatDate(post) }}`, `:to="getAccountRoute(account)"`, `@click="save(item)"`) and calls in a destructuring declaration at the top of `<script setup>` or of a TypeScript or JavaScript file (`const { t } = useI18n()`) now link to the function they call, so `callers` and impact for a composable or helper used this way are no longer mostly empty. A `v-for` item or slot prop is never taken for a project function, a handler like `@click="open = false"` no longer links to an unrelated function of the same name, and a component's own function named like a Nuxt helper, such as `clearError`, is now the one its calls reach. Thanks @deniskern for the report, and @drakeo338 and @L0garithmic. (#2340)
- In COBOL, asking `codegraph_explore` (or `codegraph explore` and `codegraph context`) about a copybook by name, such as `CVACT01Y` or a member brought in with `EXEC SQL INCLUDE`, now answers with the copybook's own source when it is indexed and every `COPY` or `EXEC SQL INCLUDE` statement that includes it, and says so when the copybook's source isn't part of the project. Before, it answered "No relevant code found", or showed an unrelated program whose name was a letter or two away. Thanks @popolusiak for the report. (#2342)
## [1.6.2] - 2026-10-03
+182
View File
@@ -0,0 +1,182 @@
/**
* COBOL copybooks are findable by name in explore (#2342).
*
* `COPY X` and `EXEC SQL INCLUDE X` are indexed as `import` nodes, and the
* context builder's default kind filter drops imports (for JavaScript an
* import statement is noise). So `codegraph_explore "MYCOPYBOOK"` — and the
* CLI `explore`, which runs the same handler — answered "No relevant code
* found" while `query`/`node` found the include at once. A copybook name with
* a digit in it (`CVACT01Y`) fared worse: the text search fell through to its
* fuzzy fallback and answered with an unrelated program one edit away.
*
* Now a query that names a copybook gets the copybook's own source (pinned,
* first) and every statement that includes it; a member whose source is not
* indexed says so. Other languages' imports stay out of explore.
*/
import { afterEach, describe, expect, it } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import CodeGraph from '../src/index';
import { ToolHandler } from '../src/mcp/tools';
let dir: string;
let cg: CodeGraph;
async function index(files: Record<string, string>): Promise<void> {
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cobol-copybook-'));
for (const [name, source] of Object.entries(files)) {
fs.mkdirSync(path.dirname(path.join(dir, name)), { recursive: true });
fs.writeFileSync(path.join(dir, name), source);
}
cg = CodeGraph.initSync(dir);
await cg.indexAll();
}
async function explore(query: string): Promise<string> {
const result = await new ToolHandler(cg).execute('codegraph_explore', { query });
expect(result.isError).toBeFalsy();
return result.content[0]!.text;
}
afterEach(() => {
cg?.destroy();
if (dir) fs.rmSync(dir, { recursive: true, force: true });
});
const lines = (...l: string[]): string => l.join('\n') + '\n';
/** The issue's reproduction, verbatim: an EXEC SQL INCLUDE whose member is not indexed. */
const TESTPROG = lines(
' IDENTIFICATION DIVISION.',
' PROGRAM-ID. TESTPROG.',
' PROCEDURE DIVISION.',
' UP-INCLUDE-COPYBOOK SECTION.',
' UP-INCLUDE-COPYBOOK-10.',
' EXEC SQL INCLUDE MYCOPYBOOK END-EXEC.',
);
/** An indexed data copybook, COPY'd by two programs. */
const COPYBOOK_PROJECT = {
'cpy/CVACT01Y.cpy': lines(
' 01 ACCOUNT-RECORD.',
' 05 ACCT-ID PIC 9(11).',
' 05 ACCT-CURR-BAL PIC S9(10)V99.',
),
'cbl/ACCTUPDT.cbl': lines(
' IDENTIFICATION DIVISION.',
' PROGRAM-ID. ACCTUPDT.',
' DATA DIVISION.',
' WORKING-STORAGE SECTION.',
' COPY CVACT01Y.',
' 01 WS-COUNT PIC 9(4) VALUE 0.',
' PROCEDURE DIVISION.',
' MAIN-PARA.',
' MOVE 1 TO ACCT-ID',
' STOP RUN.',
),
'cbl/ACCTVIEW.cbl': lines(
' IDENTIFICATION DIVISION.',
' PROGRAM-ID. ACCTVIEW.',
' DATA DIVISION.',
' WORKING-STORAGE SECTION.',
' COPY CVACT01Y.',
' PROCEDURE DIVISION.',
' MAIN-PARA.',
' DISPLAY ACCT-CURR-BAL',
' STOP RUN.',
),
// One edit away from the copybook's name, and includes nothing: the
// program the fuzzy fallback used to answer `CVACT01Y` with.
'cbl/CBACT01C.cbl': lines(
' IDENTIFICATION DIVISION.',
' PROGRAM-ID. CBACT01C.',
' PROCEDURE DIVISION.',
' MAIN-PARA.',
' DISPLAY "BATCH"',
' STOP RUN.',
),
};
describe('explore finds COBOL copybooks by name (#2342)', () => {
it('surfaces an EXEC SQL INCLUDE whose copybook is not indexed (the issue repro)', async () => {
await index({ 'TESTPROG.cbl': TESTPROG });
const text = await explore('MYCOPYBOOK');
expect(text).not.toContain('No relevant code found');
// The include site, listed and rendered.
expect(text).toContain('`TESTPROG.cbl:6` — EXEC SQL INCLUDE');
expect(text).toMatch(/6\t\s+EXEC SQL INCLUDE MYCOPYBOOK END-EXEC\./);
// And the honest note that there is no copybook source here to look for.
expect(text).toContain('no indexed source');
});
it('pins an indexed copybook ahead of every include site, and lists all of them', async () => {
await index(COPYBOOK_PROJECT);
const text = await explore('CVACT01Y');
expect(text).not.toContain('No relevant code found');
expect(text).toContain('source `cpy/CVACT01Y.cpy`; included at 2 sites');
expect(text).toContain('`cbl/ACCTUPDT.cbl:5` — COPY');
expect(text).toContain('`cbl/ACCTVIEW.cbl:5` — COPY');
// The copybook's own source renders, before any including program.
const copybookAt = text.indexOf('**`cpy/CVACT01Y.cpy`**');
expect(copybookAt).toBeGreaterThan(-1);
expect(text).toContain('01 ACCOUNT-RECORD.');
for (const program of ['cbl/ACCTUPDT.cbl', 'cbl/ACCTVIEW.cbl']) {
const at = text.indexOf(`**\`${program}\`**`);
expect(at).toBeGreaterThan(copybookAt);
}
expect(text).toContain('COPY CVACT01Y.');
expect(text).toContain('1 file pinned from the query');
// Not the program one edit away from the copybook's name.
expect(text).not.toContain('CBACT01C');
});
it('feeds the same entry points to codegraph context (buildContext)', async () => {
await index({ 'TESTPROG.cbl': TESTPROG, ...COPYBOOK_PROJECT });
const missing = await cg.buildContext('MYCOPYBOOK', { format: 'markdown' });
expect(missing).toContain('EXEC SQL INCLUDE MYCOPYBOOK END-EXEC');
const indexed = await cg.buildContext('CVACT01Y', { format: 'markdown' });
expect(indexed).toContain('ACCOUNT-RECORD');
expect(indexed).toContain('COPY CVACT01Y.');
expect(indexed).not.toContain('CBACT01C');
});
it('leaves the copybook out when an explicit kind filter excludes imports', async () => {
await index({ 'TESTPROG.cbl': TESTPROG });
const subgraph = await cg.findRelevantContext('MYCOPYBOOK', { nodeKinds: ['function'] });
expect([...subgraph.nodes.values()].some((n) => n.kind === 'import')).toBe(false);
});
it('keeps imports of other languages out of explore', async () => {
await index({
'main.ts': "import { CVACT01Y } from './lib';\nexport function run() { return CVACT01Y; }\n",
'lib.ts': 'export const CVACT01Y = 1;\n',
});
const subgraph = await cg.findRelevantContext('CVACT01Y ./lib');
expect([...subgraph.nodes.values()].some((n) => n.kind === 'import')).toBe(false);
const text = await explore('CVACT01Y');
expect(text).not.toContain('COBOL copybook');
expect(text).toContain('export const CVACT01Y = 1;');
});
});
describe('copybook member tokens (#2342)', () => {
// Imported lazily so the behavioural cases above load (and fail) on a build
// that predates the module.
const tokens = async (query: string): Promise<string[]> =>
(await import('../src/graph/cobol-copybooks')).copybookMemberTokens(query);
it('keeps COBOL-shaped words and skips plain lowercase English', async () => {
expect(await tokens('where is CVACT01Y included from cust-rec and dfh$aid'))
.toEqual(['CVACT01Y', 'cust-rec', 'dfh$aid']);
});
it('dedupes case-insensitively and drops short or digit-only tokens', async () => {
expect(await tokens('SQLCA sqlca IO 0042 Sqlca')).toEqual(['SQLCA']);
});
it('takes a one-word query as a name whatever its case', async () => {
expect(await tokens('lgpolicy')).toEqual(['lgpolicy']);
expect(await tokens('the lgpolicy')).toEqual([]);
});
});
+94 -1
View File
@@ -27,6 +27,14 @@ import { logDebug } from '../errors';
import { validatePathWithinRoot, isConfigLeafNode } from '../utils';
import { isTestFile, extractSearchTerms, scorePathRelevance, getStemVariants, isDistinctiveIdentifier } from '../search/query-utils';
import { LOW_CONFIDENCE_MARKER } from './markers';
import { findNamedCopybooks, isCopybookInclude } from '../graph/cobol-copybooks';
/**
* Most top-level declarations of one copybook that become entry points — its
* record layouts, or the first paragraphs of a procedure copybook. The rest of
* the copybook is one hop away (explore pins the whole file).
*/
const MAX_COPYBOOK_DECLARATION_ENTRIES = 3;
/**
* Extract likely symbol names from a natural language query
@@ -619,13 +627,32 @@ export class ContextBuilder {
}
}
// Step 2d: COBOL copybook members (#2342). `COPY CVACT01Y` / `EXEC SQL
// INCLUDE MYCOPYBOOK` are `import` nodes, which the default kind filter
// drops — so a query naming a copybook found nothing at all, though the
// include statement and the copybook are exactly what it asks for. Gathered
// apart from the kind-filtered channels and admitted ahead of them below,
// like an exact filename. Honors an explicit kind filter that leaves
// imports out.
const allowsIncludes = options.nodeKinds === undefined
|| options.nodeKinds.length === 0
|| options.nodeKinds.includes('import');
const copybooks = allowsIncludes
? this.findCopybookEntries(query)
: { entries: [], members: new Set<string>() };
// Step 3: Run text search for natural language term matching
// This catches file-name and node-name matches that semantic search may miss,
// which is critical for template-heavy codebases (e.g., Liquid/Shopify themes)
// where file names are the primary identifiers.
let textResults: SearchResult[] = [];
try {
const searchTerms = extractSearchTerms(query);
// A copybook member Step 2d resolved is not searched again as text: its
// only FTS hits are the import nodes this channel excludes, so the search
// fell through to the fuzzy fallback and returned whatever program name
// sat within edit distance (`CVACT01Y` → `CBACT01C`).
const searchTerms = extractSearchTerms(query)
.filter((term) => !copybooks.members.has(term.toUpperCase()));
if (searchTerms.length > 0) {
// Search each term individually to get broader coverage,
// then boost results that match multiple terms
@@ -1026,6 +1053,30 @@ export class ContextBuilder {
];
}
// A named copybook leads too: its own declarations, then its include
// statements. When the query also matched other things, the copybook takes
// at most half the entry slots so they are not capped away.
const copybookIds = new Set(copybooks.entries.map((r) => r.node.id));
const leafEntryIds = new Set<string>();
if (copybooks.entries.length > 0) {
const matchedElsewhere = new Set(filteredResults.map((r) => r.node.id));
const others = filteredResults.filter((r) => !copybookIds.has(r.node.id));
// Include statements in a file the rest of the query also matched come
// first ("COACTUPC CVACT01Y" wants COACTUPC's COPY, not the first one by
// path). Stable, so the declarations stay ahead of every include.
const otherFiles = new Set(others.map((r) => r.node.filePath));
const rank = (r: SearchResult): number =>
!isCopybookInclude(r.node) ? 0 : otherFiles.has(r.node.filePath) ? 1 : 2;
const ordered = [...copybooks.entries].sort((a, b) => rank(a) - rank(b));
const kept = others.length > 0 ? ordered.slice(0, Math.ceil(opts.searchLimit / 2)) : ordered;
// The copybook's own entries are not walked: an include statement has
// nothing to walk but the program around it, and a record layout's
// neighbourhood is whichever program happens to MOVE into it — walking
// either spent the answer on unrelated paragraphs of an including program.
for (const r of kept) if (!matchedElsewhere.has(r.node.id)) leafEntryIds.add(r.node.id);
filteredResults = [...kept, ...others];
}
// Cap entry points so traversal budget isn't spread too thin.
// With 36 entry points and maxNodes=120, each gets only 3 nodes — useless.
// Cap to searchLimit so each entry point gets a meaningful traversal budget.
@@ -1048,6 +1099,8 @@ export class ContextBuilder {
symbolsFromQuery.filter(isDistinctiveIdentifier).map(s => s.toLowerCase())
);
const anyStrong = filteredResults.some(r => {
// A copybook the query named is an exact identifier match, not prose.
if (copybookIds.has(r.node.id)) return true;
if (distinctive.has(r.node.name.toLowerCase())) return true;
const nameLower = r.node.name.toLowerCase();
const dirSegs = path.dirname(r.node.filePath).toLowerCase().split('/');
@@ -1127,6 +1180,7 @@ export class ContextBuilder {
// Traverse from each entry point
for (const result of filteredResults) {
if (leafEntryIds.has(result.node.id)) continue;
const traversalResult = this.traverser.traverseBFS(result.node.id, {
maxDepth: opts.traversalDepth,
edgeKinds: opts.edgeKinds && opts.edgeKinds.length > 0 ? opts.edgeKinds : undefined,
@@ -1272,6 +1326,45 @@ export class ContextBuilder {
return { nodes: finalNodes, edges: finalEdges, roots, confidence };
}
/**
* Entry points for the COBOL copybooks a query names (#2342): each indexed
* copybook's top-level declarations (its record layouts, or the paragraphs
* of a procedure copybook), then the include statements themselves — taken
* round-robin across the named members, so one widely-included copybook
* cannot crowd out another the query also names. `members` holds the
* matched member names, upper-cased. Both are empty when the query names no
* copybook, which is every query on a project without COBOL.
*/
private findCopybookEntries(query: string): { entries: SearchResult[]; members: Set<string> } {
const named = findNamedCopybooks(this.queries, query);
const members = new Set(named.map((c) => c.member.toUpperCase()));
const entries: SearchResult[] = [];
const seen = new Set<string>();
const add = (node: Node): void => {
if (seen.has(node.id)) return;
seen.add(node.id);
entries.push({ node, score: 1 });
};
for (const copybook of named) {
for (const file of copybook.files) {
const declarationIds = this.queries.getOutgoingEdges(file.id, ['contains']).map((e) => e.target);
[...this.queries.getNodesByIds(declarationIds).values()]
.filter((n) => n.kind !== 'import' && n.kind !== 'export')
.sort((a, b) => a.startLine - b.startLine || a.id.localeCompare(b.id))
.slice(0, MAX_COPYBOOK_DECLARATION_ENTRIES)
.forEach(add);
}
}
const longest = Math.max(0, ...named.map((c) => c.includes.length));
for (let i = 0; i < longest; i++) {
for (const copybook of named) {
const include = copybook.includes[i];
if (include) add(include);
}
}
return { entries, members };
}
/**
* Get the source code for a node
*
+49
View File
@@ -303,6 +303,9 @@ export class QueryBuilder {
getFileNodesByNamePrefix?: SqliteStatement;
getNodesByQualifiedNameExact?: SqliteStatement;
getNodesByLowerName?: SqliteStatement;
hasFilesOfLanguage?: SqliteStatement;
getCobolIncludesByMember?: SqliteStatement;
getCobolFilesByStem?: SqliteStatement;
getUnresolvedCount?: SqliteStatement;
getUnresolvedBatch?: SqliteStatement;
getUnresolvedBatchAfter?: SqliteStatement;
@@ -1488,6 +1491,52 @@ export class QueryBuilder {
return rows.map(rowToNode);
}
/** Does the index hold any file of this language? One seek on idx_files_language. */
hasFilesOfLanguage(language: string): boolean {
if (!this.stmts.hasFilesOfLanguage) {
this.stmts.hasFilesOfLanguage = this.db.prepare('SELECT 1 FROM files WHERE language = ? LIMIT 1');
}
return this.stmts.hasFilesOfLanguage.get(language) !== undefined;
}
/**
* COBOL copybook includes naming `member`: the `import` node the COBOL
* extractor makes for every `COPY member` and `EXEC SQL INCLUDE member`,
* matched case-insensitively (COBOL names ignore case), in file/line order.
*
* Seeks idx_nodes_lower_name. The unary `+` keeps the planner off the kind
* and language indexes, which would otherwise read every import (or every
* COBOL node) in the project to find the few with this name.
*/
getCobolIncludesByMember(member: string, limit: number): Node[] {
if (!this.stmts.getCobolIncludesByMember) {
this.stmts.getCobolIncludesByMember = this.db.prepare(
"SELECT * FROM nodes WHERE lower(name) = lower(?) AND +kind = 'import' AND +language = 'cobol'"
+ ' ORDER BY file_path, start_line, id LIMIT ?'
);
}
const rows = this.stmts.getCobolIncludesByMember.all(member, limit) as NodeRow[];
return rows.map(rowToNode);
}
/**
* COBOL file nodes whose basename stem is `member`, case-insensitively —
* `member.cpy`, `MEMBER.CPY`, `member.cbl`. Every such basename sorts into
* [`member.`, `member/`) under lower(), so this is a range seek on
* idx_nodes_lower_name; the unary `+` stops the planner from preferring a
* walk of every file node in idx_nodes_kind to save the (tiny) sort.
*/
getCobolFilesByStem(member: string): Node[] {
if (!this.stmts.getCobolFilesByStem) {
this.stmts.getCobolFilesByStem = this.db.prepare(
"SELECT * FROM nodes WHERE lower(name) >= lower(?) || '.' AND lower(name) < lower(?) || '/'"
+ " AND +kind = 'file' AND +language = 'cobol' ORDER BY file_path"
);
}
const rows = this.stmts.getCobolFilesByStem.all(member, member) as NodeRow[];
return rows.map(rowToNode);
}
/**
* Search nodes by name using FTS with fallback to LIKE for better matching
*
+122
View File
@@ -0,0 +1,122 @@
/**
* COBOL copybooks a query names — one derivation of "the agent asked about
* copybook X" (#2342).
*
* The COBOL extractor records every `COPY X` and `EXEC SQL INCLUDE X` as an
* `import` node named X sitting on the include statement, and the resolver
* links X to the indexed copybook file (`X.cpy`, …) with an `imports` edge
* from the including scope. That `import` node is not the low-information
* line an `import { Foo } from './foo'` is in JavaScript: a copybook member
* name is how COBOL code is talked about, and the include statement is half of
* the answer to a question about it. So a query token that names a copybook
* asks for two things:
*
* - the copybook's own source, when it is indexed (the record layout, the SQL
* host variables, the shared paragraphs), and
* - every statement that includes it — where that layout is pulled in, which
* is what changing it would touch.
*
* A member whose source is not in the project (a DB2 DCLGEN member, a
* compiler-supplied copybook such as SQLCA or DFHAID) has include sites and no
* file, and saying so is part of the answer too: it tells the reader there is
* no copybook source here to go looking for.
*
* Shared by the context builder (entry points for `codegraph_explore` and
* `codegraph context`) and by explore's renderer (pinning the copybook, listing
* its include sites), so the two read one answer and cannot drift apart.
*/
import type { Edge, EdgeKind, Node } from '../types';
/** The queries this derivation needs — satisfied by `QueryBuilder`. */
export interface CopybookLookup {
hasFilesOfLanguage(language: string): boolean;
getCobolIncludesByMember(member: string, limit: number): Node[];
getCobolFilesByStem(member: string): Node[];
getIncomingEdges(targetId: string, kinds?: EdgeKind[]): Edge[];
}
/** One copybook member the query names, with what the index knows about it. */
export interface NamedCopybook {
/** The member as the query spelled it. */
member: string;
/**
* The indexed copybook file(s) the member resolves to — a `.cpy` file, or
* any COBOL file an include actually resolved to. Empty when the member's
* source is not indexed.
*/
files: Node[];
/** Every `COPY` / `EXEC SQL INCLUDE` statement naming the member, in file/line order. */
includes: Node[];
}
/**
* Most include statements gathered per member. A shared copybook in a large
* estate is included by hundreds of programs; past this the list is a count.
*/
export const MAX_COPYBOOK_INCLUDES = 200;
/** Most query tokens looked up — a pasted paragraph must not become 200 probes. */
const MAX_MEMBER_TOKENS = 16;
/**
* A COBOL word: letters, digits, hyphens and the national characters `$ # @`
* (`CVACT01Y`, `CUST-REC`, `DFH$AID`). Wider than the identifier shapes the
* generic symbol extraction knows — a copybook name with a digit in it
* (`CVACT01Y`) was never extracted as a symbol at all.
*/
const COBOL_WORD = /[A-Za-z0-9$#@][A-Za-z0-9$#@-]*/g;
/**
* The query's tokens that could name a copybook member. In a longer query a
* plain lowercase word is skipped: `customer` in a prose question is English,
* while a member name is written the way the code writes it (`CUSTOMER`,
* `cvact01y`) — the same rule explore's own symbol seeding applies to bare
* words. A query that is one word is a name lookup whatever its case
* (`lgpolicy`, after the file `lgpolicy.cpy`).
*/
export function copybookMemberTokens(query: string): string[] {
const words = (query.match(COBOL_WORD) ?? []).map((raw) => raw.replace(/-+$/, ''));
const lone = words.length === 1;
const seen = new Set<string>();
const out: string[] = [];
for (const token of words) {
if (token.length < 3 || !/[A-Za-z]/.test(token) || (!lone && /^[a-z]+$/.test(token))) continue;
const key = token.toUpperCase();
if (seen.has(key)) continue;
seen.add(key);
out.push(token);
if (out.length >= MAX_MEMBER_TOKENS) break;
}
return out;
}
/**
* Is this node a COBOL copybook include — the statement `COPY X` /
* `EXEC SQL INCLUDE X` — rather than an import in any other language?
*/
export function isCopybookInclude(node: Pick<Node, 'kind' | 'language'>): boolean {
return node.kind === 'import' && node.language === 'cobol';
}
/**
* The copybooks `query` names. A token qualifies when the index holds a COBOL
* include of that member or a copybook file with that name. A project without
* COBOL answers after one index seek, so every other language's queries are
* untouched.
*/
export function findNamedCopybooks(lookup: CopybookLookup, query: string): NamedCopybook[] {
const tokens = copybookMemberTokens(query);
if (tokens.length === 0 || !lookup.hasFilesOfLanguage('cobol')) return [];
const named: NamedCopybook[] = [];
for (const member of tokens) {
const includes = lookup.getCobolIncludesByMember(member, MAX_COPYBOOK_INCLUDES);
// A same-named PROGRAM (`COACTUPC.cbl` for the query `COACTUPC`) is not a
// copybook: only a `.cpy` file, or a file some include resolved to, is.
const files = lookup.getCobolFilesByStem(member).filter((file) =>
/\.cpy$/i.test(file.filePath) || lookup.getIncomingEdges(file.id, ['imports']).length > 0);
if (includes.length === 0 && files.length === 0) continue;
named.push({ member, files, includes });
}
return named;
}
+10
View File
@@ -52,6 +52,7 @@ import {
} from './resolution';
import { hasSynthesisPattern } from './resolution/callback-synthesizer';
import { GraphTraverser, GraphQueryManager } from './graph';
import { findNamedCopybooks, type NamedCopybook } from './graph/cobol-copybooks';
import { ContextBuilder, createContextBuilder } from './context';
import { Mutex, FileLock } from './utils';
import { FileWatcher, WatchOptions, PendingFile, LockUnavailableError } from './sync';
@@ -1785,6 +1786,15 @@ export class CodeGraph {
return this.queries.getExploreMissDiagnostics(query);
}
/**
* The COBOL copybooks a query names (`CVACT01Y`, `MYCOPYBOOK`): each
* member's indexed copybook file(s) and every `COPY` / `EXEC SQL INCLUDE`
* statement that includes it. Empty for a query that names none.
*/
findNamedCopybooks(query: string): NamedCopybook[] {
return findNamedCopybooks(this.queries, query);
}
/**
* Graph-derived prompt matching for the front-load hook's MEDIUM tier:
* which indexed symbols do these prose words name? "state machine des
+82 -8
View File
@@ -53,6 +53,7 @@ import { clamp, validatePathWithinRoot, validateProjectPath, isConfigLeafNode, C
import { guardLabel, guardsForFileSync, siteKey, supportsBranchGuards, warmBranchGuardGrammars } from '../graph/branch-guards';
import { findDynamicBoundaries, type BoundarySite } from '../graph/dynamic-boundary-report';
import { countImplementers } from '../graph/type-hierarchy';
import { isCopybookInclude, MAX_COPYBOOK_INCLUDES, type NamedCopybook } from '../graph/cobol-copybooks';
import {
findAllSymbols,
resolveNamedSymbolFlow,
@@ -3934,6 +3935,46 @@ export class ToolHandler {
].join('\n');
}
/**
* The COBOL copybooks an explore query named (#2342): where each one's source
* lives, and every `COPY` / `EXEC SQL INCLUDE` statement that pulls it in —
* locations only, one per line. The source section can afford a few include
* windows; this list is what answers "what includes X / what does changing X
* touch" without a grep for `COPY X`. A member whose source is not indexed
* says so, so nobody goes hunting for a copybook file that is not here.
*/
private buildCopybookSection(copybooks: NamedCopybook[]): string {
const MAX_LISTED_SITES = 25;
const blocks: string[] = [];
for (const copybook of copybooks) {
// Spelled as the code spells it — an include's member, else the file's
// stem — not as the query happened to type it.
const name = copybook.includes[0]?.name
?? copybook.files[0]?.name.replace(/\.[^.]*$/, '')
?? copybook.member;
const source = copybook.files.length > 0
? `source ${copybook.files.map((f) => `\`${f.filePath}\``).join(', ')}`
: 'no indexed source (typically a DB2 DCLGEN member, a compiler-supplied copybook such as SQLCA, or one kept outside this project)';
const count = copybook.includes.length;
if (count === 0) {
blocks.push(`**COBOL copybook \`${name}\`** — ${source}; nothing indexed includes it (no \`COPY\` or \`EXEC SQL INCLUDE\` of it).`, '');
continue;
}
const sites = `${count}${count >= MAX_COPYBOOK_INCLUDES ? '+' : ''} site${count === 1 ? '' : 's'}`;
blocks.push(`**COBOL copybook \`${name}\`** — ${source}; included at ${sites}:`, '');
for (const include of copybook.includes.slice(0, MAX_LISTED_SITES)) {
const statement = /^\s*EXEC\b/i.test(include.signature ?? '') ? 'EXEC SQL INCLUDE' : 'COPY';
blocks.push(`- \`${include.filePath}:${include.startLine}\` — ${statement}`);
}
if (count > MAX_LISTED_SITES) {
const files = new Set(copybook.includes.slice(MAX_LISTED_SITES).map((n) => n.filePath)).size;
blocks.push(`- … +${count - MAX_LISTED_SITES} more site${count - MAX_LISTED_SITES === 1 ? '' : 's'} in ${files} file${files === 1 ? '' : 's'}`);
}
blocks.push('');
}
return blocks.join('\n');
}
/**
* Test-coverage note for a blast-radius entry whose DIRECT callers include no
* test file. A helper called only by production code can still be exercised
@@ -4120,6 +4161,22 @@ export class ToolHandler {
}
} catch { /* path pinning must never fail an explore call */ }
}
// A COBOL copybook the query names (`CVACT01Y`) names its FILE as surely as
// `app/cpy/CVACT01Y.cpy` would (#2342), so it is pinned the same way: the
// copybook's own source is admitted, ranked first and funded first. Looked
// up on the path-stripped query — a query that already pinned the file by
// path gets exactly the answer it got before. The include sites are listed
// in their own section below.
let namedCopybooks: NamedCopybook[] = [];
try {
namedCopybooks = cg.findNamedCopybooks(matchQuery);
} catch { /* copybook pinning must never fail an explore call */ }
for (const copybook of namedCopybooks) {
for (const file of copybook.files) {
if (pinnedFiles.length >= maxFiles) break;
if (!pinnedFiles.includes(file.filePath)) pinnedFiles = [...pinnedFiles, file.filePath];
}
}
// A same-named file the span did not pin is named in the summary line, so
// an agent that did mean it sees where it went instead of a silent drop.
const setAsideNote = setAsideMatches.map((s) => {
@@ -4654,9 +4711,20 @@ export class ToolHandler {
}
}
// Import/export nodes add noise without information — except a COBOL
// include the query named (#2342). `COPY CVACT01Y` is where the copybook is
// pulled in, half of what a question about the copybook asks, so it is
// grouped, rendered (a window around the statement) and listed like any
// symbol. The context builder makes one an entry point only when the query
// names its member, so no other import ever passes this test.
const namedIncludeIds = new Set(subgraph.roots.filter((id) => {
const root = subgraph.nodes.get(id);
return root !== undefined && isCopybookInclude(root);
}));
const isListedSymbol = (n: Node): boolean =>
n.kind !== 'export' && (n.kind !== 'import' || namedIncludeIds.has(n.id));
for (const node of subgraph.nodes.values()) {
// Skip import/export nodes — they add noise without information
if (node.kind === 'import' || node.kind === 'export') continue;
if (!isListedSymbol(node)) continue;
// SECURITY (#383): never render the on-disk source of a config-leaf
// (Spring application.{yml,properties} key) — its line is `key = <secret>`,
// so whole-file/cluster rendering here would push secrets into context
@@ -5073,6 +5141,12 @@ export class ToolHandler {
];
const summaryLineIdx = 2;
// A named COBOL copybook's include sites, ALL of them (#2342) — the source
// below has room for a few include windows, and "what includes X" is the
// question a copybook name asks.
const copybookSection = this.buildCopybookSection(namedCopybooks);
if (copybookSection) lines.push(copybookSection);
// Blast radius (always-on, compact): for the entry symbols, who depends on
// them + which tests cover them — locations only, no source — so the agent
// knows what to update/verify before editing without a separate call.
@@ -5440,7 +5514,7 @@ export class ToolHandler {
const hit = overheadCache.get(filePath);
if (hit !== undefined) return hit;
const names = [...new Set(
nodes.filter((n) => n.kind !== 'import' && n.kind !== 'export')
nodes.filter(isListedSymbol)
.map((n) => `${n.name}(${n.kind})`),
)].slice(0, budget.maxSymbolsInFileHeader);
// header + blank, then ```lang / body / ``` / blank around the source.
@@ -5978,7 +6052,7 @@ export class ToolHandler {
// The file node is not one of them: no section delivers "the file" as a
// symbol, and it spans trailing lines no render prints.
const wantedFrom = (nodes: readonly Node[]): ExploreWantedSpan[] => nodes
.filter((n) => n.kind !== 'import' && n.kind !== 'export' && n.kind !== 'file' && n.startLine > 0)
.filter((n) => isListedSymbol(n) && n.kind !== 'file' && n.startLine > 0)
.map((n) => ({
name: n.name,
kind: n.kind,
@@ -6015,7 +6089,7 @@ export class ToolHandler {
&& !anchorSpans.has(filePath)
&& (onSpineGodFile || (!hasSpineNode && isPolymorphicSibling(group.nodes) && !spared))) {
const syms = group.nodes
.filter(n => n.kind !== 'import' && n.kind !== 'export' && n.startLine > 0)
.filter(n => isListedSymbol(n) && n.startLine > 0)
.sort((a, b) => a.startLine - b.startLine);
// Pass 1: choose which symbols get a FULL body, by priority, greedily within
// a per-file body cap — so one huge family file can't body every named method
@@ -6158,7 +6232,7 @@ export class ToolHandler {
}
}
if (skel.length > 0) {
const names = [...new Set(group.nodes.filter(n => n.kind !== 'import' && n.kind !== 'export').map(n => n.name))]
const names = [...new Set(group.nodes.filter(isListedSymbol).map(n => n.name))]
.slice(0, budget.maxSymbolsInFileHeader).join(', ');
// Steer the agent to codegraph_explore for an elided body — NEVER to
// Read. The old "Read for more" / "Read for a full body" tags invited
@@ -6317,7 +6391,7 @@ export class ToolHandler {
const wholeSection = ddWhole.parts.map((p) => p.text).join(GAP_MARKER);
const uniqSymbols = [...new Set(
group.nodes
.filter(n => n.kind !== 'import' && n.kind !== 'export')
.filter(isListedSymbol)
.map(n => `${n.name}(${n.kind})`)
)];
const headerNames = uniqSymbols.slice(0, budget.maxSymbolsInFileHeader);
@@ -7496,7 +7570,7 @@ export class ToolHandler {
const g = fileGroups.get(fp);
if (!g) return sum;
return sum + new Set(
g.nodes.filter((n) => n.kind !== 'import' && n.kind !== 'export').map((n) => n.id),
g.nodes.filter(isListedSymbol).map((n) => n.id),
).size;
}, 0);
let summaryLine = survivors.length > 0