Files
codegraph/docs/design/explore-budget-allocation.md
Colby Mchenry 003305f1e7 fix(explore): claim complete source only for sections that are complete (#2077)
* fix(explore): claim complete source only for sections that are complete

On the tiers with a completeness signal (>= 500 indexed files) every
codegraph_explore response ended with "Complete source for N files is
included above — do NOT re-read them", whatever the render had cut. A
windowed spine method, a shrunk or dropped cluster and the per-symbol view
all went out under that line; the oversize-spine window never even set the
trim flag the small tiers key off, which is how a 62-line slice of vscode's
968-line rpcProtocol.ts was called complete. The line also said "Reserve
Read for a single specific line range", which explore output must never
say.

Completeness is now measured at emission from the ranges actually sent
(plus back-referenced spans) against the symbols each section set out to
deliver. A trimmed response keeps the verbatim / already-Read guarantee,
names the trimmed files and the most relevant elided symbols, and steers to
another codegraph_explore. The note is offered from most to least specific
and fitted to the leftover room (CG-26); a trimmed note yields the pointer
list's first entry. The lost-pointer, cut and truncation notes drop
"complete" the same way, at no greater length, so the epilogue floor is
unchanged. The small tiers' trimmed note now also fires for the cuts the
flag missed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(explore): offer an elided method as Owner.member in the completeness note

The note offered `as_sql` for django's elided SQLCompiler.as_sql — one of
110 definitions — so agents appended a path and line to disambiguate it,
and that query shape returns none of the body. `SQLCompiler.as_sql` alone
returns most of it. Methods are now offered as `Owner.member` from the
indexed qualified name; everything else keeps its bare name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(explore): fit the completeness note and pointer list in one tested function

A trimmed note's optional detail (the files, the elided names) never costs a
pointer entry the least specific note would have left. The fit moves out of
handleExplore into fitExploreEpilogue so both rules are pinned by tests; the
second-try branch it had could never fire and is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(explore): completeness note wording after #2068 flags the spine window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(explore): record the completeness-note replay and agent A/B

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(explore): address review of the completeness note

- Judge completeness AFTER the dedup fold: a remainder too small to fence
  is sent in neither call, so a section with one is trimmed, not complete.
  The whole-file render now declares its symbols too (clamped to the lines
  it prints; the file node is not one), so a fold there registers.
- Label trimmed files unique among every path the response can name, the
  pointer list included.
- "1 file", and no "0 files" when every section was held from an earlier call.
- The lost-pointer, cut and truncation notes are one exported table,
  EXPLORE_FALLBACK_NOTES, with the length rule tested; the lost note picks
  its wording from trimmedShown instead of re-joining the response.
- The design doc's follow-up list marks the overclaim fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:54:00 +00:00

68 KiB
Raw Permalink Blame History

Explore budget allocation — the instrument and the baseline

codegraph_explore has a fixed byte envelope (getExploreOutputBudget().maxOutputChars, hard-capped at 25K so the host never externalizes the result). How that envelope gets divided among files is decided by a long chain of gates, tiers and caps spread across handleExplore — and until CG-4 that chain was unobservable. You could read an explore response and guess; you could not say "this file took 16% and that one took 20%."

This document covers the diagnostic that makes it measurable, and the baseline it recorded.

The diagnostic

Set CODEGRAPH_EXPLORE_DEBUG and every codegraph_explore call (MCP tool or codegraph explore CLI) emits one report:

value sink
1 / true / on / yes / stderr human-readable table on stderr
json one pretty-printed JSON report on stderr
anything else treated as a path — one JSON report per line, appended (JSONL)
unset / 0 / false / off / no / empty off

Per file it reports: relevance score, graph (RWR) mass, distinct query-term hits, ranking flags (named / entry / central / spine / low-value / generated), render mode, bytes of source allocated, bytes actually delivered, both shares, and whether it was clipped. For files that never rendered it reports why (max-files, budget-90pct, budget-whole-file, budget-clusters, unreadable, no-ranges). Totals cover the envelope (delivered vs allocated vs maxOutputChars vs hard ceiling), the source/meta split, the file-selection funnel at each stage, and the thresholds applied (score floor, graph-relevance gate).

It is off by default and produces byte-identical output when off — it ships in the product binary, and a diagnostic that perturbs the response by one byte would invalidate every A/B measurement taken with it on. ExploreDiagnostics.start() returns null unless the env var is set, so every call site is a diag?. no-op. Pinned by __tests__/explore-diagnostics.test.ts.

Two envelope numbers, deliberately kept separate:

  • allocated — what the render loop chose to emit, before the final hard-ceiling cut. This is the allocator's own decision, and the number budget work is about.
  • delivered — what the agent actually received.

They diverge exactly when the ceiling truncates. Conflating them is how a dropped trailing file goes unnoticed.

Baseline (2026-08-03, this repo at main)

codegraph explore "how does explore allocate its output budget across files" --path .

469 files indexed → small tier (maxOutputChars 18,000, maxCharsPerFile 3,800, defaultMaxFiles 5). Envelope: 23,196 delivered / 23,193 allocated against an 18,000 budget — 29% over, absorbed only because the 25K hard ceiling sits above it.

# share bytes score graph hits flags render file
4 21.2% 4,928 10 0.125 1 entry whole scripts/agent-eval/offload-eval-hook.mjs
5 20.1% 4,665 10 0.125 1 entry whole scripts/agent-eval/offload-eval-metrics.mjs
3 19.8% 4,585 22 0.125 1 entry central whole scripts/agent-eval/parse-session.mjs
1 15.8% 3,659 54 0.322 4 entry central clusters* src/mcp/tools.ts
2 15.0% 3,479 34 0.082 2 entry clusters* src/index.ts

* clipped. Ranked but never rendered: scripts/agent-eval/offload-eval-cost.mjs (#6) and src/resolution/lru-cache.ts (#7), both cut by maxFiles.

Files: 17 grouped → 10 past the score floor (≥3) → 10 past the low-value filter → 7 past the relevance gate (graph ≥ 0.0193, 6% of max 0.3215) → 5 in the output.

What the baseline shows

Score does not drive allocation. src/mcp/tools.ts — the file that actually answers the query — carries 5.4× the relevance score, 2.6× the graph mass and 4× the term hits of any .mjs script, and gets a smaller share than each of them. The three agent-eval scripts take 61% of the envelope between them; the answer file takes 16%.

The mechanism is that the two allocation paths are decided by file size, not relevance: a small file clears WHOLE_FILE_MAX_LINES/WHOLE_FILE_MAX_CHARS and ships entirely, while a large file falls through to cluster selection and is clipped at maxCharsPerFile. So a weakly-relevant 130-line script gets 100% of itself; the strongly-relevant 5,000-line file gets 3,800 chars. Rank ordering is correct (tools.ts sorts #1) and buys nothing, because rank has no effect on how many bytes a file receives.

The envelope is over-subscribed. 23,193 allocated against an 18,000 budget means the per-file caps do not compose into the total cap; the total is enforced only by the 25K ceiling silently dropping whole trailing sections. Under a slightly different index state (one more candidate file) the same query allocated 27,518 chars and the ceiling dropped a 7,678-char section — the single largest allocation in the response — with the only trace being the truncation notice at the end.

This is the gap the rest of the epic closes: relevance-proportional allocation with a relative cliff (CG-12), on top of scoring that stops rewarding incidental name collisions (CG-10 — landed; see below).

CG-10 — relevance scoring

CG-10 changes what gets into the response, ahead of how bytes are split among what's in. Four levers, all multiplicative so they compose without ordering surprises.

1. Kind weighting

The tier a symbol reached us by (named seed +50, query match +10, adjacent to one +3, peripheral +1) says how it got here; RELEVANCE_KIND_WEIGHT says whether the match is evidence. Callables and types weigh 1.0, members ~0.5, and constant/variable/ parameter 0.15–0.35 — a local named explore is a name collision until something corroborates it.

Isolation. For a weak-kind symbol in the top two tiers, "is anything using it?" is the corroboration: no usage edge anywhere in the graph (contains excluded — lexical nesting is not usage) drops it to 0.08. Cost is bounded — only weak kinds in the tiers whose weight can carry a file pay for the probe, and the subgraph's own edges answer most cases for free. Measured: no latency change (210 vs 211 ms/call, n=12 interleaved).

Peripheral cap. Nodes ≥2 hops from any match now accumulate into a separate bucket capped at 5. Uncapped, every such node added a flat +1, so a file grew more relevant by being bigger — parse-session.mjs reached score 22 off one incidental constant plus twelve unrelated symbols. Size is not evidence.

2. Relative score floor

score >= 3 admits noise on any repo where the top file scores 50+. The floor is now clamp(topScore × 0.2, 1, 10):

  • relative — on a diffuse question no file dominates, every candidate sits near the top, and the whole spread survives; on a precise one it cuts the tail.
  • capped at 10 — one direct query match on a callable. A single full-strength match is never incidental, so no amount of concentration elsewhere may exclude it. Without this cap, one named-seed-heavy file pushed the floor to 21 and dropped a file the agent had named by class name (classes enter at +10, not +50 — named seeds are callables).
  • backfill — if fewer than 3 files survive, the best of what the floor cut comes back, but only from files with real evidence (≥ the absolute floor). If nothing survives, the backfill drops that requirement: returning "no relevant code found" when the gather did find candidates sends the agent straight back to grep.

3. Generated status in the score, not the tiebreak

rankPenalty(file) multiplies both the relevance score and the graph mass by 0.3 for generated files (0.5 for low-value ones). Applying it to the score alone would not have fixed #1500: the generated CRUD carries more graph mass than the hand-written use-case, and graph mass outranks score in the comparator. The penalty is self-normalizing — in an all-generated repo everything scales together and relative ranking is untouched — and it never hard-excludes: ask about the generated API by name and the named-seed tier still puts it first.

4. excludeLowValueFiles — the finding

The per-tier flag the task asked to reconsider was dead config: declared on ExploreOutputBudget and set per tier, but read nowhere. A later change had already made the test/spec/icon/i18n exclusion unconditional at all tiers. The flag is removed.

The substantive gap was in the detector, not the gating: isLowValue matched /\/(tests?|__tests?__|spec)\//, anchored on a leading slash, so a repo-root test/ directory — express, cobra, and most of npm and Go — never matched. Express's "how does express route a request to a handler?" spent 59% of its envelope on three test files while lib/application.js was clipped. Anchored at ^ as well, that query now returns lib/application.js + lib/response.js and no tests.

Two related changes: the filter now runs before the score floor and judges "are there other candidates?" on the whole gather rather than the post-floor set (judging it after was how the floor's keep-minimum pulled test files back in as the "spread"); and low-value files that survive the filter's ≥2 non-test candidates escape hatch are down-weighted via rankPenalty rather than left at full strength.

Measured effect

Before/after on the same indexes, deterministic (CODEGRAPH_EXPLORE_DEBUG diagnostic, both arms same build system, baseline = bd86ad2):

repo · query before after
this repo · self-query fixture 72% to eval scripts, tools.ts 18.5% scripts 0%, tools.ts #1
this repo · handleExplore buildFlowFromNamedSymbols … 82% to eval scripts tools.ts 48% + index.ts 32%
this repo · "how is error handling done" 58% to eval scripts, tools.ts delivered 0 transport/tools/cobol/api
this repo · "what languages does codegraph support" 63% to scripts/add-lang/* grammars/index/cli
this repo · "main components of the indexing pipeline" — byte-identical
payroll-go fixture generated 57.4%, answer 25.6% answer 61.5%, generated 23.5%
express · route a request 59% to test/* application.js + response.js
cobra · 3 queries — byte-identical

The two byte-identical rows are the control: where the answer was already concentrated, the new floor prunes the same tail earlier and cheaper and arrives at the same response.

Known thin case. Express's "how does the app object get created and what does it expose" drops from 4 files (top one an examples/ file at 38%) to lib/express.js alone, 2.6 KB against a 13 KB budget. lib/application.js matched on nothing but an unused file-scope var app — indistinguishable, at the symbol level, from the eval scripts' unused const explore; express models its API surface as properties assigned to that object, which the graph has no edges for. That is extraction coverage, not ranking. Backfilling it was tried and rejected: node-count ties handed the slot to examples/route-middleware instead, at 48% of the envelope. Thin-and-precise beats padded-with-noise — a wrong file does not save the agent the follow-up call it would pad against.

CG-12 — score-proportional allocation

CG-10 fixed what gets into the response. This fixes how the bytes are split among what got in — which, until now, was not really decided at all. Every admitted file was capped at the same flat maxCharsPerFile, and the whole-file rule handed anything under maxCharsPerFile × 3 its entire contents. So the envelope followed file size:

  • self-query: memory-budget.ts (score 18) shipped whole at 5,672 and took 51.2%; tools.ts (score 41, 4× the graph mass, 3× the term hits — it literally holds the allocator) was clipped at 3,800 and got 32.9%.
  • payroll-go: two generated CRUD files shipped whole at ~4.5 KB each and consumed two of the tier's four file slots, so BuildPayslip — the hand-written "calculate" half of the question — ranked #6 and never rendered at all.

The model

allocateExploreBudget (src/mcp/tools.ts) runs once, after ranking, before anything renders. It reserves each file a share of the envelope; the render loop then spends a reservation instead of racing for whatever the files above it left.

  1. Weight = score × worth × (spine ? 2 : 1). worth is rankPenalty applied a second time: ranking answers "is this file about the query", allocation answers "will these bytes teach the agent anything". Generated CRUD can legitimately rank — it name-collides on every domain word, and it is big and densely self-referential, so it scores on the structural keys the comparator leads with — while its bytes stay mechanical boilerplate. That second penalty is what finally sinks it.
  2. Relative cliff at 15% of the top weight, itself capped at SCORE_FLOOR_MAX. A file under it gets zero source — path, symbols and line numbers only. It costs ~100 chars instead of ~4,500, and it does not consume a maxFiles slot, so the slot passes to a file that earns its bytes. That slot hand-off is what got BuildPayslip into the response. The cap matters as much as the fraction: one 500-scoring god-file would otherwise put the cliff at 75 and silence every peer the score floor had just admitted.
  3. Floor then split. Every admitted file gets MIN_CHARS (700 — enough for one complete method); the remainder splits by weight. The floor is what keeps a diffuse survey question returning a spread; the remainder is what concentrates a precise one.
  4. Safety valve, not a per-file cap: no file exceeds 70% of the envelope. The flat per-file cap is retired as the primary guard — the proportional split already bounds a file by its weight share.

The reservation then governs every render path — whole-file, clusters, focused/skeleton — where before the whole-file branch was 3× more generous than the cluster branch, which is the 3× swing that decided the split by file size.

Two supporting changes were needed to make the reservation actually bite:

  • Oversize clusters shrink by member. A cluster is a merge of whole symbol ranges, and on a densely-packed file every symbol merges into one blob spanning the file (cycle.go's 209-line Service). The old rule took the top-ranked cluster whole however big it was, so a single-cluster file simply ignored its budget — it took ~40% more than allotted and the file below it was then dropped for lack of room. Shrinking drops whole members by importance, so a body is still never cut.
  • The arrival-order stops are gone. budget-90pct and the `!fileNecessary && totalChars

    maxOutputChars` checks dropped files by the order they were reached: whichever files ranked first spent the envelope, and everything after them was cut on a cap it had no say in. Only an absolute hard-ceiling stop remains.

Measured effect (CG-12)

repo · query before (post-CG-10) after
this repo · self-query fixture tools.ts 32.9%, memory-budget.ts 51.2% tools.ts 60.6%, memory-budget 17.2%
payroll-go fixture answer 61.5%, generated 23.5%, BuildPayslip absent answer 78.7%, generated 0%, BuildPayslip delivered
express · route a request 2 files, top 43.1% 1 file, top 82.3% (response.js cliffed — 0 term hits)
express · app registers middleware 1 file, 71.6% byte-identical
cobra · parse flags and execute 2 files, command.go 40.1% 2 files, command.go 77.2%
cobra · DIFFUSE "main components" 3 files, top 48.1% 3 files, top 50.0% — spread preserved
gin · request reaches a handler 3 files, top ginS/gins.go 48.8% 3 files, top routergroup.go 53.8%
gin · DIFFUSE "what it provides" 3 files, top recovery.go 43.0% 4 files, top context.go 33.3%

The two diffuse rows are the over-correction control: file counts hold (3→3, 3→4), so a survey question still gets a spread. The two gin rows also moved the top file to a more apt one — routergroup.go over the thin ginS singleton wrapper, context.go over recovery.go — because concentration is decided by weight rather than by which file happened to be small.

Exception to "no previously-unclipped file becomes clipped". memory-budget.ts was unclipped-whole at 5,672 and now clusters within its 3.1 KB reservation. That is the epic's own diagnosis of the bug rather than a regression: it scored 18 against tools.ts's 58 and was taking the larger slice purely for being small enough to ship whole. The guarantee holds where it was meant to — no file loses bytes to a tighter cap; the only files that lose are ones the proportional split says were over-served.

The regression fixtures (CG-6)

Two fixtures pin the failure mode so it can never silently return. They were written to fail — that is what they were for. CG-10 closed the ranking half of both and CG-12 the byte-split half; both now pass and are live regressions. The numbers quoted below are the pre-CG-10 baseline; see the two "Measured effect" tables above for where they stand.

They are declared in scripts/agent-eval/allocation-fixtures.json and run by scripts/agent-eval/probe-allocation.mjs, which drives the CG-4 diagnostic through a JSONL sidecar (so it measures the shipping allocator, not a re-derivation), groups the rendered files into answer vs incidental, and checks declared share thresholds. Needs a current npm run build; exits 1 while any assertion fails.

node scripts/agent-eval/probe-allocation.mjs                # both
node scripts/agent-eval/probe-allocation.mjs payroll-go     # one
node scripts/agent-eval/probe-allocation.mjs --json         # machine-readable

1. payroll-go — the reporter's shape

__tests__/fixtures/payroll-go/ is a synthetic Go service: generated FKIT CRUD beside a hand-written payroll use-case, entered from an HTTP route. Full description in that directory's README. The essentials:

  • Generated files with ordinary names carrying // Code generated ... DO NOT EDIT. — invisible to path-only detection, which is what makes this a #1500 fixture rather than a .pb.go one — beside payrollpb/*.pb.go covering the path-detectable channel.
  • Deliberate collisions: BuildPayslip, Upsert and Store each exist twice, generated and hand-written, and the generated layer name-collides on every query term.
  • cycle.go (227 lines) sits above the whole-file window so it clips; the generated files sit below it so they ship whole.

Query — an architecture question naming none of the answering symbols: "how does payroll cycle create and calculate payslips?"

allocated delivered
hand-written 48.4% 25.6% (all of it domain types)
generated CRUD 39.9% 57.4%

cycle.go is allocated the single largest slice (7,052 chars, 30.6%) and delivers zero — the 19,500 hard ceiling drops its whole section. payslip_builder.go (rank #8) never renders. So runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert never reach the agent, and every byte that did arrive describes either CRUD or types.

This fixture is hermetic: the probe copies the tree to a temp dir and re-indexes per run, so two runs on one build are byte-identical (verified). __tests__/explore-allocation-1500.test.ts runs the same assertions in vitest.

After CG-10 the generated files rank #3/#4 instead of #1/#2, cycle.go delivers 38.9% (it delivered nothing), and runPayrollCycleAll + the real s.store.Upsert(ctx, slip) reach the agent. Those assertions are now live regressions. What remains it.fails is payslip_builder.go: it ranks #6, the tier's maxFiles is 4, and the render loop still spends by file size — CG-12's job.

Finding, deliberately left unfixed: runPayrollCycleAll calls s.store.Upsert on a *payslipstore.Store, but the graph resolves that edge to the generated internal/gen/fkit/payroll/store.go Store.Upsert. Same-name method resolution across two packages that both define Store.Upsert picks the wrong receiver. It is upstream of allocation — a wrong edge pulls the generated store into the subgraph. CG-10 mitigates the symptom (the generated store is penalized on both score and graph mass, so it no longer displaces the real one) without fixing the resolution bug itself, which belongs with the same-name method resolution work (see samename-method-resolution-1079).

2. self-query — the same bug with no generated code in sight

The baseline above, promoted to a fixture: this repo, "how does explore allocate its output budget across files". scripts/agent-eval/*.mjs mention explore and BUDGET incidentally — they are eval harnesses, not the allocator — and they are small enough to ship whole, while src/mcp/tools.ts is large enough to be clipped.

At 493 indexed files (small tier): the script corpus takes 71.8% of the delivered envelope (79.4% allocated) against tools.ts's 18.5%, despite tools.ts scoring 46 vs 10, carrying 2.3× the graph mass and 3× the distinct term hits.

This fixture reads the live index of this repo, so unlike payroll-go its exact numbers move as the repo changes. Its assertions are relative for that reason (answer group vs incidental group, largest delivered file), never fixed percentages. Two things to know:

  • The <500-file tier boundary is close. This repo indexes 493 files including the new fixture; crossing 500 flips maxOutputChars 18,000 → 24,000, maxFiles 5 → 8 and maxCharsPerFile 3,800 → 6,500, which moves every number in the table above. Re-baseline after the crossing rather than treating the drift as a regression.
  • Adding the payroll-go fixture itself moved the count 472 → 493. Its Go files match none of this query's terms, so they change the tier arithmetic and nothing else.

Reproducing

The query explores this repo, so uncommitted edits to src/mcp/tools.ts change the result — the index picks them up and scores shift (the same query on the CG-4 working tree reported tools.ts at 13–19% depending on the sync state). Measure against a clean tree: restore src/mcp/tools.ts from main, remove src/mcp/explore-diagnostics.ts, codegraph sync, then run the built dist/ binary (which still carries the instrument). Restore afterwards.


CG-14 — locking the allocation down

The allocation change is one function plus three render-loop bounds, and every way it can regress is silent: nothing throws, the response just gets less useful and the agent falls back to Read. So the coverage is built around the question "would this test go red if the lever were removed?" rather than around line coverage.

Where the coverage lives

File Owns
__tests__/explore-proportional-allocation.test.ts allocateExploreBudget in isolation — the split, the cliff, the tier invariant, envelope safety, spine weighting, degenerate inputs
__tests__/explore-allocation-e2e.test.ts The same behaviours through the real render loop on real indexed projects — the self-query fixture's shape, degenerate result sets, the diffuse-query control
__tests__/explore-allocation-1500.test.ts The reporter's Go shape (CG-6 fixture 1), plus the hard-ceiling stress case
scripts/agent-eval/probe-allocation.mjs Both CG-6 fixtures against the built dist/, including the live self-query arm that reads this repo's own index

The split between the last two is deliberate. The self-query fixture reads a moving target (this repo), so its exact numbers drift with the tree and it belongs out of band, where a drift is a number to re-baseline rather than a red suite. npm test owns a synthetic mirror of it instead: same three roles, same size asymmetry, fixed.

The two bounds are not the same bound

Worth stating once, because a test that conflates them looks right and passes for the wrong reason:

  • maxOutputChars bounds the RESERVATIONS. sum(allowances) <= pool <= maxOutputChars, exactly, at every tier and every candidate shape.
  • hardCeiling — min(maxOutputChars * 1.5, 25000) — bounds the RESPONSE. The render loop is allowed a bounded overshoot (the whole-file grace, an oversize first cluster), so a response legitimately exceeds the envelope. payroll-go does exactly that: 19.3K delivered against a 13,000 envelope and a 19,500 ceiling.

Only the 25K is absolute. Above it the host writes the result to a file the agent Reads back, which is the failure the tool exists to prevent.

Mutation-tested, not just green

Each lever was removed from src/mcp/tools.ts in turn and the suite re-run. Every one is covered by at least one failing test — a lever with no red test is a lever that can be deleted by accident:

Mutation Tests that go red
Render loop reverted to pre-CG-12 (fileBudget = maxCharsPerFile, whole-file bound maxCharsPerFile * 3) 5 e2e + 2 payroll
Proportional split replaced with an equal one 4 unit
Cliff disabled (cliffAt = 0) 7 unit + 3 payroll
Spine boost and cliff exemption removed 3 unit
MIN_CHARS floor removed 2 unit
MAX_SHARE ceiling removed 4 unit

The first row is the one that matters most: it reproduces #1500 on the synthetic fixture verbatim, with the half-as-relevant file taking the larger share purely on size.

file score pre-CG-12 CG-12
src/mcp/allocator.ts 77.5 4,843 (39.7%) 9,335 (80.1%)
src/util/budget-math.ts 36.0 6,079 (49.8%) 1,037 (8.9%)

Two defects the coverage surfaced

Both were found by writing the invariant rather than by reading the code:

  1. Rounded shares could exceed the pool. Math.round on each file's proportional slice let the reservations sum past pool by up to half a char per file — small, but it made "reservations fit the envelope" false rather than exact. Both terms now floor.
  2. A non-finite score produced a NaN allowance. An Infinity weight makes every share Infinity / Infinity. Scores are finite sums in the pipeline so it was unreachable, but the failure mode is a NaN handed to the render loop. weightOf now fails safe to 0.

Calibration vs. invariant

EXPLORE_ALLOCATION is exported so the tests can read it. Invariant tests (envelope safety, tier monotonicity, the floor) reference the constants and hold at any value; one test pins the literals, so re-tuning a constant is a visible decision that says re-run the probe rather than a silent re-calibration of the fixtures.

CG-15 — what the agent A/B found: a reservation can go unspent

The agent A/B (full record: ../benchmarks/explore-allocation-ab-1500.md) passed on both medium repos — client-go and excalidraw at Read 0 in every run, excalidraw 34s → 24s median with one fewer explore call, the generated clientsets that took 10.5% of a baseline envelope gone from every new run — and failed on the small control, express, in 1 run of 3: 4 Reads of lib/utils.js where the baseline made 1 Read of a different file.

It is not agent variance. Replaying that run's own query deterministically:

lib/utils.js (5,293 B, 272 lines) baseline CG-12
delivered 6,380 (46.1%) whole 583 (7.7%) cluster stub
source envelope (budget 13,000) 13,849 9,241

The diagnostic says the allocator was right and the render loop was not:

allocation 12,398 reserved of 12,400 pool · nothing cliffed
 #  deliv%   bytes  reserved  score   flags                render    file
 1    5.7%     583     3,870   56.0   named entry central  clusters  lib/utils.js

utils.js is the top-ranked file and was reserved 3,870 chars — of which it spent 583. The whole-file bound (src/mcp/tools.ts:4008) is allowance + min(GRACE_MAX, allowance * GRACE_FRACTION) = 3,870 + 580 = 4,450, just under the file's 5,293 bytes, so the whole-file render is declined; the fallback cluster render has three matched symbols to work with and emits 583 chars. The remaining 3,287 chars of the reservation are not redistributed — they are lost, which is why the response shrank by a third against an unchanged budget.

This is CG-12's own acceptance criterion ("no file that was previously unclipped becomes clipped") failing. CG-14 recorded one instance as a documented exception (memory-budget.ts); this is the same defect in the wild, where it costs an agent round-trip. It is not systemic: on both medium repos the render loop saturates ([over budget] [TRUNCATED], 23,599 of a 23,600 pool reserved) so there is nothing left to lose. It needs a file whose reservation lands below its own size while its matched-symbol set is thin — likeliest on small repos, where per-file reservations are smallest.

The two candidate fixes

Widening the envelope is explicitly not one of them — that is the dial iter2 already proved doesn't work, and the bytes here were reserved for this file already.

  1. Let a large-enough reservation buy the whole file. Render whole when allowance >= k * fileSize for some k < 1 (utils.js sits at 0.73), letting the bounded overshoot the hard ceiling already tolerates absorb the difference. Smaller change, matches the observed shape.
  2. Redistribute the shortfall. Once the render loop knows a file's realised size, hand what it cannot spend to the next-ranked file. The stronger invariant — the pool is spent — and it fixes the shrinking-envelope symptom directly rather than case by case.

They compose; (1) alone would have carried this case. Whichever lands needs the express shape as a hermetic fixture — a mid-sized top-ranked file with few matched symbols, sized just above its reservation — because nothing in the current suite has that shape, which is how it shipped.

CG-21 — spending the reservation

Both fixes landed, because they cover different halves of the same failure and neither is sufficient alone. The unifying rule: a reservation is a promise the render loop has to keep, not a cap it may quietly under-use.

1. A reservation that has already bought most of the file buys the rest

WHOLE_FILE_BUY_FRACTION (0.6). The pre-existing grace is calibrated as a sliver — it rescues a file that essentially fits. Between "fits" and "several times its reservation" there was a hole, and lib/utils.js (reserved/size = 0.73) sat squarely in it. So the test is not does the file fit the reservation but has the reservation already bought most of the file: above 0.6 the loop pays at most two-thirds of a reservation extra rather than lose the whole thing, on a file that already earned those bytes.

Funding is the part that is easy to get wrong. The merit test is a ratio, so wherever several files sit near it they all qualify — and N independent overshoots inflate the response until the render ceiling drops whatever is last. Funding each buy from the file's own headroom was tried and measured on the payroll fixture: three files bought whole and payslip_builder.go — the file that computes the payslip the question asks about — was dropped entirely so three higher-ranked files could each ship their final sliver. A dropped section is strictly worse than a clustered one.

So the buy rule is two independent tests, and keeping them apart is the whole design:

reads answers
merit the file's own reserved did this file's relevance earn most of itself?
funding one shared overshoot pool (WHOLE_FILE_BUY_OVERSHOOT_FRACTION, 15% of the envelope) do the bytes exist, with every reservation below it still payable?

Merit reads reserved rather than the post-carry allowance, so borrowed slack can never promote a weak file to whole. Funding is measured against what the allocator promised rather than against renderCeiling — the ceiling sits 50% above the envelope and says nothing about who is owed what, so funding a buy out of it just moves the shortfall onto whichever file the loop reaches last. The owedBelow term is what makes it a displacement guard rather than a size cap, and it is self-limiting: each buy grows sourceSpent, so the pool cannot be spent twice.

2. What a file cannot spend goes to the file below it

Below the buy fraction the shortfall is real — the file is several times its reservation and clustering is the right render — but the bytes still must not evaporate. The render loop carries two running totals, everything promised so far and everything emitted so far, and their gap is slack the next file may add to its own reservation.

Two totals rather than a spent variable threaded through the loop's dozen continues, so no exit path can forget to account — unreadable, drifted off disk, skipped for the ceiling, thin matched set. It is symmetric: a buy that overshoots makes sourceSpent outrun reservedSoFar, which suppresses slack until a later under-spend covers the debt, so the pool is conserved in both directions and no file is ever cut below what it was promised. Slack flows in rank order (the only file a single-pass loop can still pay), clamped by MAX_SHARE so an under-spending leader cannot hand a weak tail file the whole response.

Measured effect (CG-21)

Express, the reproducer, same index and same 13,000 budget:

lib/utils.js (5,293 B, 272 lines) baseline CG-12 CG-21
delivered 6,380 (46.1%) whole 583 (7.7%) stub 6,268 (39.3%) whole
source envelope 13,849 9,241 14,505

And the self-query, where the epic's own acceptance criterion was outstanding:

file pre-CG-12 CG-12 CG-21
src/mcp/tools.ts (score 58) 32.9% 60.6% 52.6% (10,945, clusters)
src/resolution/memory-budget.ts (score 18) 51.2% whole 17.2% clustered 27.3% whole (5,672)

The memory-budget.ts exception is resolved, not re-justified. CG-14 recorded it as a documented exception to "no file that was previously unclipped becomes clipped"; its reserved/size ratio is 0.73 — the same window as express's utils.js — so the buy rule covers it. The answer file still wins the envelope and nothing that used to ship whole is clipped, which is the first time both halves of CG-12's acceptance criterion hold at once.

The synthetic mirror in explore-allocation-e2e.test.ts deliberately does not move: its helper sits at ratio ~0.17, far below the buy fraction, so it still clusters — correctly. That divergence is the point of the fraction: memory-budget.ts was over-served by a sliver, the synthetic helper by a multiple.

Coverage

Two new hermetic fixtures, one per lever, because the existing suite could not see either. The payroll and self-query fixtures both saturate ([over budget] [TRUNCATED], 23,599 of a 23,600 pool reserved), and a saturated response has no unspent reservation to lose.

Fixture Shape Guards
CG-21 — a reservation under the file size still buys the file mid-sized rank-1 file, thin matched set, sized just above its reservation and outside the grace bound the buy rule
CG-21 — an unspendable reservation flows to the next file down rank-1 file far too big to buy (ratio 0.24) under-spends; a dense rank-2 file absorbs it the carry-forward

Each carries a fixture shape block that asserts the window it depends on (0.6 × size <= reserved < size, outside the grace bound, inside the 220-line whole-file cap). Those are load-bearing, not scaffolding: every gate passes vacuously if a target ever drifts small enough for grace to cover it, which is precisely the way this defect hid.

Mutation-tested, same method as CG-14:

Mutation Tests that go red
Buy arm removed (buysWhole = fileContent.length <= graceBound) 3 (new buy fixture)
Carry-forward removed (allowance = reserved) 2 (new carry-forward fixture)
Funding guard removed (… && true) 4 — including payroll's payslip_builder.go dropped

Two traps the first drafts fell into, both of which made a test pass on the defect:

  • The spine ceiling hides a per-file assertion. SPINE_CEILING already lets a flow-path cluster reach 1.5× its allowance, so "delivered > reserved" is true without any carry-forward. The carry-forward fixture is built spine-free and asserts a 1.1× margin — measured 9,297 vs 7,479 across the mutation.
  • "Spend the whole pool" is not the invariant. A file smaller than its reservation legitimately under-spends it (the fixture's response.ts: 1,635 delivered of 5,292 reserved). The assertion is per-file — delivered >= min(reservation, file size) — which is what express's utils.js violated and a pool-sum assertion does not express.

A third condition, found by review rather than by a test

A buy must also fit the render ceiling. The whole-file branch refuses to slice a file mid-method, so a whole render that overruns renderCeiling is skipped entirely — meaning a buy approved by the funding pool but refused by the ceiling trades a clustered section for no section. That is the same trade the funding pool exists to refuse, arriving by another route.

It is reachable only on the 24K tiers, which is why neither new fixture can see it:

tier envelope renderCeiling = min(1.5x, 25000) - 600 funding line = reservedTotal + 0.15x
small 13,000 18,900 ~14,350 — cannot cross
medium/large 24,000 24,400 ~27,200 when saturated — crosses by ~2.8K

So buysWhole carries totalChars + size + FILE_OVERHEAD <= renderCeiling as well; failing it drops through to the cluster path, which is bounded by headroom and always renders something. The grace arm is deliberately untouched — a file within a sliver of its reservation that still does not fit is genuinely at the end of a full response, and that behaviour predates the epic. Verified inert on all three A/B repos (excalidraw and client-go byte-identical across 3 queries each, express reproducer unchanged), so it did not invalidate the measurement below.

The agent A/B (CG-15's gate, re-run)

The epic's gate is CG-22, not this section. This run was measured by the task that wrote the fix; CG-22 re-ran it at CG-15's exact setup (RUNS=3, fresh clones, baseline pinned to 49c11fc by SHA) and it passes all four bars there too — Read = 0 in all 12 new-arm runs while the baseline reads in 3 of 3 express runs, and the deterministic reproducer holds with both builds re-measured in one session (lib/utils.js 6,380 B whole on both, envelope 13,849 → 14,913). See ../benchmarks/explore-allocation-ab-1500.md § CG-22. The two counter-points that section records — excalidraw's answer share running below its baseline, and client-go's +1.5s median — are unresolved-but-attributed, not hidden.

Full record: ../benchmarks/explore-allocation-ab-1500.md § "Re-run after CG-21". All four bars pass, at n=6 per arm on express and excalidraw:

  • Read = 0 in all 15 new-arm runs. The express regression that routed the defect here (4 Reads of lib/utils.js) does not reproduce in 6 attempts — and the baseline reads in 4 of 6, so the control now beats the arm it previously lost to. Median 24.5s → 21.5s.
  • client-go — the reporter's shape — holds 92.7–96.2% answer share against a baseline run at 53.8%.
  • Excalidraw's ~8s median gap is not attributable to the change. Explore's own latency is 374 ms vs 372 ms (n=5, same query and index); deterministic responses differ by +2% with one byte-identical; and the unchanged main build's own median moved 34s → 26.5s between the CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is noise-dominated at this sample size, which is the known shape (host-model thinking dominates, not tool latency).

Named members before incidental ones (2026-09-28)

A query naming five independent functions in one file got two to four of them back. Repro, small tier: lib/response.ts holds sendBody, sendJson, renderView, redirectTo and sendFileStream, ~36 lines each, with three unrelated headerSlotN(res, value) helpers between each pair and ~45 more after. Query: the five names. On main, sendBody rendered 0 of 36 lines and sendFileStream 4 while helper bodies rendered in full. The same five functions written contiguously all rendered.

Two places the helpers were paid first

Between clusters. rankedClusters orders clusters of equal max importance by density, summed member importance over span. Density counts every member, so the helpers merged around a named function (a definition at 1, each setHeader edge line at 2) raised the density of its cluster, and a cluster holding a named function alone ranked below it: 0.39 against 0.95 and 0.84 in the repro. Taken whole, the two filler-heavy clusters spent the file's 9,100-char reservation and the isolated sendBody got nothing. Reordering does not fix this. Rank sendBody first and the victim moves to whichever named cluster is now last, because each cluster above it still pays for its helpers before the one below gets a turn.

Inside one cluster, at its ceiling. Every render is bounded by a ceiling, and for a cluster taken after the first, and for every re-render the ceiling fit makes, that ceiling is its room exactly. The member shrink estimates on raw source, which runs well under the rendered size (padding and line numbers). Incidental members were kept on that estimate, the render overran the ceiling, and the window back to it cut in source order, through the tail of a named body. That is gin's handleHTTPRequest at 61 of 71 lines once the first fix had given it room, and, with the named-concentration branch merged, vscode's _remoteCall at 23 of 36 when that branch's regrow re-rendered rpcProtocol.ts. When the 12-line minimum windows overran the room instead, the loop dropped the whole cluster, named members and all, and the room went to clusters with nothing named in them.

The rule

A member is protected if the query named it or it is an entry point (importance >= 9), or it sits on the rendered call path. Everything else in a cluster is incidental.

  • Before selection, protectedCoreCost prices each cluster's protected members rendered on their own, after the session history. owedFrom sums that cost over every cluster ranked below each position.
  • A cluster's incidental members may only use what is left of its room once the clusters below it are paid their protected cost. Its own protected members are chosen first, against the same cap as before, and are never held back. So when named bodies themselves do not all fit, the trade-off between them is still the one rank makes.
  • In any cluster holding a protected member, incidental members are also never kept past the render's ceiling, in rendered size. That is renderCluster's own rule, so it covers selection and every re-render alike: the render fits without being windowed, and nothing named is cut to make space for its neighbours. For the first cluster the ceiling is the 1.5× spine bound, so the rule only bites near the point where the render would be windowed anyway (a little before it, since each incidental member's padded cost is an over-estimate).
  • A cluster with nothing protected in it renders byte-for-byte as before, and so does one that fits its cap with room left for everything owed below it. A lone named function keeps the helpers merged around it (the test's control).

The incidental cap is measured against fileBudget even for a spine cluster: the clusters below it that are not on the spine are held to fileBudget, so room a spine cluster's incidental members take past it is room they never get.

Rejected: ranking by named density. Dividing only protected members' importance by span, in the named tier, stops filler from raising a cluster's rank. Measured on top of the hold-back (before the ceiling guard was added), it changed 4 of 47 responses, left complete named bodies where they were (218), and put tokio-pin over the 25K inline cap (25,007). Rank is left alone.

Measured effect

Deterministic, 47 fixed queries over the 7 README repos plus express: the README questions, the suite's prose questions, precise symbol bags, pinned-file flows and named-concentration stress queries (the set the unmerged named-concentration work was measured on). Baseline main e63fe2ec, same index for both builds. Named coverage means every definition of every queried name in every rendered file.

main this change
named bodies complete 198 234 (17 responses up, 0 down)
named lines delivered 6,912 7,631 (+10%)
byte-identical responses — 17 of 47
responses over the 25K inline cap 5 1
responses truncated by the final cut 2 3 (see below)

No definition that rendered whole on main renders partially, except one 5-line overload: tokio's park.rs park (5 → 2), in a response that completes three other named bodies while the 46-line park beside it grows (11 → 20). Otherwise what gets smaller is the un-named part of a whole-class container too big to keep as one member, whose body had been arriving as incidental members: Alamofire's Session (133 → 103 of 1,301 lines), vscode's RPCProtocol (194 → 129) beside a now-complete MessageIO class, ChannelClient (109 → 63) where the file's incidental lines are now chosen by importance rather than by source order. No named method inside those classes loses a line.

query complete named bodies named lines
vscode ipc.net.ts pinned flow 3 → 8 139 → 331
alamofire Request.swift pinned 2 → 6 18 → 90
tokio pb-worker2 7 → 10 104 → 168
tokio worker.rs pinned 6 → 9 182 → 278
gin gin.go ServeHTTP handleHTTPRequest addRoute 2 → 4 138 → 223
django symbol bag (execute_sql ×2) 5 → 7 201 → 261
excalidraw symbol bag (_renderStaticScene) 3 → 3 32 → 79

The one new truncation is django-bag, and it is not this rule's. The named file now spends 157 chars less than its reservation, the carry-forward moves that down the rank order, and the rank-3 file's funding rises from 963 to 1,777 because owedPayableBelow stops holding a partial share for the last file once that share falls under MIN_CHARS + overhead. A 32-char shift upstream crossed that threshold. The last file, a 151-char skeleton, is dropped, and the pointer list after it is cut by the final trim because the epilogue is never paid for in the render loop. The named-concentration branch reserves the epilogue; with both, the same query is not truncated and renders as many files as that branch does alone (below).

On the named-concentration branch

claude/inspiring-grothendieck-8a7b3e (fd8eab23, unmerged) was cut before #2057, and its own tip is the wrong base to measure against: there sectionText still measures gaps with their names while the window to a ceiling measures them bare, so a bound render that "fits" overruns its room and the cluster is dropped. Merged onto main (one conflict: keep the branch's buildSection and main's bare-gap sectionText), then this change on top (four hunks in shrinkCluster, renderCluster and the selection loop, all mechanical), the same 47 queries:

main + named concentration + this change
named bodies complete 235 279 (19 responses up, 1 down)
named lines delivered 9,330 9,965 (+7%)
responses over the 25K inline cap 0 0
responses truncated by the final cut 0 0

The response that goes down is tokio's pinned stress query, the same 5-line park overload as on main. That branch's own tests (explore-merged-spine-cluster, explore-named-file-valve) pass with this change merged in. The two hold-backs apply together to a merged spine cluster: that branch's limits the whole first cluster to leave the other clusters' top members room, and this one limits only incidental members, of any cluster. On vscode's rpcProtocol.ts that moves room from RPCProtocol's un-named methods to the named MessageBuffer class (66 → 187 of 187 lines, RPCProtocol 383 → 220), with every named method of RPCProtocol still whole.

On that branch a lone named file's reservation rises to 12,800, which absorbs the filler of the original repro, so the hermetic fixture uses wider helper lines. It is red there without this change (sendJson 0 of 26 lines).

Coverage

__tests__/explore-isolated-named-cluster.test.ts, one generated file in the repro's shape (26-line bodies, three wide helpers between each pair). Its shape block checks that helpers sit between every pair, that no named function calls another (so no call path ranks one), that the file is more than three times its reservation, that the named bodies cost under 90% of it, and that the same functions written contiguously all render whole under the same reservation.

Mutation Red
main's tools.ts five-name gate: sendBody 0/26
main + named concentration, without this change five-name gate: sendJson 0/26
No hold-back for clusters below (ceiling guard kept) five-name gate: sendBody 21/26
No ceiling guard on incidental members (hold-back kept) renderView redirectTo: renderView 22/26
Incidental members always dropped control: the helper above a lone renderView
Protected members not sorted first in the guarded shrink none — and all 47 sweep responses are byte-identical

The last lever is kept on reasoning alone. Named members already sort first by importance, so the sort only matters for a call-path member below importance 9 that shares a guarded cluster with a higher-importance incidental one. There it keeps the incidental member from claiming the cap first, and it keeps the size the incidental test starts from exact: that is the rendered cost of the protected set only when every protected member has been weighed before the first incidental one.

With the repro's exact geometry (36-line bodies), five padded bodies come to more than the 9,100 reservation. This change renders four whole and sendBody at 29 of 36, the lowest-ranked named cluster windowed to what is left. That is the budget, not the defect: with the named-concentration branch as well, all five render whole.

Found, not fixed

  • owedPayableBelow's partial hold has a cliff at MIN_CHARS + overhead. Just above it, the last admitted file is held its partial share. Just below it, it is held nothing, and the file above takes ~800 chars the last file needed. Any upstream shift of a few dozen chars can flip it (django-bag above).

Named concentration — merged spine clusters and the valve (2026-09-27)

Found in a pre-release agent A/B on microsoft/vscode (14,698 files, the 24K tier), identical on v1.6.0 and main. The query

MessageType enum rpcProtocol.ts _receiveOneMessage MessageIO serializeRequest serializeReplyOK serializeReplyErr serializeCancel serializeAcknowledged

came back at 8.6K. rpcProtocol.ts rendered lines 20–24 and 269–325: none of the five MessageIO.serialize* bodies, not the MessageType enum, and not the Flow section's own step 2. The agent Read the file five times.

It was not the per-file cap

The report's hypothesis was that the named file hit its per-file cap while the call budget sat unspent. The diagnostic said otherwise: rpcProtocol.ts was reserved 16,800 (the MAX_SHARE valve), funded 18,948, and spent 2,910, with the render reported as clusters, unclipped. The bytes were there. The render path threw them away.

buildSection windows an oversize spine method (n8n's 962-line processRunExecutionData) to its signature head plus ±28 lines of its next-hop call. It decided that on the cluster's span. A cluster merges every symbol within gapThreshold lines of the next, so a pinned file (whose whole symbol list enters the gather), or any dense gather, becomes one cluster spanning the file around a short spine method. rpcProtocol.ts was one cluster over lines 20–968 around a 77-line _receiveOneMessage: head 20–24, call site 297 ± 28. The window fit the cap, so nothing reported it as clipped, and the large tiers' completeness line told the agent the file was complete.

The window is a member rule

The size test is now on the spine member itself. A cluster that is long only because it merged renders plainly and, if it overruns, shrinks by member like any other cluster. A god-method inside a merged cluster is windowed where it sits, with its neighbours rendered around it. Six consequences surfaced while validating that change; each is fixed:

# Consequence Fix
1 The window elided source and reported the section unclipped A windowed spine member marks the cluster shrunk (clipped in the diagnostic, the trimmed note on small tiers)
2 Shrink ranked members by importance alone, so an unnamed spine bridge lost to smaller incidental members (vscode's _receiveRequest: 4/36 lines) Spine members first, as rankedClusters already ranks spine clusters first → 36/36
3 Past the 300-node pin cap a pinned file splits. Its spine cluster (receiveOne plus 300 unrelated methods) ranks first and, shrunk against its own members only, filled the cap with filler while the named serializers one cluster down rendered nothing For a merged spine cluster only, room for the other clusters' top members (spine, importance ≥ 9, never a container's whole span, never under MIN_CHARS) is held back; what they don't spend returns to it
4 Every budget upstream charges a gap as a bare GAP_MARKER, but gaps are named (#1711). A cluster shrunk to scattered members overran even the last-resort re-render, and the file was skipped whole The last resort drops whole parts (non-focus first, never the final part) until the real text fits
5 The adaptive (focused/skeleton) path had no ceiling fit test. With the named file now spending its reservation, tail skeletons ran past the ceiling and the final cut took them and the pointer list Same fit test as the other paths: trim trailing signatures, then trailing bodies; skip only if nothing fits
6 The summary line ("Found N symbols across M files. …") sits in lines as an empty placeholder and is filled in after the final cut, so nothing paid for it: responses ran up to 70 chars past the 25K inline cap, or squeezed out the lost-pointer note Reserved at its worst case (the gather's counts bound the shown ones) in the epilogue floor and in the epilogue fit

Two alternatives for #3 were measured and rejected. An unscoped hold-back (every first cluster) re-orders a trade-off that predates this change, and measured net-negative: tokio's run_task, a spine method, fell from 60 lines to 2. Charging containers (a named 1,022-line class) for their whole span over-reserved, and vscode's RPCProtocol lost 44 lines to room nobody used.

The valve, for a file the query named

This is the lever the report asked for. It turned out to matter less than the render bug, but it is real: on express, "response.js res.send res.json res.render …" admits one file, clamps it at 9,100 with 3,700 of the pool unreserved, and cuts the named send at 55 of 97 lines in a 10.9K response to a 13K budget. MAX_SHARE hedges against a mis-ranked dominant file, and it never redistributed: the clamped excess was left unreserved, and the carry-forward only moves reservations.

  • Allocation: a file the query named (the named-first tier, namedSeedFiles) or pinned is exempt from the valve; its proportional share stands. Reservations still fit the pool exactly.
  • Render: the part of that share above the valve is a soft share. The loop guarantees min(reserved, valve) (the fundedHeadroom floor, and what owedPayableBelow holds for it). The file may spend past the old rule only into spare budget: the call's maxOutputChars left after every file below is paid its full reservation, payable or not. The spine's 1.5× overshoot applies to the old rule's allowance, not to the extension. A soft share enters the carry-forward ledger once the loop has moved past the file, and only as far as it was spent.

Each part of the guard was the fix for a measured failure:

Variant What happened
Guarantee the whole uncapped reservation django (saturated 24K): two admitted files dropped so the named query.py could ship 3K more
Bound the extension by fundedHeadroom It holds back only the payable prefix of what is owed below (CG-31), so on vscode the room it released still belonged to a partially-payable rank-4 file, which was dropped
Measure spare against the render ceiling, not the budget The ceiling exists to absorb the render's own overshoot (named gaps, never-empty windows). django's named file overran its bound by 561 and a lower file was cut to 53% of its reservation
Book the soft share in the ledger up front What the file couldn't spend flowed down as slack that existed only in the ledger

Measured effect

Deterministic, 47 queries over the 7 README repos plus express. They are the README questions, the suite's prose questions, precise symbol bags, pinned-file flows, and named-concentration stress queries. Baseline is main 1cbac0af. Named coverage means every definition of every queried name in every rendered file.

baseline this change
named bodies complete 140 183 (14 responses up, 0 down)
named lines delivered 5,397 7,876 (+46%; 2 responses −3 and −9 lines, the summary-line reserve)
responses truncated by the final cut 3 0
responses over the 25K inline cap 5 (max 25,032) 0 (max 24,999)
byte-identical responses — 15 of 47

The reported query and the shapes it stands for:

query complete named bodies named lines response
vscode, the report 0 → 8 of 8 (+ the Flow's _receiveRequest) 46 → 356 8.6K → 24.9K
same, without the path 0 → 8 of 8 46 → 356 11.0K → 24.1K
django query.py pinned flow 1 → 6 87 → 317 10.9K → 19.3K
okhttp RealCall named concentration 4 → 9 187 → 732 13.8K → 24.7K
okhttp RealInterceptorChain.kt pinned 2 → 3 107 → 356 24,058 → 24,978
express response.js (the valve) 5 → 6 192 → 234 10.9K → 13.8K

Files fall in four responses: django-pin 6 → 5, vscode-bag 5 → 4, okhttp-pb-chain 6 → 4, excalidraw-pin 6 → 5 (that one was over the 25K cap on baseline). In each, the rank-1 named file used to spend a fraction of its own reservation because of the windowing bug, and the under-spend was passing down to incidental files. Of the five files that drop out, four are still named in the pointer list. The fifth (okhttp's ExchangeCodec.kt) is counted in its "… and 24 more files" line, which is CG-26's elastic list running out of room.

Coverage

File Guards
__tests__/explore-merged-spine-cluster.test.ts the member rule (pinned rpcProtocol shape), a real god-method still windowed with its named neighbours whole and the file clipped, the merged-spine hold-back past the pin cap, the last-resort trim at the 24K tier
__tests__/explore-named-file-valve.test.ts the valve for a named file far from its budget, and the spare-budget guard on a saturated 24K response
__tests__/explore-proportional-allocation.test.ts the allocator exemption: peers unchanged, pool fit at every tier, tier monotonicity

Mutation-tested, same method as CG-14:

Mutation Red
main's tools.ts 9 of the 19 new e2e tests (6 of 11 render, 3 of 8 valve)
Windowing not reported as clipped god-method clipped
No hold-back for other clusters past-the-pin-cap named bodies
No last-resort trim 24K file skipped (0 chars)
Allocator exemption removed, or render extension removed far-from-spent: 3 of 5 named bodies
Extension bounded by fundedHeadroom saturated: lower file dropped
Spare measured against the render ceiling saturated: lower file cut below its reservation

Four levers are validated only by the deterministic sweep. No hermetic fixture reproduced them without over-fitting:

  • spine-first member order: vscode's _receiveRequest, 4/36 → 36/36
  • top-up of unspent hold-back: alamofire's Session.swift, 876 chars and a named method short without it
  • adaptive-path fit test and summary-line reserve: 3 truncated and 5 over-cap responses → 0

Agent A/B

scripts/agent-eval/ab-new-vs-baseline.sh, baseline pinned to 1cbac0af by SHA, Sonnet --effort high, RUNS=2, prompts of the form Use codegraph to answer: <question>. Raw logs are not committed. The three questions: vscode's rpcProtocol serialization and dispatch (the reported shape); django's SQLCompiler.as_sql / pre_sql_setup / get_select; and tokio's worker run / run_task / next_task / steal_work / park.

repo Read (new / base) explore calls duration median do the builds differ on these queries?
vscode 0, 0 / 1, 0 3, 3 / 2, 3 69s / 48s yes: +82 to +320 source lines on the rpcProtocol.ts / MessageIO queries
django 0, 2 / 1, 1 3, 2 / 2, 2 43s / 30s no: 4 of 9 byte-identical, the rest ≤1 line apart except one tail-file swap
tokio 0, 2 / 0, 1 3, 3 / 2, 2 177s [49–306] / 31s no: 0–1 lines apart on 7 of 10, 22–46 on 3

Reads tie overall (4 / 4). On vscode, the one repo where the builds actually return different bytes, the baseline's Read was the reported symptom itself (rpcProtocol.ts lines 299 to 355 after an explore). The new arm's answers covered the named serializers in both runs (11 of 13 key symbols vs 9 and 11).

The new arm was slower in every repo. Attribution, per the CG-22 rule (never quote a wall-clock gap without the explore-latency and response-size probes):

  • vscode: model time was equal or lower in the new arm (30.7s / 23.4s vs 31.6s / 30.2s). The gap sat inside explore calls (39s per run vs 12–17s). Replaying the new arm's six queries, CPU per call was within 1–4% across new/base build × new/base index, while the machine's load average was 7–10 from other sessions. Load, not the change.
  • django, tokio: the two builds return near-identical responses to these agents' own queries, so a difference between the arms there cannot be the change. tokio's 306s run is a model-side stall (302s of API time, 1.2s in tools).
  • The new arm made 3 explore calls in all 6 runs vs 2.3 per run for baseline. On django and tokio that is sampling for the same reason. A larger n would settle whether vscode's richer responses prompt a deeper follow-up.

Every Read in the django runs, on both builds, is a separate pre-existing gap: SQLCompiler.as_sql's 226-line body is never returned, not even to a query that names it and its line number.

Found, not fixed

  • SQLCompiler.as_sql is never returned (above). It is the cause of every django Read in the A/B, on both builds.
  • Density ranking starves an isolated named cluster. Among clusters of equal max importance, rankedClusters orders by summed member importance ÷ span. Edge-line members inflate the density of a named-function-plus-filler cluster, so a cluster holding only a named function loses the room. Repro: five named functions interleaved with helpers, where two render 0 lines on both builds. Fixed by #2062 ("Named members before incidental ones", above).
  • A pinned file past the 300-node pin cap drops named symbols beyond the cap. Identical on baseline. The named symbols were in fact gathered (a search entry at importance 10) and injected; the render dropped them, by the same density starvation as the item above. Fixed by #2062; __tests__/explore-pinned-cap-named.test.ts (#2064) guards the pinned entry path.
  • The large tiers' completeness line ("Complete source for N files is included above") is unconditional, so it overclaims whenever a section was trimmed. Fixed by #2077: completeness is measured from the ranges each section actually sent (elidedWantedSpans), and a trimmed response says which files were trimmed; see docs/benchmarks/explore-completeness-note.md.

Landing on main (2026-09-28)

This work sat unmerged while #2062 and #2063 landed in the same render path, so it was re-applied onto main rather than merged as written:

  • The merged-spine hold-back (#3 above) and its top-up were dropped. #2062 holds back, for every cluster, room for the protected members (named, entry point, spine) of the clusters ranked below it. That is #3 without the scoping. The scoping existed because an unscoped hold-back of top-member room cost tokio's run_task its body; #2062's form never holds back a cluster's own protected members, only its incidental ones, which is why it can apply everywhere. The past-the-pin-cap test above (a merged spine cluster leaves room for the file's other clusters) passes on #2062's hold-back.
  • #2063's exact-target rule moves to the member. main skipped the spine window for any cluster holding an exact target (a qualified name, a line anchor). With the window now a member rule, a spine method is not windowed if it is, or contains, an exact target, so an anchored lines 900-1003 inside it is never elided with the rest of the body.
  • The shrink orders exact targets ahead of spine members. #2063's exact branch keeps members in sort order until the cap, so a long spine method sorted first would price out the body the agent singled out. Order: protected (when a hold-back applies), exact, spine, importance.
  • Exact pricing sees member windows. renderedSizeOfKept prices the spans buildSection renders (sectionRangesOf), so an oversize spine member costs its window.
  • The final fit shortens the file header before it cuts source. Every file is funded for source against an estimated header (sectionOverhead, the file's own symbol names). The real header also lists edge-line targets (filter(calls)) and can run past the estimate. Once #2063 gave exact targets the spine's cap, a named file with qualified targets spent right up to its funded line, and the saturated valve fixture's lower file lost 109 chars of source to its own header (922 of 1,031; 995 on main). Where a section no longer fits, the header now lists fewer names until it is back within its estimate, and only then is source cut. The valve test asserts funding in full and delivery to the line: #2062's exact rendering stops at the last whole line that fits, so a lower file may come in under its reservation by less than one line.

Measured on the same 47-query replay, main 7506dbff vs this re-application, one index:

main main + named concentration
named bodies complete 230 272 (13 responses up, 1 down)
named lines delivered 7,986 10,282 (+29%)
responses over the 25K inline cap 2 0
byte-identical responses — 16 of 47

The one response down is okhttp's pinned RealCall.kt, which drops the mock server's QueueDispatcher.kt, holding an enqueue namesake, as the named file spends its reservation. The named definitions that lose lines all sit in responses that gain more: vscode's MessageBuffer class (187 → 65, with three more named bodies complete and +258 named lines), alamofire's Session.request (17 → 12, +48), and one swap: vscode's stress query renders _doActivateExtension whole (0 → 33) where main rendered _startExtensionHost (19 → 0), four complete bodies either way. Files also fall where the record above says they do (okhttp's RealInterceptorChain.kt pin, 6 → 4).

A second replay: 17 queries that pin a real file past the 300-node pin cap and name symbols beyond it (vscode's editorOptions.ts, chatService.ts, languages.ts, extHostTypes.ts, extHost.protocol.ts; excalidraw's App.tsx; okhttp's URLConnectionTest.kt), most with an unpinned twin: 48 → 49 complete named bodies, 0 responses down, 753 → 760 named lines, and 3 → 0 over the 25K cap (the summary-line reserve).