mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-22 09:35:22 +08:00
Colby Mchenry
37cf566dcf
* feat(mcp): steer agents to explore-first; fix Kotlin/Swift test detection Two changes from diagnosing why Claude Code's Explore agent wasn't using codegraph_explore on a benchmark run (37 calls / ~90k tokens via search+Read+grep, vs a general-purpose agent that led with explore: 13 calls / ~55k tokens for the same question). 1. Tool guidance reframed across server-instructions.ts, instructions-template.ts, and .cursor/rules/codegraph.mdc (+ the explore/search tool descriptions): codegraph_explore is the workhorse for understanding/architecture/"how does X work" questions. Seed it with the key symbol names (a quick search/context first if the question names nothing concrete), read its output, and fill gaps with node/Read — instead of searching then Reading each file. The old "search first to find names, then explore" wording was short-circuiting: agents searched, got file:line locations, and Read them, never reaching explore. 2. isTestFile now recognizes Kotlin (*Test.kt, jvmTest/commonTest/ androidTest source sets), Swift (*Tests.swift), and other camelCase test conventions, so test code is deprioritized in explore/context ranking. Previously only Java/JS/Python were known, letting tests dominate Kotlin/Swift exploration (OkHttp "trace a request" went from 8/9 test files to surfacing Call.kt/OkHttpClient.kt/Request.kt/Response.kt). Capital-led matching keeps latest.kt/manifest.kt unflagged. An IDF common-term down-weighting was prototyped for the cold-query case but dropped — it was a measured no-op (the "common" terms weren't actually common in the test indexes); the test-detection gap was the real cause. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(agent-eval): add agent-behavior eval harness for codegraph MCP usage Tooling to measure how a Claude Code agent actually uses the codegraph MCP tools on a real repo — does it lead with codegraph_explore, how many Read/Grep follow-ups, token cost — for validating tool-guidance changes (server-instructions, tool descriptions) against real agent behavior. - itrun.sh drives the real interactive TUI via tmux (the faithful Explore path). Hardened for unattended runs: type-and-verify prompt delivery (the ❯ glyph is drawn ~6s before the input accepts keys), auto-accepts the "trust this folder" dialog, busy-detection keys on the universal "(Ns · …)" spinner so the pre-stream thinking phase counts as busy, and fails loudly instead of capturing an empty pane. - parse-session.mjs reports the tool breakdown + token accounting (gen / fresh-in / cached-in / billable) from the session and subagent logs, consistent across main-thread and subagent runs; counts main-thread Bash in the grep verdict. - run-agent.sh / parse-run.mjs are the headless stream-json complement (exact per-tool tokens/cost via claude -p). - run-interactive-test.md documents how to run it and how completion is detected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
132 lines
5.8 KiB
Markdown
132 lines
5.8 KiB
Markdown
# Running the agent-behavior test (how agents actually use codegraph)
|
||
|
||
This explains how to measure **how a Claude Code agent uses the codegraph MCP
|
||
tools** on a real repo — which tools it calls (does it lead with
|
||
`codegraph_explore`?), how many follow-up `Read`/`Grep`s it does, and the token
|
||
cost. Use it when changing tool guidance (`server-instructions.ts`,
|
||
`instructions-template.ts`, tool descriptions) or retrieval, to verify the
|
||
change actually shifts agent behavior.
|
||
|
||
Scripts live in `scripts/agent-eval/`.
|
||
|
||
## Why two harnesses (read this first)
|
||
|
||
| | Interactive (`itrun.sh`) | Headless (`run-agent.sh`) |
|
||
|---|---|---|
|
||
| Drives | the real TUI via tmux | `claude -p` print mode |
|
||
| Subagent it picks | **Explore** (matches real UX) | general-purpose (diverges) |
|
||
| Metrics | tool breakdown (from session logs) + `Done(…)` token summary | exact per-tool calls + tokens/cost (stream-json) |
|
||
| Cost | Claude Max subscription | API $ (`total_cost_usd`) |
|
||
|
||
**Headless `claude -p` does NOT reproduce what users see** — it silently picks
|
||
the general-purpose subagent, while interactive sessions delegate to the
|
||
read-first **Explore** subagent. So for "what does my session actually do," use
|
||
the interactive harness. For a clean per-tool/token breakdown in one shot, use
|
||
headless (and ask for the Explore subagent in the prompt if you want that path).
|
||
|
||
## Prerequisites
|
||
|
||
- **tmux 3.0+**
|
||
- A logged-in `claude` CLI (Claude Max or API).
|
||
- codegraph configured as an MCP server (`claude mcp list` shows `codegraph`).
|
||
The interactive harness uses your global config, so it runs whatever
|
||
`codegraph` resolves to — point that at your dev build (`npm link` / the
|
||
symlinked global) to test local changes.
|
||
- A target repo, cloned and indexed:
|
||
```bash
|
||
git clone --depth 1 https://github.com/square/okhttp /tmp/corpus/okhttp
|
||
cd /tmp/corpus/okhttp && codegraph init -i
|
||
```
|
||
Good scale spread for a sweep: Alamofire (~100 files), Excalidraw (~600),
|
||
OkHttp (~640), VS Code (~10k).
|
||
|
||
## Interactive test (the faithful one)
|
||
|
||
```bash
|
||
scripts/agent-eval/itrun.sh <repo-path> <label> "<question>"
|
||
```
|
||
|
||
Example:
|
||
```bash
|
||
scripts/agent-eval/itrun.sh /tmp/corpus/vscode vscode \
|
||
"How does the extension host communicate with the main process?"
|
||
```
|
||
|
||
It opens `claude` in a tmux session, types the question, waits for the agent to
|
||
finish, then prints:
|
||
- the `Done (N tool uses · Xk tokens · Ym)` subagent summary (from the pane),
|
||
- the `Context Xk/1.0M` main-session size,
|
||
- a **tool breakdown** parsed from the session logs (main + subagents), ending
|
||
in a `VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N` line.
|
||
|
||
### Startup robustness (so unattended runs don't silently no-op)
|
||
|
||
Two things bite an unattended driver before the prompt even runs:
|
||
- **The `❯` glyph is drawn ~6s before the input accepts keystrokes.** Waiting
|
||
for `❯` is necessary but not sufficient. The harness sends the prompt, then
|
||
**verifies a chunk of it actually landed in the input box**, retrying until it
|
||
does — so it can't type into a not-yet-live input and submit nothing.
|
||
- **First time claude opens a repo it shows "Is this a project you trust?"**
|
||
(which also contains `❯`). The harness detects that dialog and presses Enter
|
||
to accept it before typing.
|
||
|
||
If the prompt never lands or work never starts, the harness now **fails loudly**
|
||
(non-zero exit) instead of capturing an empty pane and reporting a bogus run.
|
||
|
||
### How completion is detected (the tricky part)
|
||
|
||
Claude's TUI redraws in place, so you can't just wait for output to stop. The
|
||
harness polls `tmux capture-pane` and treats the pane as **busy** when it shows
|
||
the spinner's elapsed-time-in-parens — `(8s · …)` / `(1m 3s · …)`, matched by
|
||
`\(([0-9]+m )?[0-9]+s ·`. That's the *universal* working signal: it shows during
|
||
the pre-stream **thinking** phase (`(8s · thinking with max effort)`, which has
|
||
no token arrow yet) *and* during streaming. The `↓ N`/`↑ N` token arrow,
|
||
`esc to interrupt`, and `Initializing…` are OR'd in as belt-and-braces (some TUI
|
||
versions show one but not the others). It declares **idle** when the `❯` prompt
|
||
is present and not busy for 10 consecutive polls (~5s, long enough to ride out
|
||
mid-conversation thinking gaps that briefly drop the spinner). (Technique
|
||
adapted from devpit's `WaitForIdle`.)
|
||
|
||
### Where the breakdown comes from
|
||
|
||
`parse-session.mjs` reads the newest session log under
|
||
`~/.claude/projects/<escaped-cwd>/<session>.jsonl` and its subagent transcripts
|
||
under `<session>/subagents/*.jsonl`. The **subagent** file is where the real
|
||
tool calls are — the main log only shows the `Agent` delegation. You can run it
|
||
standalone:
|
||
```bash
|
||
node scripts/agent-eval/parse-session.mjs /tmp/corpus/vscode
|
||
```
|
||
|
||
## Headless test (clean tokens, forceable Explore path)
|
||
|
||
```bash
|
||
scripts/agent-eval/run-agent.sh <repo-path> <label> "<question>"
|
||
```
|
||
Writes stream-json and prints the tool sequence + exact tokens/cost. To
|
||
reproduce the Explore-subagent path headlessly, ask for it:
|
||
`"Use an Explore subagent to investigate, then answer: …"`.
|
||
|
||
## Running a sweep
|
||
|
||
Single runs vary a lot (the VS Code question has ranged 26–37 tool uses /
|
||
88–105k tokens across runs). For a real signal, run N≥3 and take the median:
|
||
```bash
|
||
for i in 1 2 3; do
|
||
scripts/agent-eval/itrun.sh /tmp/corpus/vscode "vscode-$i" "<question>"
|
||
done
|
||
```
|
||
|
||
## What "good" looks like
|
||
|
||
After the explore-first guidance (PR #191), an understanding question should
|
||
show the agent **leading with `codegraph_explore`** and using `search`/`node`
|
||
to fill gaps — not a wall of `Read`/`Grep`. Example faithful run:
|
||
`VERDICT: codegraph_explore used 3x | Read 8 | Grep/Bash 1`. If `explore` is 0
|
||
and `Read`/`Grep` dominate, the guidance regressed.
|
||
|
||
## Output artifacts
|
||
|
||
Transcripts and logs go to `$AGENT_EVAL_OUT` (default `/tmp/agent-eval/`):
|
||
`itrun-<label>.txt` (pane capture), `run-<label>.jsonl` (headless stream-json).
|