mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-19 23:20:31 +08:00
Colby Mchenry
37cf566dcf
* feat(mcp): steer agents to explore-first; fix Kotlin/Swift test detection Two changes from diagnosing why Claude Code's Explore agent wasn't using codegraph_explore on a benchmark run (37 calls / ~90k tokens via search+Read+grep, vs a general-purpose agent that led with explore: 13 calls / ~55k tokens for the same question). 1. Tool guidance reframed across server-instructions.ts, instructions-template.ts, and .cursor/rules/codegraph.mdc (+ the explore/search tool descriptions): codegraph_explore is the workhorse for understanding/architecture/"how does X work" questions. Seed it with the key symbol names (a quick search/context first if the question names nothing concrete), read its output, and fill gaps with node/Read — instead of searching then Reading each file. The old "search first to find names, then explore" wording was short-circuiting: agents searched, got file:line locations, and Read them, never reaching explore. 2. isTestFile now recognizes Kotlin (*Test.kt, jvmTest/commonTest/ androidTest source sets), Swift (*Tests.swift), and other camelCase test conventions, so test code is deprioritized in explore/context ranking. Previously only Java/JS/Python were known, letting tests dominate Kotlin/Swift exploration (OkHttp "trace a request" went from 8/9 test files to surfacing Call.kt/OkHttpClient.kt/Request.kt/Response.kt). Capital-led matching keeps latest.kt/manifest.kt unflagged. An IDF common-term down-weighting was prototyped for the cold-query case but dropped — it was a measured no-op (the "common" terms weren't actually common in the test indexes); the test-detection gap was the real cause. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(agent-eval): add agent-behavior eval harness for codegraph MCP usage Tooling to measure how a Claude Code agent actually uses the codegraph MCP tools on a real repo — does it lead with codegraph_explore, how many Read/Grep follow-ups, token cost — for validating tool-guidance changes (server-instructions, tool descriptions) against real agent behavior. - itrun.sh drives the real interactive TUI via tmux (the faithful Explore path). Hardened for unattended runs: type-and-verify prompt delivery (the ❯ glyph is drawn ~6s before the input accepts keys), auto-accepts the "trust this folder" dialog, busy-detection keys on the universal "(Ns · …)" spinner so the pre-stream thinking phase counts as busy, and fails loudly instead of capturing an empty pane. - parse-session.mjs reports the tool breakdown + token accounting (gen / fresh-in / cached-in / billable) from the session and subagent logs, consistent across main-thread and subagent runs; counts main-thread Bash in the grep verdict. - run-agent.sh / parse-run.mjs are the headless stream-json complement (exact per-tool tokens/cost via claude -p). - run-interactive-test.md documents how to run it and how completion is detected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Headless Claude Code run against a repo with codegraph MCP, capturing the
|
|
# full stream-json so we can see tool calls + token usage. Complements the
|
|
# interactive itrun.sh: headless gives a clean per-tool breakdown + exact
|
|
# tokens/cost, but defaults to the general-purpose subagent (not Explore).
|
|
# To force the Explore path, ask for it in the prompt.
|
|
#
|
|
# Usage: run-agent.sh <repo-path> <label> "<prompt>"
|
|
# Env: AGENT_EVAL_OUT (default /tmp/agent-eval), CG_BIN (codegraph dist binary)
|
|
set -uo pipefail
|
|
|
|
REPO="$1"; LABEL="$2"; PROMPT="$3"
|
|
CG_BIN="${CG_BIN:-$(command -v codegraph || echo /usr/local/bin/codegraph)}"
|
|
OUT_DIR="${AGENT_EVAL_OUT:-/tmp/agent-eval}"; mkdir -p "$OUT_DIR"
|
|
OUT="$OUT_DIR/run-${LABEL}.jsonl"
|
|
|
|
MCP_CONFIG=$(cat <<JSON
|
|
{"mcpServers":{"codegraph":{"command":"${CG_BIN}","args":["serve","--mcp","--path","${REPO}"]}}}
|
|
JSON
|
|
)
|
|
|
|
echo "→ running [$LABEL] in $REPO"
|
|
cd "$REPO" || exit 1
|
|
|
|
claude -p "$PROMPT" \
|
|
--output-format stream-json --verbose \
|
|
--permission-mode bypassPermissions \
|
|
--model opus \
|
|
--max-budget-usd 2 \
|
|
--strict-mcp-config --mcp-config "$MCP_CONFIG" \
|
|
> "$OUT" 2>"$OUT_DIR/run-${LABEL}.err"
|
|
|
|
echo "exit: $? | wrote $OUT ($(wc -l < "$OUT") lines)"
|
|
node "$(cd "$(dirname "$0")" && pwd)/parse-run.mjs" "$OUT" 2>/dev/null || true
|