mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-18 13:53:01 +08:00
Colby Mchenry
a192df40f9
Three bugs in the self-hosted telemetry services (telemetry-worker/ and
telemetry-dashboard/), reported against 6560052a and all still on main.
1. An index run that uploads late never counted toward activation. The
dashboard's funnel reads machine_first_seen.first_index_day, which only the
nightly rollup wrote, and the rollup re-rolls just the last three days plus
days with no daily_machines row. Ingest accepts timestamps up to 30 days old
and the client keeps an event's original timestamp when it re-queues it, so
an index event 4+ days late landed on a day nothing revisited and
first_index_day stayed NULL. The ingest upsert of machine_first_seen now
lowers first_index_day the way it lowers first_day:
coalesce(min(old, new), old, new), because SQLite's min() is NULL if either
side is. It is the same statement and row, with no new index and no
migration (the column exists since 0002). The rollup still re-derives it
from raw events, which covers events stored before this change.
2. ingest_stalled read only max(day) FROM events, but usage counters have
lived in usage_daily since 0003, so a day with usage and no lifecycle events
read as "The ingest worker ... is not storing anything". /api/meta now also
reads max(day) FROM usage_daily (a primary-key lookup) and returns
latest_usage_day and latest_ingest_day. Ingest counts as stalled only when
there is no lifecycle event from yesterday or later and no usage counter
from the day before yesterday or later, because clients upload a day's
counters only after that day ends. The banner names latest_ingest_day.
3. runNightly rolled up the 3 regular days, then up to 31 missed days, and ran
purgeOldEvents last. Each day first folds its legacy usage_rollup rows,
which takes minutes on a heavy day, so a backlog could run past Cloudflare's
15-minute Cron Trigger limit and skip the purge and the summary line night
after night. The purge now runs first: it is bounded and touches only days
past the window, which no rollup reads. After that, rollups stop starting
new days or fold chunks 10 minutes in (NIGHTLY_BUDGET_MS). A day stopped
mid-fold gets no rollup at all, so it stays a missed day and the next night
continues from its last committed chunk. The summary line counts those days
as `deferred`, and `caught_up` now counts missed days actually rolled up.
Verification: __tests__/telemetry-services.test.ts (new; the root vitest
config picks it up) runs the worker's fetch/cron code and the dashboard API
against the checked-in migrations in in-memory node:sqlite through a small
D1-shaped adapter. With the source fix reversed, all 5 tests fail (activation
0, first_index_day null, ingest_stalled true, and "nightly rollup incomplete:
4 day(s) failed, purge failed" once statements pass the 15-minute mark). With
the fix, all 5 pass. `npm run check` is clean in both packages. Against
wrangler dev, run locally only: smoke:rollup 71/71, smoke:cutover 62/62,
smoke:api 122/122 and render-check 87/87. smoke (ingest) is 46/47; the one
failure is the 70 KB oversized-body curl argument hitting the Windows
command-line limit, and main fails it the same way. The new smoke assertions
fail on main.
Takes effect only once both workers are redeployed; no migration.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
276 lines
14 KiB
Bash
Executable File
276 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# The cutover gate (CG-14): drives the WHOLE chain the way production will run it —
|
|
# a client POSTs a batch, the ingest worker writes D1, the nightly rollup aggregates,
|
|
# and the dashboard reads the numbers back out.
|
|
#
|
|
# Every other suite tests one link. smoke-ingest.sh stops at the `events` table,
|
|
# smoke-rollup.sh hand-checks the rollup SQL, and smoke-api.sh reads a fixture that
|
|
# was written by hand rather than by the cron. That leaves exactly the seam this
|
|
# cutover turns on unverified: the dimension names the rollup WRITES versus the ones
|
|
# the dashboard READS. Those two lists live in different workers on different
|
|
# branches, and a mismatch is silent — no error, no failed request, just a panel that
|
|
# renders zero forever. Catching that after cutover means a day of lost telemetry;
|
|
# catching it here costs a minute.
|
|
#
|
|
# Both workers declare the same D1 `database_id`, so pointing them at one
|
|
# `--persist-to` directory gives them literally the same local SQLite file. The state
|
|
# is a fresh mktemp each run, so every expected number below is exact rather than a
|
|
# lower bound.
|
|
#
|
|
# npm run smoke:cutover
|
|
#
|
|
# Expected numbers are derived from THE_BATCH below and nothing else; see the table
|
|
# in that comment block.
|
|
set -uo pipefail
|
|
|
|
cd "$(dirname "$0")/.."
|
|
WORKER_DIR="$PWD"
|
|
DASH_DIR="$(cd .. && pwd)/telemetry-dashboard"
|
|
|
|
[ -d "$DASH_DIR" ] || { echo "cannot find telemetry-dashboard/ next to telemetry-worker/"; exit 1; }
|
|
|
|
INGEST_PORT="${CUTOVER_INGEST_PORT:-8795}"
|
|
DASH_PORT="${CUTOVER_DASH_PORT:-8796}"
|
|
INGEST="http://127.0.0.1:$INGEST_PORT"
|
|
DASH="http://127.0.0.1:$DASH_PORT"
|
|
|
|
# Test-only credentials. The point is to exercise the wiring, not to keep a secret.
|
|
ADMIN_TOKEN=cutover-admin-token
|
|
DASH_PASSWORD=cutover-dashboard-password
|
|
SESSION_SECRET=cutover-session-secret
|
|
|
|
STATE="$(mktemp -d -t cg-cutover-state)"
|
|
JAR="$(mktemp -t cg-cutover-jar)"
|
|
ILOG=/tmp/cg-cutover-ingest.log
|
|
DLOG=/tmp/cg-cutover-dash.log
|
|
|
|
pass=0; fail=0
|
|
ok() { pass=$((pass + 1)); printf ' ok %s\n' "$1"; }
|
|
bad() { fail=$((fail + 1)); printf ' FAIL %s — expected %s, got %s\n' "$1" "$2" "$3"; }
|
|
is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; }
|
|
|
|
DEV_PID=""
|
|
stop_dev() {
|
|
[ -n "$DEV_PID" ] || return 0
|
|
kill "$DEV_PID" 2>/dev/null
|
|
wait "$DEV_PID" 2>/dev/null
|
|
DEV_PID=""
|
|
}
|
|
cleanup() { stop_dev; rm -rf "$STATE" "$JAR"; }
|
|
trap cleanup EXIT
|
|
|
|
# Boot a worker in <dir> on <port> against the SHARED state, wait for <readyurl>.
|
|
boot() { # boot <dir> <port> <readyurl> <log> [extra wrangler args...]
|
|
local dir="$1" port="$2" ready="$3" log="$4"; shift 4
|
|
( cd "$dir" && exec npx wrangler dev --port "$port" --ip 127.0.0.1 \
|
|
--persist-to "$STATE" "$@" ) >"$log" 2>&1 &
|
|
DEV_PID=$!
|
|
for _ in $(seq 1 90); do
|
|
curl -sf -o /dev/null "$ready" && return 0
|
|
kill -0 "$DEV_PID" 2>/dev/null || break
|
|
sleep 1
|
|
done
|
|
echo "worker in $dir never came up on :$port — log follows"; cat "$log"; exit 1
|
|
}
|
|
|
|
# Resolve a dotted path through a JSON document. Numeric segments index arrays.
|
|
jget() {
|
|
node -e '
|
|
let v = JSON.parse(process.argv[1]);
|
|
for (const k of process.argv[2].split(".")) v = v?.[k];
|
|
console.log(v === undefined ? "<missing>" : typeof v === "object" && v !== null ? JSON.stringify(v) : String(v));
|
|
' "$1" "$2"
|
|
}
|
|
|
|
day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; }
|
|
|
|
# Inside the ingest clamp window (30 days) and outside the cron's 3-day lookback.
|
|
DAY="$(day_ago 5)"
|
|
RANGE="from=$DAY&to=$DAY"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# THE_BATCH — three machines, one day. Everything asserted below follows from here.
|
|
#
|
|
# machine os arch node version ci events
|
|
# m1 darwin arm64 22 1.5.0 false install(local/fresh, [claude,cursor])
|
|
# index([typescript,python], 100-1k, 10-60s)
|
|
# usage_rollup(codegraph_explore x12, Claude Code)
|
|
# m2 linux x64 20 1.5.0 false install(global/upgrade, [codex])
|
|
# index([typescript], 1k-10k, 1-5m)
|
|
# usage_rollup(codegraph_explore x8, Codex CLI)
|
|
# m3 linux arm64 22 1.4.1 TRUE index([go], <100, <10s)
|
|
# uninstall([claude])
|
|
#
|
|
# The three deliberate traps:
|
|
# * m3 is ci=true, so it counts as active but NOT as a production user.
|
|
# * tool_calls must SUM the `count` prop (12 + 8 = 20), not count the 2 rows.
|
|
# * m3's uninstall carries targets=[claude], so a `target` breakdown that forgets
|
|
# to scope by event would report claude twice.
|
|
# ---------------------------------------------------------------------------
|
|
M1=11111111-1111-4111-8111-111111111111
|
|
M2=22222222-2222-4222-8222-222222222222
|
|
M3=33333333-3333-4333-8333-333333333333
|
|
|
|
post_batch() { # post_batch <json>
|
|
curl -s -o /dev/null -w '%{http_code}' -X POST "$INGEST/v1/events" \
|
|
-H 'content-type: application/json' --data-binary "$1"
|
|
}
|
|
|
|
batch() { # batch <machine> <os> <arch> <node> <version> <ci> <events-json>
|
|
node -e '
|
|
const [m, os, arch, node_major, v, ci, events, day] = process.argv.slice(1);
|
|
process.stdout.write(JSON.stringify({
|
|
machine_id: m, codegraph_version: v, os, arch,
|
|
node_major: Number(node_major), ci: ci === "true", schema_version: 1,
|
|
events: JSON.parse(events).map((e) => ({ ...e, ts: `${day}T12:00:00Z` })),
|
|
}));
|
|
' "$@" "$DAY"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
echo "cutover chain: client → ingest worker → D1 → rollup → dashboard"
|
|
echo
|
|
echo "migrating the shared local D1 state"
|
|
( cd "$WORKER_DIR" && npx wrangler d1 migrations apply codegraph-telemetry \
|
|
--local --persist-to "$STATE" ) >/tmp/cg-cutover-migrate.log 2>&1 ||
|
|
{ echo "migration failed:"; cat /tmp/cg-cutover-migrate.log; exit 1; }
|
|
|
|
echo "booting the ingest worker on :$INGEST_PORT"
|
|
boot "$WORKER_DIR" "$INGEST_PORT" "$INGEST/" "$ILOG" --var "ADMIN_TOKEN:$ADMIN_TOKEN"
|
|
|
|
echo
|
|
echo "ingest accepts the batch"
|
|
is "m1 batch → 204" 204 "$(post_batch "$(batch "$M1" darwin arm64 22 1.5.0 false '[
|
|
{"event":"install","props":{"scope":"local","kind":"fresh","targets":["claude","cursor"]}},
|
|
{"event":"index","props":{"languages":["typescript","python"],"file_count_bucket":"100-1k","duration_bucket":"10-60s"}},
|
|
{"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":12,"client_name":"Claude Code"}}
|
|
]')")"
|
|
is "m2 batch → 204" 204 "$(post_batch "$(batch "$M2" linux x64 20 1.5.0 false '[
|
|
{"event":"install","props":{"scope":"global","kind":"upgrade","targets":["codex"]}},
|
|
{"event":"index","props":{"languages":["typescript"],"file_count_bucket":"1k-10k","duration_bucket":"1-5m"}},
|
|
{"event":"usage_rollup","props":{"kind":"mcp_tool","name":"codegraph_explore","count":8,"client_name":"Codex CLI"}}
|
|
]')")"
|
|
is "m3 (ci) batch → 204" 204 "$(post_batch "$(batch "$M3" linux arm64 22 1.4.1 true '[
|
|
{"event":"index","props":{"languages":["go"],"file_count_bucket":"<100","duration_bucket":"<10s"}},
|
|
{"event":"uninstall","props":{"targets":["claude"]}}
|
|
]')")"
|
|
|
|
sleep 2 # let the ctx.waitUntil writes drain before rolling up
|
|
|
|
echo
|
|
echo "the nightly rollup aggregates the day"
|
|
ROLL=$(curl -s -X POST -H "x-admin-token: $ADMIN_TOKEN" "$INGEST/admin/rollup?day=$DAY")
|
|
is "POST /admin/rollup → ok" true "$(jget "$ROLL" ok)"
|
|
is "rollup wrote rows" true "$(node -e 'process.stdout.write(String((JSON.parse(process.argv[1]).rows ?? 0) > 0))' "$ROLL")"
|
|
|
|
stop_dev # free the D1 lock before the dashboard opens the same file
|
|
|
|
echo
|
|
echo "booting the dashboard on :$DASH_PORT against the same D1"
|
|
( cd "$DASH_DIR" && npm run --silent vendor ) >/dev/null 2>&1
|
|
boot "$DASH_DIR" "$DASH_PORT" "$DASH/robots.txt" "$DLOG" \
|
|
--var "ADMIN_PASSWORD:$DASH_PASSWORD" --var "SESSION_SECRET:$SESSION_SECRET"
|
|
|
|
curl -s -o /dev/null -c "$JAR" -X POST "$DASH/login" --data-urlencode "password=$DASH_PASSWORD"
|
|
api() { curl -s -b "$JAR" "$DASH/api/$1"; }
|
|
is "dashboard session established" 200 "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/health")"
|
|
|
|
# --- the big numbers -------------------------------------------------------
|
|
echo
|
|
echo "summary panel reads back what was ingested"
|
|
S=$(api "summary?$RANGE")
|
|
is "production users (ci machine excluded)" 2 "$(jget "$S" production_users)"
|
|
is "active machines" 3 "$(jget "$S" active_machines)"
|
|
is "new machines" 3 "$(jget "$S" new_machines)"
|
|
is "installs" 2 "$(jget "$S" installs)"
|
|
is "uninstalls" 1 "$(jget "$S" uninstalls)"
|
|
is "indexing runs" 3 "$(jget "$S" index_runs)"
|
|
is "tool calls SUM the count prop (12+8)" 20 "$(jget "$S" tool_calls)"
|
|
|
|
# --- every dimension the dashboard offers ----------------------------------
|
|
# This is the actual point of the suite: each of these is a distinct string that
|
|
# must match between rollup.ts and api.ts's DIMS registry. An empty `labels` means
|
|
# the dashboard is asking for a dimension the cron never writes.
|
|
echo
|
|
echo "every breakdown dimension resolves against the cron's output"
|
|
bd() { # bd <desc> <query> <expected-labels-json> <expected-data-json>
|
|
local body; body=$(api "breakdown?$RANGE&$2")
|
|
is "$1 — labels" "$3" "$(jget "$body" labels)"
|
|
is "$1 — data" "$4" "$(jget "$body" datasets.0.data)"
|
|
}
|
|
bd "os" "dim=os" '["linux","darwin"]' '[2,1]'
|
|
bd "arch" "dim=arch" '["arm64","x64"]' '[2,1]'
|
|
bd "version" "dim=codegraph_version" '["1.5.0","1.4.1"]' '[2,1]'
|
|
bd "node major" "dim=node_major" '["22","20"]' '[2,1]'
|
|
bd "language" "dim=language" '["typescript","go","python"]' '[2,1,1]'
|
|
bd "files in project" "dim=file_count_bucket" '["<100","100-1k","1k-10k","10k+"]' '[1,1,1,0]'
|
|
bd "run length" "dim=duration_bucket" '["<10s","10-60s","1-5m","5m+"]' '[1,1,1,0]'
|
|
bd "install scope" "dim=scope" '["global","local"]' '[1,1]'
|
|
bd "install kind" "dim=kind" '["fresh","upgrade"]' '[1,1]'
|
|
bd "tool name" "dim=name" '["codegraph_explore"]' '[20]'
|
|
bd "agent" "dim=client_name" '["Claude Code","Codex CLI"]' '[12,8]'
|
|
|
|
# The trap: `target` defaults to event=install, so the uninstall's own claude target
|
|
# must NOT be folded in — and must still be reachable by asking for it explicitly.
|
|
bd "agent target (install-scoped)" "dim=target" '["claude","codex","cursor"]' '[1,1,1]'
|
|
bd "agent target (uninstall)" "dim=target&event=uninstall" '["claude"]' '[1]'
|
|
|
|
# --- the remaining panels --------------------------------------------------
|
|
echo
|
|
echo "the timeseries and funnel panels see the day"
|
|
# Every entry in api.ts's SERIES registry — each one reads a different rollup table,
|
|
# so this is the second half of the write-vs-read seam the breakdowns cover above.
|
|
ts() { # ts <desc> <metric> <series-0> [series-1]
|
|
local body; body=$(api "timeseries?$RANGE&metric=$2")
|
|
is "$1 — day" "[\"$DAY\"]" "$(jget "$body" labels)"
|
|
is "$1 — series" "$3" "$(jget "$body" datasets.0.data)"
|
|
[ $# -ge 4 ] && is "$1 — second series" "$4" "$(jget "$body" datasets.1.data)"
|
|
}
|
|
ts "installs and uninstalls" installs_uninstalls '[2]' '[1]'
|
|
ts "new installs" new_installs '[3]'
|
|
ts "production users" production_users '[2]'
|
|
ts "indexing activity" indexing_activity '[3]' '[3]'
|
|
ts "tool calls (sums the prop)" tool_calls '[20]' '[2]'
|
|
|
|
MET=$(api "meta")
|
|
is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)"
|
|
is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)"
|
|
# Usage lands in usage_daily rather than events; the stalled-ingest check reads both.
|
|
is "meta reports the last day ingest stored" "$DAY" "$(jget "$MET" latest_ingest_day)"
|
|
|
|
# The funnel reads machine_first_seen.first_index_day. The ingest worker sets it as
|
|
# each index event is stored, and the nightly rollup re-derives it from raw `index`
|
|
# events (smoke-rollup.sh pins that half, on events it seeds past the ingest path).
|
|
# This is the seam: ingest writes the column, the dashboard reads it. If neither
|
|
# writer set it, "activated" here would read 0.
|
|
#
|
|
# Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine
|
|
# that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never
|
|
# sent an install event, but it is new and it indexed, so it belongs in both legs.
|
|
# Reading 2 here would mean the funnel had quietly become an install-event ratio.
|
|
ACT=$(api "activation?$RANGE&window=1")
|
|
is "funnel counts new machines, not install events" 3 "$(jget "$ACT" installs)"
|
|
is "all three indexed within the window" 3 "$(jget "$ACT" activated)"
|
|
is "nobody dropped out" 0 "$(jget "$ACT" dropped)"
|
|
is "cohorts are counted through the rolled-up day" "$DAY" "$(jget "$ACT" covered_through)"
|
|
|
|
is "retention endpoint answers" 200 \
|
|
"$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/retention?$RANGE")"
|
|
|
|
# --- the guarantee the cutover is selling ----------------------------------
|
|
echo
|
|
echo "the no-third-party guarantee still holds"
|
|
is "ingest worker makes no outbound fetch" 0 \
|
|
"$(grep -E 'fetch\(' "$WORKER_DIR"/src/*.ts | grep -vc 'async fetch(request' || true)"
|
|
is "ingest worker names no third-party analytics endpoint" 0 \
|
|
"$(grep -rEil 'https?://[a-z0-9.-]+/(batch|capture|collect|track|ingest)' \
|
|
"$WORKER_DIR"/src "$WORKER_DIR"/wrangler.jsonc 2>/dev/null | wc -l | tr -d ' ')"
|
|
|
|
echo
|
|
if [ "$fail" -eq 0 ]; then
|
|
echo "$pass passed, 0 failed — the chain is whole; safe to cut over"
|
|
else
|
|
echo "$pass passed, $fail failed"
|
|
fi
|
|
[ "$fail" -eq 0 ]
|