mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-18 05:41:18 +08:00
Colby Mchenry
217190680d
The telemetry database hit D1's 10 GB cap on 2026-08-11 and refused nearly every write for seven weeks: the dashboard kept showing Aug 9 as the latest day, and its activation query (a cohort join over raw events, ~55 s per week) stalled D1's single query lane until every panel failed. Root cause was the client: usage counters were aggregated per process and appended on exit, so every `serve` launch, CLI command and prompt hook uploaded its own `count: 1` line - 30.3M of the 30.8M event rows. - Client: merge count lines per (day, kind, name, client) on append, stale-claim recovery (which also bypassed the size cap) and send. - Ingest: usage counters ADD into usage_daily, one row per machine x day x tool (migration 0003), so storage no longer depends on upload frequency. - Rollup: reads usage from usage_daily, folds legacy usage rows out of events in 50k-row transactions, maintains machine_first_seen.first_index_day (0002), catches up on days a failed run missed, purges usage_daily with retention. 0004 drops the now-unused events_machine_day index. - Dashboard: ranges end today; rolled-up series return null past the rollup's last day instead of a phantom zero; no panel reads raw events (activation reads first_index_day); min/max lookups stay index-only; one retry on transient 5xx; banners when ingest stalls or the rollup falls behind. - scripts/backfill-rollup.sh re-runs the rollup over a day range. Already applied in production: DB 10 GB -> 467 MB, all usage counts kept, rollups current through 2026-10-02. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
34 lines
1.6 KiB
Bash
Executable File
34 lines
1.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Re-runs the nightly rollup over a range of days against the DEPLOYED ingest worker,
|
|
# one day per request. Each day: legacy usage rows still in `events` are folded into
|
|
# `usage_daily`, `first_index_day` is set, and the day's rollups are recomputed. All of
|
|
# it is idempotent, so a range can be repeated or overlapped safely.
|
|
#
|
|
# ./scripts/backfill-rollup.sh 2026-07-04 2026-10-02 [token-file]
|
|
#
|
|
# The admin token comes from the token file when one is given, else from $ADMIN_TOKEN.
|
|
# Stops at the first day that does not answer 200, so a rerun can start from there.
|
|
set -euo pipefail
|
|
|
|
FROM=${1:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]}
|
|
TO=${2:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]}
|
|
if [ -n "${3:-}" ]; then ADMIN_TOKEN=$(tr -d '\n' < "$3"); fi
|
|
: "${ADMIN_TOKEN:?set ADMIN_TOKEN or pass a token file}"
|
|
BASE="${ROLLUP_BASE:-https://telemetry.getcodegraph.com}"
|
|
|
|
days=$(node -e '
|
|
const [from, to] = process.argv.slice(1);
|
|
for (let t = Date.parse(`${from}T00:00:00Z`); t <= Date.parse(`${to}T00:00:00Z`); t += 864e5)
|
|
console.log(new Date(t).toISOString().slice(0, 10));' "$FROM" "$TO")
|
|
|
|
for day in $days; do
|
|
started=$(date +%s)
|
|
# A day with millions of legacy rows takes minutes; never let one hang forever.
|
|
response=$(curl -sS --max-time 1800 -X POST -H "x-admin-token: $ADMIN_TOKEN" \
|
|
-w $'\n%{http_code}' "$BASE/admin/rollup?day=$day") || { echo "$day request failed"; exit 1; }
|
|
code=${response##*$'\n'}
|
|
body=${response%$'\n'*}
|
|
printf '%s %s %ss %s\n' "$day" "$code" "$(( $(date +%s) - started ))" "$body"
|
|
[ "$code" = 200 ] || { echo "stopping at $day"; exit 1; }
|
|
done
|