Ops & Runbook

Day-to-day operational checks and recovery steps for XMDB.

Quick checks

• Health: curl -s http://localhost:8090/health
• Metrics (admin token required): curl -H "Authorization: Bearer $XMDB_API_TOKEN" http://localhost:8090/metrics

Backup & restore

Snapshots are deterministic, verifiable exports for a workspace + scope. Derived tables are rebuilt after import.

Export snapshot

xmdbctl snapshot export --base http://localhost:8080 \
  --workspace demo-ws --scope demo-scope --out snapshot.json --token "$XMDB_API_TOKEN"

Restore + rebuild

xmdbctl snapshot import --base http://localhost:8081 \
  --workspace demo-ws --scope demo-scope --in snapshot.json --token "$XMDB_API_TOKEN"

curl -s -X POST "http://localhost:8081/admin/rebuild?workspace_id=demo-ws&scope_id=demo-scope" \
  -H "Authorization: Bearer $XMDB_API_TOKEN"

Verification drill

# Verify signatures on import (validation only)
xmdbctl snapshot import --base http://localhost:8080 \
  --workspace demo-ws --scope demo-scope --in snapshot.json --verify-sigs --token "$XMDB_API_TOKEN"

# Verify full ledger
xmdbctl admin verify --base http://localhost:8080 --workspace demo-ws --scope demo-scope --token "$XMDB_API_TOKEN"

Pilot cadence: daily snapshot export, monthly restore drill, weekly xmdbctl admin verify.

SLO / telemetry snapshot

Run the daily SLO snapshot script to capture query latency, errors, ingest latency, DB/index size, and item count.

# Manual run
bash ~/xmdb/scripts/slo-snapshot.sh

Default output: ~/clawd/memory/flywheel/metrics/YYYY-MM-DD.json (appends entries).

Recorded fields:

• query_p50_ms, query_p95_ms
• query_errors
• ingest_ms
• db_size_bytes, index_size_bytes
• item_count

Automation (pilot setup):

• Runs daily at 2:30 AM PST via launchd
• LaunchAgent: com.lily.xmdb-slo-snapshot
• Plist: ~/Library/LaunchAgents/com.lily.xmdb-slo-snapshot.plist
• Log: /tmp/xmdb-slo-snapshot.log
• Manual trigger: launchctl start com.lily.xmdb-slo-snapshot