Prevention Checklists
Incidents repeat when fixes stay in chat logs. These checklists convert post-mortem action items into monitors, CI gates, and quarterly drills so failed transactions, RPC outages, and exploit paths are caught before users tweet about them.
How to Use This Checklist
- Run Tier 1 before any mainnet program or frontend promotion
- Add Tier 2 items within 30 days of every Sev-1/2 incident
- Re-run Tier 3 quarterly and after major Agave or Anchor upgrades
- Record pass/fail in your release ticket; block deploy on Tier 1 failures
Tier 1 - Detection & Monitoring
-
TVL / vault balance velocity: Alert when balance drops > X% in Y minutes on program PDAs.
- Owner: Backend / indexer team
- Tool: Custom indexer or Helius webhook
-
Transaction landing p95: Synthetic send every 5 minutes; alert if p95 > 30s for 10 min.
- Owner: SRE
- Tool:
@solana/kitcron job + Grafana
-
Program error rate: Ratio of failed
Customerrors to total txs for your program ID.- Owner: On-call eng
- Tool: Explorer API or Geyser filter
-
RPC provider health: Slot lag, 429 rate, WS disconnect count per provider.
- Owner: Infra
- Tool:
rpc-health.sh+ PagerDuty
-
Priority fee sanity: Alert if median fee bid drops to zero during known high-traffic windows.
- Owner: Client team
- Tool: App metrics
-
Upgrade authority changes: Page on any
set-upgrade-authorityfor production program IDs.- Owner: Security
- Tool: Indexer rule or Squads notification
-
Pause flag drill: Quarterly test
set_paused(true)on devnet clone in < 5 min.- Owner: Program lead
- Tool: Runbook
incident-pause.sh
Tier 2 - Tests & CI Gates
-
LiteSVM / Surfpool regression: Replay known exploit txs against every program PR.
- CI:
anchor test+ LiteSVM 0.6.x fixture
- CI:
-
IDL sync gate: Client codegen fails if IDL hash differs from deployed devnet program.
- CI: Hash compare in GitHub Actions
-
Verifiable build on release:
solana-verifyhash published in release notes.- CI: Required artifact on tag
-
Simulation smoke: Top 10 user flows simulate without error on devnet pre-promote.
- CI: Nightly + pre-mainnet manual gate
-
Multisig upgrade path: No direct laptop deploy to mainnet; only Squads proposal.
- Process: Release checklist sign-off
-
Blockhash refresh test: Client integration test asserts retry uses new blockhash.
- CI: Jest/Vitest with mocked RPC
-
Rate limit budget: Load test read path at 2x expected launch traffic against staging RPC.
- Tool: k6 or locust
Tier 3 - Runbooks & Culture
- Sev definitions documented: Sev-1 includes fund loss; escalation tree in Notion/git.
- On-call runbook links: Every alert links to a runbook section, not a wiki home page.
- Blameless post-mortem template: Stored in repo; facilitator rotation scheduled.
- Break-glass keys: Location, custodians, and expiry review every 90 days.
- Comms templates: Pre-written pause, rollback, and resolved messages for Discord/X.
- Third-party contacts: RPC vendor TAM, auditor emergency line, bounty platform.
- Incident drill log: Last drill date and gaps filed as tickets.
- User support macros: "Transaction stuck" and "Failed simulation" with explorer links.
- Genesis hash check in deploy pipeline: Wrong cluster blocked at CI.
- Backup RPC in client bundle: Env vars for primary/secondary validated at build.
- Post-incident action item SLA: 90% closed within 30 days or exec review.
Applying the Checklist in Order
- Tier 1 (1-7): Observability - without detection, incidents are user-reported
- Tier 2 (8-14): Prevention in CI - catch regressions before mainnet
- Tier 3 (15-25): Human systems - tools fail without practiced runbooks
FAQs
How many alerts is too many?
If on-call acknowledges < 20% as actionable, tune thresholds. Start strict after incidents, relax with data.
What synthetic tx should run?
Minimal transfer or no-op program ix against production RPC with priority fee - measures landing only.
Do devnet monitors matter?
Yes for pre-promotion; mainnet monitors are mandatory for user funds.
Who owns checklist updates?
Incident commander updates within 48 hours of post-mortem; eng manager reviews quarterly.
Should checklist block hotfix?
Tier 1 pause/mitigation never blocked; full checklist applies before re-enabling traffic.
How tie to DORA metrics?
Change-fail rate and MTTR improve when detection and rollback items pass.
What is a good TVL alert threshold?
Start 3-5% in 5 minutes for lending; tune per protocol size and noise.
Are public RPCs OK for synthetics?
Use the same provider tier production uses, or results mislead.
How document drill failures?
File ticket per failed step; failed drill counts as incident near-miss.
When retire a checklist item?
When superseded by automation with 90 days clean history; document in post-mortem.
Related
- Debugging Failed Transactions - decode failures alerts surface
- Blameless Post-Mortems & RCA - source of new items
- DORA Metrics for Web3 - measure improvement
Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, anchor-lang 0.32.1, Rust 1.91.1, @solana/kit 7.0.0, Surfpool 0.12.0, and LiteSVM 0.6.x.