Prevention Checklists

Incidents repeat when fixes stay in chat logs. These checklists convert post-mortem action items into monitors, CI gates, and quarterly drills so failed transactions, RPC outages, and exploit paths are caught before users tweet about them.

How to Use This Checklist

• Run Tier 1 before any mainnet program or frontend promotion
• Add Tier 2 items within 30 days of every Sev-1/2 incident
• Re-run Tier 3 quarterly and after major Agave or Anchor upgrades
• Record pass/fail in your release ticket; block deploy on Tier 1 failures

Tier 1 - Detection & Monitoring

1. TVL / vault balance velocity: Alert when balance drops > X% in Y minutes on program PDAs.

   • Owner: Backend / indexer team
   • Tool: Custom indexer or Helius webhook

2. Transaction landing p95: Synthetic send every 5 minutes; alert if p95 > 30s for 10 min.

   • Owner: SRE
   • Tool: @solana/kit cron job + Grafana

3. Program error rate: Ratio of failed Custom errors to total txs for your program ID.

   • Owner: On-call eng
   • Tool: Explorer API or Geyser filter

4. RPC provider health: Slot lag, 429 rate, WS disconnect count per provider.

   • Owner: Infra
   • Tool: rpc-health.sh + PagerDuty

5. Priority fee sanity: Alert if median fee bid drops to zero during known high-traffic windows.

   • Owner: Client team
   • Tool: App metrics

6. Upgrade authority changes: Page on any set-upgrade-authority for production program IDs.

   • Owner: Security
   • Tool: Indexer rule or Squads notification

7. Pause flag drill: Quarterly test set_paused(true) on devnet clone in < 5 min.

   • Owner: Program lead
   • Tool: Runbook incident-pause.sh

Tier 2 - Tests & CI Gates

8. LiteSVM / Surfpool regression: Replay known exploit txs against every program PR.

   • CI: anchor test + LiteSVM 0.6.x fixture

9. IDL sync gate: Client codegen fails if IDL hash differs from deployed devnet program.

   • CI: Hash compare in GitHub Actions

10. Verifiable build on release: solana-verify hash published in release notes.

    • CI: Required artifact on tag

11. Simulation smoke: Top 10 user flows simulate without error on devnet pre-promote.

    • CI: Nightly + pre-mainnet manual gate

12. Multisig upgrade path: No direct laptop deploy to mainnet; only Squads proposal.

    • Process: Release checklist sign-off

13. Blockhash refresh test: Client integration test asserts retry uses new blockhash.

    • CI: Jest/Vitest with mocked RPC

14. Rate limit budget: Load test read path at 2x expected launch traffic against staging RPC.

    • Tool: k6 or locust

Tier 3 - Runbooks & Culture

15. Sev definitions documented: Sev-1 includes fund loss; escalation tree in Notion/git.
16. On-call runbook links: Every alert links to a runbook section, not a wiki home page.
17. Blameless post-mortem template: Stored in repo; facilitator rotation scheduled.
18. Break-glass keys: Location, custodians, and expiry review every 90 days.
19. Comms templates: Pre-written pause, rollback, and resolved messages for Discord/X.
20. Third-party contacts: RPC vendor TAM, auditor emergency line, bounty platform.
21. Incident drill log: Last drill date and gaps filed as tickets.
22. User support macros: "Transaction stuck" and "Failed simulation" with explorer links.
23. Genesis hash check in deploy pipeline: Wrong cluster blocked at CI.
24. Backup RPC in client bundle: Env vars for primary/secondary validated at build.
25. Post-incident action item SLA: 90% closed within 30 days or exec review.

Applying the Checklist in Order

• Tier 1 (1-7): Observability - without detection, incidents are user-reported
• Tier 2 (8-14): Prevention in CI - catch regressions before mainnet
• Tier 3 (15-25): Human systems - tools fail without practiced runbooks

FAQs

Related

• Debugging Failed Transactions - decode failures alerts surface
• Blameless Post-Mortems & RCA - source of new items
• DORA Metrics for Web3 - measure improvement

Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, anchor-lang 0.32.1, Rust 1.91.1, @solana/kit 7.0.0, Surfpool 0.12.0, and LiteSVM 0.6.x.