Incident Response In Depth
Production Solana products fail in ways that look similar to users (stuck wallets, empty balances, failed sends) but require opposite first moves from engineers.
A rate-limited RPC provider is not a vault exploit. A Custom program error after a bad upgrade is not a blockhash expiry. Treating every red dashboard as "retry harder" burns minutes and sometimes funds.
This page is the section umbrella: how to classify incidents, gather evidence, contain damage, mitigate without rewriting history, and close the loop with prevention and blameless RCA for Solana programs, dApps, and backends.
Summary
- Solana incident response is a discipline that separates symptom class (failed tx, RPC outage, on-chain exploit, bad deploy, client UX) from containment (pause, failover, feature flag) and forward-only mitigation (redeploy, compensate, communicate), then locks learning into monitors and runbooks.
- Insight: Finalized ledger state cannot be rewound; wrong pause or wrong retry can amplify loss; multi-provider RPC and upgrade authority are part of the product, not side ops. A shared map keeps on-call from guessing under pressure.
- Key Concepts: severity, incident commander, evidence chain (signature, slot, program id, RPC provider), simulation, pause switch, upgrade authority, multi-provider failover, verifiable rollback, feature flag, blameless postmortem, RCA.
- When to Use: First orientation for production on-call; writing runbooks before mainnet; triage during Sev-1/2; post-incident review design; aligning program, client, and infra owners on one playbook language.
- Limitations/Trade-offs: Runbooks do not replace security design; pause flags must be pre-deployed; multi-provider failover has cost and consistency trade-offs; blameless culture requires leadership buy-in, not only a markdown template.
- Related Topics: Debugging failed transactions, RPC and infrastructure outages, on-chain incident response, rollback and mitigation, prevention checklists, and blameless postmortems with RCA.
Foundations
Solana products straddle immutable chain state and mutable off-chain systems.
On-chain: programs, accounts, mints, vault PDAs, and upgrade authority live on a cluster under Agave consensus. Once a transaction is finalized, the ledger does not offer an "undo" API. Mitigation is pause, freeze, redeploy, migrate, or compensate.
Off-chain: RPC providers, indexers, frontends, backends, wallets, and ops tooling decide what users can see and submit. Most "outages" users report start here.
An incident is any unexpected degradation of safety, funds control, correctness, or availability that needs coordinated response beyond normal debugging. Severity usually tracks user funds at risk, duration, and blast radius:
| Sev | Typical signal | Example |
|---|---|---|
| 1 | Active fund loss or critical pause needed | Exploit drain, compromised authority |
| 2 | Major product path broken, funds not actively draining | Bad deploy spike of program errors |
| 3 | Partial degradation with workaround | Single-region RPC lag, one instruction failing |
| 4 | Localized or cosmetic | Explorer mismatch, non-critical UI |
Roles beat heroics. Name an incident commander (decides sequence), a comms lead (users and stakeholders get facts, not speculation), and domain leads (program, client, infra). One channel for decisions; one timeline in UTC with signatures and slots.
Evidence is the Solana-specific glue between classes of failure. Capture early:
- User-visible symptom and cluster (mainnet-beta vs others).
- Transaction signature (if any), slot, and error class.
- Program id and instruction name (from IDL or logs).
- RPC URL / provider and HTTP vs WebSocket path.
- Deploy version, feature flags, and recent upgrade authority activity.
- Whether funds moved, and to which addresses.
Without that chain, people argue anecdotes while the attacker or the 429 storm continues.
The section's sibling pages each own one face of the machine:
- Failed and non-landing transactions: decode before retry.
- RPC and infrastructure: treat providers as critical path.
- On-chain exploits: pause, authority, upgrade-under-fire.
- Rollback and mitigation: redeploy prior verified bytes and client flags.
- Prevention: monitors, drills, and checklists.
- Blameless postmortems: systems fixes with owners and dates.
This page keeps the whole loop in view.
Mechanics & Interactions
Detection to classification
Incidents enter through alerts (TVL velocity, error rate, slot lag, 429s), user reports, whitehat messages, or explorer anomalies. First job is classify, not fix:
Symptom
|
v
Evidence pack (sig, slot, program, RPC, funds?)
|
+-- Tx / sim / Custom / CU / accounts --> failed-tx path
+-- 429, lag, WS drop, multi-user blank UI --> RPC path
+-- Unexpected outflows, CPI abuse, authority txs --> on-chain path
+-- Post-deploy error spike, wrong program id --> deploy/client path
Failed transactions leave logs, simulation output, and Anchor or native error codes. Simulate with @solana/kit 7.0.0 (simulateTransaction, logs, unitsConsumed) before rebroadcast. Refresh blockhash; map Custom: N to the program enum; separate user error from program bug from compute budget. Blind retry with the same stale message multiplies noise and fees.
RPC and infrastructure outages present as empty balances, "account not found," stuck confirms, or mass WebSocket disconnects. Primary actions: health-check providers (slot lag, genesis hash, error budgets), fail over reads and writes, open circuit breakers so retries do not cascade, and avoid declaring an on-chain exploit when the chain is fine and your edge is not.
On-chain incidents (exploits, authority compromise, critical logic bugs) demand pre-built pause paths, multisig or cold upgrade authority, and evidence preservation. Draining transactions cannot be reversed; containment is stop the bleed, snapshot addresses, and prepare a verified patch or authority rotation. Assume active exploitation until logs prove otherwise.
Bad deploys and client mismatches sit between program and frontend: wrong cluster env, feature flag on for a broken instruction, or upgrade that passes tests but fails under mainnet account shapes. Frontend kill-switches are often faster than program rollback; both may be required.
Containment and mitigation (forward-only)
Solana "rollback" is a product metaphor, not a ledger rewind. Practical levers:
| Lever | Speed | Scope | Notes |
|---|---|---|---|
| Client feature flag / kill switch | Fast | New user flows | Stops new bad paths |
| Multi-provider RPC failover | Fast | Availability | Does not fix program bugs |
| Program pause / mode flag | Medium | Mutating ix | Must exist before crisis |
| Authority rotation | Medium | Control plane | Break-glass multisig |
Redeploy prior verified .so | Medium-slow | Program logic | Verifiable build hash |
| State migration / compensation | Slow | Balances, claims | Needs design and comms |
Order of operations under pressure:
- Safety: protect remaining funds and keys (pause, freeze, authority).
- Stability: restore truthful UX (failover RPC, disable broken UI paths).
- Correctness: deploy known-good program bytes or a reviewed fix.
- Truth: communicate what happened, what users should do, and what is still unknown.
- Learning: freeze the timeline for RCA; do not debate blame in the bridge.
Upgrade authority and pause admin are production features. If they live only on a hot laptop with no runbook, you do not have incident response; you have hope.
Communication under load
Internal bridge: UTC timestamps, signature hashes, program ids, decisions and owners. External: short factual updates (impact, mitigation status, user actions). Do not publish unconfirmed exploit theories or private keys in "debug" screenshots. For Sev-1, plan a customer-facing summary within a fixed window after containment (sibling RCA page details the template).
Advanced Considerations & Applications
Product architecture that survives 3 a.m.
Build for response, not only for happy path:
- Pause and mode flags checked on every mutating instruction (Anchor 0.32.1
GlobalConfigpatterns and native equivalents). - Upgrade authority on multisig with break-glass runbooks; alert on any authority change.
- Verifiable builds so "roll back to v1.2.3" means a known hash, not a mystery binary.
- RPC SLO design: separate read, write, and WebSocket endpoints; synthetic landing checks; provider error budgets in PagerDuty.
- Client send path: simulate, set compute budget and priority fees thoughtfully, confirm through commitment, classify errors before retry.
- Indexers and webhooks for TVL velocity and anomalous CPI, not only explorer refresh.
Teams shipping only feature work without these controls eventually invent them during an incident, at maximum cost.
Decision matrix for first hour
| Question | If yes | Primary playbook |
|---|---|---|
| Are funds leaving unexpectedly? | Sev-1 contain | On-chain incident response |
| Are many users failing on one provider only? | Failover | RPC and infrastructure outages |
| Does simulation show Custom / CU / accounts? | Decode | Debugging failed transactions |
| Did error rate jump right after deploy? | Mitigate | Rollback and mitigation |
| Is the path flaky under load only? | Fees / landing / RPC | Failed txs + RPC pages |
Run the matrix in order; parallelize only after classification so eng and security do not thrash each other.
Closing the loop
When the bridge ends, work is not done. A blameless postmortem separates proximate cause from contributing factors, records what went well, and assigns action items with owners and due dates. Prevention checklists convert those items into Tier-1 deploy gates, Tier-2 30-day fixes, and Tier-3 quarterly drills (pause drills, synthetic landing, authority monitoring).
Incident response maturity is measurable: mean time to classify, mean time to pause or failover, fraction of Sev-1s with completed action items, and drill pass rate. Stack versions matter for drills too: re-validate scripts against Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, Rust 1.91.1, and @solana/kit 7.0.0 after toolchain bumps so runbooks do not rot.
Security and legal adjacency
Whitehat reports, law enforcement, insurers, and auditors may need the same evidence pack your engineers use. Preserve logs, signatures, and authority history; do not "clean up" explorers or rotate keys without recording what was compromised. Program security design (account checks, CPI hygiene) reduces incident frequency; incident response reduces severity when design fails. Both are required.
Common Misconceptions
- "sendTransaction returned a signature, so the user is fine." A signature is a handle; durability is commitment. Incidents often start with optimistic UI that never confirmed.
- "If the app is broken, the program must be exploited." RPC lag, 429s, wrong cluster, and frontend flags produce identical user panic. Classify evidence first.
- "We can reverse a bad mainnet transaction." Finalized state is not rewound. Mitigation is pause, redeploy, migrate, or compensate.
- "Retry with higher priority fees fixes every failure." Fees help landing under congestion; they do not fix Custom program errors, missing accounts, or dead RPC.
- "Pause can be added after the first exploit." If the instruction path does not exist on-chain before the crisis, you cannot invent it in time.
- "Blameless means no accountability." Blameless means fix systems and process; action items still have owners and deadlines.
- "One premium RPC provider is enough." Single points of failure become Sev-2s on provider maintenance days. Failover and health checks are product requirements.
- "Postmortems are optional if we already patched." Without RCA and prevention, the same class of failure returns under a new signature list.
FAQs
What is Solana production incident response in one sentence?
It is the coordinated practice of classifying chain and off-chain failures, containing fund and UX damage with prebuilt levers, mitigating forward without ledger rewind, and turning each event into durable prevention.
How is a failed transaction different from an outage?
A failed transaction has a signature or simulation result and a program or runtime error class; an outage is often multi-user unavailability caused by RPC, network, or dependent infrastructure without a single program root cause.
What should on-call capture in the first five minutes?
Cluster, symptom, signatures and slots, program id, RPC provider, whether funds moved, and recent deploys or authority changes; then assign commander, eng, and comms roles.
When do I pause the program versus only fail over RPC?
Pause when funds or critical state are at risk from on-chain logic or exploitation; fail over RPC when evidence points to provider health, rate limits, or slot lag with healthy on-chain balances.
Can I roll back a Solana program like a web server deploy?
You can redeploy a prior verified binary to the same program id if you hold upgrade authority; you cannot erase finalized account changes that already executed under the bad version.
Why does simulation matter during incidents?
Simulation surfaces logs and compute usage without spending mainnet fees on blind retries, and it separates preflight-detectable bugs from landing and confirmation problems.
How do multi-provider RPC setups help incidents?
They let you route around 429s, regional failures, and single-vendor maintenance; health checks and circuit breakers prevent cascading retries that make outages worse.
What belongs in an on-chain exploit playbook?
Pre-deployed pause, break-glass admin keys or multisig, upgrade authority procedure, evidence capture for exploiter addresses, comms templates, and a path to verified patch under audit pressure.
How soon should we write a postmortem?
Draft the timeline while memory is fresh (same day when possible); complete blameless RCA with action items after containment, and publish customer-facing facts on a fixed Sev-1 SLA.
What makes a postmortem blameless?
It explains system and process conditions that made the failure likely, without shaming individuals, and still assigns concrete fixes with owners and dates.
How do prevention checklists relate to incidents?
Every meaningful Sev should leave monitors, CI gates, or drills on a checklist so the next release proves the gap is closed before users rediscover it.
Where do feature flags fit?
Client and backend flags disable broken instruction paths faster than on-chain upgrades; pair them with on-chain pause when funds can still move through other clients.
Does @solana/kit change incident response?
Kit 7.0.0 improves typed RPC, simulation, and send paths, but classification, commitment, failover, and pause authority remain operational and program design concerns.
What toolchain should runbooks assume?
Pin scripts and examples to Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, Rust 1.91.1, and @solana/kit 7.0.0, and re-drill after upgrades.
Where should I go next after this page?
Use Debugging Failed Transactions and RPC & Infrastructure Outages for the two most common non-exploit paths; open On-Chain Incident Response and Rollback & Mitigation for fund-risk events; close the loop with Prevention Checklists and Blameless Post-Mortems & RCA.
Related
- Debugging Failed Transactions - logs, simulation, Custom errors, and safe retry discipline
- RPC & Infrastructure Outages - multi-provider failover, health checks, and rate-limit storms
- On-Chain Incident Response - pause, authority lockdown, and upgrade-under-fire
- Rollback & Mitigation - verifiable redeploy, client flags, and user communications
- Prevention Checklists - monitors, drills, and deploy gates from past Sevs
- Blameless Post-Mortems & RCA - timeline, root cause, and owned action items
Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, Rust 1.91.1, and @solana/kit 7.0.0.