Delivery Best Practices

Enterprise Solana delivery means releases are reversible, bytecode is verifiable, and observability proves health within minutes of deploy. These practices reduce change-fail rate and MTTR.

How to Use This List

Gate mainnet promotion on sections A and B
Apply C before any public launch or TVL milestone
Review D monthly with platform and security leads

A - Verifiable & Reversible

Build with anchor build --verifiable in CI. Artifact hash published on every tag.
Retain last three .so files per program. Rollback does not depend on one engineer's laptop.
Run solana-verify before and after mainnet deploy. Hash matches release notes.
Document rollback vs forward-fix decision tree. Linked from every deploy ticket.
Never deploy mainnet from unmerged branch. Squads proposal references git SHA.

B - Authority & Governance

Upgrade authority on multisig before launch. Hot keys only on break-glass doc.
Separate deploy wallets per environment. Devnet keys never touch mainnet programs.
Timelock on community-visible upgrades when policy requires. Emergency path documented.
Pause instruction tested quarterly. Containment under 5 minutes on drill.
IDL semver in release notes. Clients pin compatible @coral-xyz/anchor 0.32.1 types.

C - Coordinated Release Train

Single change window for breaking IDL updates. Program + frontend + indexer in sync.
Devnet smoke mandatory. Top flows simulate and send before mainnet.
Feature flags for new instruction paths. On-chain guard for security-critical flags.
Env validation fails build on missing PROGRAM_ID. Preview cannot target mainnet.
Indexer team notified pre-deploy. Backfill plan for new accounts or events.

D - Observable Deploys

Synthetic landing tx within 15 minutes post-deploy. Alert on failure.
Program error rate dashboard per release tag. Compare to pre-deploy baseline.
RPC failover configured in production client. Primary and secondary URLs tested.
DORA metrics reviewed monthly. Deploy frequency, change-fail, MTTR trends.
Post-mortem action items close within 30 days. Or exec escalation.

FAQs

Which section blocks hotfix?

B pause authority never blocked; full A-D required before re-enabling full traffic after fix.

Minimum team size for this?

Two engineers can follow A-B; C-D scale with launch size.

Verifiable builds slow CI?

Small cost vs audit and incident cost; cache target dir.

Immutable programs?

Still apply client and ops practices; skip upgrade-specific items.

How prove observability?

Link Grafana dashboard and synthetic check run in deploy ticket.

Agave upgrade in train?

Add SIMD review item; test on Surfpool 0.12.0 before mainnet program deploy.

External audit gate?

No mainnet user funds until audit findings Sev-High closed or accepted in ADR.

Who owns list compliance?

Tech lead signs deploy ticket; security reviews B monthly.

Relation to DORA?

These practices directly improve change-fail rate and MTTR.

Startup MVP skip?

Never skip A-B if mainnet holds user funds; C-D can phase in.

DORA Metrics for Web3 - measure outcomes
Program Upgrade Strategy - upgrade detail
Prevention Checklists - incident prevention

Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, anchor-lang 0.32.1, Rust 1.91.1, @solana/kit 7.0.0, Surfpool 0.12.0, and LiteSVM 0.6.x.

Delivery Best Practices

Enterprise Solana delivery means releases are reversible, bytecode is verifiable, and observability proves health within minutes of deploy. These practices reduce change-fail rate and MTTR.

How to Use This List

Gate mainnet promotion on sections A and B
Apply C before any public launch or TVL milestone
Review D monthly with platform and security leads

A - Verifiable & Reversible

Build with anchor build --verifiable in CI. Artifact hash published on every tag.
Retain last three .so files per program. Rollback does not depend on one engineer's laptop.
Run solana-verify before and after mainnet deploy. Hash matches release notes.
Document rollback vs forward-fix decision tree. Linked from every deploy ticket.
Never deploy mainnet from unmerged branch. Squads proposal references git SHA.

B - Authority & Governance

Upgrade authority on multisig before launch. Hot keys only on break-glass doc.
Separate deploy wallets per environment. Devnet keys never touch mainnet programs.
Timelock on community-visible upgrades when policy requires. Emergency path documented.
Pause instruction tested quarterly. Containment under 5 minutes on drill.
IDL semver in release notes. Clients pin compatible @coral-xyz/anchor 0.32.1 types.

C - Coordinated Release Train

Single change window for breaking IDL updates. Program + frontend + indexer in sync.
Devnet smoke mandatory. Top flows simulate and send before mainnet.
Feature flags for new instruction paths. On-chain guard for security-critical flags.
Env validation fails build on missing PROGRAM_ID. Preview cannot target mainnet.
Indexer team notified pre-deploy. Backfill plan for new accounts or events.

D - Observable Deploys

Synthetic landing tx within 15 minutes post-deploy. Alert on failure.
Program error rate dashboard per release tag. Compare to pre-deploy baseline.
RPC failover configured in production client. Primary and secondary URLs tested.
DORA metrics reviewed monthly. Deploy frequency, change-fail, MTTR trends.
Post-mortem action items close within 30 days. Or exec escalation.

FAQs

Which section blocks hotfix?

B pause authority never blocked; full A-D required before re-enabling full traffic after fix.

Minimum team size for this?

Two engineers can follow A-B; C-D scale with launch size.

Verifiable builds slow CI?

Small cost vs audit and incident cost; cache target dir.

Immutable programs?

Still apply client and ops practices; skip upgrade-specific items.

How prove observability?

Link Grafana dashboard and synthetic check run in deploy ticket.

Agave upgrade in train?

Add SIMD review item; test on Surfpool 0.12.0 before mainnet program deploy.

External audit gate?

No mainnet user funds until audit findings Sev-High closed or accepted in ADR.

Who owns list compliance?

Tech lead signs deploy ticket; security reviews B monthly.

Relation to DORA?

These practices directly improve change-fail rate and MTTR.

Startup MVP skip?

Never skip A-B if mainnet holds user funds; C-D can phase in.

DORA Metrics for Web3 - measure outcomes
Program Upgrade Strategy - upgrade detail
Prevention Checklists - incident prevention

Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, anchor-lang 0.32.1, Rust 1.91.1, @solana/kit 7.0.0, Surfpool 0.12.0, and LiteSVM 0.6.x.

Delivery Best Practices

Search Documentation

Delivery Best Practices