Indexing Blueprint
Solana indexing is the practice of turning slot-ordered chain events into durable, queryable application data so products can answer "who did what, when, and how much?" without hammering JSON-RPC for every screen. Once you separate live chain access (RPC, WebSockets) from shaped history (your database and API), Geyser plugins, Yellowstone gRPC, webhooks, IDL parsers, and BI dashboards stop looking like unrelated tools and start looking like layers of one data plane.
Indexing Basics is the hands-on entry; Geyser Plugins, Yellowstone gRPC, Webhooks, Building an Indexer, Parsing Program Data, and Analytics & Dashboards each zoom into one stage. This page sits underneath: why indexers exist, how the pieces connect, and when direct RPC is still the right answer.
Summary
- An indexer continuously ingests Solana transactions and account updates, parses them with program layouts, and persists domain rows so apps and analytics query a database instead of replaying the chain on every request.
- Insight: RPC polling and pagination cannot power SQL volume, wallet history at whale scale, or multi-tenant product APIs; without indexing, product features collapse into rate limits, incomplete history, and fragile log scraping.
- Key Concepts: backfill vs live stream, Geyser / Yellowstone, webhooks, slot cursor, idempotent upserts, IDL discriminators, inner instructions, commitment and reorgs, serving API, analytics warehouse.
- When to Use: Product search and history UIs, DeFi metrics, portfolio snapshots beyond DAS, alerts and ops dashboards, or any feature that needs joins, aggregations, or multi-day retention over program activity.
- Limitations/Trade-offs: You own schema, lag, parser versioning, and ops cost; streams add infra; webhooks can miss events; self-hosted Geyser ties you to validator lifecycle; direct RPC remains better for simple "current balance" reads.
- Related Topics: indexing basics, Geyser plugins, Yellowstone gRPC, webhooks, building an indexer, parsing program data, analytics and dashboards.
Foundations
Solana RPC answers point questions: current account data, a signature status, a recent block, or a page of signatures for an address. That model is correct for wallets and transaction submission. It is a poor model for product history, aggregations, and cross-entity queries.
An indexer exists because the chain is an append-only execution log optimized for validators, not for your app's SQL or GraphQL shape. You need a second store that rewrites "raw slots and bytes" into "swaps, deposits, mints, positions" with indexes your product can afford to query.
Think of the data plane as layers from consensus out to product surfaces:
+------------------------------------------+
| Analytics & dashboards (BI, metrics) | <- reads shaped tables
+------------------------------------------+
| Serving API (REST / GraphQL / workers) |
+------------------------------------------+
| Durable store (Postgres, warehouse) | <- idempotent domain rows
+------------------------------------------+
| Parser (IDL, Borsh, Codama, events) |
+------------------------------------------+
| Ingest (Geyser, Yellowstone, webhooks, |
| RPC backfill / archives) |
+------------------------------------------+
| Chain (Agave validators, slots, txs) |
+------------------------------------------+Direct RPC is still essential for submission, confirmation, and small interactive reads (getAccountInfo, getBalance, status of a just-sent signature). @solana/kit (this site: 7.0.0) sits there for clients. Indexing does not replace RPC; it offloads history and analytics so RPC stays available for latency-sensitive paths.
Why polling fails at scale is mechanical. getSignaturesForAddress and getTransaction walk history as many HTTP round trips; endpoints rate-limit bulk history; there is no durable join for "all deposits last week" unless you write those rows once. Indexing Basics shows the pipeline; the foundation is write once at the tip, query many from your store.
Geyser is the validator-side hook: an Agave plugin (.so) receives account and transaction updates, often filtered by owner or mention. Yellowstone gRPC exposes that firehose over the network without requiring you to run stake. Webhooks invert the connection: a provider matches filters and POSTs HTTP payloads. All three feed the same pipeline; they differ in latency, ops burden, and delivery guarantees.
Parsing is the contract between chain bytes and product language: Anchor 8-byte discriminators, Borsh or zero-copy layouts, IDL/Codama decoders. Analytics is the last mile: materialized views, documented metrics, and dashboards that never call mainnet RPC per panel.
Mechanics & Interactions
Backfill and live stream share one cursor
Production systems almost always run two modes on the same schema.
- Backfill walks historical slots or archived transactions (RPC
getBlock/getTransaction, provider archives, or ledger exports) until a stored cursor reaches near the tip. - Live stream tails new slots via Geyser, Yellowstone, or webhooks, advancing the same cursor.
[historical slots] ---- backfill ----+
v
last_slot cursor
^
[tip stream] -------- live tail -----+
|
v
parse -> upsert -> APICrash recovery is "replay from last durable slot with idempotent writes." Natural keys such as (signature, instruction_index) or (pubkey, write_version) make replays safe. Without that, retries and stream reconnects double-count volume and break finance metrics.
Ingest paths: Geyser, Yellowstone, webhooks, RPC
| Path | Source of truth | Typical fit |
|---|---|---|
| Geyser plugin | Your Agave validator hooks | Self-hosted, max control, filter at source |
| Yellowstone gRPC | Provider or self-hosted Geyser distribution | High throughput remote consumers (Rust/Go) |
| Webhooks | Provider-matched HTTP push | Serverless, alerts, moderate volume |
| RPC backfill | getBlock / signatures APIs | Catch-up, gap fill, small histories |
Geyser plugins load into the validator process on Agave 4.1.1 with a config JSON and must stay non-blocking; slow hooks risk validator lag. Selectors on program owners and transaction mentions cut bandwidth before your network pays for the full mainnet firehose. Details: Geyser Plugins.
Yellowstone-style gRPC moves filtering and fan-out off your product servers while preserving slot-ordered streams. Webhooks invert the connection: the provider calls you. Fast 2xx ACKs plus a queue keep retries from becoming duplicate work; always verify HMAC or provider auth. Details: Webhooks.
RPC remains the gap filler: after outages, compare stream cursor to chain tip and backfill missing slots. Heavy mainnet history should use archives or dedicated nodes, not unrestricted public endpoints.
Parse with layouts, not log strings
Instruction data and account blobs are the canonical inputs. Match program id, then discriminator or enum tag, then deserialize fields. Walk inner instructions in transaction meta for CPI-heavy DeFi; many "business" events only appear as nested invokes.
transaction
outer instructions --> your program?
meta.innerInstructions --> CPI stack (often where value moves)
account keys (+ ALT resolution) --> metas for decode context
logs --> optional diagnostics, never sole truthPin IDL or layout versions to program deploy tags so upgrades introduce a new parser version rather than silent mis-decodes. Store unknown rows for forward-compatible discriminators. See Parsing Program Data.
Persist, serve, then analyze
Building an Indexer centers the pipeline: ingest → normalize → parse → upsert → API. Fact tables (events, swaps, transfers) and optional account snapshots power product endpoints with cursors on (slot, signature).
Analytics & Dashboards sit on top of those tables: materialized views for daily volume, documented metric definitions (DAU, fees, retention), and BI tools on read-only replicas. Charts that hit JSON-RPC per refresh recreate the rate-limit problem indexing was meant to solve.
Commitment, reorgs, and "fresh enough"
Streams can deliver data at processed or confirmed speed while finance-critical rows should mark finalized only under your policy. Design provisional vs finalized flags or delayed promotion rather than assuming every webhook payload is eternal. Reorgs are rare on modern Solana after confirmation, but idempotent keys and slot rollback handling still belong in production designs.
How the pieces interact in one day of work
- Choose ingest: webhook MVP, Yellowstone for volume, or self-hosted Geyser if you already operate nodes.
- Define schema and natural keys; implement upserts before fancy metrics.
- Generate or hand-test parsers against fixture transactions (include inner ix).
- Backfill a bounded window; start the live consumer; store
last_slot. - Expose a thin API for product; connect dashboards to a replica.
- Alert on lag (tip slot minus indexed slot), parse error rate, and queue depth.
Client reads of current balances can stay on RPC via kit; historical portfolio and admin analytics go through the index.
Advanced Considerations & Applications
Teams do not need a full warehouse on day one. Match product risk, volume, and team size to an ingest and storage path, then deepen parsers and analytics as questions get harder.
| Path | What you run | Strengths | Weaknesses | Best fit |
|---|---|---|---|---|
| RPC + light cache | kit/RPC, short TTL cache, little history | Minimal ops; fine for "now" UIs | No deep history; rate limits on whales | MVPs, wallets with provider-backed history, one-off status checks |
| Webhook + Postgres | Provider webhooks, queue, Postgres, API | Fast to ship; simple HTTP ops | Provider filters and caps; gap risk | Alerts, moderate program activity, serverless teams |
| Yellowstone / Geyser + indexer | Stream consumer, parsers, durable DB, lag monitors | Throughput; slot-ordered control; custom programs | Infra and parser ownership; ABI/version coupling for plugins | High-volume DeFi, proprietary metrics, multi-tenant products |
| Full analytics stack | Above + warehouse/MVs, metric layer, BI | Trusted business metrics; joins at scale | Definition drift; cost of storage and freshness SLAs | Growth, ops, risk, executive reporting |
RPC + light cache is still correct when the question is "what is this account right now?" Do not index for its own sake.
Webhook + Postgres optimizes for team velocity. Pair with periodic reconciliation jobs so downtime does not become permanent holes.
Yellowstone / Geyser + indexer is the default serious path once instruction volume or custom program complexity outgrows provider parse templates. Filter early; keep hooks and stream handlers thin; put business logic in workers.
Full analytics stack treats metric definitions as product: USD prices at block time, bot filtering, retention cohorts, and refresh SLAs. Public tools (Dune-style) help open protocols; private programs need your own store.
Do not conflate layers: kit is not an indexer, webhooks are not complete history, DAS is not arbitrary instruction indexing, and a green dashboard query is not proof the stream is caught up. For design reviews, answer first: which ingest path, which natural keys, which commitment for money, which lag budget, and who owns IDL version bumps after program upgrade.
Common Misconceptions
- "We can power the product entirely with getSignaturesForAddress." Pagination and rate limits collapse under real user history and analytics; that path is for thin clients and gap fill, not the whole product database.
- "WebSockets alone are an indexer." Subscriptions deliver live updates; without durable storage, cursors, and parsers you still cannot serve SQL history after reconnect.
- "Geyser is only for validator operators who care about staking." Geyser is the low-level stream interface; product teams consume it via plugins or Yellowstone even when they never run consensus.
- "Log messages are a stable schema." Logs change with program refactors; instruction data and account layouts are the durable contract.
- "Idempotency is optional if the stream is reliable." Retries, restarts, and provider redelivery will double-insert without natural keys.
- "Backfill is a one-time launch chore." Gaps from outages, filter changes, and new instruction types require ongoing replay tools.
- "Provider enhanced payloads remove the need to understand the program." They accelerate common SPL patterns; custom programs still need your own decode rules and upgrade policy.
- "Dashboards should query mainnet RPC for freshness." Freshness belongs in the indexer lag budget; dashboards should read shaped tables on a replica.
FAQs
What is the single most important idea in Solana indexing?
Separate live chain access (RPC for current truth and submission) from a durable, parsed, queryable store so products scale without replaying the ledger on every request.
When is direct RPC enough without an indexer?
When you only need current balances, small account reads, transaction send/confirm, or tiny histories that fit provider APIs and rate limits.
How does Geyser relate to Yellowstone?
Geyser is the validator plugin interface for account and transaction updates; Yellowstone-style gRPC systems distribute those streams to remote consumers with filters.
Do I need my own validator to index mainnet?
No. Managed Yellowstone, webhooks, and archive RPC cover most teams; self-hosted Geyser is for control, custom sinks, or existing node ops.
What is a slot cursor?
A durable record of the last fully processed slot (per source) so restarts and gap fills resume without reprocessing the entire chain.
Why are primary keys on signature plus instruction index common?
Signatures identify transactions; instruction index (including careful handling of inner ix) identifies a single logical event for idempotent upserts.
Should I parse inner instructions?
Yes for CPI-heavy programs. Business-relevant token moves and program calls often appear only inside meta.innerInstructions.
How do webhooks differ from a full indexer?
Webhooks are an ingest transport. You still need parsing, storage, gap reconciliation, and a serving layer for complete product history.
What breaks when a program upgrades?
New discriminators, account layouts, or CPI patterns can fail old parsers. Pin IDL per deploy, version event rows, and dual-run parsers during migrations.
How should I handle stream lag?
Monitor tip slot minus indexed slot, alert on thresholds, and auto-backfill gaps rather than silently serving stale APIs as if they were live.
Is Postgres enough for analytics?
Often for early and mid scale. Move heavy aggregations to a warehouse or OLAP engine when dashboard queries and retention outgrow the OLTP replica.
Where does @solana/kit fit this blueprint?
Kit drives RPC client paths for backfill helpers, confirmation, and app reads of current state. High-volume stream consumers are often Rust or provider SDKs beside kit.
Can DAS replace a custom indexer?
DAS excels at digital asset reads and compressed NFT surfaces. It does not replace arbitrary custom-program instruction indexing or your private metric definitions.
What commitment level should indexed finance rows use?
Document a policy: many systems ingest early for UX but only treat rows as final after confirmed or finalized, with explicit handling for rare rollbacks.
How do I start if I only need alerts today?
Ship webhooks with verified signatures, idempotent handlers, and a small events table; add Yellowstone and deeper parsers when volume or history requirements grow.
Related
- Indexing Basics - why RPC alone fails and the core pipeline shape
- Geyser Plugins - validator-side streaming hooks and selectors
- Webhooks - HTTP push ingest and verification patterns
- Building an Indexer - persistence, upserts, and serving APIs
- Parsing Program Data - IDL, discriminators, and decode rules
- Analytics & Dashboards - metrics, materialized views, and BI
Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, Rust 1.91.1, and @solana/kit 7.0.0.