Data Analysis

Analytics workflows pull Solana data from RPC snapshots, indexer exports, or provider APIs into pandas or Polars for aggregation, joins, and visualization. Python is the default glue between Agave 4.1.1 chain data and notebooks, dashboards, and batch reports.

Recipe

Quick-reference recipe card - copy-paste ready.

import polars as pl
 
df = pl.DataFrame(
    {
        "signature": [],
        "slot": [],
        "fee_lamports": [],
        "success": [],
    }
)
summary = df.group_by("success").agg(pl.col("fee_lamports").mean())

When to reach for this:

• Exploratory analysis in Jupyter on transaction samples.
• Daily treasury or protocol metrics (volume, fees, unique signers).
• Joining on-chain events with off-chain CRM or product data.
• Building training sets for monitoring and anomaly detection.

Working Example

import asyncio
import polars as pl
from solders.pubkey import Pubkey
from solana.rpc.async_api import AsyncClient
 
PROGRAM = Pubkey.from_string("<PROGRAM_ID>")
RPC = "https://api.devnet.solana.com"
 
async def collect_signatures(limit: int = 200):
    async with AsyncClient(RPC) as client:
        resp = await client.get_signatures_for_address(PROGRAM, limit=limit)
        rows = [
            {
                "signature": str(item.signature),
                "slot": item.slot,
                "err": item.err is not None,
                "block_time": item.block_time,
            }
            for item in resp.value
        ]
    return pl.DataFrame(rows)
 
async def enrich_with_fees(df: pl.DataFrame):
    async with AsyncClient(RPC) as client:
        fees = []
        for sig in df["signature"].to_list():
            tx = await client.get_transaction(sig, max_supported_transaction_version=0)
            meta = tx.value.transaction.meta if tx.value else None
            fees.append(meta.fee if meta else None)
    return df.with_columns(pl.Series("fee_lamports", fees))
 
async def main():
    df = await collect_signatures()
    df = await enrich_with_fees(df)
    print(df.group_by("err").agg(pl.len(), pl.col("fee_lamports").mean()))
 
asyncio.run(main())

What this demonstrates:

• Signature history seeds a Polars frame without heavy deps.
• Per-transaction enrichment fetches meta fee fields.
• Group-by aggregates success vs failure fee averages.
• Async RPC suits batch enrichment scripts.

Deep Dive

How It Works

• Extract: RPC (getSignaturesForAddress, getTransaction), Geyser exports, or warehouse tables.
• Normalize: stable schemas (signature, slot, program_id, instruction_index, amount).
• Transform: pandas/Polars for joins, window functions, resampling by slot/time.
• Load: Parquet to object storage, DuckDB, BigQuery, or dashboard tools.

Data Source Trade-offs

Python Notes

# Prefer Polars for large scans
df = pl.scan_parquet("s3://bucket/txs/*.parquet").filter(pl.col("slot") > 250_000_000).collect()
 
# Integer amounts only
df = df.with_columns((pl.col("amount_raw") / 10**decimals).alias("amount"))  # watch float policy
 
# pandas interop
pdf = df.to_pandas()

Gotchas

• RPC backfill at scale - slow and rate-limited. Fix: indexer or provider bulk export.
• Float token amounts - precision loss on 9-decimal mints. Fix: keep raw u64 strings/integers in frames.
• Unbounded getTransaction loops - expensive. Fix: batch signatures; cache responses to Parquet.
• Schema drift across program versions - breaks joins. Fix: version column from IDL/discriminator map.
• Timezone confusion on block_time - Unix seconds UTC. Fix: explicit datetime conversion in Polars.

Alternatives

FAQs

Related

• Reading Chain Data - RPC and DAS reads
• Building an Indexer - production pipelines
• Pagination & Efficiency - RPC batching
• Python for Solana Best Practices - service hygiene

Stack versions: This page was written for Agave 4.1.1, Solana CLI 3.0.10, Anchor 0.32.1, anchor-lang 0.32.1, Rust 1.91.1, @solana/kit 7.0.0, Surfpool 0.12.0, and LiteSVM 0.6.x.