Polymarket Substreams

A high-performance Substreams package for extracting and processing Polymarket prediction market data on Polygon, with built-in support for PostgreSQL, ClickHouse, and SQL sinks.

Features

• 13 Event Types: Full coverage of CTF Exchange events (order fills, matches, cancellations, fees, token registrations, admin/operator changes, factory updates, trading pauses)
• SQL Sink Ready: db_out module produces DatabaseChanges for direct use with substreams-sink-sql
• PostgreSQL & ClickHouse: Schemas included for both databases with indexes and ClickHouse materialized views
• Stateful Stores: Cumulative order counts and trading volume per market via BigInt stores
• Block Filtering: Efficient indexing using ethcommon:index_events filters

Contracts Indexed

Architecture

                    ┌─────────────────────────┐
                    │  sf.ethereum.type.v2.Block  │
                    └────────────┬────────────┘
                                 │
              ┌──────────────────┼──────────────────┐
              ▼                  ▼                  ▼
   map_ctf_exchange_events  map_ctf_events    map_usdc_events
              │
     ┌────────┼────────┐
     ▼        ▼        ▼
  store_     store_   db_out ──▶ DatabaseChanges
  total_     total_              │
  order_     volume         ┌────┴────┐
  count                     ▼         ▼
                        PostgreSQL  ClickHouse

Modules

Map Modules

Store Modules

Quick Start

Prerequisites

1. Install the Substreams CLI
2. Get an API key from The Graph Market

Build

git clone https://github.com/PaulieB14/Polymarket-Substreams.git
cd Polymarket-Substreams
substreams build

Stream Events

# Stream CTF Exchange events
substreams gui substreams.yaml map_ctf_exchange_events -e polygon.substreams.pinax.network:443

# Stream database changes (for sink testing)
substreams gui substreams.yaml db_out -e polygon.substreams.pinax.network:443

SQL Sink Setup

PostgreSQL

# 1. Setup the database schema
substreams-sink-sql setup "psql://user:password@localhost:5432/polymarket" ./substreams.yaml

# 2. Run the sink
substreams-sink-sql run "psql://user:password@localhost:5432/polymarket" ./substreams.yaml

ClickHouse

# 1. Setup using ClickHouse schema
substreams-sink-sql setup "clickhouse://user:password@localhost:9000/polymarket" ./substreams.yaml --schema ./schema.clickhouse.sql

# 2. Run the sink
substreams-sink-sql run "clickhouse://user:password@localhost:9000/polymarket" ./substreams.yaml --engine clickhouse

Note: Pass the DSN on the command line, not in the manifest. The schema.sql (PostgreSQL) and schema.clickhouse.sql (ClickHouse) files define the table structures.

Database Tables

Event Tables

Aggregate Tables (from Store Deltas)

ClickHouse Materialized Views

• mv_hourly_volume: Hourly trading volume, trade count, and fees per market
• mv_trader_stats: Per-trader stats (trade count, total volume, fees paid)

Example Queries

PostgreSQL: Top Markets by Order Volume

SELECT maker_asset_id, COUNT(*) AS fills, SUM(CAST(maker_amount_filled AS NUMERIC)) AS volume
FROM order_fills
GROUP BY maker_asset_id
ORDER BY fills DESC
LIMIT 20;

PostgreSQL: Most Active Traders

SELECT taker, COUNT(*) AS trades, SUM(CAST(fee AS NUMERIC)) AS total_fees
FROM order_fills
GROUP BY taker
ORDER BY trades DESC
LIMIT 20;

ClickHouse: Hourly Volume (from materialized view)

SELECT hour, trade_count, total_maker_volume
FROM mv_hourly_volume
WHERE maker_asset_id = '<MARKET_ID>'
ORDER BY hour DESC
LIMIT 24;

Development

Building from Source

# Install Rust and wasm target
rustup target add wasm32-unknown-unknown

# Build
cargo build --target wasm32-unknown-unknown --release

# Or use the Substreams CLI
substreams build

Project Structure

├── abi/                          # Contract ABI JSON
├── proto/
│   └── contract.proto            # Protobuf definitions
├── src/
│   ├── abi/                      # Generated ABI bindings
│   ├── pb/                       # Generated protobuf code
│   └── lib.rs                    # Module handlers (maps, stores, db_out)
├── schema.sql                    # PostgreSQL DDL
├── schema.clickhouse.sql         # ClickHouse DDL with materialized views
├── substreams.yaml               # Module manifest with sink config
├── Cargo.toml                    # Rust dependencies
└── build.rs                      # ABI code generation

Adding New Event Types

1. Add the contract ABI to abi/
2. Define protobuf messages in proto/contract.proto
3. Add a map module in src/lib.rs and substreams.yaml
4. Add corresponding tables in schema.sql and schema.clickhouse.sql
5. Wire the new events into the db_out function

Publishing

# Package the substreams
substreams pack

# Publish to the registry
substreams publish

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

License

MIT - see LICENSE.

Links

• Substreams Documentation
• Polymarket
• The Graph
• Built with Substreams by StreamingFast