SQD Data Processing Crates
This repo contains Rust crates for blockchain data processing, indexing, and querying. They are used in various SQD components.
| Crate |
Description |
| sqd-primitives |
Basic types, range types, and SID (sequence ID) primitives used throughout the system |
| sqd-dataset |
Dataset descriptions, table schemas, and schema options for defining data structures |
| sqd-array |
Arrow array operations including building, slicing, sorting, I/O, and chunking utilities |
| sqd-bloom-filter |
Bloom filter implementation for arrow buffers |
| Crate |
Description |
| sqd-data |
Data models for various blockchain protocols. Used in the query engine and parquet writers |
| sqd-data-core |
Core data processing utilities: chunk building, table processing, serialization, and sorting algorithms |
| sqd-data-client |
HTTP client for fetching data from SQD data sources with streaming support |
| sqd-data-source |
Abstractions for standard and mapped data sources |
| Crate |
Description |
| sqd-query |
Query engine used by the Worker |
| sqd-query-example |
Example demonstrating how to use the query engine with Parquet files |
| sqd-storage |
RocksDB-based storage layer with key-value store and table management |
| sqd-polars |
Polars DataFrame integration for high-performance data analysis |
| Crate |
Description |
| sqd-archive |
Archive service for ingesting and storing data to S3 with layout management, progress tracking, and Prometheus metrics |
| sqd-bds |
Big Data Service (WIP) - Cassandra-based data storage |
| sqd-hotblocks |
Hotblocks database with portal-like API |
| sqd-hotblocks-retain |
Retention coordinator that applies Hotblocks retain points based on scheduling status updates |
The "docker" workflow should be triggered manually with the following inputs:
- target — one of hotblocks, hotblocks-retain, or sqd-archive.
- tag — a 8-byte hash of the commit. This will be the published docker tag.
- platforms — platforms to build for. Using only linux/amd64 instead of default values can save a lot of building time.