Getting started

This guide will instruct you through:

• Creating an API token needed for pipelines to authenticate with your data catalog.
• Creating your first pipeline with a simple ecommerce schema that writes to an Apache Iceberg table managed by R2 Data Catalog.
• Sending sample ecommerce data via HTTP endpoint.
• Validating data in your bucket and querying it with R2 SQL.

1. Sign up for a Cloudflare account .
2. Install Node.js .

Node.js version manager

Use a Node version manager like Volta or nvm to avoid permission issues and change Node.js versions. Wrangler, discussed later in this guide, requires a Node version of 16.17.0 or later.

Pipelines must authenticate to R2 Data Catalog with an R2 API token that has catalog and R2 permissions.

{
	"fields": [
		{
			"name": "user_id",
			"type": "string",
			"required": true
		},
		{
			"name": "event_type",
			"type": "string",
			"required": true
		},
		{
			"name": "product_id",
			"type": "string",
			"required": false
		},
		{
			"name": "amount",
			"type": "float64",
			"required": false
		}
	]
}

npx wrangler pipelines setup

npx wrangler pipelines setup --name ecommerce

Send ecommerce events to your pipeline's HTTP endpoint:

curl -X POST https://{stream-id}.ingest.cloudflare.com \
  -H "Content-Type: application/json" \
  -d '[
    {
      "user_id": "user_12345",
      "event_type": "purchase",
      "product_id": "widget-001",
      "amount": 29.99
    },
    {
      "user_id": "user_67890",
      "event_type": "view_product",
      "product_id": "widget-002"
    },
    {
      "user_id": "user_12345",
      "event_type": "add_to_cart",
      "product_id": "widget-003",
      "amount": 15.50
    }
  ]'

Replace {stream-id} with your actual stream endpoint from the pipeline setup.

Set up your environment to use R2 SQL:

export WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN

Or create a .env file with:

WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN

Where YOUR_API_TOKEN is the token you created in step 1. For more information on setting environment variables, refer to Wrangler system environment variables.

Query your data:

npx wrangler r2 sql query "YOUR_WAREHOUSE_NAME" "
SELECT
    user_id,
    event_type,
    product_id,
    amount
FROM default.ecommerce
WHERE event_type = 'purchase'
LIMIT 10"

Replace YOUR_WAREHOUSE_NAME with the warehouse name noted during pipeline setup. You can find it in the Cloudflare dashboard under R2 object storage > your bucket > Settings > R2 Data Catalog.

You can also query this table with any engine that supports Apache Iceberg. To learn more about connecting other engines to R2 Data Catalog, refer to Connect to Iceberg engines.

Streams

Pipelines

Sinks

Examples