This repository handles the HTTP Archive data pipeline, which takes the results of the monthly HTTP Archive run and saves this to the httparchive dataset in BigQuery.

The pipelines are run in Dataform service in Google Cloud Platform (GCP) and are kicked off automatically on crawl completion and other events. The code in the main branch is used on each triggered pipeline run.

Tag: crawl_complete

• Crawl dataset httparchive.crawl.*

  Consumers:

  • public dataset and BQ Sharing Listing

• Blink Features Report httparchive.blink_features.usage

  Consumers:

  • chromestatus.com

Tag: crux_ready

• httparchive.reports.cwv_tech_* and httparchive.reports.tech_*

  Consumers:

  • HTTP Archive Tech Report

1. Install dependencies:

   npm install

2. Available Scripts:

   • npm run format - Format code and fix Markdown issues
   • npm run lint - Run linting checks on Markdown files, and compile Dataform configs

This repository uses:

• Markdownlint for Markdown file formatting
• Dataform's built-in compiler for SQL validation