firebolt-db/firebolt-core: Firebolt Core is a free, self-hosted edition of Firebolt's distributed query engine (https://www.firebolt.io/); it provides high-performance data warehousing capabilities that can be deployed anywhere from a single laptop to enterprise datacenters. · GitHub

Deploy anywhere, from a single laptop to your own datacenter

• 🚀 Powerful. Firebolt Core ships with all key performance and usability features of Firebolt's managed Cloud data warehouse, including a state-of-the-art query optimizer, distributed query execution engine, Iceberg support, and many more.
• 🆓 Free to use. Firebolt Core is free to use, forever (see the LICENSE for details).
• 📈 No usage limits. Firebolt Core has no usage limits. Process unlimited data, scale to as many nodes as you need, and run as many queries as you like.
• 🛢️ Postgres compliant. Firebolt's SQL dialect is Postgres compliant. We offer powerful extensions for analytical workloads, such as lambda functions for array processing. For a complete reference, see the SQL reference documentation.
• 🛠️ Self-contained. Firebolt Core comes packaged as a single Docker image (ghcr.io/firebolt-db/engine:dev) that contains everything needed to run it.
• 🏠 Self-hosted. You can deploy Firebolt Core anywhere you want, from your personal workstation to large on-premise clusters or VPCs.
• 📊 First-class support with updates and active community support via GitHub Discussions. We encourage you to join the conversation!
• 🤖 AI-ready architecture optimized for modern data and ML applications.
• 🎯 Designed for demanding applications. Powering real-time analytics, embedded analytics, and large-scale data processing workloads.
• 🔄 Workload compatibility. Many workloads run interchangeably with managed Firebolt.

Start Core on your machine with:

curl -fsSL https://get.firebolt.io/ | bash

If you want to work with Docker directly, you can also run on Linux:

docker run -i --rm \
        --ulimit memlock=8589934592:8589934592 \
        --security-opt seccomp=unconfined \
        -p 127.0.0.1:3473:3473 \
        -v ./firebolt-core-data:/var/lib/firebolt \
        ghcr.io/firebolt-db/engine:dev

Or on MacOS:

mkdir -p -m 777 firebolt-core-data
# On macOS the Docker Desktop file-sharing backend cannot stat the engine's Unix
# domain socket when it lives on the bind-mounted data directory, so relocate the
# socket onto an in-memory tmpfs at /run/firebolt via a config file.
cat > firebolt-core-data/config.yaml <<'EOF'
schema_version: "1.0"
endpoints:
  http:
    listeners:
      - type: tcp
        port: 3473
      - type: unix
        path: /run/firebolt/query_endpoint
EOF
docker run -i --rm \
        --user root \
        --ulimit memlock=8589934592:8589934592 \
        --security-opt seccomp=unconfined \
        -p 127.0.0.1:3473:3473 \
        -v ./firebolt-core-data:/var/lib/firebolt \
        --tmpfs /run/firebolt:rw,mode=2770,uid=3473,gid=0 \
        ghcr.io/firebolt-db/engine:dev

You can also start a single node cluster by cloning this repository and then run the following command within the repository root directory:

docker compose up

Use this setup if you want to leverage the computing power of multiple hosts.

1. Add in config.json one entry (either hostname or IP address) for each of the nodes; node 0 is the first node in the array:

   {
       "nodes": [
           {
               "host": "ip-or-host-of-node-0"
           },
           {
               "host": "ip-or-host-of-node-1"
           },
           {
               "host": "ip-or-host-of-node-2"
           }
       ]
   }

2. Make sure that this repository and your config.json are present on each of the nodes.

3. Create the volume directory and assign correct permissions:

   mkdir -p ./firebolt-core-data
   sudo chown -R 3473:3473 ./firebolt-core-data

4. Activate node 0:

   docker compose up

5. Activate all further nodes by running on each host:

   NODE=1 docker compose up

   Increase NODE for each further node.

If your host is macOS then you will need to run the Core containers as root:

CORE_USER=root docker compose up

See also:

• Docker Compose

For Kubernetes-based deployments, see the Firebolt self-managed documentation.

Software for your host OS:

• Docker Engine, with the Docker Compose plugin if you want to use docker compose; if you use curl -fsSL https://get.firebolt.io/ | bash the Docker engine will be installed automatically.
• cURL or any other HTTP client in order to send SQL queries to Firebolt Core.

Software for your Docker host:

• Linux kernel version >= 6.1. Firebolt Core internally uses the io_uring kernel API for fast network and disk I/O, and some required features of this API have only been released in Linux 6.1.

Resources for each node (either a local machine or a VM instance):

• An amd64 CPU supporting at least SSE 4.2, or an arm64 CPU All published Firebolt Core Docker images are multi-arch images suitable for both amd64 and arm64.
• At least 16 GB of RAM are recommended in order to run basic queries.
• At least 25 GB of SSD space are recommended in order to run basic queries.
• At least 10 GBit/s of inter-node network bandwidth is recommended for multi-node deployments.
• TCP port 3473 open when using a single node.
• TCP ports 3473, 1717, 3434, 5678, 6500, 8122, 16000 open when using multiple nodes.

You can submit queries to a Firebolt Core cluster using any HTTP client (like cURL), the official Firebolt CLI as a standalone binary, or by invoking the CLI from within the Core Docker container.

# Use the firebolt CLI available within the container
docker exec -ti firebolt-core firebolt -c "SELECT 42;"

# Run the standalone fb CLI
fb --core "SELECT 42;"

# Use cURL 
curl -s "http://localhost:3473/?output_format=psql" --data-binary "SELECT 42";

See also:

• Example Queries

1. Download the Firebolt Core UI Docker image:

docker pull ghcr.io/firebolt-db/firebolt-core-ui

2. Run the UI container, linking it to your Firebolt Core instance:

FIREBOLT_CORE_URL="http://localhost:3473"
if [ "$(uname)" = "Darwin" ]; then
  FIREBOLT_CORE_URL="http://host.docker.internal:3473"
fi
docker run --rm --name firebolt-core-ui -p 9100:9100 -e FIREBOLT_CORE_URL=$FIREBOLT_CORE_URL ghcr.io/firebolt-db/firebolt-core-ui

3. Open your browser and navigate to http://localhost:9100 to access the UI.

• For assistance, join the GitHub Discussions.

See LICENSE and NOTICE.