8000 GitHub - firebolt-db/firebolt-core: Firebolt Core is a free, self-hosted edition of Firebolt's distributed query engine (https://www.firebolt.io/); it provides high-performance data warehousing capabilities that can be deployed anywhere from a single laptop to enterprise datacenters.
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Firebolt Core is a free, self-hosted edition of Firebolt's distributed query engine (https://www.firebolt.io/); it provides high-performance data warehousing capabilities that can be deployed anywhere from a single laptop to enterprise datacenters.

License

Notifications You must be signed in to change notification settings

firebolt-db/firebolt-core

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

19 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Firebolt Core logo

Firebolt Core is a free, self-hosted edition of Firebolt's high-performance distributed query engine, designed to power the data infrastructure behind today's most demanding applications.

Discord chat Firebolt Core documentation Deployment and Operational Guide Release GitHub Issues GitHub Stars

Deploy anywhere, from a single laptop to your own datacenter

Key Features

  • πŸš€ Powerful. Firebolt Core ships with all key performance and usability features of Firebolt's managed Cloud data warehouse, including a state-of-the-art query optimizer, distributed query execution engine, Iceberg support, and many more.
  • πŸ†“ Free to use. Firebolt Core is free to use, forever (see the LICENSE for details).
  • πŸ“ˆ No usage limits. Firebolt Core has no usage limits. Process unlimited data, scale to as many nodes as you need, and run as many queries as you like.
  • πŸ›’οΈ Postgres compliant. Firebolt's SQL dialect is Postgres compliant. We offer powerful extensions for analytical workloads, such as lambda functions for array processing. For a complete reference, see the SQL reference documentation.
  • πŸ› οΈ Self-contained. Firebolt Core comes packaged as a single Docker image (ghcr.io/firebolt-db/firebolt-core:preview-rc) that contains everything needed to run it.
  • 🏠 Self-hosted. You can deploy Firebolt Core anywhere you want, from your personal workstation to large on-premise clusters or VPCs.
  • πŸ“Š First-class support with documentation, updates, and active community support via GitHub Discussions. We encourage you to join the conversation!
  • πŸ€– AI-ready architecture optimized for modern data and ML applications.
  • 🎯 Designed for demanding applications. Powering real-time analytics, embedded analytics, and large-scale data processing workloads.
  • πŸ”„ Workload compatibility. Many workloads run interchangeably with managed Firebolt.

Get Started

Start Core on your machine with:

bash <(curl -s https://get-core.firebolt.io/)

If you want to work with Docker directly, you can also run:

docker run -i --rm \
        --ulimit memlock=8589934592:8589934592 \
        --security-opt seccomp=unconfined \
        -p 127.0.0.1:3473:3473 \
        -v ./firebolt-core-data:/firebolt-core/volume \
        ghcr.io/firebolt-db/firebolt-core:preview-rc

Caution

This will create a local firebolt-core-data directory, owned by root, where data, metadata, logs and diagnostic information are persisted.

You can also start a single node cluster by cloning this repository and then run the following command within the repository root directory:

docker compose up

See also:

Multi-Node via Docker Compose

Use this setup if you want to leverage the computing power of multiple hosts.

  1. Add in config.json one entry (either hostname or IP address) for each of the nodes; node 0 is the first node in the array:

    {
        "nodes": [
            {
                "host": "ip-or-host-of-node-0"
            },
            {
                "host": "ip-or-host-of-node-1"
            },
            {
                "host": "ip-or-host-of-node-2"
            }
        ]
    }
  2. Make sure that this repository and your config.json are present on each of the nodes.

  3. Activate node 0:

    docker compose up
  4. Activate all further nodes by running on each host:

    NODE=1 docker compose -f compose.yaml -f compose.nodeN.yaml up

    Increase NODE for each further node.

See also:

Multi-Node via Kubernetes

You can deploy Firebolt Core on Kubernetes (v1.19+) by following these steps:

  1. Create a dedicated namespace:
kubectl create namespace firebolt-core
  1. Customize the values for the chart (see helm/README.md), for example by setting nodesCount to 3.
  2. Install the Helm chart in such namespace:
helm install helm/ --generate-name --namespace firebolt-core
  1. Verify that pods are running:
kubectl get pods

Expected output:

NAME                              READY   STATUS    RESTARTS   AGE
helm-1748880880-firebolt-core-0   0/1     Running   0          5m32s
helm-1748880880-firebolt-core-1   0/1     Running   0          5m32s
helm-1748880880-firebolt-core-2   0/1     Running   0          5m32s

See also:

Requirements

Software for your host OS:

Software for your Docker host:

  • Linux kernel version >= 6.1. Firebolt Core internally uses the io_uring kernel API for fast network and disk I/O, and some required features of this API have only been released in Linux 6.1.

Resources for each node (either a local machine or a VM instance):

  • An amd64 CPU supporting at least SSE 4.2, or an arm64 CPU All published Firebolt Core Docker images are multi-arch images suitable for both amd64 and arm64.
  • At least 16 GB of RAM are recommended in order to run basic queries.
  • At least 25 GB of SSD space are recommended in order to run basic queries.
  • At least 10 GBit/s of inter-node network bandwidth is recommended for multi-node deployments.
  • TCP port 3473 open when using a single node.
  • TCP ports 3473, 1717, 3434, 5678, 6500, 8122, 16000 open when using multiple nodes.

Note

There is no universally correct amount of RAM and disk space for running Firebolt Core, and the above are simply rough guidelines for running some simple queries as a way to get started. The ideal amount of RAM and disk space depends heavily on the specific workload that you are running against a Firebolt Core deployment (see Deployment and Operational Guide for details).

Run Queries on Firebolt Core

You can submit queries to a Firebolt Core cluster using any HTTP client (like cURL), the official Firebolt CLI as a standalone binary, or by invoking the CLI from within the Core Docker container.

Suggested change
# Use the fbcli script available within the container
docker exec -ti firebolt-core fbcli "SELECT 42;"
# Run the standalone fb CLI
fb --core "SELECT 42;"
# Use cURL 
curl -s "http://localhost:3473/?output_format=psql" --data-binary "SELECT 42";

See also:

Troubleshooting & Support

Detailed information about Firebolt Core is available in the documentation.

  • Encountering issues? Check the Troubleshooting Guide for common problems and solutions and the FAQs
  • For further assistance, join the GitHub Discussions.
  • For best practices on securing your deployment and information on backing up, consult the Operational Guide.
  • Curious about what's next for Firebolt Core? Check out the Roadmap to see planned features and improvements.

Verifying Firebolt Core Docker Images

All Firebolt Core published images are signed using cosign.

To verify:

  1. Download the public key: cosign.pub
  2. Run:
cosign verify --key cosign.pub ghcr.io/firebolt-db/firebolt-core:preview-rc

You should see output confirming the signature is valid:

Verification for ghcr.io/firebolt-db/firebolt-core:preview-rc --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
[...]

License

See LICENSE and NOTICE.

About

Firebolt Core is a free, self-hosted edition of Firebolt's distributed query engine (https://www.firebolt.io/); it provides high-performance data warehousing capabilities that can be deployed anywhere from a single laptop to enterprise datacenters.

Topics

Resources

License

Security policy

Stars

Watchers

Forks

Packages

 
 
 

Contributors 6

0