Why only S3? #127
-
|
Hi Dan. This project looks great. I read the blog https://blog.danthegoodman.com/icedb-v2 , and somehow felt that I could relate with every single word written over there. So I am about to try Icedb . But one question that came to mind (as someone who googled for parquet database and stumble upon this 5 minutes before ) is Why only S3? Can we run this locally ? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 2 replies
-
|
You can run this locally using minio and docker quite easy. See https://github.com/danthegoodman1/icedb/blob/main/docker-compose.yml which is what is used for running all the examples and tests. Very simple to just run Note that this repo represents an evolution of https://blog.danthegoodman.com/icedb-v3--third-times-the-charm There's an open issue to add an "on-disk" mode but I've not needed it yet so I haven't spent time on it (although it shouldn't be very hard). As to why: Because the intention was to not need any state outside of usage-based billable infra like S3. To use local disks would suggest you might as well use something like duckdb or clickhouse directly. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Dan for your reply. This is great work here. |
Beta Was this translation helpful? Give feedback.
-
|
how do you connect to s3? Is it all based on duckdb and httpfs? You might consider using fsspec (or pyarrows own filesystem, pyarrow.fs) instead, which would easily allow to use other filesystems/storage backends (including local filesystem). Furthermore, I found, that in many scenarios, duckdb + fsspec is way faster than duckdb + https to run queries on s3. |
Beta Was this translation helpful? Give feedback.
-
|
S3 needs to be configured for both the LogIO clients, and the IceDB client (uses duckdb for writing). You can see how that's done in the examples, and in https://github.com/danthegoodman1/icedb?tab=readme-ov-file#quick-start, but that does use the older method in duck 8000 db. I'm not familiar with fsspec but I'll take a look, do you have references to where it can be faster than duckdb using S3 directly? |
Beta Was this translation helpful? Give feedback.
-
|
You can easily get started with fsspec filesystems from the duckdb docs: https://duckdb.org/docs/guides/python/filesystems.html Here is a simple example for s3. Note, s3fs is based on fsspec. import duckdb
import s3fs
# or:
# from fsspec import filesystem
# e.g. connecting to minio
fs = s3fs.S3FileSystem(access="my_key", secret="my_secret", endpoint_url="https://localhost:9000")
# or:
# fs = filesystem(protocol="s3", access="my_key", secret="my_secret", endpoint_url="https://localhost:9000")
con = duckdb.connect()
con.register_filesystem(fs)
res=con.sql("SELECT * FROM read_parquet('s3://path/to/parquet/files/**/*.parquet', hive_partitioning=True'))Using fsspec has the additional benefit, that you have the full range of filesystem operations (like, copy, move, delete, list,...) available. One further recommendation. I also found, that using [pyarrow dataset]s(https://arrow.apache.org/docs/python/dataset.html) and query them using duckdb, instead of using duckdbs read_parquet was also faster in my personal benchmarks. See here for a good blog post from duckdb: https://duckdb.org/2021/12/03/duck-arrow.html Here is how you can do that import pyarrow.dataset as pds
import s3fs
import duckdb
fs = s3fs.S3FileSystem(access="my_key", secret="my_secret", endpoint_url="https://localhost:9000")
con = duckdb.connect()
con.register_filesystem(fs)
ds = pds.dataset("path/to/parquet/files", partitioning="hive", filesystem=fs)
res = con.sql("SELECT * FROM ds")
# or:
con.register("my_dataset", ds)
res=con.sql("SELECT * FROM my_dataset")And finally, when you track a The above is exactly, what I am doing in pydala2. The reasons for developing this are very similar to yours for building iceDB. However, I do miss a lot of things you are doing with your log and IceDB is therefore a huge inspiration for me. Thanks and keep up your awesome work! legout |
Beta Was this translation helpful? Give feedback.
You can run this locally using minio and docker quite easy. See https://github.com/danthegoodman1/icedb/blob/main/docker-compose.yml which is what is used for running all the examples and tests. Very simple to just run
docker compose up -dand then run tests. Makes it easier to tear it down and reset all state as well withdocker compose down -v.Note that this repo represents an evolution of https://blog.danthegoodman.com/icedb-v3--third-times-the-charm
There's an open issue to add an "on-disk" mode but I've not needed it yet so I haven't spent time on it (although it shouldn't be very hard).
As to why: Because the intention was to not need any state outside of usage-based billable infra…