This project adds a basic high availability and consistent hash layer to InfluxDB v2.
NOTE: influx-proxy must be built with Go 1.21+ with Go module support, don't implement udp.
NOTE: InfluxDB Cluster - open source alternative to InfluxDB Enterprise has been released, which is better than InfluxDB Proxy.
We used InfluxDB Relay before, but it doesn't support some demands. We use grafana for visualizing time series data, so we need add datasource for grafana. We need change the datasource config when influxdb is down. We need transfer data across idc, but Relay doesn't support gzip. It's inconvenient to analyse data with connecting different influxdb. Therefore, we made InfluxDB Proxy. More details please visit https://github.com/shell909090/influx-proxy.
Forked from the above InfluxDB Proxy, after many improvements and optimizations, InfluxDB Proxy v1 has released, which no longer depends on python and redis, and supports more features.
Since the InfluxDB Proxy v1 is limited by the only ONE database and the KEYMAPS configuration, we refactored InfluxDB Proxy v2 with high availability and consistent hash, which supports multiple databases and tools to rebalance, recovery, resync and cleanup.
InfluxDB Proxy v3 is aimed at InfluxDB v2.
- Support query and write.
- Support /api/v2 endpoints.
- Support flux language query.
- Support some cluster influxql.
- Filter some dangerous influxql.
- Transparent for client, like cluster for client.
- Cache data to file when write failed, then rewrite.
- Support data sharding with consistent hash.
- Support custom hash key and shard key of database sharding.
- Load config file and no longer depend on python and redis.
- Support influxdb-java, and influxdb shell.
- Support prometheus monitor with /metrics.
- Support authentication and https.
- Support health status check.
- Support version display.
- Support gzip.
- Golang >= 1.21 with Go module support
- InfluxDB >= 2.0 (For InfluxDB 1.x, please visit branch master)
Download docker-compose.yml, proxy.json and setup.sh from docker/quick
$ docker-compose up -d --scale influx-proxy=0
$ bash setup.sh
$ docker-compose up -dAn influx-proxy container (port: 7076) and 4 influxdb containers will start.
$ git clone https://github.com/chengshiwen/influx-proxy.git
$ cd influx-proxy
$ make
$ ./bin/influx-proxy -config proxy.json$ ./bin/influx-proxy -h
Usage of ./bin/influx-proxy:
-config string
proxy config file with json/yaml/toml format (default "proxy.json")
-version
proxy version$ # build current platform
$ make build
$ # build linux amd64
$ make linux
$ # cross-build all platforms
$ make releaseBefore developing, you need to install and run Docker
$ ./script/setup.sh # start 4 influxdb instances by docker
$ make run
$ ./script/write.sh # write data
$ ./script/query.sh # query data
$ ./script/remove.sh # remove 4 influxdb instancesThe architecture is fairly simple, one InfluxDB Proxy instance and two consistent hash circles with two InfluxDB instances respectively. The Proxy should point HTTP requests with organization, bucket and measurement to the two circles and the four InfluxDB servers.
The setup should look like this:
┌──────────────────┐
│ writes & queries │
└──────────────────┘
│
▼
┌──────────────────┐
│ │
│ InfluxDB Proxy │
│ (only http) │
│ │
└──────────────────┘
│
▼
┌──────────────────┐
│ org,bucket │
│ & measurement │
│ consistent hash │
└──────────────────┘
| |
┌─┼──────────────┘
│ └────────────────┐
▼ ▼
Circle 1 Circle 2
┌────────────┐ ┌────────────┐
│ │ │ │
│ InfluxDB 1 │ │ InfluxDB 3 │
│ InfluxDB 2 │ │ InfluxDB 4 │
│ │ │ │
└────────────┘ └────────────┘
The configuration file supports format json, yaml and toml, such as proxy.json, proxy.yaml and proxy.toml.
The configuration settings are as follows:
circles: circle listname: circle name,requiredbackends: backend list belong to the circle,requiredname: backend name,requiredurl: influxdb addr or other http backend which supports influxdb line protocol,requiredtoken: influxdb token,requiredwrite_only: whether to write only on the influxdb, default isfalse
dbrp: dbrp mapping config for 1.x compatibilityseparator: the separator of the key-value pair mapping, for 1.x compatibility, default is/mapping: the key-value pair mapping fromdb/rptoorg/bucket, for 1.x compatibility, default isnil
listen_addr: proxy listen addr, default is:7076data_dir: data dir to save .dat .rec, default isdatahash_key: backend key template for consistent hash, which containing%idx, likebackend-%idx, default is%idx, once changed rebalance operation is necessaryshard_key: data shard key template for hash, which containing%org,%bkor%mm, likeshard-%org-%bk-%mm, default is%org,%bk,%mmwhich meansorg,bucket,measurement, once changed rebalance operation is necessaryflush_size: default is10000, wait 10000 points writeflush_time: default is1, wait 1 second write whether point count has bigger than flush_size configcheck_interval: default is1, check backend active every 1 secondrewrite_interval: default is10, rewrite every 10 secondsrewrite_threads: default is5, rewrite under 5 threadsconn_pool_size: default is20, create a connection pool which size is 20write_timeout: default is10, write timeout until 10 secondsidle_timeout: default is10, keep-alives wait time until 10 secondstoken: proxy token, default isemptywhich means no authping_auth_enabled: enable authentication on/pingand/metrics, default isfalsewrite_tracing: enable logging for the write, default isfalsequery_tracing: enable logging for the query, default isfalsepprof_enabled: enable/debug/pprofHTTP endpoint, default isfalsehttps_enabled: enable https, default isfalsehttps_cert: the ssl certificate to use when https is enabled, default isemptyhttps_key: use a separate private key location, default isemptytls: configuration settings for tlsciphers: set of cipher suite IDs to negotiate when https is enabled, referring to ciphersMap, default is[]min_version: minimum version of the tls protocol when https is enabled, includingtls1.0,tls1.1,tls1.2andtls1.3, default isemptymax_version: maximum version of the tls protocol when https is enabled, includingtls1.0,tls1.1,tls1.2andtls1.3, default isempty
hash_key and shard_key together control which influxdb instance the data should be written to.
hash_key is backend key template for consistent hash, which containing %idx (%idx is the index of the influxdb backend in the circle), like backend-%idx.
shard_key is data shard key template for hash, which containing %org, %bk or %mm (%org, %bk and %mm are the org, bucket and measurement in the data respectively), like shard-%org-%bk-%mm.
To avoid data skew (i.e. uneven data distribution), both hash_key and shard_key need to be set appropriately. Before setting, influx-tool hashdist can help simulate and test the hash distribution. For example, execute
$ head -n 3 table.csv
org1,bucket1,cpu1
org1,bucket2,cpu2
org2,bucket3,cpu3
$ influx-tool hashdist -v v2 -n 6 -k backend-%idx -K shard-%org-%bk-%mm -f table.csv -D -
node total: 6, hash key: backend-%idx, shard key: shard-%org-%bk-%mm, total hits: 40
node index: 0, hits: 5, percent: 12.5%, expect: 16.7%
node index: 1, hits: 5, percent: 12.5%, expect: 16.7%
node index: 2, hits: 9, percent: 22.5%, expect: 16.7%
node index: 3, hits: 7, percent: 17.5%, expect: 16.7%
node index: 4, hits: 7, percent: 17.5%, expect: 16.7%
node index: 5, hits: 7, percent: 17.5%, expect: 16.7%NOTE: Once one of hash_key and shard_key is changed, rebalance operation is necessary.
- /api/v2/write v2 supported
- /write v1 compatibility supported
- /api/v2/query v2 supported
- /query v1 compatibility supported
Note: _measurement must be specified
from(bucket: "example-bucket")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "example-measurement" and r.tag == "example-tag")
|> filter(fn: (r) => r._field == "example-field")
Note: dbrp mapping must be specified like
"dbrp": {
"separator": "/",
"mapping": {
"mydb/myrp": "myorg/mybucket"
}
}
Only support match the following commands, more details please see InfluxQL support.
select from(read-only)show databasesshow seriesshow measurementsshow tag keysshow tag valuesshow field keysdelete fromdrop measurementon clausefrom clauselikefrom <db>.<rp>.<measurement>
There are three tools for benchmarking InfluxDB, which can also be applied to InfluxDB Proxy:
- influx-stress is a stress tool for generating artificial load on InfluxDB.
- influxdb-comparisons contains code for benchmarking InfluxDB against other databases and time series solutions.
- tsbs (Time Series Benchmark Suite) is a tool for comparing and evaluating databases for time series data.
There is a tool for InfluxDB and InfluxDB Proxy:
- influx-tool: high performance tool to rebalance, recovery, resync, cleanup and compact. most commands do not require InfluxDB to start
MIT.