Home
WARNING: This is not yet released. It is a work in progress!
Create Kafka style consumer groups in other technologies. Rebalanser was born of the need for consumer groups with RabbitMQ. But Rebalanser is completely technology agnostic and will balance activity over any group of resources across a group of participating nodes.
- Create Kafka-like "consumer groups" with messaging technologies like RabbitMQ, SQS, etc.
- Consume a group of resources such as file shares, FTPs, S3 buckets between the instances of a scaled out application.
- Single Active Consumer / Active-Backup
- Create an application cluster that consumes a single resource in a highly available manner. The cluster leader (Coordinator) consumes the single resource and the slaves (Followers) remain idle in backup in case the leader dies.
Rebalanser consists of (or will consist of when complete):
- a set of protocols for different resource allocation algorithms
- a set of TLA+ specifications that verify the protocols
- a set of code libraries for multiple languages and backends
The important terms are:
- Resource Group = Group of Nodes + Group of Resources
When a node becomes active it notifies the other nodes. One node is the Coordinator (leader) and the rest are Followers. The Coordinator has the job to assign resources to nodes. It monitors the coming and going of nodes, as well as changes to the number of resources available to the resource group. When any change happens to the nodes or resources then the Coordinator triggers a rebalancing.
Different rebalancing algorithms exist:
- Leader based resource barrier
- Leader based global barrier
With an RDBMS we use the global barrier algorithm where:
- The Coordinator orders all Followers to stop activity.
- Once activity has stopped the Coordinator distributes the resource identifiers equally between the Followers and itself.
- The final step is that the Coordinator notifies each Follower of the resources it has been assigned and can start its activity (consuming, reading, writing etc).
Other backends are suited to either. With Apache ZooKeeper we can use the faster Resource Barrier algorithm as is documented in the rebalanser-net-zookeeper readme.
Leader election determines who the Coordinator is. If the Coordinator dies, then a Follower takes its place. Leader election and meta-data storage is performed via a consensus service (ZooKeeper, Etcd, Consul) or an RDBMS with serializable transaction support (SQL Server, Oracle, PostgreSQL). All communication between nodes is also performed via this backing meta-data store.
Rebalanser is a suite of code libraries. It must be implemented in your language in order to use it. Also, different backends will be available.
See this blog post for instructions on how to use Rebalanser in a C# project (when it is finished)
When you use Rebalanser in a .NET project you'll always use Rebalanser.Core. This contains the code you'll interact with in your application. You'll also pull in a backend library:
- C# library for SQL Server (under development) - rebalanser-net-mssql
- C# library for ZooKeeper (under development) - rebalanser-net-zookeeper
- C# lib for Consul (not yet under development)
I will be working on Java, Python and C++ clients at some point.
If anyone is interested in this project, then please contact me on jack.vanlightly@gmail.com