This is a working-in-progress Elixir implementation of Tango1 for educational purposes. This library provides the abstraction of a replicated, in-memory data structure (such as a map or a tree) backed by a shared log2 (which, for simplicity, is implemented as an in-memory log that is persisted to the local file system).
Full API documentation is available at jonaprieto.github.io/tango.
To generate documentation locally:
# Using the Makefile
make docs
# Or with the interactive preview script (starts a local web server)
./docs_preview.sh- Elixir 1.18.3 or higher
- Erlang/OTP 25.3.2 or higher
To check your Elixir version:
elixir --versionIf you need to install or update Elixir, visit the official installation guide.
You can add Tango as a dependency in your mix.exs file:
def deps do
[
{:tango, "~> 0.1.0"}
]
endFor experimentation:
- Clone the repository:
git clone https://github.com/jonaprieto/tango.git
cd tango- Start the interactive Elixir shell:
iex -S mix- Start Tango:
{:ok, _pid} = Tango.start()You'll see initialization logs like:
[info] Starting Tango system
[info] Initializing Tango supervision tree
[info] Using log implementation: standard
[info] Initializing Tango sequencer
[info] Tango runtime subscribed to Tango.Log
[info] Tango supervisor started successfully
[info] Tango system started successfully with log implementation: standard
Logging Configuration
By default, Tango is configured to show only warning and error messages to keep the output clean. The logging levels from most to least verbose are:
:debug(most verbose):info:warning(default):error(least verbose)
To change the logging level temporarily in your IEx session:
Logger.configure(level: :debug) # Show all messages including debug
Logger.configure(level: :info) # Show info, warnings, and errors
Logger.configure(level: :error) # Show only errorsThe logging level is environment-specific:
- Development:
:infolevel (shows info, warnings, and errors) - Test:
:errorlevel (shows only errors) - Production:
:warninglevel (shows warnings and errors)
Let's explore the core features of Tango through examples. First, let's add some helpful aliases:
alias Tango.Objects.Counter
alias Tango.Objects.Map# Create a counter
Tango.create_counter(name: JonaCounter)
# Basic operations
Counter.get(JonaCounter) # => 0
Counter.increment(JonaCounter) # => 1
Counter.increment(JonaCounter, 5) # => 6 # Increment by 5
Counter.decrement(JonaCounter) # => 5
Counter.decrement(JonaCounter, 3) # => 2 # Decrement by 3# Create a map
Tango.create_map(name: JonaMap)
# Basic operations
Map.put(JonaMap, :key1, "value1") # => :ok
Map.put(JonaMap, :key2, "value2") # => :ok
Map.get(JonaMap, :key1) # => "value1"
Map.get_all(JonaMap) # => %{key1: "value1", key2: "value2"}
Map.delete(JonaMap, :key1) # => :okExecute operations atomically across multiple objects:
Tango.transaction(fn ->
Counter.increment(JonaCounter)
Map.put(JonaMap, :counter_update, "incremented")
{:ok, :success}
end)
# => {:ok, {:ok, :success}}For read-only operations:
Tango.read_transaction(fn ->
counter_value = Counter.get(JonaCounter)
Map.get_all(JonaMap)
counter_value
end)View log entries and system state:
# View all log entries
Tango.get_log_entries()
# View entries for specific objects
Tango.get_object_log_entries(JonaMap)
# Get object info
Tango.get_object_info(JonaMap)
# Get system status
Tango.get_system_status()Streams provide a way to subscribe to object changes:
alias Tango.Stream
# Create a stream
{:ok, stream} = Tango.create_stream(stream_id: JonaStream)
# Basic operations
Stream.read_next(stream)
{:ok, tail_pos} = Stream.sync(stream)
entries = Stream.get_entries(stream)
Stream.subscribe(stream, self())This guide shows how to quickly test Tango in a distributed environment.
Follow these steps on two terminal windows to try Tango in a distributed setup. You'll need to execute specific commands on both a server and client node.
-
Start the Elixir nodes with distribution enabled
- Server Node:
iex --name server@127.0.0.1 --cookie tango -S mix
- Client Node:
iex --name client@127.0.0.1 --cookie tango -S mix
- Server Node:
-
Start Tango on both nodes with the same log configuration
- On both nodes:
Tango.start(log_storage: :file, log_path: "shared_log")
- Expected output:
%Tango.OperationResult{ reason: nil, value: #PID<0.195.0>, status: :ok }
- On both nodes:
-
Set up aliases for convenience
- On both nodes:
alias Tango.Objects.Counter
- On both nodes:
-
Connect the nodes
- Client Node:
Node.connect(:"server@127.0.0.1")
- Server Node (verify connection):
Node.list()
- Expected output on server:
["client@127.0.0.1"]
- Client Node:
-
Check initial system status
- Client Node:
Tango.get_system_status
- Expected output:
%Tango.SystemStatus{ system_running: true, log_impl: Tango.Log, runtime_info: %{active_transaction: false}, log_metrics: %{ tail_position: 0, total_entries: 0 }, active_objects: 0 }
- Client Node:
-
Create a shared counter on the server
- Server Node:
Tango.create_counter(name: {:global, :shared_counter})
- Expected output:
%Tango.OperationResult{ reason: nil, value: #PID<0.207.0>, status: :ok }
- Server Node:
-
Increment the counter from client
- Client Node:
Counter.increment({:global, :shared_counter}, 10)
- Client Node:
-
Verify the changes on the server
- Server Node:
Tango.get_system_status
- Expected output:
%Tango.SystemStatus{ system_running: true, log_impl: Tango.Log, runtime_info: %{active_transaction: false}, log_metrics: %{ tail_position: 1, total_entries: 1 }, active_objects: 1 }
- Server Node:
-
View log entries on server
- Server Node:
Tango.get_log_entries
- Expected output:
[ %Tango.StreamEntry{ metadata: %{token: 1}, timestamp: 1747710171986690, streams: nil, tx_id: nil, payload: {:inc, 10}, stream: {:global, :shared_counter}, pos: 1 } ]
- Server Node:
-
Get counter value on server
- Server Node:
Counter.get({:global, :shared_counter})
- Expected output:
10
- Server Node:
Key Points
- Always use
{:global, name}syntax for distributed objects- All nodes must share the same log configuration
- Changes on one node are immediately visible on all nodes
Advanced Log Configuration Options
For production or advanced testing, Tango supports multiple log configuration options:
Enhanced Log Implementation:
Tango.start(
log_impl: :enhanced, # Includes backpointers for efficient stream traversal
log_storage: :file,
log_path: "shared_log"
)SQLite Storage Option:
Tango.start(
log_storage: :external,
log_config: %{
adapter: :sqlite,
database: "tango_sqlite.db"
}
)PostgreSQL Storage Option:
Tango.start(
log_storage: :external,
log_config: %{
adapter: :postgres,
hostname: "db.example.com",
username: "tango",
password: "password",
database: "tango_log"
}
)For production deployments, consider using libraries like libcluster to automate node discovery and connections.
To inspect and debug the in-memory log while using Tango:
# In an IEx session, first import the debug utilities
import_file("debug_log.exs")
# View current log contents in the console
TD.dump_log()
# Save log to a file
TD.dump_log(output: "tango_log.txt")
# Watch log entries as they occur
TD.watch_log()This implementation currently provides the following core Tango functionalities:
-
Shared Log Abstraction: Objects are built on a shared, append-only log.
- Standard log implementation.
- Enhanced log implementation with backpointers (need testing).
-
Log Persistence Options for the demo: Recall the Tango paper assumed an external distributed shared log, so here, for demonstration purposes, the shared log is implemented as an in-memory log that is persisted to the local file system.
- In-memory storage.
- File-based storage (requires testing).
- External storage (SQLite, but requires testing).
-
Core Object Types:
- Replicated Counter (increment, decrement, etc.).
- Replicated Map (put, get, delete, etc.).
-
Transactions:
- ACID-like transactions for operations on single or multiple objects.
- Optimistic concurrency control with conflict detection.
- Read-only transactions.
-
Object State Management:
- Object-level check pointing for faster state recovery (object-specific).
- Log trimming to manage log size.
-
Streams API:
- Objects expose their updates as streams.
- Ability to read and subscribe to object streams.
-
Basic Distributed Capabilities:
- Nodes can connect and share Tango objects.
- Requires all nodes to point to the same external log instance (e.g., shared file or database).
-
Sequencer Implementation:
- Centralized sequencer for global ordering of log entries.
- Per-stream token tracking for efficient stream traversal.
- Automatic recovery from sequencer failures (future work).
-
Debugging Utilities: Tools for inspecting log contents.
-
Dashboard: A dashboard for monitoring Tango objects and streams.
You might want to check the notes directory for more information about the implementation status and design of Tango. Specially, the Tango requirements and design document is here.
There are still many features to be added to achieve a truly fault-tolerant and robust system as envisioned by the original Tango and Corfu papers. However, notice that the Tango paper assumed and abstracted away the details of the shared log implementation, so the following list is not exhaustive.
- Log Replication & Redundancy:
- Implement multi-node replication for the shared log itself (e.g., using Raft or Paxos).
- Introduce a quorum-based approach for log reads and writes to ensure consistency across replicas.
- Add mechanisms to handle log replica failures and redirect operations to healthy nodes.
- Data Integrity & Corruption Detection:
- Incorporate checksums or hashes for each log entry.
- Implement verification of these checksums upon reading entries.
- Develop strategies for handling corrupted entries (e.g., repair from other replicas if log replication is in place).
- Advanced Automatic Recovery & Reconfiguration:
- Implement dynamic reconfiguration of the sequencer set if a node is permanently lost.
- Design automatic recovery for log replicas (e.g., re-replicating data from healthy nodes).
- Comprehensive Fault Injection Testing:
- Create tests that simulate abrupt node crashes (not just clean restarts).
- Introduce tests for I/O errors during log persistence.
- Develop tests that simulate data corruption in the log store.
Footnotes
-
Balakrishnan, M., Malkhi, D., Wobber, T., Wu, M., Prabhakaran, V., Wei, M., Davis, J. D., Rao, S., Zou, T., & Zuck, A. (2013). "Tango: Distributed Data Structures over a Shared Log." In Proceedings of the 24th ACM Symposium on Operating Systems Principles (SOSP '13). ACM. ↩
-
Balakrishnan, M., Malkhi, D., Prabhakaran, V., Wobber, T., Wei, M., & Davis, J. D. (2012). "CORFU: A Shared Log Design for Flash Clusters." In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI '12). USENIX Association. ↩