This is the implementation repository of Ditto: An Adaptive Memory-Disaggregated Caching System. This artifact provides the source code of Ditto and scripts to reproduce experiment results in our paper.
- Hardware Requirements: Mellanox ConnectX-6 NIC
- Software Requirements: Ubuntu 18.04, Mellanox OFED 4.9-5.1.0.0, libmemcached-dev, cmake 3.16.8, libgtest-dev, memcached, python-memcached, redis, libhiredis, redis++
We strongly recommend you execute Ditto using r650 nodes of CloudLab with Ubuntu 18.04, since the code has been thoroughly tested there. We haven't done any tests on any other hardware environment.
- Log into your CloudLab account. You will need to apply for an account if you don't have it.
- After logging into the CloudLab console, create an experiment profile with 10 r650 nodes with the following steps:
- Click
Experiments |--> Create Experiment Profile. - Click
Uploadto upload the provided profile. - Name your profile and click
Create.
- Click
- Click
Instantiateto create a 10-node cluster with the profile. If there are not 10 r650 nodes available, please submit a reservation request viaExperiments |--> Reserve Nodesand wait for approval. Feel free to contact us if you have trouble reserving nodes. - Try to log into each node and check each node using SSH commands provided in the
List Viewon CloudLab. If you find some nodes have broken shells (which sometimes happens in CloudLab), you can reload them viaList View |--> Reload Selected
After logging into all 10 r650 CloudLab nodes, use the following command to clone this GitHub repository in the home directory of all nodes:
git clone https://github.com/dmemsys/Ditto.git
Our experiments use 10 r650 nodes. We use node-0 of the cluster as a coordinator to start processes on other nodes and collect experiment results, and we use node-1 as our memory node. Different types of nodes need different setup steps. We mark the nodes that the following steps should be executed on at the beginning of each step.
- All nodes: Install required libraries and software on all nodes.
We provide you with a shell script to install all required libraries and software.Warning: This script is written for r650 nodes. It repartitions the disk partition of the node to expand its file system. You will need to disable line 121 to line 141 to avoid partitioning your disk.cd Ditto/scripts source ./setup-env.sh # Takes about 15 minutes. Execute this command in tmux to prevent network interruption.
- All nodes: Expand the file system on all nodes.
We have re-partitioned the disk in the previous step to have more space for our workloads. We need to reboot all machines to make the modified partition take effect.After the nodes reboot, execute the following command:sudo reboot
sudo resize2fs /dev/sda1
- Node-0: Setup Memcached on node-0.
We use Memcached on node-0 to coordinate compute and memory nodes. Modify the following settings of/etc/memcached.confon node-0 in the cluster:Add the following two configurations to the file:-l <IP address of node-0> # In the form of 10.10.1.X
Execute the following command after modifying the configuration file:-I 128m -m 2048
sudo service memcached restart - All nodes: Setup memory node IP and RNIC to conduct experiments on all nodes.
cd Ditto/experiments python modify_config.py memory_ip_list=\[\"<node-1 IP>\"\] python modify_config.py ib_dev_id=3 python modify_config.py conn_type=\"ROCE\" python modify_config.py ib_gid_idx=3
- All nodes: Setup Memcached IP on all nodes.
Setmemcached_ip=<node-0 IP>inDitto/experiments/scripts/shell_settings.sh - Node-1: Setup hugepages on node-1.
Execute the following commands on node-1:Note: The hugepage setting losses every time we reboot the machine, so we need to re-execute this step every time we reboot the machines.sudo su echo 10240 > /proc/sys/vm/nr_hugepages exit
- Your Laptop/PC: Setup SSH keys on all nodes from your own Laptop/PC.
By default, nodes on CloudLab cannot SSH each other. We need to add the public key of node-0 to all other nodes to enable node-0 to access other nodes with SSH. We provide a python script to achieve this. The following steps should be executed on your own machine that has the added public key on CloudLab.- Modify cluster user: Modify the
uservariable inDitto/scripts/setkey.pyto your username of the CloudLab cluster. - Modify cluster setting: Modify the
server_listinDitto/scripts/setkey.pyaccording to theSSH Commandson theList Viewof the CloudLab experiment. - Execute
python setkey.py. (You will need to approve record keys to your local machine when executing the python script.)
Note: The authorized SSH keys losses every time we reboot the machine, so we need to re-execute this step every time we reboot the machines.
- Modify cluster user: Modify the
Use the following commands to download workloads for all machines:
- Node-0: Download workload from Internet:
cd Ditto/experiments/workloads ./download_all.sh - Node-0: Start an HTTP server:
cd Ditto/experiments/workloads python -m http.server - Nodes-1~9: Download workload from node-0:
cd Ditto/experiments/workloads ./download_all_from_peer.sh node-0:8000 - Node-0: Stop the HTTP server after other nodes have downloaded the dataset.
- Node-1: Set hugepage:
sudo su echo 10240 > /proc/sys/vm/nr_hugepages exit - Your Laptop/PC: Set SSH keys on all nodes:
- Modify the
uservariable inDitto/scripts/setkey.pyto your user name of CloudLab. - Modify the
server_listinDitto/scripts/setkeys.pyaccording to theSSH Commandson theList Viewof the CloudLab experiment. - Execute
python setkeys.pyon your own Laptop/PC to finish the setting. (You will need to approve record keys to your local machine when executing the python script.)
- Modify the
- Node-0: Go to the root directory of Ditto (
~/Ditto) and execute the following commands:The kick-the-tires script is used ascd experiments/scripts python kick-the-tires.py 256 ycsbc # The command executes about 70 seconds.
python kick-the-tires.py <num_clients> <ycsb-workload>. It automatically compiles Ditto on all nodes and executes th 6000 e corresponding YCSB workload with the assigned number of client threads. The throughput and latency will be printed on the command line. You can change the command to execute Ditto with different numbers of clients and with other YCSB workloads (ycsba/ycsbb/ycsbc/ycsbd).
We provide code and scripts in Ditto/experiments/scripts to reproduce our experiments. Please refer to experiments/scripts/README.md for more details.
- The absolute hit rates in Figures 18, 20, and 21 are supplemented in experiments/results.
- The supplementary results of the proposed techniques under different workloads are shown in experiments/scripts/more_results/.