Cinched is a cryptographic service designed to help facilitate the development of applications which need to implement encryption of data at rest.
Why use a microservice for encryption?
- It is simpler to audit the correctness of the encryption implementation if it is centralized.
- A developer friendly API
- Backend storage agnostic
- Centralized secret key management
- Cinched is a clustered service with no single point of failure
- Cinched uses a highly available storage backend with automatic data replication.
- The client API uses TLS encryption with both client and server certificates.
- Intra-cluster traffic is encrypted using TLS
- Data is encrypted via randomly generated keys using strong encryption algorithms.
- Only the service can decrypt the data when provided with the data key and encrypted payload.
- All data encryption keys at rest are stored encrypted.
- Only designated user groups (based on SELinux roles) can manage the service.
- Service files (library code, logs, data files) can only be accessed through a single security context, which has a well defined entry point. No users have direct filesystem access to any of the service files or write access to any of the executables.
- The policy defends against attempts to read keys from RAM (denying the ptrace() syscall, preventing the dynamic loading of kernel modules to dump RAM, executable mem/stack/mod/heap).
- Once enabled, the SELinux policy can be made immutable.
- Audit logs are encrypted using rotating encryption keys.
- All encryption operations performed on data and keys use authenticated encryption. Any attempts to alter the encrypted payloads will result decryption failures.
- The storage subsystem used in Cinched can detect and repair silent on-disk corruption via read-repair.
- Audit logs are tamper-resistant and can be verified via a hash chain. Any attempts to alter the logs can be immediately detected.
- In order to ensure consistency in a clustered environment, writes to the storage subsystem employ a multi-paxos consensus algorithm to ensure agreement of cluster state.
- CPU intensive processes are bounded to a configurable number to partially mitigate denial of service conditions.
- Encryption keys stored in the Cinched cluster are replicated to a configurable number of nodes.
- Replication groups are spread across all nodes, and can tolerate the failure of nodes based on a quorum of peer replicas remaining online per replication group.
This list is a first stab at identifying attack vectors which are not in the current scope or implementation. If anyone sees something I've missed, please get in touch with me.
This implementation assumes TLS Erlang distribution using unencrypted x509 private keys. The private keys are protected via SELinux policy.
Should an attacker manage to gain a trusted x509 certificate/key and the configured Erlang cookie (also protected by SELinux policy), he/she would be able to j 9E81 oin a rogue node to the Erlang cluster assuming the attacker can run the attack code from the same subnet.
Once joined to the cluster, subverting all the other controls is relatively easy.
I believe the proper long-term solution to this would be to implement an attestation protocol that would run prior to erlang distribution being setup between nodes. The only way I an see this working in a provably secure way is to leverage TPM hardware and kernel integrity checking hooks.
There is limited defense against DoS attacks in the form of limiting the number of CPU intensive processes. Exceeding configured capacity will lead to clients waiting on available slots. Eventually the service will experience resource exhaustion when all file descriptors used for socket handling are used up.
It is possible for a root user to kill the service process without being a member of the correct SELinux context.
As all operations are logged, it would be possible for an authenticated client to exhaust disk space by crafting bogus requests. This will result in a DoS condition.
I've encountered several issues when attempting to lock memory inside Cinched and have unfortunately had to disable memory locking.
This means that should there be memory pressure, it's possible for data held in RAM by Cinched to be paged to swap, and thus be vulnerable to a variety of attacks.
- Disable swap (swapoff).
- Run the Cinched cluster in it's own subnet.
- Use a firewall between the service consumers and the cluster nodes.
- Enable a password on grub as well as set a BIOS boot password if possible. This will prevent an unattended reboot (which would be required to change SELinux policy).
- Cinched splits up the key namespace into partitions which are mapped to a consistent hash ring. Each partition in the space is then mapped to nodes based on the desired replication factor. These groups of nodes form the consensus/replication group for that partition.
- Data for each partition is stored in a LevelDB database.
- Data is replicated to each node in the consensus group using the multi-paxos consensus algorithm.
- Data is encrypted prior to being stored/replicated.
All operations in Cinched are logged to an audit log. The following items are logged:
- Connection metadata
- Peer IP/Port
- Peer certificate information (CN, Serial)
- Operation
- API endpoint (
/doc,/blob) - Type (
encrypt,decrypt)
- API endpoint (
- Administrative commands
- User performing the action
- Check/View/Rotate logs
- Application provided request metadata
Log entries are made tamper resistant by creating a hash chain. Any attempt to alter log entries can be identified immediately.
Individual log entries are encrypted using a rotating encryption key, ensuring that should the other protections be compromised, log contents are still confidential, and their integrity can be ensured.
Cinched uses both DAC (unix permissions) and MAC (type enforcement via SELinux) to protect the confidentiality, integrity, and availability of the service.
Cinched must be started as the root user in order to be able to setup an appropriate runtime environment. During the service startup, Cinched:
- Sets the system resource limit to ensure sufficient file descriptors are available.
- Disables kernel core dumps
- Resets all filesystem file/folder permissions and ownership to the configured Cinched user
- Drops privileges to the
cincheduser
The following SELinux booleans are set during initial node setup using the cinched setup command:
- ssh_sysadmin_login: on
- deny_ptrace: on
- deny_execmem: on
- secure_mode_insmod: on (optional)
- selinuxuser_execstack: off
- selinuxuser_execmod: off
- selinuxuser_execheap: off
- secure_mode_policyload: on (optional)
Cinched policy is stored in /usr/lib64/cinched/policy.
The policy relies on SELinux reference policy and implements a restrictive security domain. This policy should restrict operators from being able to directly access all data and library files used by the service.
There are three roles:
- Operators (can interact with the service)
- Admins (can label files into the security context)
- Security admins (have full access to the security context).
The policy currently only performs an association for the operators (no admins/secadmins). It's on the TODO to create the additional mappings.
This service uses authenticated symmetrical encryption implemented via libsodium. The alogorithms used are Salsa20 for encryption/decryption, and Poly1305 for message authentication.
All encryption keys used in Cinched are generated using libsodium's randombytes_buf function, which in turn will leverage the Linux kernel's getrandom() syscall. To ensure adequate amounts of entropy are always available, it is recommended that the Haveged daemon be installed on all nodes participating in the cluster.
The following sections describes the various types of keys that are part of Cinched.
This key is generated during the initial cluster setup in response to an operator running the cinched init-cluster command. The key is then split into several shards using Shamir secret sharing and the shards are output to the operator terminal.
By default, 10 shards are created, with a threshold of three shards required to reconstruct the original key (therefore no more than 2 shards should be distributed to each shard custodians to ensure that more than one operator is required for node start-up).
This key is held in memory only during the initial cluster setup while encrypting the service key and node startup phases, to decrypt the service key. It is never persisted to disk.
Rotating this key is possible using the cinched rotate-operator-key command.
This key is generated during the initial cluster setup in response to an operator running the cinched init-cluster command.
It is encrypted with the OK and persisted to the storage subsystem.
During cluster start-up, the encrypted key is loaded and decrypted with the OK. Once loaded, it is held in memory and used to encrypt and decrypt Master Keys.
A new Master Key is automatically generated once per day on-demand during Data Key generation. It is encrypted with the SK and persisted in the storage system.
Master Keys are used to encrypt the Data Keys generated during the course of the day they are created, and decrypt the Data Keys thereafter.
Master Keys are associated with an integer number representing the number of days since the begining of the Unix Epoch (Jan 1, 1970).
Data keys are generated on demand either through the key API (/key/data-key) or when encryption API endpoints are called without the x-cinched-data-key HTTP header provided.
The keys are encrypted with the MK for the date at which they are generated and returned to calling clients who are responsible for storing them.
It is recommended that the Data Keys are stored separately from the data they are protecting, for example in a different database server with different access credentials.
To make key management easier for calling applications, a crypto-period identifier is also returned to calling applications. This identifier can be stored and used to quickly find keys which have aged beyond application requirements.
Data keys are not persisted in Cinched.
The network transport uses TLSv1.2 and requires client and server x509 certificates.
The list of allowed ciphers, in order of preference:
- ECDHE-ECDSA-AES256-SHA384
- ECDHE-RSA-AES256-SHA384
- ECDH-ECDSA-AES256-SHA384
- ECDH-RSA-AES256-SHA384
- DHE-RSA-AES256-SHA256
- DHE-DSS-AES256-SHA256
- ECDHE-ECDSA-AES128-SHA256
- ECDHE-RSA-AES128-SHA256
- ECDH-ECDSA-AES128-SHA256
- ECDH-RSA-AES128-SHA256
- DHE-RSA-AES128-SHA256
- DHE-DSS-AES128-SHA256
Some important things to keep in mind:
- Server and client certificates must be issued from the same certificate authority.
- If using DNS for load balancing, make sure you setup Subject Alternative Names when creating your x509 certificates that include each node in your cluster, otherwise some client libraries might get confused and refuse to connect. An alternative is to use a wildcard for the cluster (eg: *.cinched.mydomain.com)
- Cinched will validate certificates for both client calls as well as when establishing intra-cluster communication.
- Checks CRLs only for erlang distribution
- Checks only OCSP for client calls, and therefore requires a valid OCSP URL in the Authority Information Access field of the x509 certificates. OCSP check results are cached for a configurable amount of time (defaults to 60s).
A single node cluster is fine for testing and prototyping, however Cinched is designed to work in a cluster of servers. I strongly recommend you setup a minimum of five nodes, with a replication factor of 3.
At this time Cinched does not support dynamic cluster resizing or dynamic node membership.
It is possible to resize the cluster, however this cannot be done online.
All encryption/decryption operations in Cinched are performed in memory. Thus, you need to account for that when provisioning machines. A napkin calculation: Average size of documents * number of concurrent requests being handled * 2.
As encryption/decryption is CPU intensive, provision as many cores as you can, Cinched can use them all.
I have not performed much in the way of performance testing or tuning at this point. It's on the TODO list.
Note: Cinched supports sub-command autocompletion. To trigger it, double-TAB:
[root@dev1 cinched]# cinched s<TAB-TAB>
send-key-shard setup start status stop
See instructions here for setting up the repository for libocsp and cinched.
Cinched requires libsodium, which can be installed from the EPEL repo:
yum -y install epel-release
Next we install the Cinched package, which will pull it's dependancies (libsodium, haveged, libocsp):
yum -y install cinched
Repeat this process for each node that will be part of the cluster before proceeding.
One of the built-in protections offered during the setup wizard is to lock down dynamic loading of kernel modules.
It is recommended that this be enabled, however note that most users are running kernels that load modules during system initialization or at runtime for things like NIC drivers and so on.
Enabling the kernel module lock-down from the wizard will break your system unless you've built a monolithic kernel that includes all the modules you need.
The down-side of not locking down kernel module loading is that it's trivial to write a kernel module that can dump the contents of memory (thus allowing an attacker to get a copy of the encryption keys from RAM).
So if you're comfortable with compiling a custom kernel, stop here. Recompile the kernel on your nodes with all the modules you need built-in, then come back here.
If you don't know how to do this, select 'N' when prompted to lock down kernel module loading.
I've include crash-course like instructions at the end of this document for those who want to experiment.
Once the software is installed, it must be configured.
Given the sensitive nature of this service, it's configuration cannot be changed once it has been set without bringing down the cluster. Furthermore, some of the configuration parameters cannot be changed after initial configuration without re-initializing the state, which would result in losing all stored keys.
The list of possible configuration parameters is below. Please ensure you have documented all values you wish to use prior to continuing with the setup process.
| Item | Description | Default value |
|---|---|---|
| Certificate | This is the path to a PEM encoded x509 certificate that will be used for TLS encryption between cluster node peers as well as consumers of the API. You'll need a copy of this certificate on each server in the cluster. The setup wizard will prompt for filesystem location of the certificate and copy it. | none |
| Certificate private key | This is the path to the PEM encoded x509 certificate private key. You'll need a copy of this key on each server in the cluster. The setup wizard will prompt for the filesystem location of the certificate private key and copy it. Note: The private key should not be encrypted. After the private key is copied by the wizard it should be removed from the nodes. | none |
| CA certificate | This is the path to the PEM encoded CA certificate that issued the service certificate. | none |
| Erlang cookie | A unique string configured in an Erlang cluster. All nodes in the cluster must use the same value. The setup wizard will offer to generate a random cookie value, or will prompt to input a value | none |
| Key cache TTL | The number of seconds a master key is cached in memory | 86400 |
| Key cache size | The size in bytes of the master key cache | 1048576 |
| Crypto worker processes | The number of parallel processes each server will spawn to process cryptographic operations | 50 |
| Partitions | The number of data partitions, or ring size. (should be 16-1024) | 64 |
| Key shards | The number of shards to generate when creating the operator key | 10 |
| Key shard recovery | The number of key shards required to reconstruct the operator key | 3 |
| OCSP TTL | The OCSP cache time-to-live in seconds | 120 |
| OCSP cache size | The size of the OCSP cache in bytes | 1048576 |
| Replication factor | The number of replicas to store for each master key | 3 |
| IP address | The IP address that will be used to bind the service to | 192.168.101.1 |
| REST API port | The port to use for the REST API | 55443 |
| Backup port | The port to use for Cinched backup traffic | 55442 |
| Cluster node list | A comma separated list of hosts in this cluster | 192.168.101.1,192.168.101.2,192.168.101.5 |
Configuration values should be identical on all nodes (except for the IP address).
After you've noted the desired configuration settings as well as copied the x509 certificates to all the nodes, it's time to run the setup wizard.
The wizard will generate the configuration file as well as setup SELinux policy. Before running it, you must ensure that you are running in permissive SELinux mode.
< F438 div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[root@dev2 ~]# setenforce 0">[root@dev2 ~]# setenforce 0