8000 GitHub - jrapoport/chestnut: 🌰 Chestnut is a powerful encrypted storage library for Go, featuring Sparse Encryption, a novel technique for selectively encrypting struct fields. It supports Chained Encryption, custom encryption (AES256-CTR), multiple storage backends (BBolt, NutsDB), and built-in compression (Zstandard), offering unmatched flexibility for secure data storage.
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

🌰 Chestnut is a powerful encrypted storage library for Go, featuring Sparse Encryption, a novel technique for selectively encrypting struct fields. It supports Chained Encryption, custom encryption (AES256-CTR), multiple storage backends (BBolt, NutsDB), and built-in compression (Zstandard), offering unmatched flexibility for secure data storage.

License

Notifications You must be signed in to change notification settings

jrapoport/chestnut

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

18 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

🌰  Chestnut

GitHub Workflow Status Go Report Card Codecov branch GitHub go.mod Go version GitHub

Buy Me A Coffee

Chestnut is encrypted storage for Go. The goal was an easy to use encrypted store with helpful features that was quick to set up, but highly flexible.

Chestnut is written in pure go and designed not to have strong opinions about things like storage, compression, hashing, secrets, or encryption. Chestnut is a storage chest, and not a datastore itself. As such, Chestnut must be backed by a storage solution.

Currently, Chestnut supports BBolt and NutsDB as backing storage.

Table of Contents

Getting Started

Installing

To start using Chestnut, install Go (version 1.11+) and run go get:

$ go get -u github.com/jrapoport/chestnut

Importing Chestnut

To use Chestnut as an encrypted store, import as:

import (
  "github.com/jrapoport/chestnut"
  "github.com/jrapoport/chestnut/encryptor/aes"
  "github.com/jrapoport/chestnut/encryptor/crypto"
  "github.com/jrapoport/chestnut/storage/nuts"
)

// use nutsdb for storage
store := nuts.NewStore(path)

// use AES256-CFB for encryption
opt := chestnut.WithAES(crypto.Key256, aes.CFB, mySecret)

cn := chestnut.NewChestnut(store, opt)
if err := cn.Open(); err != nil {
    return err
}

defer cn.Close()

Requirements

Chestnut has two requirements:

  1. Storage that supports the storage.Storage interface (with a lightweight adapter).
  2. Encryption which supports the crypto.Encryptor interface.

Storage

Chestnut will work seamlessly with any storage solution (or adapter) that supports thestorage.Storage interface.

Built-in

Currently, Chestnut has built-in support for BBolt and NutsDB.

BBolt

https://github.com/etcd-io/bbolt Chestnut has built-in support for using BBolt as a backing store.

To use bbolt for a backing store you can import Chestnut's bolt package and call bolt.NewStore():

import "github.com/jrapoport/chestnut/storage/bolt"

//use or create a bbolt backing store at path
store := bolt.NewStore(path)

// use bbolt for the storage chest
cn := chestnut.NewChestnut(store, ...)

NutsDB

https://github.com/nutsdb/nutsdb
Chestnut has built-in support for using NutsDB as a backing store.

To use nutsDB for a backing store you can import Chestnut's nuts package and call nuts.NewStore():

import "github.com/jrapoport/chestnut/storage/nuts"

//use or create a nutsdb backing store at path
store := nuts.NewStore(path)

// use nutsdb for the storage chest
cn := chestnut.NewChestnut(store, ...)

Planned

Other K/V stores like LevelDB.

GORM (probably not) Gorm is an ORM, so while it's not a datastore per se, it could be adapted to support sparse encryption and would mean automatic support for databases like mysql, sqlite, etc. However, most (if not all) of those DBs already support built-in encryption, so w/o some compelling use-case that's not already covered I don't see a lot of value-add.

Encryption

Chestnut supports several flavors of AES out of the box:

  • AES128-CFB, AES192-CFB, and AES256-CFB
  • AES128-CTR, AES192-CTR, and AES256-CTR
  • AES128-GCM, AES192-GCM, and AES256-GCM

You can add AES encryption to Chestnut by passing the chestnut.WithAES() option:

opt := chestnut.WithAES(crypto.Key256, aes.CFB, mySecret)

AES256-CTR

For encryption we recommend using AES256-CTR. We chose AES256-CTR based in part on this helpful analysis from Shawn Wang, PostgreSQL Database Core.

Custom Encryption

Chestnut supports drop-in custom encryption. A struct that supports the crypto.Encryptor interface can be used with the chestnut.WithEncryptor() option.

Supporting crypto.Encryptor interface is straightforward and mainly consists of vending the following two methods:

// Encrypt returns data encrypted with the secret.
Encrypt(plaintext []byte) (ciphertext []byte, err error)

// Decrypt returns data decrypted with the secret.
Decrypt(ciphertext []byte) (plaintext []byte, err error)

Chained Encryption

Chestnut also supports chained encryption which allows data to be arbitrarily transformed by a chain of Encryptors in a FIFO order.

A chain of crypto.Encryptors can be passed to Chestnut with the chestnut.WithEncryptorChain option:

opt := chestnut.WithEncryptorChain(
    encryptor.NewAESEncryptor(crypto.Key128, aes.CFB, secret1),
    encryptor.NewAESEncryptor(crypto.Key192, aes.CTR, secret2),
    encryptor.NewAESEncryptor(crypto.Key256, aes.GCM, secret3),
)

or by using a crypto.ChainEncryptor with the chestnut.WithEncryptor option:

encryptors := []crypto.Encryptor{
    encryptor.NewAESEncryptor(crypto.Key128, aes.CFB, secret1),
    encryptor.NewAESEncryptor(crypto.Key192, aes.CTR, secret2),
    encryptor.NewAESEncryptor(crypto.Key256, aes.GCM, secret3),
}
chain := crypto.NewChainEncryptor(encryptors...)
opt := chestnut.WithEncryptor(chain)

If you use both the chestnut.WithEncryptor and the chestnut.WithEncryptorChain options, the crypto.Encryptor from chestnut.WithEncryptor will be prepended* to the chain.

Sparse Encryption

Chestnut supports the sparse encryption of structs.

Sparse encryption is a transparent feature of saving structs with Chestnut.Save(), Chestnut.Load(), and Chestnut.Sparse(); or structs that support the value.Keyed interface with Chestnut.SaveKeyed(), Chestnut.LoadKeyed(), and Chestnut.SparseKeyed().

What is "sparse" encryption?

With sparse encryption, only struct fields marked as secure will be encrypted. The remaining "plaintext" fields are encoded and stored separately.

This allows you to load a "sparse" copy of the struct by calling Chestnut.Sparse() or Chestnut.SparseKeyed() (if you have a value.Keyed value) and examine the plaintext fields without the overhead of decryption. When a sparse struct is loaded, the contents of struct fields marked as secure are replaced by empty values.

Enabling Sparse Encryption

Chestnut uses struct tags to indicate which specific struct fields should be encrypted. To enable sparse encryption for a struct, add the secure tag option to the JSON tag of at least one struct field:

SecretKey string `json:",secure"` // 'secure' option (bare minimum)

like so:

type MySparseStruct struct {
    SecretValue string `json:"secret_value,secure"` // <-- add 'secure' here
    PublicValue string `json:"public_value"`
}

Using Sparse Encryption

Structs can be sparsely encrypted by calling Chestnut.Save(), or if the struct supports the value.Keyed interface, Chestnut.SaveKeyed(). Chestnut will automatically detect the secure tag and do the rest.

If no secure fields are found, Chestnut will encrypt the entire struct.

sparseObj := &MySparseStruct{
    SecretValue: "this is a secret",
    PublicValue: "this is public",
}

err := cn.Save("my-namespace",  []byte("my-key"), sparseObj)

When MySparseStruct is saved, Chestnut will detect the secure struct field and only encrypt those fields. Any remaining fields will be encoded as plaintext. In the case of MySparseStructthis means that SecretValuewill be encrypted prior to being encoded, and PublicValuewill not be encrypted.

Sparse Loading

A sparse struct can be loaded by calling Chestnut.Sparse(), or if the struct supports the value.Keyed interface, Chestnut.SparseKeyed(). When these methods are called to load a sparsely encrypted struct, a partially decoded struct will be returned, but the no decryption will occur. Secure fields will instead be replaced by empty values.

sparseObj := &MySparseStruct{}

err := cn.Sparse("my-namespace",  []byte("my-key"), sparseObj)

Examining the struct will reveal that the secure fields were replaced with empty values, and not decrypted.

*MySparseStruct{
    SecretValue: ""
    PublicValue: "this is public"
}

Only sparsely encrypted structs can be sparsely loaded
If Chestnut.Sparse() or Chestnut.SparseKeyed() is called on a struct that was not sparsely encrypted, the fully decrypted struct will be returned.

Decryption

A sparsely encrypted struct can be fully decrypted by calling Chestnut.Load(), or if the struct supports the value.Keyed interface, Chestnut.LoadKeyed(). When any of those methods are called on a sparsely encrypted struct, a fully decrpted copy of the struct is returned.

Secrets

Chestnut secrets are handled through the crypto.Secret interface. The crypto.Secret interface is designed to provide a high degree of flexibility around how you store, retrieve, and manage the secrets you use for encryption.

While Chestnut currently only comes with AES symmetric key encryption, the crypto.Secret interface can easily be adapted to support other forms of encryption like a private key-based crypto.Encryptor.

Chestnut currently provides three basic immplementations of the crypto.Secret interface which should cover most cases.

TextSecret

crypto.TextSecret provides a lightweight wrapper around a plaintext string.

textSecret := crypto.NewTextSecret("a-secret")

ManagedSecret

crypto.ManagedSecret provides a unique ID alongside a plaintext string secret. You can use this id to securely track the secret if you use external vaults or functionality like rollover.

managedSecret := crypto.NewManagedSecret("my-secret-id", "a-secret")

SecureSecret

crypto.SecureSecret provides a unique id for a secret alongside an openSecret() callback which returns a byte representation of the secret for encryption and decryption on SecureSecret.Open(). When crypto.SecureSecret calls openSecret() it will pass a copy of itself as a crypto.Secret. This allows for remote loading of the secret based on its id, or using a secure in-memory storage solution for the secret like memguarded.

openSecret := func(s crypto.Secret) []byte {
	// fetch the secret 
    mySecret := getMySecretFromTheVault(s.ID())
    return mySecret
}
secureSecret := crypto.NewSecureSecret("my-secret-id", openSecret)

Compression

Chestnut supports compression of the encoded data. Compression takes place prior to encryption.

Compression can be enabled through the chestnut.WithCompression option and passing it a supported compression format:

opt := chestnut.WithCompression(compress.Zstd)

Data compressed while chestnut.WithCompression is active with a supported compression format will continue to be correctly decompressed when read even if compression is no longer active (i.e. chestnut.WithCompression is no longer being used). This is not true with custom compression. Data compressed using custom compression cannot be decompressed if that custom compression is disabled.

Zstandard

Chestnut currently supports Zstandard compression out of the box with the compress.Zstd format option. To enable Zstandard compression, call chestnut.WithCompression passing compress.Zstd as the compression format:

opt := chestnut.WithCompression(compress.Zstd)

Please Note: I have no affiliation with Facebook (past or present) and just liked this compression format.

Custom Compression

If you wish to supply your own compression routines you can do so easily with the chestnut.WithCompressors option:

opt := chestnut.WithCompressors(myCompressorFn, myDecompressorFn)

Your two custom compression functions, a compressor compress.CompressorFunc, and a decompressor compress.DecompressorFunc must have the following format:

Compressor(data []byte) (compressed []byte, err error)

Decompressor(compressed []byte) (data []byte, err error)

Compression + Sparse Encryption

Enabling compression will not affect sparse encryption. Sparsely encrypted values compress their secure and plaintext encodings independently.

Operations

Chestnut supports all basic CRUD operations with a few extras.

All WRITE operations: Chestnut.Put(), Chestnut.Save(), & Chestnut.SaveKeyed(), will encrypt data prior to it being stored.

All READ operations: Chestnut.Get(), Chestnut.Load(), & Chestnut.LoadKeyed(), will decrypt data prior to it being returned.

All SPARSE operations: Chestnut.Sparse(), & Chestnut.SparseKeyed(), will not decrypt data prior to it being returned.

In all cases no record of the plaintext data is kept
(even with DebugLevel logging enabled).

Basic Operations

Put

To save an encrypted value to a namespaced key in the storage chest, use the Chestnut.Put() function:

err := cn.Put("my-namespace", []byte("my-key"), []byte("plaintext"))

This will set the value of the "my-key" key to the encrypted ciphertext of "plaintext" in the my-namespace namespace. If a namespace does not exist, it will be automatically created.

If the key already exists, and the storage chest was initialized with the chestnut.OverwritesForbidden option, this call will fail with ErrForbidden.

To retrieve this value, we can use the Chestnut.Get() function:

Get

To retrieve a decrypted value from a namespaced key in the storage chest, we can use the Chestnut.Get() function:

plaintext, err := cn.Get("my-namespace", []byte("my-key"))

Delete