Trade Data Database #27

sneilan · 2023-11-27T02:30:36Z

Implementation Notes

Use sqlite write ahead log to journal trades. Leverage SQLite for WAL and atomic commits instead of apache avro. Can tune settings for performance.
Rotate trade journals by storing no more than 100k trades per file. Filename should be unique. Can put trade id / date etc in filename but rely on database because storing metadata in filenames is less scalable.
Take checkpoints of market depth and event store in sqlite too.
Track checkpoints in database to say which trade ID was latest when checkpoint taken.
Restore checkpoints in sqlite to memory by deserialization.
Do not save / restore checkpointed data by pure memory because the pointers to mmap buffers will misalign on reload. Plus saving memory data directly will also save any memory fragmentation.
Replaying trade journals should result in correct system state.
Add tests to make sure journals / replay work.
Define data structures to store this.

sneilan added the enhancement New feature or request label Nov 27, 2023

sneilan self-assigned this Nov 27, 2023

Provide feedback