Change fog-ledger to support getting ledger data from a remote mobilecoind #3701

eranrund · 2023-11-09T00:11:46Z

Motivation

Right now every instance of fog-ledger-router and fog-ledger-store run their own mobilecoind and sync the ledger and watcher databases. This is inefficient in terms of storage, and increases our ongoing operations costs.

The changes in this PR:

Add and extend some mobilecoind endpoints to accommodate for the type of queries fog-ledger needs to do
Add a new mc-fog-block-provider crate that provides an abstraction over the database queries fog-ledger is doing, and two implementations: one that uses the local lmdb databases and one that uses a remote mobilecoind
Modify the two fog-ledger server binaries to allow either using local databases or a remote mobilecoind.

Performance

To verify the performance of thi 8000 s I ran the fog-test-client locally (for my future self: MC_CONTINUOUS=true MC_TRANSFER_PERIOD=15 MC_TELEMETRY=true RUST_LOG=trace SGX_MODE=HW IAS_MODE=DEV CONSENSUS_ENCLAVE_CSS=$PWD/keys/libconsensus-enclave.css VIEW_ENCLAVE_CSS=$PWD/keys/libview-enclave.css INGEST_ENCLAVE_CSS=$PWD/keys/libingest-enclave.css LEDGER_ENCLAVE_CSS=$PWD/keys/libledger-enclave.css cargo run --release -p mc-fog-test-client -- --chain-id mc-fog-mobilecoind --key-dir keys/fog_keys --consensus mc://node1.mc-fog-mobilecoind.development.mobilecoin.com --num-clients 4 --consensus-wait 300 --transfer-amount 20 --fog-view fog-view://fog.mc-fog-mobilecoind.development.mobilecoin.com --fog-ledger fog-ledger://fog.mc-fog-mobilecoind.development.mobilecoin.com --admin-listen-uri insecure-mca://0.0.0.0:8001/) and queried the Prometheus metrics endpoint (grpcurl -d '' -plaintext -protoset ./admin.protoset 127.0.0.1:8001 admin.AdminAPI.GetPrometheusMetrics | jq -r .metrics) after it did some iterations.

The local mobilecoind ("stock" but with the support for remote mobilecoind - e.g. switching to Box<dyn BlockProvider> but using the local lmdbs implementation) did 52 iterations before I stopped it, and the remote one did 124 iterations - this has nothing to do with performance, this is just how long I let it ran. I was aiming for 50 iterations but got distracted and didn't Ctrl+C the remote version early enough, so we got more data points.

Here's my analysis:

(The table might be a bit hard to read, https://docs.google.com/spreadsheets/d/1dwfzj3RQ99ubUNWRTFxpstucmHLAhdTgkyJ3GDHon4k/edit?usp=sharing might be easier to look at. The numbers under "Local mobilecoind" and "Remote mobilecoind" show how many out of how many iterations took less than a certain amount of time, e.g. 40/53 <2s reads as 40 out of 53 iterations took less than 2 seconds).

1	Metric name	Metric description	Local mobilecoind	Remote mobilecoind
2	tx_build_time	Time to build a transaction (before sending) Includes time it takes to query fog ledger for rings and merkle proofs	39/53 <1.7s 40/53 <2s 53/53 <2.5s	114/124 <1.7s 119/124 <2s 124/124 <2.5s
Appears to perform identically
3	tx_confirmed_time	Check if a TxOut appears in ledger by querying Fog Ledger (untrusted)	5/52 <1s 52/52 <2s	99/124 <2s 124/124 <2.5s
Appears to perform almost identically - majority of iterations took <2s, however no <1s measurements were seen for the remote mobilecoind
4	tx_received_time	Fog client sees correct up to date balance	9/52 <2.5s 40/52 <3s 44/52 <4s 49/52 <4.5s 52/52 <16s	19/124 <2.5s 76/124 <3.5s 88/124 <4s 121/124 <4.5s 124/124 <6.5s
Surprisingly, remote mobilecoind appears to perform better at the tail end. Majority of measurements were <4.5s in both cases but tail end latency was lower in remote mobilecoind (upper bound 6.5s vs 16s)
5	tx_send_time	Time it takes to send a build transaction to consensus	40/52 <0.2s 52/52 <0.5s	119/124 <0.2s 124/124 <0.5s
Appears to perform identically (expected since this doesn't touch fog)

The raw data is available here: https://gist.github.com/eranrund/8e89d5d8ad6b6d5c79d72a3e6011d1bb

So, it appears that at least in the dev network configuration this performs fine and will allow us to save costs.

Future room for improvement

Get fog-ingest to also use a remote mobilecoind, or to expose its own mobilecoind for fog-view to use.

TODO

Deploy this as is, just to make sure the local databases still work as expected.
Change the deployment manifests so that the fog-ledger servers connect to a centralized mobilecoind and ensure things still work as expected.
Measure performance to see if it is significantly affected by this change.

fog/api/proto/ledger.proto

fog/block_provider/src/local.rs

fog/block_provider/src/mobilecoind.rs

fog/ledger/server/src/block_service.rs

mobilecoind/api/proto/mobilecoind_api.proto

fog/block_provider/src/lib.rs

eranrund · 2023-11-14T23:32:20Z

Annoyingly renaming a branch on GH closes the PR instead of moving it to a new branch. I'll bring it back once I'm done working on it.

This reverts commit 6bb3692.

This reverts commit c086ea0.

.internal-ci/helm/fog-test-client/values.yaml

nick-mobilecoin

a few nits and some questions on how missing/unfound blocks are handled.

fog/block_provider/src/mobilecoind.rs

mobilecoind/src/service.rs

mobilecoind/api/proto/mobilecoind_api.proto

fog/block_provider/src/lib.rs

eranrund added 13 commits November 7, 2023 14:54

add timestamp to GetBlockResponse

a07abf3

initial work on getting fog ledger to access the database via a trait

7eb466a

merkle proof service uses block provider

61d50ae

untrusted tx out service uses block provider

02adcee

block contents -> block data

0c6ac98

8000 add timestamp to get_block_data

e0b37af

block_service uses block provider

b6dbcb0

initial work on mobilecoind block provider

a75c683

implement get_latest_block

6afc442

add get_blocks_data

b953b70

poll_block_timestamp

1fe37f1

get_tx_out_and_membership_proof_by_index

5c941e1

clippy

947d15c