Update FVR client for integration test #2448
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 29, 2022
|
Current dependencies on/for this PR:
This comment was auto-generated by Graphite. |
This was referenced Aug 29, 2022
1865e70 to
97ce652
Compare
97ce652 to
d33be82
8000
Compare
3de39cd to
6bcffaa
Compare
1 task
nick-mobilecoin
approved these changes
Sep 1, 2022
jcape
reviewed
Sep 2, 2022
jcape
suggested changes
Sep 2, 2022
jcape
approved these changes
Sep 2, 2022
awygle
approved these changes
Sep 2, 2022
samdealy
added a commit
that referenced
this pull request
Sep 13, 2022
NotGyro
added a commit
that referenced
this pull request
Sep 27, 2022
* Create helper methods to obliviously collate shard responses (#2252) * Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline * Collate shard response in constant time (obliviously) (#2250) * Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors * Implement epoch sharding via trait (#2350) * Use sharding strategy to tell view what blocks to process (#2351) * Return minimum highest processed block to client (#2387) * Add config option for sharding strategy (#2352) * Add readiness API to ShardingStrategy (#2353) * Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method * Add report fetching logic to FVR (#2454) * Fix responder ID usage in FVR (#2446) * Create different types of View Servers for store and client-facing (#2447) * Messages for noise protocol exchanges with explicit nonces (#2461) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages (#2462) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages * Update FVR client for integration test (#2448) * Create gRPC FogViewRouterAdminAPI (#2360) * Create FogViewRouterAdminUri (#2361) * Add NonceAuthRequest, NonceAuthResponse, NonceSession (#2463) * Add NonceAuthRequest, NonceAuthResponse, NonceSession * Remove EnclaveNonceMessage in favor of EnclaveMessage<NonceSession> * Manually implement Hash to resolve clippy warning * Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Starting branch for ledger router service. * Adapted more of fog view router code for ledger router. * Revert mistake in shard_responses_processor.rs * Adapting copied fog view router code, continuing to fill in a first draft. * Ledger router bin improvements. * EnclaveCall changes * Adding ledger_store_server and service * Adding new enclave methods to ecall_dispatcher() * Clarifying names * Further disentangling of parts of the ledger, plus filling out more enclave-related code * More reorganization * Rewrite create_key_image_store_query to use sealed messages (without changing logic) * Plumb decrypt_and_seal_query enclave call * Enable enclave call for multi-query creation * Implement shard response processing as in Fog View Router * Enable store authentication from router * Implement retry logic * First-pass implementation of response collation Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> Co-authored-by: James Cape <james@mobilecoin.com> Co-authored-by: Millie C <gyro@pop-os.localdomain> Co-authored-by: NotGyro <gyrocoder@gmail.com>
samdealy
added a commit
that referenced
this pull request
Sep 28, 2022
awygle
added a commit
to awygle/mobilecoin
that referenced
this pull request
Oct 4, 2022
* Create helper methods to obliviously collate shard responses (mobilecoinfoundation#2252) * Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline * Collate shard response in constant time (obliviously) (mobilecoinfoundation#2250) * Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors * Implement epoch sharding via trait (mobilecoinfoundation#2350) * Use sharding strategy to tell view what blocks to process (mobilecoinfoundation#2351) * Return minimum highest processed block to client (mobilecoinfoundation#2387) * Add config option for sharding strategy (mobilecoinfoundation#2352) * Add readiness API to ShardingStrategy (mobilecoinfoundation#2353) * Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method * Add report fetching logic to FVR (mobilecoinfoundation#2454) * Fix responder ID usage in FVR (mobilecoinfoundation#2446) * Create different types of View Servers for store and client-facing (mobilecoinfoundation#2447) * Messages for noise protocol exchanges with explicit nonces (mobilecoinfoundation#2461) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages (mobilecoinfoundation#2462) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages * Update FVR client for integration test (mobilecoinfoundation#2448) * Create gRPC FogViewRouterAdminAPI (mobilecoinfoundation#2360) * Create FogViewRouterAdminUri (mobilecoinfoundation#2361) * Add NonceAuthRequest, NonceAuthResponse, NonceSession (mobilecoinfoundation#2463) * Add NonceAuthRequest, NonceAuthResponse, NonceSession * Remove EnclaveNonceMessage in favor of EnclaveMessage<NonceSession> * Manually implement Hash to resolve clippy warning * Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Starting branch for ledger router service. * Adapted more of fog view router code for ledger router. * Revert mistake in shard_responses_processor.rs * Adapting copied fog view router code, continuing to fill in a first draft. * Ledger router bin improvements. * EnclaveCall changes * Adding ledger_store_server and service * Adding new enclave methods to ecall_dispatcher() * Clarifying names * Further disentangling of parts of the ledger, plus filling out more enclave-related code * More reorganization * Rewrite create_key_image_store_query to use sealed messages (without changing logic) * Plumb decrypt_and_seal_query enclave call * Enable enclave call for multi-query creation * Implement shard response processing as in Fog View Router * Enable store authentication from router * Implement retry logic * First-pass implementation of response collation Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> Co-authored-by: James Cape <james@mobilecoin.com> Co-authored-by: Millie C <gyro@pop-os.localdomain> Co-authored-by: NotGyro <gyrocoder@gmail.com>
joekottke
pushed a commit
that referenced
this pull request
Apr 24, 2023
* Fix Fog View Connection Cargo.toml dependencies * Implement create_multi_view_query_data API for Fog View Router (#2115) * Implement create_multi_view_store_query_data API * Address remoun's comments * Implement MultiViewStoreQuery API for Fog View (#2133) * Create MultiViewStoreQuery API for Fog View Store * Implement multi_view_store_query in Fog View Store * Use mut_decryption_error * Implement enclave APIs that facilitate Fog View Router dynamic store discovery (#2200) * Add client_init and client_connect enclave methods * Move multiview query construction to ake enclave * Implement Remoun's suggestions * Implement second round of remoun's suggestions * Remove duplicate wasi files * Re-add older wasi version * Implement query / dynamic shard component discovery in Fog View Router (#2189) * Remove FogViewStoreUri * Implement core Fog Router Service logic * Refactor module structure and add tests * Use where clauses * Implement nick's suggestions * Implement second round of nick's suggestions * Implement lint suggestions * Implement james's suggestions * Fix naming error * Fix whitespace * Split Fog View API into Client-facing and Store (Fog Router) APIs (#2265) * Add FogViewStoreUri * Split Fog View API in two and implement * Implement nick's suggestions * Create helper methods to obliviously collate shard responses (#2252) * Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline * Collate shard response in constant time (obliviously) (#2250) * Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors * Implement epoch sharding via trait (#2350) * Use sharding strategy to tell view what blocks to process (#2351) * Return minimum highest processed block to client (#2387) * Add config option for sharding strategy (#2352) * Add readiness API to ShardingStrategy (#2353) * Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method * Add report fetching logic to FVR (#2454) * Fix responder ID usage in FVR (#2446) * Create different types of View Servers for store and client-facing (#2447) * Messages for noise protocol exchanges with explicit nonces (#2461) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages (#2462) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages * Update FVR client for integration test (#2448) * Create gRPC FogViewRouterAdminAPI (#2360) * Create FogViewRouterAdminUri (#2361) * update cargo lock * Sealed messages for clients with backends (#2449) * Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Update lockfiles again after rebase Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Implement shard management API sans auth (#2354) * Implement shard management API * Implement James's suggestions * Re-apply 2463. (#2547) * Re-apply PR #2473 (was accidentally removed) (#2548) * Reapply changes from #2473 * Add inbound frontend connection support crypto-ake-enclave (#2549) * Add frontends * Fix traits on NonceSession * Make constants and comments slightly friendlier/more consistent (#2550) * Re-apply 2463. (#2547) * Fix traits on NonceSession * Minor comment and constant consistency/friendliness changes * Fix compile error. * Fix deleted Eq from NonceSession. * fix cargo lock * Sort deps * Fix cargo lock files * Spurious commit to trigger CI * Remove `get_nonce` from NonceSession, rename `peek_nonce`->`nonce` (#2638) Also sort Cargo.toml files due to precommit hook. * Replace client with nonce paradigm for backends (#2625) * "Replace client with nonce paradigm for backends" * Sort cargo deps * Fix clone * Fix clippy * Modify MVQ to use NonceMessage (#2617) * Fix view proto * Save work * Implement nick's suggestions + fix lint * Fix fog conformance OOM failure by reducing `frontends` allocation (#2660) * Fix CI and remove Cargo.lock changes * Fix CI and remove Cargo.lock changes * Create query_backend enclave methods (#2627) * Create query_backend enclave methods * Implement nick's suggestions * Rename to QueryStore * Implement nonce based auth for Fog View Store (#2653) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped (#2692) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped * Close gRPC channel when client is dropped (#2694) * Create fog view router test utils (#2698) * Create fog view router test utils * Implement nick's suggestions * Add block range to MVQ response (#2720) * Add block range to MVQ response * Implement James's suggestions * Update BlockTracker to use ShardingStrategy (#2711) * Create MVQ internal type (#2721) * Create MVQ internal type * Remove explicit vec type * Add unknown status * Implement nick's suggestions * Implement new highest processed block count logic (#2722) * Implement highest processed block count logic based on gaps * Implement James's suggestions * Fix comment * Set block-related data more accurately (#2726) * Set block-related data more accurately * Change last known block logic * Implement nick's suggested test cases * Create Fog View Router integration test (#2375) * Create FVR integration tests * Implement nick's suggestions * Calculate last known block txo count and add tests (#2766) * Calculate last known block txo count and add tests * Implement Nick's suggestions * Implement James's suggestions * Collate remaining QueryResponse fields (#2781) * Create types module * Create shared data * Implement Nick's suggestions * Implement James's suggestions * Clean up response creation (#2783) * Make query batch logic compatible with sharding strategy (#2845) * Fix DbFetcher logic and modify recovery db method to use BlockRange * Implement Remoun's suggestions * Re-submit 2803 (#2866) * Replace ciphertext delta with explicit length field * Implement Nick's suggestions * Make payload_length fixed int * Make FVR integration test more complex (#2804) * Make FVR integration test more complex * Implement Nick's + Andrew's suggestions * Add unary apis to FogViewRouter (#2839) * Add unary apis to FogViewRouter * Implement Nick's suggestions * Account for epochs in block tracker highest processed block count logic (#2857) * Account for epochs in block tracker highest processed block count logic * Create and use block range method for ingress keys * Skip a key if the shard's epoch doesn't overlap * Implement Chris's suggestion * Incorporate missed blocks into calculation for highest processed block count (#2858) * Incorporate missed blocks into calculation for highest processed block count * Implement Nick's suggestions * Fix blank query sealing issue (#2904) * Fix blank query sealing issue * Implement Nick's suggestions * Rename seal message * Remove FogViewApi from FogViewService (#2852) * Remove FogViewApi from FogViewService * Fix tests, parametrize, create util methods * Remove FogViewRouterAdminUri * Modify fog conformance and local tests * Implement Nick's suggestions * Implement James's suggestions * Rename tests (#2905) * Rename tests * Implement Nick's suggestions * Implement James's suggestions * Create InternalTxOutSearchResult (#2880) * Create FixedTxOutSearchResult * Replace TxOutSearchResult with FixedTxOutSearchResult * Implement tx out search result conversion * Fill TxOutSearchResult field and update tests * Implement Nick's suggestions * Implement James's comments * Fix response processing loop logic (#2946) * Implement prometheus metrics for FVR (#2976) * Remove unnecessary logs * Create helper fn for response status * Create methods that don't rely on ctx * Implement correct prometheus metrics for FVR * Implement nick's suggestions * implement eran's suggestions * fix: bump postgresql chart version * fix: update scheme * Implement optelemetry tracing for FVR (#2979) * Implement optelemetry tracing for FVR * implement eran's suggestions * Merge master * Fix lock files * Ensure shards provide correct blocks (#2981) * Add sharding_strategies field to config * Plumb block range from config down to shard processor * Throw error if block ranges do not match * Fix python tests * Implement James's and Andrew's suggestions * Fix tests * Run cargo fmt * Fix python tests * Fix default parse * Add admin uri env var * Fix typo * Make Fog View Store return externally-addressable URI to Router (#3081) * Fix store uri by using responder id * Add log for uri * Fix query param * Fix test urls * Use host and responder id in router service * Add logging for ResponderId in router and store * Add more logging * Add more logging * Re-add urls push * Remove logging * Fix clippy * Fix clippy * Fix e_tx_out_store (#3082) * Fix e_tx_out_store * Make default payload_length == 0 * Run format * Fix clippy * Implement James's suggestions * Make find_record constant time (#3097) * Make find_record constant time * Implement Chris's suggestion * Improve error logging for epoch sharding parsing (#3141) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma (#3142) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma * Fix clap feature param * Add uri logging * Add uri logging * Fix deps * Remove custom readiness check * Fix bad merge * Use mc-rand * Re-add edl dep * Re-add all deps taken out by merge * Update pem in mc-fog-view-server * Update cargo lock * Update Cargo.lock * Implement Nick's suggestions * Fix modified test * Fix lint * Add MC_SHARDING_STRATEGY env var --------- Co-authored-by: James Cape <james@mobilecoin.com> Co-authored-by: awygle <awygle@gmail.com> Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> Co-authored-by: Aaron Weisberg <aaronweisberg@gmail.com> Co-authored-by: Sam Dealy <samdealy@users.noreply.github.com>
joekottke
pushed a commit
that referenced
this pull request
Apr 24, 2023
* Fix Fog View Connection Cargo.toml dependencies * Implement create_multi_view_query_data API for Fog View Router (#2115) * Implement create_multi_view_store_query_data API * Address remoun's comments * Implement MultiViewStoreQuery API for Fog View (#2133) * Create MultiViewStoreQuery API for Fog View Store * Implement multi_view_store_query in Fog View Store * Use mut_decryption_error * Implement enclave APIs that facilitate Fog View Router dynamic store discovery (#2200) * Add client_init and client_connect enclave methods * Move multiview query construction to ake enclave * Implement Remoun's suggestions * Implement second round of remoun's suggestions * Remove duplicate wasi files * Re-add older wasi version * Implement query / dynamic shard component discovery in Fog View Router (#2189) * Remove FogViewStoreUri * Implement core Fog Router Service logic * Refactor module structure and add tests * Use where clauses * Implement nick's suggestions * Implement second round of nick's suggestions * Implement lint suggestions * Implement james's suggestions * Fix naming error * Fix whitespace * Split Fog View API into Client-facing and Store (Fog Router) APIs (#2265) * Add FogViewStoreUri * Split Fog View API in two and implement * Implement nick's suggestions * Create helper methods to obliviously collate shard responses (#2252) * Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline * Collate shard response in constant time (obliviously) (#2250) * Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors * Implement epoch sharding via trait (#2350) * Use sharding strategy to tell view what blocks to process (#2351) * Return minimum highest processed block to client (#2387) * Add config option for sharding strategy (#2352) * Add readiness API to ShardingStrategy (#2353) * Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method * Add report fetching logic to FVR (#2454) * Fix responder ID usage in FVR (#2446) * Create different types of View Servers for store and client-facing (#2447) * Messages for noise protocol exchanges with explicit nonces (#2461) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages (#2462) * attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages * Update FVR client for integration test (#2448) * Create gRPC FogViewRouterAdminAPI (#2360) * Create FogViewRouterAdminUri (#2361) * update cargo lock * Sealed messages for clients with backends (#2449) * Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Update lockfiles again after rebase Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Implement shard management API sans auth (#2354) * Implement shard management API * Implement James's suggestions * Re-apply 2463. (#2547) * Re-apply PR #2473 (was accidentally removed) (#2548) * Reapply changes from #2473 * Add inbound frontend connection support crypto-ake-enclave (#2549) * Add frontends * Fix traits on NonceSession * Make constants and comments slightly friendlier/more consistent (#2550) * Re-apply 2463. (#2547) * Fix traits on NonceSession * Minor comment and constant consistency/friendliness changes * Fix compile error. * Fix deleted Eq from NonceSession. * fix cargo lock * Sort deps * Fix cargo lock files * Spurious commit to trigger CI * Remove `get_nonce` from NonceSession, rename `peek_nonce`->`nonce` (#2638) Also sort Cargo.toml files due to precommit hook. * Replace client with nonce paradigm for backends (#2625) * "Replace client with nonce paradigm for backends" * Sort cargo deps * Fix clone * Fix clippy * Modify MVQ to use NonceMessage (#2617) * Fix view proto * Save work * Implement nick's suggestions + fix lint * Fix fog conformance OOM failure by reducing `frontends` allocation (#2660) * Fix CI and remove Cargo.lock changes * Fix CI and remove Cargo.lock changes * Create query_backend enclave methods (#2627) * Create query_backend enclave methods * Implement nick's suggestions * Rename to QueryStore * Implement nonce based auth for Fog View Store (#2653) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped (#2692) * Implement nonce based auth for Fog View Store * Implement nonce based auth * Close gRPC channel when client is dropped * Close gRPC channel when client is dropped (#2694) * Create fog view router test utils (#2698) * Create fog view router test utils * Implement nick's suggestions * Add block range to MVQ response (#2720) * Add block range to MVQ response * Implement James's suggestions * Update BlockTracker to use ShardingStrategy (#2711) * Create MVQ internal type (#2721) * Create MVQ internal type * Remove explicit vec type * Add unknown status * Implement nick's suggestions * Implement new highest processed block count logic (#2722) * Implement highest processed block count logic based on gaps * Implement James's suggestions * Fix comment * Set block-related data more accurately (#2726) * Set block-related data more accurately * Change last known block logic * Implement nick's suggested test cases * Create Fog View Router integration test (#2375) * Create FVR integration tests * Implement nick's suggestions * Calculate last known block txo count and add tests (#2766) * Calculate last known block txo count and add tests * Implement Nick's suggestions * Implement James's suggestions * Collate remaining QueryResponse fields (#2781) * Create types module * Create shared data * Implement Nick's suggestions * Implement James's suggestions * Clean up response creation (#2783) * Make query batch logic compatible with sharding strategy (#2845) * Fix DbFetcher logic and modify recovery db method to use BlockRange * Implement Remoun's suggestions * Re-submit 2803 (#2866) * Replace ciphertext delta with explicit length field * Implement Nick's suggestions * Make payload_length fixed int * Make FVR integration test more complex (#2804) * Make FVR integration test more complex * Implement Nick's + Andrew's suggestions * Add unary apis to FogViewRouter (#2839) * Add unary apis to FogViewRouter * Implement Nick's suggestions * Account for epochs in block tracker highest processed block count logic (#2857) * Account for epochs in block tracker highest processed block count logic * Create and use block range method for ingress keys * Skip a key if the shard's epoch doesn't overlap * Implement Chris's suggestion * Incorporate missed blocks into calculation for highest processed block count (#2858) * Incorporate missed blocks into calculation for highest processed block count * Implement Nick's suggestions * Fix blank query sealing issue (#2904) * Fix blank query sealing issue * Implement Nick's suggestions * Rename seal message * Remove FogViewApi from FogViewService (#2852) * Remove FogViewApi from FogViewService * Fix tests, parametrize, create util methods * Remove FogViewRouterAdminUri * Modify fog conformance and local tests * Implement Nick's CEB7 suggestions * Implement James's suggestions * Rename tests (#2905) * Rename tests * Implement Nick's suggestions * Implement James's suggestions * Create InternalTxOutSearchResult (#2880) * Create FixedTxOutSearchResult * Replace TxOutSearchResult with FixedTxOutSearchResult * Implement tx out search result conversion * Fill TxOutSearchResult field and update tests * Implement Nick's suggestions * Implement James's comments * Fix response processing loop logic (#2946) * Implement prometheus metrics for FVR (#2976) * Remove unnecessary logs * Create helper fn for response status * Create methods that don't rely on ctx * Implement correct prometheus metrics for FVR * Implement nick's suggestions * implement eran's suggestions * fix: bump postgresql chart version * fix: update scheme * Implement optelemetry tracing for FVR (#2979) * Implement optelemetry tracing for FVR * implement eran's suggestions * Merge master * Fix lock files * Ensure shards provide correct blocks (#2981) * Add sharding_strategies field to config * Plumb block range from config down to shard processor * Throw error if block ranges do not match * Fix python tests * Implement James's and Andrew's suggestions * Fix tests * Run cargo fmt * Fix python tests * Fix default parse * Add admin uri env var * Fix typo * Make Fog View Store return externally-addressable URI to Router (#3081) * Fix store uri by using responder id * Add log for uri * Fix query param * Fix test urls * Use host and responder id in router service * Add logging for ResponderId in router and store * Add more logging * Add more logging * Re-add urls push * Remove logging * Fix clippy * Fix clippy * Fix e_tx_out_store (#3082) * Fix e_tx_out_store * Make default payload_length == 0 * Run format * Fix clippy * Implement James's suggestions * Make find_record constant time (#3097) * Make find_record constant time * Implement Chris's suggestion * Improve error logging for epoch sharding parsing (#3141) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma (#3142) * Improve error logging for epoch sharding parsing * Fix string format * Fix clippy lint re: expect statement * Make shard uris delimiter be a comma * Fix clap feature param * Add uri logging * Add uri logging * Fix deps * Remove custom readiness check * Fix bad merge * Use mc-rand * Re-add edl dep * Re-add all deps taken out by merge * Update pem in mc-fog-view-server * Update cargo lock * Update Cargo.lock * Implement Nick's suggestions * Fix modified test * Fix lint * Add MC_SHARDING_STRATEGY env var * Pull in protobuf changes from milliec/ledger-router-dev (#2889) * Pull in protobuf changes from milliec/ledger-router-dev * Move AddShardRequest to fog_common.proto, resolving #2888 * Clarifying comments and names, implementing changes requested for PR. * Fixing old copy/paste mistakes with 'view' * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * URIs for Fog Ledger Router (#2897) * URIs for Fog Ledger Router * Use AdminUri for Fog Ledger Router Admin URI * PR feedback from James * Protobuf conversion functions for Fog Ledger Router (#2891) * URIs for Fog Ledger Router * Use AdminUri for Fog Ledger Router Admin URI * Protobuf conversion functions for Fog Ledger Router * Key Image Router Service (#2898) * Key Image Router Service * Remove unneeded dead_code annotation * Update fog/ledger/server/src/error.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Clean up commented-out code Co-authored-by: Nick Santana <nick@mobilecoin.com> * Fix misnamed type in a comment Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Address error in loop termination logic. * Parameterize allowed number of retries for query loop * Update based on changes from previous PRs * Don't create 'groups' in `mod` or `use` declarations. Co-authored-by: NotGyro <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Oblivious collation of key image store responses (#2913) * Oblivious collation of key image store responses * Refactor tests to be more readable, per PR feedback * Update fog/ledger/enclave/impl/src/oblivious_utils.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Fog Ledger Router Admin service (#2920) * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Fix redundant mod declaration * Update to use local SVC_COUNTERS * Key Image Router Server + Binary (#2921) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Remove dead_code declaration on router service constructor * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary Co-authored-by: Emily C <gyrocoder@gmail.com> * Key image store service and server for router/store system (#2954) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Apply suggestions from code review Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> * Implement Sharding Strategy (#2873) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Fog Ledger Router Admin service * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Implement ShardingStrategy into DbFetcher * Split out DbFetcher creation and starting for ease of integration * Add a DbFetcher to the ledger store server. * Start the db_fetcher in the Fog Ledger Server to reflect the new behavior * Run cargo fmt * Run cargo clippy * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Remove some unwraps * Lint again * rebase fixups * PR feedback --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Implement non-KeyImage Functionality for Fog Ledger Router (#3084) * Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Implement ShardingStrategy into DbFetcher * Add a DbFetcher to the ledger store server. * Run cargo fmt * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Refactor router dispatch into match statement Uses ugly names from rust-protobuf. rust-protobuf version 3.* will do the Rusty thing and put the generated enums into modules, but we're on version 2.* and they do the C thing of really_long_snake_case_names. * Add merkle proof service to router server * Add untrusted tx out service to router server * Add block service to router server * Fixups from rebase --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Porting existing Fog Ledger tests to Router (#3138) * Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from code review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Pull request feedback --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Test key image retrieval via unary API on the router (#3163) * Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Nick Santana <nick@mobilecoin.com> * Test key image retrieval via unary API on the router. * Cargo fmt * Fix a merging mistake. * Fix some additional merge mistakes * Sort dependencies --------- Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> * Fix up Clippy issues from new Rust version * Make Fog Ledger Store return externally-addressable URI to Router (#3192) * Port fog local network and conformance tests to use ledger router (#3144) * Initial port of python tests, needs cleanup * Fix ledger and watcher DB paths in conformance * Fixups from rebase and add path arguments to key image store * Pull request feedback * Rebase fixups for unary test * Rename new_from_config to new, drop old new * Fog Ledger Router integration test (#3188) * Router integration test * Clippy updates after rebase * Implement Drop for TestEnvironment * Eliminate temp dirs after test * Remove unused GRPC arguments * Load environment config from JSON file. Might delete later. * Revert "Load environment config from JSON file. Might delete later." This reverts commit a677b59. * Remove magic numbers * Remove client-facing API from key-image store (#3217) * Remove client-facing API from key-image store * Cargo fmt * Remove fog_ledger_key_images_test from connection.rs (equivalent exists in router_connection.rs) * Cargo fmt * Pull out old code for the single-server design which will not be used anymore. * Fix cargo.toml. * Optelemetry tracing for FLR (#3264) * Optelemetry tracing for FLR * Cargo fmt * Apply James' suggested span names Co-authored-by: James Cape <james@mobilecoin.com> * Cargo fmt --------- Co-authored-by: James Cape <james@mobilecoin.com> * Process all blocks, but only add our blocks to the ORAM (#3203) (#3262) * Prometheus metrics for FLR (#3267) * Prometheus metrics for FLR * Cargo fmt * Post-rebase cleanups * Use default readiness check for FLR * Remove mc-crypto-rand from FLR branch * Address PR feedback * Clean up Cargo.locks * Add MC_SHARDING_STRATEGY --------- Co-authored-by: Sam Dealy <sam.dealy@gmail.com> Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> Co-authored-by: James Cape <james@mobilecoin.com> Co-authored-by: Aaron Weisberg <aaronweisberg@gmail.com> Co-authored-by: Sam Dealy <samdealy@users.noreply.github.com> Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The preexisting router client was created to demo dummy requests to FVR. It didn't do any attestation / make any legit query requests. This change adds this functionality.
The logic is taken from the preexisting fog/view/connectoin/lib.rs code, and the only difference is that it supports streaming.