Mark/first iteration of artifact manager #20
Conversation
cargo.toml
Outdated
| "artifact_manager", | ||
| "pyrsia-node" |
There was a problem hiding this comment.
We should be consistent with - and _
There was a problem hiding this comment.
I agree. The Rust naming convention says that module names should be snake case. Perhaps you have time to change the persia-node to persia_node to be compliant.
|
@prince-chrismc @betarelease @efrisch Please take another look at my pull request for artifact manager. I think I have addressed previously stated concerns. I need approvals. |
|
@mgrand Can you change the title and description of this PR? |
| fn file_path_for_new_artifact(&self, expected_hash: &Hash) -> PathBuf { | ||
| let mut base_path: PathBuf = expected_hash.base_file_path(self.repository_path); | ||
| // for now all artifacts are unstructured | ||
| base_path.set_extension("file"); |
There was a problem hiding this comment.
I still wonder why the same artifact with two different hashes will be duplicated on disk?
Taken from your example in the comments
pyrsia-artifacts/SHA256/68efadf3184f20557aa2bbf4432386eb79836902a1e5aea1ff077e323e6ccbb4
pyrsia-artifacts/SHA512/3ac33f26b78b5167d8c2334e8755c7a38e8985f1f62bb943861e20d0777b3d5225cb497c0220922e34bf3c25c44f143b3657289d693226b8f5321862da878b3c
Both folder structures have the same "content" seems a little overkill
There was a problem hiding this comment.
After speaking with @mgrand, there's some background that explains why.
➡️ we have no way of knowing what an artifact is
The only internal knowledge is it's unique hash which is why we start with the Hash name and separate
| // algorithm used to compute the hash and a file name that is the hash, encoded as hex | ||
| // (base64 is more compact, but hex is easier for troubleshooting). For example | ||
| // pyrsia-artifacts/SHA256/68efadf3184f20557aa2bbf4432386eb79836902a1e5aea1ff077e323e6ccbb4 | ||
| // TODO To support nodes that will store many files, we need a scheme that will start separating files by subdirectories under the hash algorithm directory based on the first n bytes of the hash value. |
There was a problem hiding this comment.
I want to raise maybe using an existing library here. This seems to be a problem that has already supported in our tech stack
https://docs.ipfs.io/concepts/content-addressing/#identifier-formats
https://crates.io/crates/cid is the crate that handles this already
|
@prince-chrismc Since your questions are answered, please approve this pull request. |
| } | ||
| } | ||
|
|
||
| fn tmp_dir_name(prefix: &str) -> String { |
There was a problem hiding this comment.
Let's replace this with mktemp.
| let mut hash_array: [u8; 64] = [0; 64]; | ||
| self.result(&mut hash_array); | ||
| let mut i = 0; | ||
| while i < 64 { |
| let mut hash_array: [u8; 32] = [0; 32]; | ||
| self.result(&mut hash_array); | ||
| let mut i = 0; | ||
| while i < 32 { |
| } | ||
|
|
||
| fn hash_size_in_bytes(&self) -> usize { | ||
| return 256/8; |
There was a problem hiding this comment.
Odd. Why didn't rustfmt run on this line?
| @@ -7,16 +7,24 @@ edition = "2021" | |||
| # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | |||
|
|
|||
| [dependencies] | |||
| anyhow = "1.0.45" | |||
| blake3 = "1.2.0" | |||
| env_logger = "0.9.0" | |||
There was a problem hiding this comment.
Remove. We have pretty_env_logger coming in on 17/21.
| let mut hash_array: [u8; 64] = [0; 64]; | ||
| self.result(&mut hash_array); | ||
| let mut i = 0; | ||
| while i < 64 { |
There was a problem hiding this comment.
hash_array.len() in stead of hardcoding.
| hash_buffer: &'b mut [u8; 128], | ||
| digester: &mut Box<dyn Digester>, | ||
| ) -> &'b mut [u8] { | ||
| let buffer_slice: &mut [u8] = &mut hash_buffer[..digester.hash_size_in_bytes()]; |
| ok | ||
| } | ||
|
|
||
| fn tmp_dir_name(prefix: &str) -> String { |
There was a problem hiding this comment.
Can we replace this with mktemp or similar?
* Removal of OpenSSL dependencies - Signed_struct and Struct * Zap openssl (#20) * Keep all logs in release build and default to DEBUG log output for demo (part of #442) (#480) * Provide and retrieve artifacts via the kademlia p2p network (#477) * lookup blobs via kademlia content provisioning * use dial and identify to add new node to dht * add local peer_id to status * Update source folder readme with more terminology and sub-crate (#446) * Bump clap from 3.1.5 to 3.1.6 (#474) Bumps [clap](https://github.com/clap-rs/clap) from 3.1.5 to 3.1.6. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v3.1.5...v3.1.6) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-patch ... * Bump once_cell from 1.9.0 to 1.10.0 (#485) Bumps [once_cell](https://github.com/matklad/once_cell) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/matklad/once_cell/releases) - [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md) - [Commits](matklad/once_cell@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: once_cell dependency-type: direct:production update-type: version-update:semver-minor ... * Introduce the `KeyBox` for consensus engine (#464) * delete signed structs * more traces of openssl * cargo fmt * add `now_as_iso8601_string` for testing * Resolution of errors * Restoring ArtifactBuilder * Removal of _hash * Suggested changes - Chris * Update Dockerfile * fix clippy warning by move import to correct spot * fix import * Update src/docker/v2/handlers/manifests.rs * Update src/metadata_manager/metadata.rs * Update src/docker/v2/handlers/manifests.rs * Updates to signed functions * Cargo format fix * Updates to dev-dep * Changes of methods to fields Signed-off-by: Abhijith Ganesh <67182544+AbhijithGanesh@users.noreply.github.com> Co-authored-by: Erwin Morrhey <erwin@lodgon.com> Co-authored-by: Joeri Sykora <joeri@sertik.net> Co-authored-by: Chris Mc <christopherm@jfrog.com> Co-authored-by: Sudhindra Rao <41690+betarelease@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Mc <prince.chrismc@gmail.com> Co-authored-by: Joeri Sykora <joeri.sykora@gluonhq.com>
I am creating a Draft pull request so people can see what I am up to.