Add support for maven requests #791

jperedadnr · 2022-06-20T15:48:10Z

Description

Fixes pyrsia #781

This PR adds support for Maven requests by adding a new route and processing the artifacts accordingly.

Any Maven client could add a repository like:

<repositories>
        <repository>
            <id>Pyrsia</id>
            <url>http://localhost:7888/maven2</url>
        </repository>
    </repositories>

and then requests dependencies from that repository.

A unit test is included, but it doesn't show the full end-to-end integration with a Maven client.

This requires extra steps that are not included in this PR.

Let's say we have a Java project that can be published to Maven local or Central, creating the artifact test:test:1.0.
The files that are installed under ~/.m2/repository/test/test/1.0 are:

test-1.0.jar
test-1.0.jar.md5
test-1.0.jar.sha1
test-1.0.pom
test-1.0.pom.md5
test-1.0.pom.sha1

The first step will be adding those files to the transparency log.

Copy the files to the pyrsia node and remove local dependencies:

cp ~/.m2/repository/test/test/1.0/test-1.0.* $PYRSIA_HOME/nodeA/pyrsia/SHA256/
rm -rf ~/.m2/repository/test

Calculate the files' SHA and rename them:

for each file:

$ sha256sum test-1.0.* 
mv test-1.0* $hash.file

Add the files to the transparency log in main.rs:

For instance, like this:

    debug!("Create transparency log");
    let mut transparency_log = TransparencyLog::new();

    debug!("Add mock artifacts to transparency log");
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.jar", "e11c16ff163ccc1efe01d2696c626891560fa82123601a5ff196d97b6ab156da")
        .unwrap_or_else(|_| debug!("Test jar already existed."));
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.jar.md5", "22f5cc2e4eefd02ae58767f6d63cba3ff10df306d27293f8be6abb14c5ebec61")
        .unwrap_or_else(|_| debug!("Test jar.md5 already existed."));
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.jar.sha1", "4900f2fbe79149fe50458afb6385feefdcbba7a9041477b9a9dbfeeca8f03dbf")
        .unwrap_or_else(|_| debug!("Test jar already existed."));
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.pom", "65e543bb8d7855e4cc0ef8d58d86edc8dcbd02cda1f0df18af530e355a280603")
        .unwrap_or_else(|_| debug!("Test pom already existed."));
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.pom.md5", "1d6fe2d458bc09ce7e0feb24a95ab742f7ab28bfb3cea9140b157f9740ada6db")
        .unwrap_or_else(|_| debug!("Test pom.md5 already existed."));
    transparency_log.add_artifact("MAVEN2/FILE/test/test/1.0/test-1.0.pom.sha1", "7d87a44563c1b4d424e20c62e0ba510e9532fb6e5b4b374dd083ca1c42fcfcd0")
        .unwrap_or_else(|_| debug!("Test pom.sha1 already existed."));

build and run nodeA

$ cargo build --workspace
cp target/debug/pyrsia_node nodeA
rm -rf nodeA/transparency_log
DEV_MODE=on RUST_LOG="pyrsia=debug,info"  nodeA/pyrsia_node -H 0.0.0.0 -p 7888 -L /ip4/0.0.0.0/tcp/44001

Final test

In a Java project, add the dependency and repository:

<dependencies>
        <dependency>
            <groupId>test</groupId>
            <artifactId>test</artifactId>
            <version>1.0</version>
        </dependency>
    </dependencies>

    <repositories>
        <repository>
            <id>Pyrsia</id>
            <url>http://localhost:7888/maven2</url>
        </repository>
    </repositories>

when building the project, note that the dependency is retrieved:

$ mvn compile
...
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from Pyrsia: http://localhost:7888/maven2/test/test/1.0/test-1.0.pom
Downloaded from Pyrsia: http://localhost:7888/maven2/test/test/1.0/test-1.0.pom (898 B at 22 kB/s)
Downloading from Pyrsia: http://localhost:7888/maven2/test/test/1.0/test-1.0.jar
Downloaded from Pyrsia: http://localhost:7888/maven2/test/test/1.0/test-1.0.jar (2.0 kB at 284 kB/s)

and also check the pyrsia logs:

 DEBUG pyrsia::java::maven2::routes      > got full path : /maven2/test/test/1.0/test-1.0.pom
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting maven artifact: /maven2/test/test/1.0/test-1.0.pom
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting artifact for id MAVEN2/FILE/test/test/1.0/test-1.0.pom
 INFO  pyrsia::artifact_service::storage               > An artifact is being pulled from the artifact manager SHA256:65e543bb8d7855e4cc0ef8d58d86edc8dcbd02cda1f0df18af530e355a280603
 DEBUG pyrsia::artifact_service::storage               > Pulling artifact from $PYRSIA_HOME/nodeA/pyrsia/SHA256/65e543bb8d7855e4cc0ef8d58d86edc8dcbd02cda1f0df18af530e355a280603.file
 INFO  pyrsia_registry                                 > 127.0.0.1:50850 "GET /maven2/test/test/1.0/test-1.0.pom HTTP/1.1" 200 "-" "Apache-Maven/3.8.6 (Java 17.0.2; Mac OS X 12.4)" 2.378791ms
 DEBUG pyrsia::java::maven2::routes                    > got full path : /maven2/test/test/1.0/test-1.0.pom.sha1
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting maven artifact: /maven2/test/test/1.0/test-1.0.pom.sha1
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting artifact for id MAVEN2/FILE/test/test/1.0/test-1.0.pom.sha1
 INFO  pyrsia::artifact_service::storage               > An artifact is being pulled from the artifact manager SHA256:7d87a44563c1b4d424e20c62e0ba510e9532fb6e5b4b374dd083ca1c42fcfcd0
 DEBUG pyrsia::artifact_service::storage               > Pulling artifact from $PYRSIA_HOME/nodeA/pyrsia/SHA256/7d87a44563c1b4d424e20c62e0ba510e9532fb6e5b4b374dd083ca1c42fcfcd0.file
 INFO  pyrsia_registry                                 > 127.0.0.1:50850 "GET /maven2/test/test/1.0/test-1.0.pom.sha1 HTTP/1.1" 200 "-" "Apache-Maven/3.8.6 (Java 17.0.2; Mac OS X 12.4)" 1.304375ms
 DEBUG pyrsia::java::maven2::routes                    > got full path : /maven2/test/test/1.0/test-1.0.jar
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting maven artifact: /maven2/test/test/1.0/test-1.0.jar
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting artifact for id MAVEN2/FILE/test/test/1.0/test-1.0.jar
 INFO  pyrsia::artifact_service::storage               > An artifact is being pulled from the artifact manager SHA256:e11c16ff163ccc1efe01d2696c626891560fa82123601a5ff196d97b6ab156da
 DEBUG pyrsia::artifact_service::storage               > Pulling artifact from $PYRSIA_HOME/nodeA/pyrsia/SHA256/e11c16ff163ccc1efe01d2696c626891560fa82123601a5ff196d97b6ab156da.file
 INFO  pyrsia_registry                                 > 127.0.0.1:50850 "GET /maven2/test/test/1.0/test-1.0.jar HTTP/1.1" 200 "-" "Apache-Maven/3.8.6 (Java 17.0.2; Mac OS X 12.4)" 1.245041ms
 DEBUG pyrsia::java::maven2::routes                    > got full path : /maven2/test/test/1.0/test-1.0.jar.sha1
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting maven artifact: /maven2/test/test/1.0/test-1.0.jar.sha1
 DEBUG pyrsia::java::maven2::handlers::maven_artifacts > Requesting artifact for id MAVEN2/FILE/test/test/1.0/test-1.0.jar.sha1
 INFO  pyrsia::artifact_service::storage               > An artifact is being pulled from the artifact manager SHA256:4900f2fbe79149fe50458afb6385feefdcbba7a9041477b9a9dbfeeca8f03dbf
 DEBUG pyrsia::artifact_service::storage               > Pulling artifact from $PYRSIA_HOME/nodeA/pyrsia/SHA256/4900f2fbe79149fe50458afb6385feefdcbba7a9041477b9a9dbfeeca8f03dbf.file
 INFO  pyrsia_registry                                 > 127.0.0.1:50850 "GET /maven2/test/test/1.0/test-1.0.jar.sha1 HTTP/1.1" 200 "-" "Apache-Maven/3.8.6 (Java 17.0.2; Mac OS X 12.4)" 832µs

Screenshots (optional)

PR Checklist

I've read the contributing guidelines.
I've read "What is a Good PR?"
I've included a good title and brief description along with how to review them.
I've linked any associated an issue.

Code Contributions

I've built the code cargo build --all-targets successfully.
I've run the unit tests cargo test --workspace and everything passes.
I've made sure my rust toolchain is current rustup update.

CLAassistant · 2022-06-20T15:48:18Z

All committers have signed the CLA.

codecov · 2022-06-20T16:48:27Z

Codecov Report

Merging #791 (447232e) into main (8e09fe2) will increase coverage by 0.13%.
The diff coverage is 80.88%.

@@            Coverage Diff             @@
##             main     #791      +/-   ##
==========================================
+ Coverage   77.79%   77.92%   +0.13%     
==========================================
  Files          29       31       +2     
  Lines        1477     1545      +68     
==========================================
+ Hits         1149     1204      +55     
- Misses        328      341      +13

Impacted Files	Coverage Δ
src/java/maven2/routes.rs	`0.00% <0.00%> (ø)`
src/java/maven2/handlers/maven_artifacts.rs	`88.70% <88.70%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e09fe2...447232e. Read the comment docs.

src/java/maven2/routes.rs

src/java/maven2/handlers/maven_artifacts.rs

Add support for maven requests

31da974

jperedadnr requested a review from a team as a code owner June 20, 2022 15:48

jperedadnr requested review from tiainen and mseabornIBM and removed request for a team June 20, 2022 15:48

github-actions bot added the in-review label Jun 20, 2022

github-actions bot assigned jperedadnr Jun 20, 2022

jperedadnr added 2 commits June 20, 2022 18:02

revert copyright changes

98773f4

format code

d6d0d7c

fix warnings

dd77ef4

tiainen reviewed Jun 21, 2022

View reviewed changes

src/java/maven2/routes.rs Outdated Show resolved Hide resolved

src/java/maven2/handlers/maven_artifacts.rs Outdated Show resolved Hide resolved

src/java/maven2/handlers/maven_artifacts.rs Show resolved Hide resolved

jperedadnr added 3 commits June 21, 2022 17:52

Address feedback from reviewer

036299f

Minor rewording

7179057

Remove test exclusions

8d28949

mseabornIBM previously approved these changes Jun 22, 2022

View reviewed changes

jperedadnr added 2 commits June 22, 2022 20:15

Merge branch 'main' into 781-maven

e0eb53b

Remove assay and update changes

447232e

jperedadnr dismissed mseabornIBM’s stale review via 447232e June 22, 2022 18:27

tiainen approved these changes Jun 23, 2022

View reviewed changes

johanvos approved these changes Jun 23, 2022

View reviewed changes

tiainen merged commit d18005c into pyrsia:main Jun 23, 2022

jperedadnr mentioned this pull request Jun 24, 2022

Pyrsia nodes handle incoming requests from a maven client #781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for maven requests #791

Add support for maven requests #791

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add support for maven requests #791

Add support for maven requests #791

Uh oh!

Conversation

Description

Screenshots (optional)

PR Checklist

Code Contributions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!