Storage of build artifacts produced by continous intergration which makes them available for automated deployment to different deployment environment
Provides a central location
Artifacts are applications built into a single file
There are different artifact formats: (JAR, WAR, ZIP, TAR, etc.)
Artifact repository needs to support this specific format
Repository for each file/artifact type
Can store many different artifact types
Which is great, because as a company you often produce different types of artifacts
Public Repository Managers
There are public repository managers
For example libraries/frameworks you use as a dependency (Maven Repository, NPM etc...)
You can make own project publically available there
Nexus Private Repository Manager
Allow to store and upload different build Artifact .
Also can retrieve (download) artifact later.
Central Storage
It makes sense if I have multiple projects in a company to have Nexus where you can centrally sotre all those artifacts in case the teams and projects want to share them among each other
I can also set up a Proxy repository for public Repository
-
For example for Maven Central Repository, you can acutally create a proxy repository in Nexus and the main reason would be to kind of consolidate all of my artifact management in 1 spot.
-
So whether you are using Company internal artifacts or you are using public artifact you can fetch everything from Nexus
Integrate with LDAP . Means that in big company where is 100 of developers, you can actually pretty easily configure the access management in Nexus just by integrating LDAP and then giving some sets of permissions
REST API endpoint . Nexus is not something we manually manage you wouldn't be building the artifacts locally on my laptop and then uploading them to Nexus manually and you will also not be downloading the artifacts manually to my deployment server . This will be part of CI/CD
Backup and Restore. When I have multiple projects and everybody's building artifacts and they are pushing that to Nexus Repository, we have some storage demand . Nexus has its own storage mechanism configured by default and it stores those artifacts on the Server, but it's also important to be able to configure easy backup and restore, or just generally managing the storage of Nexus, especially if you end up having loads of artifacts on Nexus server, then you need to increase the storage size
Multi-format support
Metada tagging (labeling and tagging artifact). Managing and helping to manage the tags and version and so on. Bcs we may have release versions and we may have development verison and so on, and Nexus Actually denpending on the repository typ may acutally help you manage those version and texts
Cleanup Policies: Image we have CI for my application, then every commit to the master branch bascially creates a new artifact of the application. And if that happens on daily bases or multiple times a day, you can just imagine how many artifacts you would generate and this will all end up on Nexus and if other teams are doing the same, then it could be that Nexus run out of storage space bcs of that Cleanup policy are important
Search functionality : If we end up having tons of different artifacts, especially of different types and different formats then you need to be able to search across repositories to find exactly the version that artifact that you are looking for
User Token Support for system user
Run Nexus on Droplet and Publish Artifact to Nexus
Technologies used:
- Nexus, DigitalOcean, Linux, Java, Gradle, Maven
Project Description:
- Install and configure Nexus from scratch on a cloud server Create new User on Nexus with relevant permissions Java Gradle Project: Build Jar & Upload to Nexus Java Maven Project: Build Jar & Upload to Nexus
I will create a Server on Digital Ocean
I will configure Firewall for this Droplet
- Accessible at Port 22
Now I can ssh to the server using : ssh root@<Public-ip>
I need to install Java openjdk-17 bcs Nexus need Java to run : apt install openjdk-17-jre-headless
I will move in to cd /opt folder and Download Nexus here .
Link to download Nexus (https://help.sonatype.com/en/download.html)
I will copy link address of the Unix Archive
Head back to the terminal and do : wget https://download.sonatype.com/nexus/3/nexus-3.81.1-01-linux-x86_64.tar.gz. With this command I just download a tar file Nexus
Now I have tar file locally nexus-3.81.1-01-linux-x86_64.tar.gz
To untar that file : tar -zxvf nexus-3.81.1-01-linux-x86_64.tar.gz
Now I should see 2 Folder nexus-3.81.1-01 and sonatype-work
nexus-3.81.1-01 : Contain a runtime and the application of Nexus itself .
nexus-3.81.1-01/binthe binary to start Nexus Application
sonatype-work/nexus3 contain my own configuration for Nexus and the data
-
This mean when you upgrade to a new version of nexus, you replace the Nexus directory, the application directory with the binaries, but you still keep the custom configuration and repository data in
sonatype-work -
Sonotype work directory is created as a sibling, to the
nexus-3.81.1-01directory and nexus knows the location ofsonatype-workdirectory through its properties file -
Sonatype-work/nexus3container number of subdirectories, depending on what you configure in Nexus itself . For examole i can install plugin in Nexus If I do different subdirectories will appear in this folder . -
This directory also contain a file has a log of all IP address that accessed or is accessing the NExus application
-
It also contains logs of Nexus application itslef .
-
It also store actual component. So the archives, the files that you are uploading to Nexus and the metadata to them many other files and subdirectories that contain different sets of information
I can use sonatype-work folder to backup all the Nexus data since this folder contain all the data all the storage, all the configuration, etc ...
NOTE: Never start services generally with root user bcs of security reason . I will create own Linux user for its own services and then start that service with that specific user that has only permission it needs to run the application and to interact with the application
To create nexus user : adduser nexus . Set password
Now I check the permissions of nexus-3.81.1-01 and sonatype-word directory ls -l, both of them own by Root user and Root Group
If I want to run Nexus application with Nexus user, I need to change both of these directories permissions or ownership to nexus user : chown -R nexus:nexus nexus-3.81.1-01 and chown -R nexus:nexus sonatype-work
I will set Nexus configuration so that it will run as a Nexus user vim nexus-3.81.1-01/bin/nexus.rc I will set run_as_user="nexus"
Now I will switch to nexus user su - nexus
Then I will start Nexus /opt/nexus-3.81.1-01/bin/nexus start
Now I can check If nexus running ps aux | grep nexus
Now I can see nexus is running . And I can see the process id is 3478
Now I do netstat -ltpn I can see that process id 3478 is running on port 8081
That mean If I want to access Nexus from browser I have to open port 8081 in my Firewall
Once it configured I can access it from the browser <droplet-public-ip:8081>
After deployed Nexus we have auto generate admin password in /opt/sonatype-work/nexus3/admin.password with username is admin
Once Login Nexus will ask me to create my new password
I can have multiple Repositories of different formats like Helm Charts, Docker Images, Java Archive file, Javascript artifact, etc
By default I can see some Repostories automatically get created bcs properly its the most used one
Each Repository has a type (hosted, proxy, group):
Proxy Repository : This is a repository that is linked to a remote Reposiory . For example Maven Central Repositories is a remote public repository for Maven artifacts . Developers can download jar files, other artifacts and dependencies for application
-
If a component is requested from the Remote Repository by my application . For example : I am trying to download a library with a Specific Version from Maven Central, it will go through the Proxy instead of directly to remote and first check whether the component is available locally on Nexus, if yes my application that I am developing within the company will take it from Nexus . If not there yet the request will be forwarded to remote Repository . The component will then be retrived from remote and stored locally in Nexus . So Nexus will act as a cache .
-
Advantages : First of all it saves network bandwidth and time for retrieving the component from remote on every request . I just need to do it once
-
URL: http://143.198.134.109:8081/repository/maven-central/Which will use to access the repository and get some information of it -
Remote Storage : https://repo1.maven.org/maven2/This is a location of Maven Repository -
Another Advantage of using a proxy repositroy instead of fetching from the remote directly is that it is giving developer single endpoint instead of each developer team managing their own logic
Hotsed Repository: This is a Repository Primary Storage for artifacts and components
-
For example for a Company own components this is a typical use case where everything that is developed within the company, all the applications, the artifacts are being storted in Nexus hosted repository, so this is the primary location for these artifacts.
-
By default I have
maven-realeasesandmaven-snapshotsare Maven two repository formats and each one has the version policies integrated for releases and snapshot respectively and these repositories will be used by Maven or Gradle projects to store Java artifact -
The best practice is that when I am developing an Application in Java, I have this development version of my artifact, so I release a development version to be tested and that will be a Snapshot version and when I actually yest and it's ready to be released to Production, then I have actual releases and this will end in
maven-releasesrepository. So this way these 2 are separated, the name of version, the naming can also be different -
These repository types have integrated version policies for managing these artifact
-
Maven-releaseesis intended to be the repository, where you organization publishes internal releases, you can also use this repository for thrid-party component that are not available in puclic or external repositories and can therefore not be retrived via proxy repository . For examle it could be commercial library, some propriatary libraries such as Oracle, JDBC driver that maybe reference by my organization. So they are not externally available so I can create a proxy to fetch directly from them, but you can host it on thishosted repositoryas company internal component even though it not made by you company -
Maven-Snapshotis intended to be the repository, where your organization publishes internal development versions and these development version also called Snapshot
Group Repository
I may have multiple individualy repositories in NExus and each one has its own p 8000 urpose however if you need to use multiple of these repositories in my application, I don't want to configure endpoints of all of these different repositories, I want to address them using just 1 endpoint. And then have all these different repositories
This type allow me to combine multiple repositories and even other repository groups in a single repository
I click in the Create Repository I have ther list of different Repostory format and types I can choose from
Java Gradle App: (https://gitlab.com/twn-devops-bootcamp/latest/06-nexus/java-app)
Java Maven App: (https://gitlab.com/twn-devops-bootcamp/latest/06-nexus/java-maven-app)
I will upload a Jar file from Maven and Gradle Projects .
I will use a Maven hosted repository
Maven and Gradle have specific command to push to Repository
Before I can push to Repository I have to configure both tools to connect to Nexus (Nexus Repo URL + Credentials)
-
Relistic use cases is we won't give developers the admin username and password for our Nexus Repo Manager .
-
We will create Nexus User with permission to upload artifact file in certain repositories and then we will use that Nexus user to upload the jar
Go to Security -> Users -> Create Local User
Usally I would integrate User System using LDAP integration and then give it Nexus Permission
Create Roles for Nexus User : Go to Roles -> Create Roles -> Choose Nexus Role Type -> Perission is nx-repository-view-maven2-maven-snapshots-*
!!! Best Pratice when generally creating permission for users is to give the least necessary permission to a user to execute a task that is really needs to execute, bcs obivously we don't want user to have a lot of permissions
Now I can assign the Role to my User
Now I can use this user in my project to give Gradle and Maven credentials of this user to connect to our nexus
First thing I will add a plugin for publishing a Jar to Maven formatted repository : apply plugin: 'maven-publish'
!!! Note: Both Gradle and Maven Project use the same maven to format to upload the Java artifacts
This will enrich Gradle with ability to connect to Nexus and push to its Maven Repository
Now I will configure :
publishing {
publications {
maven(MavenPublication) {
artifact("build/libs/my-app-$version" + ".jar"){
extension 'jar'
}
}
}
repositories {
maven {
name 'nexus'
url "http://[nexus-ip]:[nexus port]/repository/maven-snapshots/"
allowInsecureProtocol = true
credentials {
username project.repoUser
password project.repoPassword
}
}
}
}
-
publicationsis a Jar file configuration that we will upload-
build/libs/my-app: This is a location of Artifact when we build the project, Gradle will actually create abuild/libsfolder and it will generate and put the JAR in there -
$versionVersion of the App
-
-
repositoriesis the Nexus Repository that we will upload that JAR file-
name: nexus: This is a name of the Repository Manager -
url: http://143.198.134.109:8081/repository/maven-snapshots/this is a URL of the Repository in my Repository Manager -
I will create
gradle.propertiesfile to storeusernameandpassword
-
gradle.properties
repoUser = tim
repoPassword = mypassword
In setting.gradles I have rootProject.name = my-app This will be used to generate the JAR file
- allowInsecureProtocol = true : Bcs we not access Nexus in
HTTPSthis will allow me to access the repositories usingHTTP
To build Gradle Application : gradle build
Once successed I should have a JAr file in build/lib
To upload JAR file I will execute gradle publish
- This command is not available by default . It from a
apply plugin: 'maven-publish'that we installed. And we have configured Gradle wirh all the information necessary so that publish command knows what to publish and where to publish that artifact to
Now I should see my Java application in Maven-Snapshot Repository Manager
In addition to a Jar file in there and I see a bunch of other files and metadata files and so on, which is the way NExus actually store components. So I uploaded 1 specific artifact which is JAR, bu some metadata and additional data and files get generated as information or addtional information for the jar file
With Maven we are also need to configure artifact and the address of the repository
Second part is configure the credentials so that Maven can actually upload the jar to Nexus using nexus user that we created
Configure Plugin that enable Maven to upload Jar file :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>3.1.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
</plugin>
This part will basically let me configure repository for my Snapshot.
<id>nexus-snapshots</id>: This id to identify the Repository if I have multiple of them configure. So we can reference which Repository from the List
<distributionManagement>
<snapshotRepository>
<id>nexus-snapshots</id>
<url>http://143.198.134.109:8081/repository/maven-snapshots/</url>
</snapshotRepository>
</distributionManagement>
Configure Credentials so Maven can also authenticate with it using nexus user
-
The way we configure user credentiasl for Repository in Maven is in
.m2folder in my user directory . This is a hidden folder in my home directory -
Inside I have
repositoryfolder which hold all the locally downloaded dependencies of my Maven projects -
And in
.m2folder I will create a filesettings.xml. This is a file where Maven Global credentials can be defined . This will be accessible for all my Maven projects
<settings>
<servers>
<server>
<id>nexus-snapshots</id>
<username>tim</username>
<password>My-password</password>
</server>
</servers>
</settings>
-
<settings>list of settings here and servers is bascially credentials to any server that Maven has to authenticate to -
<id>nexus-snapshots</id>the same id that I configure above -
Now I have everything ready to upload to Nexus Repository
Move to Java-Maven folder : cd Java-Maven
Build the Jar: mvn package
Once it success I can see a target folder with a Jar Artifact
To upload the Jar mvn deploy .
Once it success I can see my Maven Artifact in Nexus Repository
We need Nexus rest API endpoint to query Nexus repositories for different information like :
-
Which component are avaialbe
-
What are the version
-
Which repositoies are avaialble and so on
-
Query for different information:
-
Download an artifact
-
Upload an artifact
-
List of repositories list of component
I would need those information in CI/CD pipelines where I need to fetch the information about the available versions that maybe user can select to deploy on staging environment or production environment and so on
To access REST Endpoint :
-
Use tools like
curlorwgetto execute http request -
I also need to provide Credentiasl of Nexus User
-
Use the Nexus user with the required permissions
For example : curl -u user:pwd -X GET 'http://143.198.134.109:8081/service/rest/v1/repositories'
/service/rest/v1/repositories: This is an endpoint
Once it executed I can see a list of Repository that user have permission to fetch
If I execute with admin user I can see a whole list of repositories
Another query is listing all the components in a certain repository : curl -u user:pwd -X GET 'http://143.198.134.109:8081/service/rest/v1/components?repository=maven-snapshots'
- From this endpoint I get all a items that are in snapshots repository
To use Nexus API or any API: I should always reference the documentation. Not learn by heart, because
1: API's change
2: there are a lot of endpoints
Nexus must have some kind of storage configured to store all these assets and components
Nexus is using Blob stores as a storage for its components
Blob Store is what Nexus is using to manage the storage per repository for all of its components
How does it work?
Blob Store is internal storage mechanism for binary parts of artifacts
And blob store can be on a local file on the Server where Nexus is deployed
Or it could be a Cloud based storage like Amazon S3 .
Each Blob Store Can be use by one or multiple repositories and repository group
In my Server where I deploy Nexus ls /opt/sonatype-work/nexus3/blobs . This is a file storage that keep alot of different information about Nexus like configuration and plugins etc ... And this is where blob store storage is configured so all the components assets and so on will be store here
In /opt/sonatype-work/nexus3/blobs I have a default folder that automatic configured
In that default folder I have all the data in the content folder
Type field = Storage Backend
File represent File system-based storage
S3 allow Blobs to be stored in AWS S3 Cloud Storage
State filed = State of the Blob store
Started : Indicate it is running as expected
failed: Indicate configuration issue
If I look in the content folder the Data is actually split up in different volumes or blobs and this is number of blobs that is currently stored
So the components and the artifacts files that we upload are split into this way to be store in the Blob store
The Type can be File or S3
Path parameter should be an absolute path /opt/sonatype-work/nexus3/blobs/
Some things to consider
Blob store can't be modified
Blob store used by Repository can't be deleted
I need to decide carefully how many blob stores I create with which sizes and which one will use for Repository
The reasone for that is Once a repository is allocated to a blob store it is there permanently . Blob store can be moved from one storage device to another for example to a larger storage device If I need more space I can do it manually but blob stores can not be split and also one repository cannot use multiple blob stores
Component
Abstract, High Level definition
What we are uploading
Term "component" refers to any type or format (Docker Image, Zip, Jar)
Assets
Actual physcial package/files
1 component = 1 or more assets
Clean up polices help me decide rules that will clean up artifacts, the components from the repository, either when they are too old, or they haven't been used for a long time
Clean up old artifacts to make space. You can define logic like:
-
Delete all artifacts older than 30 days
-
Delete all artifacts, which haven't been used/downloaded for more than 10 days
-
You can create individual cleanup policy for each repository
I go to Repositorys -> Cleanup Policies -> Create Cleanup Policy
I can choose Format: maven2
I can choose Release Type :
- I can modify to clean up
Release And Pre-Release/Snapshots, orRelease, orPre-Release/Snapshot
I can configure Cleanup Criteria
I can choose Preview Reposiotry before I actually configure that policy
Now I have create Cleanup Policy .
To assign Cleanup Policy to Repository go To Repository -> Maven-Snapshot (bcs I want to associate with Maven-snapshot repositories)
At the end I have a cleanup policy I can apply that one to the repository
When will this cleanup policy work ?
Go to System -> Tasks -> I can see cleanup Service Task get created in waiting status
I can create new task manually
!!! Note : Important thing with Cleanup policies and cleaning up components from a repository is that they will not acutally get deleted . They will be marked for deletion (Soft Delete).
If we want to acutally delete those Item in order to make space we would need to Compact the blob store by using Admin - Compact Blob store task