compliance-backend is a project meant to parse OpenSCAP reports into a database, and perform all kind of actions that will make your systems more compliant with a policy. For example, you should be able to generate reports of all kinds for your auditors, get alerts, and create playbooks to fix your hosts.
This project does two main things:
- Serve as the API backend for the web UI compliance-frontend and for other consumers,
- Connect to a Kafka message queue provided by the Insights Platform.
The Insights Compliance backend comprises of these components/services:
- Rails web server — serving REST API (port 3000)
- Sidekiq — job runner connected through Redis (see app/jobs)
- Inventory Consumer (karafka) — processor of Kafka messages, mainly to process and parse reports
Before running the project, these services must be running and acessible:
- Kafka — message broker (default port 29092)
- set by
SETTINGS__KAFKA__BROKERSenvironment vari 8000 able
- set by
- Redis — Job queue and cache
- PostgreSQL compatible database
DATABASE_SERVICE_NAME=postgres- conrolled by environment variables
POSTGRES_SERVICE_HOST,POSTGRESQL_DATABASE,POSTGRESQL_USER,POSTGRESQL_PASSWORD
- Insights Ingress (also requires S3/minio)
- Insights PUPTOO — Platform Upload Processor
- Insights Host Inventory (MQ service and web service)
- Insights RBAC (Role-based access control)
Let's examine how to run the project:
Option 1: Ephemeral Environment with Bonfire
-
Log in into the ephemeral cluster
-
Set up bonfire configuration (optional)
To use local deployement confguration update the
~/.config/bonfire/config.yamlas follows:apps: - name: compliance components: - name: compliance host: local repo: ~/path/to/local/compliance-backend path: deploy/clowdapp.yaml parameters: REDIS_SSL: 'false'
bonfire deploy compliance --optional-deps-method hybrid --frontends true --source=appsre --ref-env insights-stage --timeout 900This will set up the environment with all service dependencies, 3scale gateway, frontend and platform mocks (authentication & authorization).
A custom (local) clowdapp can be used for deployment if the step 2 of the setup has been not skipped. A custom image can be used by overwriting parameters of the clowder template. Note that the container image needs to be pushed to an accessible location.
bonfire deploy compliance ... -p compliance/IMAGE=quay.io/me/myimage -p compliance/IMAGE_TAG=mytagThe frontend route and credentials can be retrieved by calling the following command:
bonfire namespace describe <ephemeral-######>The first step is to copy over the .env file from .env.example and modify the values as needed:
cp .env.example .envEither podman-compose or docker compose should work. Note that 1.0.6 and newer versions of podman-compose handle multi-level dependencies poorly which might cause problems with startup. Furthermore podman-compose does not support exec, so podman commands must be run manually against the running container, as demonstrated:
# docker
docker compose exec rails bash
# podman
podman exec compliance-backend_rails_1 bashBring up the everything, including inventory, ingress, etc.:
docker compose upAccess the rails console:
docker compose exec rails bundle exec rails consoleAttach the container to the current terminal:
docker attach compliance-backend_rails_1Debug with pry-remote:
docker compose exec rails pry-remote -wRun the tests:
# run all tests (same thing run on PRs to master)
docker compose exec rails bundle exec rake test:validate
# run a single test file
docker compose exec -e TEST=$TEST rails bundle exec rake test TEST=test/models/host_test.rb
# run a specific Rspec test
podman-compose exec -e SPEC_OPTS="-e 'V2::ReportsController /reports GET index' --color --tty --format documentation" rails bundle exec rake spec
# run Rspec tests in a specific directory (doesn't work with tests including shared_examples)
podman-compose exec -e SPEC_OPTS="-P 'spec/policies/v2/*.rb' --color --tty --format documentation" rails bundle exec rake spec
# run a single test case
docker compose exec rails bundle exec rake test TEST=test/path/to_test.rb TESTOPTS="-n '/matching test description/'"Access logs: note: podman-compose does not support the logs command, so similar to exec, it must be run against the container itself, as shown
docker compose logs -f sidekiq
# podman
podman logs -f compliance-backend_sidekiq_1This project uses build_deploy.sh to build and push this image to a remote registry.
You must set the required variables to provide authentication to the required registries.
You must set at least the Red Hat Registry credentials to be able to pull the base image.
Example:
LOCAL_BUILD=true RH_REGISTRY_USER=guybrush RH_REGISTRY_TOKEN=M0nk3y ./build_deploy.sh
Optionally, if you want to push to the remote registry you must set the remote registry credentials as well (LOCAL_BUILD defaults to false)
Example:
LOCAL_BUILD=true RH_REGISTRY_USER=guybrush RH_REGISTRY_TOKEN=monkey QUAY_USER=lechuck QUAY_TOKEN=Ela1ne ./build_deploy.sh
To seed accounts, policies, results and hosts use dev:db:seed rake task.
It is required to wait for the SSG content to be synchronized. Watch compliance-backend_import-ssg_1 container to see the synchronization progress.
It is recommeneded to run the command after a first log in was initiated, as it would generate data for that account.
bundle exec rake dev:db:seed
To create hosts in the inventory the kafka_producer.py script can be used from the inventory container:
docker compose run -e NUM_HOSTS=1000 -e INVENTORY_HOST_ACCOUNT=00001 inventory-web bash -c 'pipenv install --system --dev; python3 ./utils/kafka_producer.py;'
If there is a tags column defined in any model, it always should be a jsonb column and follow the structured representation of tags described in Insights, i.e. an array of hashes. If this convention is not kept, the controllers might break when a user tries to pass the tags attribute to a GET request.
The API documentation can be found at /api/compliance and you may access the raw OpenAPI
definition here.
Docs can be updated by running:
rake rswag:specs:swaggerizeIf you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.
This project ensures code style guidelines are followed on every pull request
using Rubocop.
Format for git commit message is described in .commitlint.yml.
The code in this project is licensed under GPL v3 license.