Description
This is an umbrella issue for the high-level goal of being able to automatically run (integration) tests so we can build new releases and deploy with confidence.
Next steps
- Fix the core unit tests #103
- Fix the api unit tests #104
- Figure out a way to run the core & api unit tests as a GitHub Action #105
- Create a run script to deploy one or more testing VMs via terraform #106
- Create Ansible playbooks to connect to these testing VMs and run tests, plus report back to the pull request the triggered them #107
- Create a run script to destroy/clean up/remove the testing VMs #108
History
When we initially started working on Morio, we created unit tests for both the core and api REST APIs.
Running these tests was always a bit of a hassle because these services run in containers that (in the case of core) spawn other containers, but we made it work, somewhat.
Things got more complicated once we added support for clustering. For starters, the core service now has 3 main tasks:
- Resolve the configuration, make sure everything is as we want it
- Run the core REST API
- Handle inter-node cluster communication and keep the cluster state
The core REST API unit tests really only cover the second item on that list, with a bit of the first.
Testing the code that handles the clustering part is non-trivial, and mocking all of the moving parts seems like it would be a lot.
So, we kicked the can down the road thinking we would run unit tests for what we can test locally, and run integration tests for the rest.
Goals
Unit tests
We should not abondon unit tests altogether. They are still the fastest feedback loop available for developers to know that their code does what it should and, crucially, that a change does not introduce a regression.
Currently, the unit tests for both the core and api REST APIs lag behind changes. Which is not so surprising since we're still in alpha.
We should get this back up and running and, crucially, we should figure out how we can run these unit tests in CI.
However, GitHub actions won't grant access to the docker socket, so we need to find a workaround for this. Either mocking everything around core, or perhaps we can bring our own runner where we can map the docker socket.
Integration tests
To get closer to a real world scenario, we also need to run end-to-end test.
I'm open to suggestions, but I imaging something like this:
Single node
- Trigger happens (we should be able to run this manually, or ideally from a pull request that has a specific label or comment)
- A new VM is deployed on AWS (For this, I think terraform is the obvious choice)
- Morio is installed on it from the build artifacts (from the
.debpackage) (perhaps ansible for this?) - We run one or many ansible playbooks to test what needs testing
- We report back the results of the tests
- We destroy the new VM that was created
Clustering
- We should be able to run all the steps above for a 3, 5, 7, and 9-node cluster
- While we probably want to test things on a 3-node cluster, we should at least have the option to test on different cluster sizes too, even if we don't do it every time.