Open
Description
We need system tests for ramenctl to detect regressions early. We can use the ramen e2e job to test also ramenctl.
ramenctl has several advantages:
- better progress
- machine readable test results that can be used for analyzing tests results
- automatic gathering on failures so we don't need to gather every run
- separating cleanup from test flow
Issues:
- ramenctl uses specific ramen/e2e version, not the current version. Can be solved by building ramenctl with local ramen checkout.
- When making incompatible e2e API changes we cannot test ramenctl without modifying it so we cannot use it for all cases.
Maybe add a daily ramenctl job running ramenctl with ramen main, or add another e2e job in ramenctl CI, using ramen main.
Related to RamenDR/ramenctl#15