Description
Hi,
In the context of #18263, and following discussions in #18319 and #18271, this is an issue to summarize the work we have been doing with workspaces tooling since Conan v1. Maybe it can be helpful for further discussions or tools development.
We are interested in the approach of a monolithic project that we could compose of any number of repositories. This use-case, as well as the motivations and overarching infrastructure, have been described in this document. Sadly it never gained traction, and I did not update it to reflect our current Conan 2 infrastructure (we lost the custom CMake variables, etc.).
An important note at this point is that recipe references matching commits on the develop branch (which are in practice the recipe references we most often use during development) are not semver: they use the 10 first characters of the commit hash for a version.
To support our use of Conan v1 workspace, we developed Python tooling named SPH. This also served as a pretext to implement a Python graphics lib for the CLI, but here are the main workflow complications we intended to address:
- Keeping the recipe references in the workspace file, the conanfiles, and the individual repositories, in sync
- Find dirty repositories among the workspace editables (requirement for pushing a consistent state)
- Manage the order of pushes and recipe references updates when a new state of the "workspace" is pushed.
Regarding point 1, I just opened issue #18329, so maybe we could address this complication at Conan level directly.
Regarding point 3, the issue is that since a recipe reference often use a shortened commit hash, it implies that to update the references in downstreams, the upstream must have successfully gone through CI first. Even with simple dependency graphs, this creates a long sequential chain of step, conducted as a lot of repetitive cycles, made worse by the wait for CI completion:
- Starting from up-most dependency, and for each repository in turn:
- edit references of upstreams in both conanfile and workspace file
- commit & push, wait for CI
- in case of CI success, get the repo commit hash as recipe reference
- propagate this new reference to downstreams, starting a new cycle
Sadly, this automation never worked well enough, and we ended up doing that mostly manually anyway.