-
Notifications
You must be signed in to change notification settings - Fork 4
Add proposal to deal with multiple build-systems using directives #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,100 @@ | ||||||
# Support multiple build-systems per package using a directive | ||||||
|
||||||
## The problem | ||||||
|
||||||
There are multiple packages that are either changing their build-system during the evolution of the project, or using different build-systems for different platforms. Spack, at the moment, provides no support to model this use case. As a result we wrote some [documentation](https://github.com/spack/spack/pull/25174) to enumerate various workarounds that people adopted - each with its own drawbacks. What we would like to have in the long term is proper support for packages that can be built using multiple build-systems. | ||||||
|
||||||
## Proposed changes | ||||||
|
||||||
What we propose here to add support for packages using multiple build-systems is to __add another directive__ to the package DSL. If we take as an example `hdf5` this would look like: | ||||||
```python | ||||||
class Hdf5(Package): | ||||||
build_system('cmake', when='@X.Y:') | ||||||
build_system('autotools') | ||||||
``` | ||||||
Each build-system can be subject to a constraint. In the example above, for instance, `cmake` can be used only if the version of `hdf5` is greater or equal than `X.Y` (while `autotools` is always available). | ||||||
|
||||||
We can also assume that there is an __implicit preference order__ based on on the order of declaration in the class. In the example above this means that, if no other requests are made by the user, `cmake` is preferred to `autotools` when available. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree with this, but this "implicit preference order based on the order of declaration in the class" is different than all other directives in Spack, so it may be good to make this very clear in the docs. |
||||||
|
||||||
Build systems can be specified in specs using a key-value pair: | ||||||
``` | ||||||
hdf5 build_system=cmake | ||||||
``` | ||||||
__which requires the keyword `build_system` to be reserved by Spack__ (similarly to what is done for `dev_path`). It is interesting to note that modeling build-systems this way make them similar to signle-valued variants with the only difference that the set of allowed values is "dynamic" and subject to constraints. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
A lot of users have been asking for this for swapping between Ninja and Makefile in CMakePackage. Would you consider Ninja vs. Makefile to be a build system? We could do There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have to think a bit about this. A solution like Using variant('generator', values=('makefile', 'ninja'), default='makefile', when='build_system=cmake') |
||||||
|
||||||
__Each package needs to have one and only one build-system__. | ||||||
|
||||||
### Overriding build-system phases | ||||||
|
||||||
Some packages may need to override phases of the build-systems they use and there must be a way to specify which phase for which build-system is overridden. What we propose to use here is a decorator to distinguish among multiple build-systems: | ||||||
```python | ||||||
class Hdf5(Package): | ||||||
build_system('cmake', when='@X.Y:') | ||||||
build_system('autotools') | ||||||
|
||||||
@cmake | ||||||
def build(self, spec, prefix): | ||||||
pass | ||||||
|
||||||
@autotools | ||||||
def build(self, spec, prefix): | ||||||
pass | ||||||
``` | ||||||
Using a decorator permits seamlessly to override phases with the same name. | ||||||
|
||||||
## Implementation at a high-level | ||||||
|
||||||
Currently the build-system specific methods are grouped in base classes and each package uses them via inheritance. There's a fair amount of machinery with metaclasses that is used to have dynamic phases for each build-system. | ||||||
|
||||||
To support multiple build-systems we should probably add, as it happens in other directives, a dictionary of builder objects: | ||||||
```python | ||||||
hdf5.build_system = { | ||||||
'cmake': cmake_builder_obj, | ||||||
'autotools': autotools_builder_obj | ||||||
} | ||||||
``` | ||||||
|
||||||
Each builder object will accept a _package object tied to a concrete spec_ as argument and contain all the methods that are relevant for building it. | ||||||
8000 |
|
|||||
The main idea is to: | ||||||
1. Move things like `phases` and other similar methods into a class of their own | ||||||
2. Use a different object for each package so that the package can override these methods if need be | ||||||
3. Have decorators that can monkey patch the object to customize the install phases | ||||||
|
||||||
Dipatching to the correct build system object happens once we have a concrete spec associated with the package (since the spec coveys information about the build system, we know which one to select). | ||||||
|
||||||
### Backward compatibility | ||||||
|
||||||
Most of the packages are currently fine with a single build-system and they inherit almost all of the related methods from a base class: | ||||||
|
||||||
```python | ||||||
class OpenJpeg(CmakePackage): | ||||||
pass | ||||||
``` | ||||||
|
||||||
To be backward compatible, if we push the methods and attributes down to a "builder" class, we can maintain the directives in the same base class: | ||||||
```python | ||||||
class CMakePackage(PackageBase): | ||||||
# Can be overridden if conditional in derived classes | ||||||
build_system('cmake') | ||||||
depends_on('cmake', type='build') | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If I write a package that supports both CMake and Autotools, will I have to manually add: depends_on('cmake', when='build_system=cmake', type='build') or can that be included in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes, that is available in the spec. There are legitimate use cases where you want to use it in recipes too, e.g. for bootstrapping tools. Think for instance depends_on('foo build_system=makefile') to avoid circular dependencies.
I think that should be implied and something similar needs to be included only if there are additional constraints to be met e.g. on the version of |
||||||
|
||||||
class CMakeBuildSystem(object): | ||||||
phases = [ | ||||||
'cmake', | ||||||
'build', | ||||||
'install' | ||||||
] | ||||||
|
||||||
@staticmethod | ||||||
def cmake(pkg, spec, prefix): | ||||||
pass | ||||||
|
||||||
@staticmethod | ||||||
def setup_build_environment(pkg, spec, prefix): | ||||||
pass | ||||||
``` | ||||||
When we construct the package object we can check the methods defined on the package to see if there are overrides for the build-system being used, without the | ||||||
need to employ a decorator. | ||||||
|
||||||
We can special case this behavior to packages where there's only a single, unconstrained build-system and otherwise require the use of decorators for more clarity. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a very good point that I didn't think about while writing spack/spack#25174, glad to see more core developers thinking about Windows support!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very much so! Consider spack/spack@5818a54, are we going to turn all
./configure
-based packages into CMake packages across all platforms just to support Visual Studio? I really hope not...zlib
is a teeny tiny package that should be able to build with 0 other deps than a shell and a compiler.