8000 Add proposal to deal with multiple build-systems using directives by alalazo · Pull Request #3 · spack/seps · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Add proposal to deal with multiple build-systems using directives #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions seps/sep-0002.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Support multiple build-systems per package using a directive

## The problem

There are multiple packages that are either changing their build-system during the evolution of the project, or using different build-systems for different platforms. Spack, at the moment, provides no support to model this use case. As a result we wrote some [documentation](https://github.com/spack/spack/pull/25174) to enumerate various workarounds that people adopted - each with its own drawbacks. What we would like to have in the long term is proper support for packages that can be built using multiple build-systems.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using different build-systems for different platforms

This is a very good point that I didn't think about while writing spack/spack#25174, glad to see more core developers thinking about Windows support!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very much so! Consider spack/spack@5818a54, are we going to turn all ./configure-based packages into CMake packages across all platforms just to support Visual Studio? I really hope not... zlib is a teeny tiny package that should be able to build with 0 other deps than a shell and a compiler.


## Proposed changes

What we propose here to add support for packages using multiple build-systems is to __add another directive__ to the package DSL. If we take as an example `hdf5` this would look like:
```python
class Hdf5(Package):
build_system('cmake', when='@X.Y:')
build_system('autotools')
```
Each build-system can be subject to a constraint. In the example above, for instance, `cmake` can be used only if the version of `hdf5` is greater or equal than `X.Y` (while `autotools` is always available).

We can also assume that there is an __implicit preference order__ based on on the order of declaration in the class. In the example above this means that, if no other requests are made by the user, `cmake` is preferred to `autotools` when available.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this, but this "implicit preference order based on the order of declaration in the class" is different than all other directives in Spack, so it may be good to make this very clear in the docs.


Build systems can be specified in specs using a key-value pair:
```
hdf5 build_system=cmake
```
__which requires the keyword `build_system` to be reserved by Spack__ (similarly to what is done for `dev_path`). It is interesting to note that modeling build-systems this way make them similar to signle-valued variants with the only difference that the set of allowed values is "dynamic" and subject to constraints.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
__which requires the keyword `build_system` to be reserved by Spack__ (similarly to what is done for `dev_path`). It is interesting to note that modeling build-systems this way make them similar to signle-valued variants with the only difference that the set of allowed values is "dynamic" and subject to constraints.
__which requires the keyword `build_system` to be reserved by Spack__ (similarly to what is done for `dev_path`). It is interesting to note that modeling build-systems this way make them similar to single-valued variants with the only difference that the set of allowed values is "dynamic" and subject to constraints.

A lot of users have been asking for this for swapping between Ninja and Makefile in CMakePackage. Would you consider Ninja vs. Makefile to be a build system? We could do build_system=cmake_ninja or build_system=cmake_makefile. Or it could be a different reserved keyword.

Copy link
Member Author
@alalazo alalazo Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to think a bit about this. A solution like build_system=cmake_ninja does not require any change in the proposed semantics, but we are effectively splitting the cmake build-system into many build-systems based on the generator we use.

Using hdf5 build_system=cmake generator=ninjaseems more appealing, but we need the ability to attach specific variants based on the build-system being used. That may require the "conditional" variant PR from @becker33:

variant('generator', values=('makefile', 'ninja'), default='makefile', when='build_system=cmake')


__Each package needs to have one and only one build-system__.

### Overriding build-system phases

Some packages may need to override phases of the build-systems they use and there must be a way to specify which phase for which build-system is overridden. What we propose to use here is a decorator to distinguish among multiple build-systems:
```python
class Hdf5(Package):
build_system('cmake', when='@X.Y:')
build_system('autotools')

@cmake
def build(self, spec, prefix):
pass

@autotools
def build(self, spec, prefix):
pass
```
Using a decorator permits seamlessly to override phases with the same name.

## Implementation at a high-level

Currently the build-system specific methods are grouped in base classes and each package uses them via inheritance. There's a fair amount of machinery with metaclasses that is used to have dynamic phases for each build-system.

To support multiple build-systems we should probably add, as it happens in other directives, a dictionary of builder objects:
```python
hdf5.build_system = {
'cmake': cmake_builder_obj,
'autotools': autotools_builder_obj
}
```

Each builder object will accept a _package object tied to a concrete spec_ as argument and contain all the methods that are relevant for building it.
8000
The main idea is to:
1. Move things like `phases` and other similar methods into a class of their own
2. Use a different object for each package so that the package can override these methods if need be
3. Have decorators that can monkey patch the object to customize the install phases

Dipatching to the correct build system object happens once we have a concrete spec associated with the package (since the spec coveys information about the build system, we know which one to select).

### Backward compatibility

Most of the packages are currently fine with a single build-system and they inherit almost all of the related methods from a base class:

```python
class OpenJpeg(CmakePackage):
pass
```

To be backward compatible, if we push the methods and attributes down to a "builder" class, we can maintain the directives in the same base class:
```python
class CMakePackage(PackageBase):
# Can be overridden if conditional in derived classes
build_system('cmake')
depends_on('cmake', type='build')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I write a package that supports both CMake and Autotools, will I have to manually add:

depends_on('cmake', when='build_system=cmake', type='build')

or can that be included in CMakeBuildSystem? Is the build_system=cmake even available in the spec?

Copy link
Member Author
@alalazo alalazo Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the build_system=cmake even available in the spec?

Yes, that is available in the spec. There are legitimate use cases where you want to use it in recipes too, e.g. for bootstrapping tools. Think for instance foo can build with cmake and makefile and is a cmake dependency. In CMake's recipe you probably want to specify:

depends_on('foo build_system=makefile')

to avoid circular dependencies.

If I write a package that supports both CMake and Autotools, will I have to manually add: [ ... ]

I think that should be implied and something similar needs to be included only if there are additional constraints to be met e.g. on the version of cmake.


class CMakeBuildSystem(object):
phases = [
'cmake',
'build',
'install'
]

@staticmethod
def cmake(pkg, spec, prefix):
pass

@staticmethod
def setup_build_environment(pkg, spec, prefix):
pass
```
When we construct the package object we can check the methods defined on the package to see if there are overrides for the build-system being used, without the
need to employ a decorator.

We can special case this behavior to packages where there's only a single, unconstrained build-system and otherwise require the use of decorators for more clarity.
0