CEP for the next evolution of Repodata (v2) and MatchSpec #111

wolfv · 2025-02-05T12:21:27Z

This defines new repodata with two additional fields and a modfied matchspec syntax:

extra (optional) dependency sets that can be activated by the user or other packages
conditional dependencies to be activated on a specific platform, or when other package versions are also pulled in (e.g. require six; if python <3.8)
flags to make it easier to select variants confidently (instead of globbing the build string)

Some initial discussion happened over the past days on:

TODO:

specify that we need to understand full-blown matchspecs in the repodata.v2 (e.g. with bracket syntax), because we want other packages to be able to request extras or flags).
Decide whether flag negation should be ~foo, !foo, or -foo.
Decide if we need to add a default flags entry to the repodata record, or another way to mark the default variant (can also still use down-priorization via track records)
Investigate ordering of flags and whether we should have some weighing e.g. for numerical flags (higher == better)? Or wether this should be decided by another order integer that the users can compute.

wolfv · 2025-02-05T12:31:10Z

Here are comments from the HackMD:

Should we consider a system like Spack has (https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies) or maybe Conan (https://docs.conan.io/2/reference/conanfile/attributes.html#options)?
Additionally, how would a package declare which flag is the default? (Edited)

For default flags I think we might want to deprecate the track_features field and instead add a priority: integer field that can hold any value. The user could mark the default this way. We could also use the inverse of priority, so that higher values are less desirable.

Or, maybe a better thought: we could also make it so that if priority: -1 we never enable this package, except if we request its flags directly.

Regarding Spack and Conan: I would be super hyped if you can distill this on how it could work for the conda ecosystem. I haven't had the time yet to read into it deeply. I am not a fan of super arbitrary code execution though.

mariusvniekerk · 2025-02-20T15:00:48Z

cep-repodata-next.md

+- `release`: only packages with `release` flag are used
+- `~release`: disallow packages with `release` flag
+- `?release`: if release flag available, filter on it, otherwise use any other
+- `gpu:*`: any flag starting with `gpu:` will be matched


can you add an example for the string matching for say blas:mkl

Would an exact match not work fine?

It would, we should just state how to do an exact match

You coudl just ask for flags = ["blas:mkl"] for an exact match :)

jaimergp · 2025-03-30T13:54:18Z

cep-repodata-next.md

+
+The proposed syntax is to extend the `MatchSpec` syntax by appending `; if <CONDITION>` after the current MatchSpec.
+
+We would like to also allow for AND and OR with the following syntax:


You probably need NOT and parentheses for precedence overrides, right?

In regular MatchSepc, we have , and | used for versions for and and or. I think we should keep thing similar, even if it means supporting and and or in versions.

If find this hard to distinguish when parsing the version. E.g.

python <3.8|>3.9 | numpy >=2.0 python <3.8|>3.9 or numpy >=2.0

I find the version with or easier to read than the one with pipes.

jaimergp · 2025-03-30T14:02:49Z

cep-repodata-next.md

+ - six; if python <3.8
+```
+
+The proposed syntax is to extend the `MatchSpec` syntax by appending `; if <CONDITION>` after the current MatchSpec.


It would be nice if we didn't need the ; because somehow conda will happily ignore the if parts while parsing MatchSpecs.

>>> from conda.models.match_spec import MatchSpec as M >>> M("python 3.8 * if python") MatchSpec("python==3.8[build=*]") >>> M("python 3.8 'if' if __win") # quote 'if' to force parse it as a build string MatchSpec("python==3.8='if'")

It will also ignore parenthesised blocks:

>>> M("python 3.8 * (__win)") MatchSpec("python==3.8[build=*]") >>> M("python 3.8 (__win)") MatchSpec("python==3.8") >>> M("python 3.8 (__win and __osx)") MatchSpec("python==3.8") >>> M("python 3.8 (if __win and __osx)") MatchSpec("python==3.8")

libmamba also ignores parentheses:

>>> from libmambapy.specs import MatchSpec as LibmambaMatchSpec >>>print(LibmambaMatchSpec.parse("python 3.8 * (__win and __osx)")) python==3.8 >>> print(LibmambaMatchSpec.parse("python 3.8 * (if __win and __osx)")) python==3.8

My suggestion would be to design the syntax like this:

name [version [build]] ('if' condition)

The if literal could be omitted, or replaced with with, if folks feel it's clearer that way. See discussion in #conda-maintainers > Conditional dependencies syntax in v2 environments & recipes @ 💬.

Another idea (not sure if a good one) could be: rather than make MatchSpecs more complex than they already are, build on the idea of selectors from the new recipe format and allow depends to contain objects like so:

depends: - python >=3.8 - if: python <3.8 then: six

In the recipe format, this is expressed via a variation on the selector to avoid being process at build time (if(run): for instance).

This disallow basically disallow conditionals outside of recipes (or format with selector), but makes a more consistent narrative around conditionals.

The idea was indeed also posted on zulip. One issue is that it then becomes harder for cli tools to adopt.

I propose we use the when syntax without a semicolon to force an error on older versions of conda that dont support it and the distinguish between the already established if syntax in recipe v1. We can use the same keyword in a more expanded form as the "build spec"

foobar when python >=3.8

E.g. in a recipe:

- if: unix then: foobar when: python >=3.8 # OR - when: python >=3.8 then: foobar # OR - "foobar when python >=3.8"

AntoinePrv · 2025-04-15T08:49:35Z

cep-repodata-next.md

+
+## Conditional dependencies
+
+Conditional dependencies are activated when the condition is true. The most straight-forward conditions are `__unix`, `__win` and other platform specifiers. However, we would also like to support matchspecs in conditions such as `python >=3`.


I very much like this idea. An implementation note: while if __unix is reasonably easy to implement because it is "static", if python <3.8 is conceptually much harder as it is not something that can be decided ahead of solving. I requires to be able to adapt the dependencies of a package as partial candidates are investigated during solve.

Yep, this is a form of boolean dependencies in rpm (already supported by libsolv I believe)

This is the PR that implements this for resolvo: prefix-dev/resolvo#136

AntoinePrv · 2025-04-15T08:59:56Z

cep-repodata-next.md

+
+However, it would be nice if we could have a flexible, powerful and simple syntax to enable or disable "flags" on packages in order to select a variant.
+
+A RepodataRecord should get a new field "flags" that is a list of strings, such as:


How do flags and extra mix and overlap? Wouldn't conditional dependencies on a flag be enough to generate the extra category?

Flags are part of a variant. So there is no variation of flags for a single variant. E.g. flags could be used to say a particular variant is using cuda and another is used to target cpu. Extras are a way to select additional dependencies for a particular variant. If a variant also adds a CLI tool it provides the extra "cli". Only if that extra is requested by another package are particular dependencies also requested.

In technical terms extras can indeed be implemented as conditional dependencies. E.g. for a package my_package we could express it as typer when my_package[cli]. If there is a package that depends on my_package[cli] typer would also be required.

baszalmstra · 2025-05-13T11:31:49Z

cep-repodata-next.md

+
+The proposed syntax is to extend the `MatchSpec` syntax by appending `; if <CONDITION>` after the current MatchSpec.
+
+We would like to also allow for AND and OR with the following syntax:


If find this hard to distinguish when parsing the version. E.g.

python <3.8|>3.9 | numpy >=2.0 python <3.8|>3.9 or numpy >=2.0

I find the version with or easier to read than the one with pipes.

baszalmstra · 2025-05-13T11:35:52Z

cep-repodata-next.md

+ - six; if python <3.8
+```
+
+The proposed syntax is to extend the `MatchSpec` syntax by appending `; if <CONDITION>` after the current MatchSpec.


I propose we use the when syntax without a semicolon to force an error on older versions of conda that dont support it and the distinguish between the already established if syntax in recipe v1. We can use the same keyword in a more expanded form as the "build spec"

foobar when python >=3.8

E.g. in a recipe:

- if: unix then: foobar when: python >=3.8 # OR - when: python >=3.8 then: foobar # OR - "foobar when python >=3.8"

baszalmstra · 2025-05-13T11:42:48Z

cep-repodata-next.md

+
+However, it would be nice if we could have a flexible, powerful and simple syntax to enable or disable "flags" on packages in order to select a variant.
+
+A RepodataRecord should get a new field "flags" that is a list of strings, such as:


Flags are part of a variant. So there is no variation of flags for a single variant. E.g. flags could be used to say a particular variant is using cuda and another is used to target cpu. Extras are a way to select additional dependencies for a particular variant. If a variant also adds a CLI tool it provides the extra "cli". Only if that extra is requested by another package are particular dependencies also requested.

In technical terms extras can indeed be implemented as conditional dependencies. E.g. for a package my_package we could express it as typer when my_package[cli]. If there is a package that depends on my_package[cli] typer would also be required.

baszalmstra · 2025-05-13T11:43:36Z

cep-repodata-next.md

+
+## Conditional dependencies
+
+Conditional dependencies are activated when the condition is true. The most straight-forward conditions are `__unix`, `__win` and other platform specifiers. However, we would also like to support matchspecs in conditions such as `python >=3`.


This is the PR that implements this for resolvo: prefix-dev/resolvo#136

baszalmstra · 2025-05-13T11:45:18Z

cep-repodata-next.md

+    - pyxpgres >=8
+```
+
+When a user, or a dependency, selects an extra through a MatchSpec, the extra and it's dependencies are "activated". This is conceptually the same as having three packages with "exact" dependencies from the "extra" to the base package: `sqlalchemy`, `sqlalchemy-sqlite` and `sqlalchemy-postgres` – which is the workaround currently employed by a number of packages on conda-forge.


I think we should clarify what happens if an extra is requested for a package but the selected variant doesnt provide that extra. E.g. what happens if I depend on a foobar[extras=["doesntexist"]]

Maybe I missed it but its also not defined how to depend on an extra?

repodata-next

3a88a3e

wolfv mentioned this pull request Feb 5, 2025

feat: implement flags conda/rattler#1040

Open

wolfv mentioned this pull request Feb 8, 2025

Optional groups of dependencies conda/conda#7502

Open

1 task

mariusvniekerk reviewed Feb 20, 2025

View reviewed changes

traversaro mentioned this pull request Mar 26, 2025

DEV: pin CUDA variant for PyTorch data-apis/array-api-extra#186

Merged

jaimergp reviewed Mar 30, 2025

View reviewed changes

weiji14 mentioned this pull request Apr 1, 2025

Zarr v3 Support and conda-forge versus pypi dependencies nsidc/earthaccess#974

Open

AntoinePrv reviewed Apr 15, 2025

View reviewed changes

weiji14 mentioned this pull request Apr 16, 2025

Packaging the optional s3 dependencies? conda-forge/rasterio-feedstock#329

Open

1 task

weiji14 mentioned this pull request Apr 27, 2025

Add NVIDIA TransformerEngine conda-forge/staged-recipes#29403

Merged

10 tasks

baszalmstra reviewed May 13, 2025

View reviewed changes

pavelzw mentioned this pull request May 21, 2025

pixi search should take down_prioritize_variant into account prefix-dev/pixi#3780

Open

h-vetinari mentioned this pull request May 22, 2025

ABI compatibility for modern Fortran packages conda-forge/flang-activation-feedstock#14

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CEP for the next evolution of Repodata (v2) and MatchSpec #111

CEP for the next evolution of Repodata (v2) and MatchSpec #111

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!


		The proposed syntax is to extend the `MatchSpec` syntax by appending `; if <CONDITION>` after the current MatchSpec.

		We would like to also allow for AND and OR with the following syntax:


		## Conditional dependencies

		Conditional dependencies are activated when the condition is true. The most straight-forward conditions are `__unix`, `__win` and other platform specifiers. However, we would also like to support matchspecs in conditions such as `python >=3`.


		However, it would be nice if we could have a flexible, powerful and simple syntax to enable or disable "flags" on packages in order to select a variant.

		A RepodataRecord should get a new field "flags" that is a list of strings, such as:

CEP for the next evolution of Repodata (v2) and MatchSpec #111

Are you sure you want to change the base?

CEP for the next evolution of Repodata (v2) and MatchSpec #111

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!