WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls #399775

TomaSajt · 2025-04-18T15:37:11Z

Draft.
Let's discuss first.

Questions:

how would this be useful?
- I couldn't find a public non-crates-io registry
  - the crates-io registries are
    - registry+https://github.com/rust-lang/crates.io-index
    - sparse+https://index.crates.io/
- so this is probably mostly useful for internal registries of companies
  - but then we'd probably also need to figure out authentication
- maybe I should host a registry myself and try it out
should crates from different registries be placed into different directories?
- if not, we could get duplicates, like what rustPlatform.fetchCargoVendor: allow duplicated dependencies #387337 tries to solve, but in the FOD part of the fetcher
  - what's the likelihood of this?
- the way crates are currently stored is not the best if we wanted to have separate directories
  - we have /tarballs currently
    - this doesn't have distinction between registries
  - it would have been better to have /registry-0, /registry-1 etc
    - the index could be based on string-sorting order
  - what should we do to keep it backwards compatible?
considering the complexity of this, should we re-add a way that uses the cargo vendor command
- we'd need to explicitly advertise it as a fetcher to only use if you want to use non-crates.io registries
- does cargo vendor even support other registries?
  - I haven't checked

should we do something similar to importCargoLock instead?

nixpkgs/pkgs/build-support/rust/import-cargo-lock.nix

Lines 22 to 31 in 32c543a

    
           # Additional registries to pull sources from 
        
           #   { "https://<registry index URL>" = "https://<registry download URL>"; } 
        
           #   or if the registry is using the new sparse protocol 
        
           #   { "sparse+https://<registry download URL>" = "https://<registry download URL>"; } 
        
           # where: 
        
           # - "index URL" is the "index" value of the configuration entry for that registry 
        
           #   https://doc.rust-lang.org/cargo/reference/registries.html#using-an-alternate-registry 
        
           # - "download URL" is the "dl" value of its associated index configuration 
        
           #   https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration 
        
           extraRegistries ? { },

this doesn't use the config.json endpoint, it requires people to manually provide the download endpoint
I think this is worse than using the config.json automatically

what can we learn from crane?
- they already have a method to handle the config.json files

TODOs:

override other registries in the config.toml using replace-with
- copy how importCargoLock does it
make separate directories in the non-FOD part
- like with rustPlatform.fetchCargoVendor: allow duplicated dependencies #387337

I don't think a decision needs to be made ASAP, since this problem is pretty niche, and people can use importCargoLock anyways.

Things done

Add a 👍 reaction to pull requests you find important.

…ate download urls

simonzkl · 2025-04-28T18:01:33Z

I couldn't find a public non-crates-io registry

The one I used to test an issue a while back was: https://crates.polomack.eu/ (https://github.com/Hirevo/alexandrie-index)

should crates from different registries be placed into different directories?

Crane does this for some reason, but cargo vendor does not.

considering the complexity of this, should we re-add a way that uses the cargo vendor command

I've seen a lot of pushback from nixpkgs maintainers to using upstream package managers to fetch dependencies (see yarn, pnpm, etc.). Unless there are some upstream stability guarantees for cargo vendor, I think this would not be accepted.

does cargo vendor even support other registries?

It does. I'm using it as a workaround right now (custom cargoDeps override).

should we do something similar to importCargoLock instead?

I think that's a fine approach, however for private registries we would need to be able to set custom headers to authenticate. Maybe something similar to the netrcPhase of pkgs.fetchurl where you can configure curlOpts="-H <auth-header>"? Obviously since we're not using curl this would be very diffe 8000 rent.

TomaSajt · 2025-05-01T00:42:12Z

does cargo vendor even support other registries?

It does. I'm using it as a workaround right now (custom cargoDeps override).

Since cargo vendor supports auth, even if not 100% guaranteed to be stable, IMO it's the best solution.
But this takes us back to the question of whether all people who need custom registries should roll their own cargo vendor-based logic, or if there should be a basic nixpkgs implementation.

As for fetchCargoVendor, I myself do not wish to implement the cargo registry authorization API logic in it.

TomaSajt · 2025-05-01T00:46:19Z

I couldn't find a public non-crates-io registry

The one I used to test an issue a while back was: https://crates.polomack.eu/ (https://github.com/Hirevo/alexandrie-index)

I have also since found https://github.com/rust-lang/staging.crates.io-index

Sadly, both of these seem to be git-based registry indexes.
It would be great if I could test the sparse protocol as well...

TomaSajt · 2025-05-04T12:59:54Z

Btw I found sparse+https://mirrors.ustc.edu.cn/crates.io-index/
just a mirror, but at least it's different...

WIP: rustPlatform.fetchCargoVendor: use registry's config.json for cr…

2e15692

…ate download urls

github-actions bot added 6.topic: rust 10.rebuild-darwin: 5001+ 10.rebuild-darwin: 501+ 10.rebuild-linux: 5001+ 10.rebuild-linux: 501+ labels Apr 18, 2025

TomaSajt mentioned this pull request Apr 20, 2025

fetchCargoVendor: support other registries #377558

Open

wip

e0cdd37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls #399775

WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls #399775

	# Additional registries to pull sources from
	# { "https://<registry index URL>" = "https://<registry download URL>"; }
	# or if the registry is using the new sparse protocol
	# { "sparse+https://<registry download URL>" = "https://<registry download URL>"; }
	# where:
	# - "index URL" is the "index" value of the configuration entry for that registry
	# https://doc.rust-lang.org/cargo/reference/registries.html#using-an-alternate-registry
	# - "download URL" is the "dl" value of its associated index configuration
	# https://doc.rust-lang.org/cargo/reference/registry-index.html#index-configuration
	extraRegistries ? { },

WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls #399775

Are you sure you want to change the base?

WIP: rustPlatform.fetchCargoVendor: use registry's config.json for crate download urls #399775

Conversation

Things done