Manage a set of Nix Haskell package overrides, somewhat automatically.
Put another way, it aids definition and validation of properties of every override in a set:
- validity (whether it breaks things)
- necessity (whether it’s needed to keep things working, along with a proof)
- actuality (whether the override status is up to date with the upstream, Hackage and Nixpkgs)
Definitions are managed somewhat conveniently, like:
nh hackage foo 1.2.14 nh hackage foo # Pull the latest one nh upstream bar nh upstream bar ghc-8.4 nh unmerged bar concerned-citizen PR-number
- Init a package DB:
nh init..which will ask you to create a config file first – follow the instuctions.
- Add overrides:
nh hackage lens # easy: there's a Hackage release, get the latestnh import microlens-th # import metadata for microlens-th from Hackage cabal file nh upstream microlens-th master # Fetch the tip of upstream's master (any refspec goes) nh chdir microlens-th microlens-th # -- in case the Cabal metadata was wrong nh set-issue microlens-th 222 # Specify the relevant upstream issue IDnh set-upstream hedgehog hedgehogqa nh unmerged hedgehog gwils 134 [GIT-REV] # the third argument to "unmerged" is the Github PR # - Observe the changes:
nh show lens nh show microlens-th nh show hedgehog..and you always can generate
overrides.nixfrom the very latest package database – although it happens transparently most of the time:nh apply nh override lens # ..to see the individual overrideThe
--disable-ghc-configurationswitch will position your override set as a replacement to the global, version-specific GHC configuration.If you set
TARGET_NIXPKGSin the.nhconfiguration file to a non-empty value,nhwill emit non-local overrides directly into the Nixpkgs GHC configuration for the selected GHC version. - Validate if the overridden packages build now:
nh acme # Build all defined overrides nh build NAME.. # ..as an alternative...and..
nh progress # ..to watch progress in another terminal - See what the audit tool thinks about the situation:
nh audit --skip-acme [NAME..] # Defaults to everything, once moreIt should complain, at the very least because the
trimpart was not run, and so there’s no proof that any of the overrides are necessary.Note that the
nt auditsubcommand provides suggestions, that can be auto-executed when the--auto-fixoption is supplied. - Let’s try trimming the set:
nh trim [NAME..]That would take a while, and also it won’t do anything to the recorded overrides – it merely collected information, as an intermediate step.
Let’s execute on that information, then:
nh execute-trims - Audit again:
nh audit..and this is probably going to fail, because the trim is often over-eager. So, some of the overrides need to be reintroduced, and the figuring-out-what-and-why part is manual (albeit assisted). Thankfully, we can record the manually-collected information in the database (which would prevent trimming of this override in the future):
nh set-explanation broken-attribute doCheck "....."Rinse, repeat – and remember that individual attributes can be rebuilt using
nh baselineornh build.Also, explanations (aka proofs of override) can be shown per-override with:
nh explain happy src
Finally: look at suite.sh for inspiration.
First, we describe the general flow of fixes, to establish a terminology.
- Fixes are generally born on a third-party Github repo, and they are expected to be submitted upstream via pull request.
- The PR gets merged upstream.
- Upstream cuts a release, bumping the package version in the cabal file.
- Upstream performs a Hackage upload.
- Nixpkgs imports Hackage, adding a versioned
package-attribute_1_2_1_0. - Nixpkgs promotest the versioned
package-attribute_1_2_1_0topackage-attribute, which completes the cycle. - Nixpkgs also supports non-source tweaks (jailbreaking out of restrictive version bounds, test and Haddock generation disables).
nh maps the above into a status, per attribute:
- unmerged
- phases #1
- upstreamed
- phases #2 and #3
- hackaged
- phase #4
- shadowed
- phase #5 (after shadow attributes – those shadowing non-versioned ones)
- config
- not-really-phase #7
nhtracks the aforementioned attribute status and content of the attribute overrides in a package database (aka PKGDB). This is just a file-system directory – but it’s better to version it in Git, to be able to recover, whennhgoes off-rails and breaks overrides.- The result is delivered in the form of a Nix file defining a GHC package
set override (customarily called
overrides.nix).This trivially-structured, generated file is then supposed to be imported into another, static Nix file called
packages.nix, which then forms a proper GHC package set. That one can be passed tonix-build. - The major package DB operations that
nhprovides are:- acme
- Build every attribute in the override set using a proxy, that depends on everything overridden (really, acme).
- trim
- Try to remove overrides, one by one, and record the results of those attempts in the package DB – trying to deducing whether these overrides are necessary. It is a heuristic.
- execute-trims
- Modify the package DB in accordance with the trim step. This effectively removes any overrides that weren’t found necessary. This is also error-prone (more things are sometimes removed than is feasible).
- audit
- Verify every attribute against a set of status-dependent
invariants, that ensure:
- the override necessity (along with the existence of proof), and
- the override being up-to-date.
nhkeeps as much build information as possible, and that includes store derivation links, store source links, override expressions and build logs for every build attempt that takes place. In particular every attribute build attempt happens in three phases, handled separately:- attribute instantiation
- dependency pre-build
- build of the attribute itself
$ nh x hackage funcmp 1.9
downloading ‘http://hackage.haskell.org/package/funcmp-1.9.tar.gz’... [0/0 KiB, 0.0 KiB/s]
path is ‘/nix/store/akhnn03wfi3jlx2rqgwjdz07qpz983iz-funcmp-1.9.tar.gz’
- 1d5appkjhajb9ndv2gwnfz8lw2w53v8baajzmrhg26ihzj1bkch8
- https://hackage.haskell.org/package/funcmp-1.9
$ nh set-explanation funcmp src
funcmp.def/meta.src.explanation: Needed for (<>) in prelude
$ nh jailbreak deepseq-generics
$ nh set-explanation deepseq-generics jailbreak
deepseq-generics.def/meta.jailbreak.explanation: https://github.com/haskell-hvr/deepseq-generics/pull/4
$ nh import securemem # this fetches metadata like repoName, upstream, chdir etc.
$ nh unmerged securemem shlevy 12 6168d90b00bfc6a559d3b9160732343644ef60fb
- 06dhx1z44j5gshpdlsb4aryr3g4was3x4c2sgv1px8j57zrvlypx
- https://github.com/vincenthz/hs-securemem/commit/6168d90b00bfc6a559d3b9160732343644ef60fb- def
- definitions
- meta
- non-override metadata
- over
- overrides
- hackage, github
- src-specific information, per-attribute-override
- cache
- override cache, per-attribute
- build
- build output information: logs, expressions, derivations
Usage: nh [--cls] [--nixpkgs] [--trace] [--debug] [--quiet] SUBCMD [SUBARGS..]
NOTE: if --nixpkgs is passed, non-local overrides instead serve as definition
for /home/deepfire/nixpkgs/pkgs/development/haskell-modules/configuration-ghc-8.4.x.nix
PKGDB:
forall-defined-edit TYPE FIELD
Interactively edit all FIELD definitions of TYPE
Metadata (non-override):
ls-meta ATTR List attribute's metadata (as opposed to overrides
meta ATTR META Print a single metadata entry of an attribute
set-meta ATTR META VAL Set a single metadata entry of an attribute
edit-meta ATTR META Edit the current attribute's meta value using readline
disable ATTR[.OVER] Disable all/single overrides for an attribute
enable ATTR[.OVER] Re-enable previously disabled overrides
with-disabled-attrs ATTR..
Disable all listed attribute overrides and pause; Re-enable on exit or newline in stdin
ls-disabled List all disabled attributes
set-explanation ATTR OVER VAL
Manually supply explanation for an override's existence
set-erdeps ATTR 'ATTR..' Set attribute's essential rev-deps that must keep working
chdir ATTR SUBDIR Change directory before build; "" removes the override
local ATTR Mark ATTR as local: not subject for Nixpkgs GHC configuration
nonlocal ATTR Remove marking of ATTR as local
Override manipulation (low level):
remove ATTR[.OVER] Remove specified overrides
ls-over ATTR List attribute's overrides
ls-input-overs ATTR List attribute's input overrides
get ATTR OVER Get an attribute's override value
set ATTR OVER VAL Set an attribute's override value; "" removes the override
edit ATTR OVER Edit the current attribute's value using readline
set-input-over ATTR INPUT VAL
Set ATTR's override for INPUT
edit ATTR OVER Edit the current attribute's value using readline
check ATTR Disable an existing dontCheck override
dontCheck ATTR Disable tests
haddock ATTR Disable an existing dontHaddock override
dontHaddock ATTR Disable Haddock generation
jailbreak ATTR Turn on jailbreaking
dontJailbreak ATTR Disable an existing jailbreak override
{library,executable,test}Haskell ATTR [ATTR..]
Specify extra *HaskellDepends; "" removes the override
add-patch ATTR SHA256 URL Add a patch to ATTR
Status:
status ATTR Print status of a single attribute
ls-shadowed List all attributes with status 'shadowed'
ls-hackaged ...'hackaged'
ls-upstreamed ...'upstreamed'
ls-unmerged ...'unmerged'
ls-config ...'config'
Nix-level inferences:
drv ATTR Store derivation for a single override
pprint-drv ATTR Pretty-print ATTR's derivation (requires nix-derivation-pretty)
src ATTR Store source derivation for a single override
src-drv ATTR Store source derivation of ATTR
src-url ATTR Source URL of ATTR
inputs ATTR ATTR's store inputs
deps | refs | references ATTR
ATTR's store drv dependencies
rdeps | referrers ATTR
6C2D
ATTR's store reverse drv dependencies
realise-drv ATTR Realise ATTR's derivation
drv-pprint STORE-DRV Pretty-print a Nix-stored .drv file
src-drv ATTR Store source derivation of ATTR
src-drv-url STORE-DRV Source URL of a Nix-stored source-.drv file
drv-inputs STORE-DRV Store inputs for a Nix-stored .drv file
drv-refs | drv-references STORE-DRV
Store .drv references for a Nix-stored .drv file
deriver-of STORE-PATH Store .drv for a Nix store path. Will fail if built non-locally
PKGDB emission to Nix overrides:
over | override | show-override ATTR
Print the attribute's override defined by PKGDB
apply [--reuse-cache] Apply all overrides via /home/deepfire/overrides.nix
cache [--require-descs] Regenerate override cache
show-cache ATTR Print the cached text of attribute's override (DEBUG)
General:
ls [REGEX] List all overridden attributes
info ATTR Overview of an attribute's PKGDB
overview [ATTR..] List overridden attributes, grouped by status + relevant info
Hackage:
import ATTR Scrape ATTR's Cabal file from Hackage for some properties
cabal ATTR Print the latest released cabal file for ATTR
hackage ATTR [RELEASE=upstream-latest]
Override to a Hackage release
Github:
github ATTR [REF] Override ATTR to its latest upstream Github commit
unmerged ATTR USER PR# [REV=HEAD]
Override to a 3rd-party Github commit
upstream ATTR [REV=HEAD] Override to an upstream Github commit
set-upstream ATTR GITHUB-USER
Specify an attribute's Github upstream username
edit-upstream ATTR Edit an attribute's Github upstream username
set-pr ATTR PR# Set the PR# of an attribute's Github override
set-issue ATTR ISSUE# Set the Issue# of an attribute's Github override
set-repoName ATTR REPO Set an attribute's Github repository name
edit-repoName ATTR Edit an attribute's Github repository name
Build & results:
instantiate [--reuse-cache] [ATTR..]
Instantiate overridden attrs (or specified subset)
acme [--reuse-cache] Build everything at once, collecting all failures
build [COMMON-OPTS] ATTR Build a single attribute with current overrides
log ATTR [OVER=baseline] Obtain trim build logs for a single override
failure ATTR [OVER=baseline]
Obtain trim failure kind of an override
failure-log ATTR [OVER=baseline]
Obtain trim failure log of an override
failure-type ATTR [OVER=baseline]
Obtain trim failure type of an override
proof ATTR [OVER] Print an override's proof of necessity. When OVER is empty, print context.
Override database maintenance:
trim [--reuse-cache] [ATTR..]
Suggest a reduction to the override set (or specified subset)
trim-override ATTR OVER Attempt trimming a specific override of a given attribute
show-trims Show the trim suggestion
execute-trims Execute the suggestion
audit [--autofix] [--autoonly] [--skip-acme] [--reuse-{overrides,cache}] [ATTR..]
Sanity check the overridden attrs (or specified subset). --autofix applies suggestions
extra-validation-attributes
Edit the set of attributes validated regardless of being overridden
edit-fixed-content Edit the static part of the GHC configuration
Nix shell:
shell Nix shell with up-to-date overrides (shell.nix required)
shell-for ATTR Nix shell for building ATTR
cabal-shell Nix shell from a cabal file (nothing else required)
clone-upstream-fixer-shell
Nix shell from a cabal file (nothing else required)
try-fix ATTR Push the current commit and try the fix
find-module NAME Convenience alias for 'ghc-pkg find-module NAME'
list-packages ... Convenience alias for 'ghc-pkg list ...
describe-package ATTR Convenience alias for 'ghc-pkg describe ATTR
package-modules ATTR List ATTR's exposed modules
phases ATTR Print ATTR's build phases
Miscellanea:
eval BASH-EXPR Passthrough, to execute anything defined.
loop-hunter Detect attribute loops: nix-shell 2>&1 | nh loop-hunter
ls-builds List active builds
progress [LOG] Live summary of new, complete and failing builds
watch Observe the current build, as it hits the logs..
ghc Shell with current GHC
prefetch-ghc GITREV Prefetch a GHC revision
less-ghc-config [NEEDLE] Run less on the Nixpkgs GHC configuration
git OPTIONS.. ARGS.. Run git inside controlled Nixpkgs
nixpkgs-diff [(base-head|base-master|head-master] [REF]
Diff of current GHC configuration