raccoon's zone!

This web is still being woven. Please don't be surprised if it changes from time to time...
Contents

last updated 2024-01-04 for nix 2.13 - 2.18, nixos 23.05 - 23.11

Writing Packages with Nix

Continuing from the introduction, this document talks about how derivations are structured, how they're used by package collections like Nixpkgs, and how to write some of our own.

This one's going to be more like a shortened reference version of what already exists in the Nixpkgs manual, rather than a guided tutorial. Ultimately, once you know the basics, it's best to start using the thousands of preexisting examples in the Nixpkgs repository to guide you.

If you have any questions, complaints, or suggestions, contact us!

derivation

In the Nix language, derivations are ultimately created by passing a set of inputs to the built-in function derivation, * detailed below.

derivation {
  # derivations need a human-readable name to affix to the end of their path
  name = "foo";

  # they can only be built on a system that matches a given platform specifier
  system = "x86_64-linux";

  # when a derivation is built, an executable called a builder is copied to the
  # store and executed to produce the output
  builder = ./builder.sh;

  # optionally, commandline arguments can be passed to the builder
  args = [ "--do-the-thing" ];

  # by default, the builder is provided with an environment variable called `out`.
  # it's the location it should produce its output at, which corresponds to
  # `/nix/store/<hash>-<name>`
  #
  # optionally, you can supply a list of environment variables to use as outputs
  # instead, which each correspond to `/nix/store/<hash>-<name>-<output>`
  outputs = [ "bin" "lib" "doc" ];

  # any other attribute is simply converted to string and provided to the builder
  # as an environment variable.
  #
  # these attributes are converted to strings, mostly as if passed to `toString`,
  # however:
  #
  # - paths are *copied to the store* as if string interpolated
  #
  # - derivations are automatically built when this one is, so their output paths
  #   are guaranteed to exist
  #
  # this is useful for passing in all the paths to sources, dependencies, etc.
  # needed for the build, e.g.
  src = /path/to/source;
  some-dependency = dependency-derivation;
}
See also Advanced Attributes (nix manual) for additional attributes that offer more niche functionality

When you produce a derivation value, you can still interact with it in a Nix expression as if it were still just its input set. However, in addition to the input attributes you give it, it'll also contain some new values:

let drv = derivation { /* ... */ }; in [
  # the string "derivation"
  drv.type

  # the original input set
  drv.drvAttrs

  # the store path for the derivation, `/nix/store/<hash>-<name>.drv`,
  # which doesn't exist until this attribute is accessed or the derivation is built
  drv.drvPath

  # the name of the output path (e.g. "out")
  drv.outputName

  # the output path as a string, which may not exist until the derivation is built
  drv.outPath

  # in the case of derivations with multiple outputs, the `outputName` and `outPath`
  # will only represent *the first one* in the list. if you want to access the
  # other output paths, you can first supply the output name as an attribute
  drv.lib.outputName # = "lib"
  drv.lib.outPath

  # you can also access a list of all of these output variants of the derivation
  # with `drv.all`
  (builtins.head drv.all).outPath
]

Importantly, you can't invoke the build process for a derivation from inside the Nix language; a derivation can be thought of as purely an I/O instruction for the commandline program itself to perform. In a situation like nix build --expr "/* ... */", the expression resolves to a derivation and then nix build builds it.

stdenv and mkDerivation

Though it's useful for understanding how derivations are fundamentally structured, invoking derivation directly can be cumbersome and impractical. You will rarely, if ever, need to use it without a wrapper.

In Nixpkgs, a derivation called stdenv is built for each platform, which bootstraps a consistent environment containing essential utilities like Bash, GNU Make, etc. The stdenv provides a function called mkDerivation, which wraps derivation to automatically provide the correct system and a featureful builder script setup.sh. This mkDerivation function, or another wrapper on top of it, is what you'll generally use.

stdenv.mkDerivation {
  # it's sometimes useful to be able to access a package's version string
  # on its own, so here it can be defined seperately from the package name.
  # the derivation's `name` will automatically be set to `"${pname}-${version}"`
  pname = "foo";
  version = "1.0";


  # instead of writing our own builder, a number of environment variables may be
  # passed to the builder script in order to control its behaviour, many of which
  # are explained below.

  ## Dependencies
  # setup.sh does automatic dependency configuration (taking care of PATH,
  # LD_LIBRARY_PATH etc. for you), and introduces the concept of Autoconf-style
  # platforms to express different kinds of dependencies more specifically.
  #
  #
  # for the purposes of setup.sh's dependency system:
  #   a derivation is built on its *build platform*,
  #   producing an output that runs on its *host platform*,
  #   and its output may itself produce output that runs on its *target platform*.
  #
  # the "target" platform is only relevant when a derivation produces something
  # like a compiler; most software doesn't itself produce platform-specific code.
  #
  # as such, we normally only care about build-time versus run-time dependencies:
  # whether a dependency runs on its dependent's build platform (which we will
  # notate as `build -> *`), or its host platform (`host -> *`).
  #
  # for this, we only need to know two dependency types:

  # `build -> host`, a build-time dependency
  nativeBuildInputs = ... ;

  # `host -> target`, a run-time dependency
  buildInputs = ... ;

  # but if we *are* building something like a cross-compiler and we really do
  # care about the target platform of a dependency, we can also use the
  # following (the platform after the arrow is the *dependency's target*):

  # `build -> build`
  depsBuildBuild = ... ;

  # `build -> target`
  depsBuildTarget = ... ;

  # `host -> host`
  depsHostHost = ... ;

  # `target -> target`
  depsTargetTarget = ... ;


  # sometimes, when a derivation is used as a dependency, one of its own
  # dependencies should be *propagated* to its dependent, as if the dependent
  # also has that dependency.
  #
  # in this case, we can use the propagated variants of the attributes:

  propagatedNativeBuildInputs = ... ;
  propagatedBuildInputs = ... ;

  depsBuildBuildPropagated = ... ;
  depsBuildTargetPropagated = ... ;
  depsHostHostPropagated = ... ;
  depsTargetTargetPropagated = ... ;

  # when these dependencies are propagated, we have to consider not just what
  # type of dependency they are to us, but also what type of dependency we are to
  # our dependent!
  #
  # once again, if you only care about the two common types of dependencies,
  # all you'll need to know is this:
  # `                                                   will be seen by
  # | this type                   | propagated by     | the dependent as  |
  # +-----------------------------+-------------------+-------------------+
  # | propagatedBuildInputs       | buildInputs       | buildInputs       |
  # | propagatedBuildInputs       | nativeBuildInputs | nativeBuildInputs |
  # | propagatedNativeBuildInputs | buildInputs       | nativeBuildInputs |
  # | propagatedNativeBuildInputs | nativeBuildInputs | NOT PROPAGATED    |
  # ` 
  # however, the full algorithm for determining the types of propagated
  # dependencies is described below if you really really need it.
  # this is advanced stuff, folks!


  # if we think of each platform as some integer offset from the dependent's host
  # platform, we can say that build = -1, host = 0, target = 1.
  #
  # then, as if using the following function:
  #
    f = host: target: input: = if input == 1 then target else host - input ;
  #
  # if our dependent sees us as the type `host -> target`,
  # and we see our propagated dependency as the type `p-host -> p-target`,
  # then our dependent will see that propagated dependency as the type
  # `f host target p-host -> f host target p-target`.
  #
  #
  # for example,
  #
  # a `nativeBuildInputs` (`-1 -> 0`)
  # with a `propagatedBuildInputs` (`0 -> 1`)
  # will propagate the latter to its dependent as if it were
  # a `f -1 0 0 -> f -1 0 1`, i.e. a `-1 -> 0`, another `nativeBuildInputs`!
  #
  # whereas a `depsBuildBuild` (`-1 -> -1`)
  # with a `depsBuildTargetPropagated` (`-1 -> 1`)
  # would hypothetically propagate to its dependent as if it were
  # a `f -1 -1 -1 -> f -1 -1 1`, i.e. a `-2 -> -1`,
  # but platform offsets outside the range `[-1, 1]` don't make sense, so it's
  # never propagated at all!


  # further reading in Specifying dependencies (nixpkgs manual)

  ## Fetchers
  # fetchers are simple derivations that produce files from some external source,
  # like the source code for some software you're building.
  #
  # they're known as *fixed-output* derivations, because we can't make guarantees
  # about the purity of the output from an external source (someday the same
  # URL might serve something different, or corruption will cause the output to
  # differ), so we need to specify the expected output's hash in advance.

  # one example of a fetcher is `fetchurl`, which simply downloads a file
  src = fetchurl {
    url = "https://www.example.com/v${version}/download";
    hash = "sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=";
  };
  # (defined here)

  # more specialized fetchers exist, too, like ones for specific git services:
  src = fetchFromGitHub {
    owner = "example";
    repo = "test";
    rev = "v1.0"; # can be a commit hash or a tag
    hash = "sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=";
  };
  # (defined here)

  # for more complicated fetchers like this, it can be difficult to know the hash
  # of the file in advance. don't worry, though: if the derivation fails, nix
  # will tell you what hash it expected. it's common to fill in a dummy hash
  # and let it fail, so that nix will tell you what you actually need to put in.

  # most of the important fetchers implemented in nixpkgs are listed in
  # Fetchers (nixpkgs manual)

  ## Phases
  # when the derivation is built, it goes through a series of phases, by default
  # the following:
  phases = [
    prePhases
    "unpackPhase" "patchPhase"
    preConfigurePhases "configurePhase"
    preBuildPhases "buildPhase" "checkPhase"
    preInstallPhases "installPhase"
    preFixupPhases "fixupPhase" "installCheckPhase"
    preDistPhases "distPhase"
    postPhases
  ];
  # you shouldn't overwrite this attribute yourself, but you *can* set the
  # `pre...Phases` and `postPhases` attributes in that list if you want to add
  # more phases.


  # each phase is a bash snippet that performs some action to process the input
  # and produce the output.
  #
  # they have too many options to list here, but they're described extensively in
  # Controlling phases (nixpkgs manual).
  #
  # you can also read the `...Phase()` function for each phase in setup.sh
  # if you want to know exactly how they're defined.

  # the gist of it is, by default:
  #
  # - `unpackPhase` unpacks the `src` (and/or the list of paths `srcs`) or
  #   just copies it if it's already a directory, then `cd`s into that directory
  #
  # - `patchPhase` applies diff patches in the list `patches` (which is
  #   initially empty, making it a no-op by default)
  #
  # - `configurePhase` runs `./configure` if present
  #
  # - `buildPhase` runs `make` if Makefile present
  #
  # - `checkPhase` runs `make check` if Makefile and `check` target present,
  #   only if `doCheck = true`
  #
  # - `installPhase` creates the directory `$prefix` (which defaults to `$out`)
  #   and then runs `make install`
  #
  # - `fixupPhase` is mainly a vessel for miscellaneous post-processing hooks;
  #   it runs the `fixupOutput` hooks for each output:
  #     - `strip` binaries,
  #     - patch RPATHs and shebangs to refer to nix store paths of `host -> *`
  #       dependencies,
  #     - automatically move stuff around to sensible locations,
  #     - etc etc
  #   and adds the hook scripts in `setupHook`/`setupHooks` to
  #   `$dev/nix-support/setup-hook` so they can be propagated to your dependents
  #
  # - `installCheck` runs `make installcheck` if Makefile and
  #   `installcheck` target present, only if `doInstallCheck = true`
  #
  # - `distPhase` runs `make dist` and copies source tarballs to `$out/tarballs`,
  #   only if `doDist = true`
  

  # you can replace the default actions by overwriting 
  # the `...Phase` attribute with your own string containing a bash snippet
  #
  # like, let's say if the program you're installing doesn't have an `install`
  # target in its makefile, but instead has a little install script you can run:
  installPhase = ''
    ./install $out
    # success!
  '';

  ## Setup Hooks
  # if you choose, each phase also includes *setup hooks* that allow you to
  # extend the phase with additional code without replacing it entirely.
  #
  # hooks can be added by name as attributes in your derivation, or they can be
  # added by your dependencies: if a dependency consists of just a file, or a
  # directory that contains the file `nix-support/setup-hook`, that file is
  # sourced by the setup script.
  # 
  # makeSetupHook is a helpful shortcut for making your own setup hook derivations.
  #
  # from there, the file may modify the setup script, typically by adding a hook
  # to one of its arrays like so:
  #
  #   ```
  #   # the array for the 'preFixup' hook
  #   preFixupHooks+=(_myFixupHook)
  #
  #   _myFixupHook () {
  #     do-cool-stuff --to $out
  #   }
  #   ```
  #
  # keep in mind that the runHook calls are inside the phase functions themselves.
  # e.g. if you replace `installPhase`, you have to include `runHook preInstall`
  # and `runHook postInstall` yourself if you want them to happen during your
  # custom `installPhase`.

  # `mkDerivation` contains several setup hooks by default in order to provide
  # a variety of conveniences.
  #
  # they're described in Package setup hooks (nixpkgs manual) and defined in
  # `nixpkgs/pkgs/build-support/setup-hooks/`

  ## Meta
  # this attribute is optional and not passed to the builder, but it contains
  # static information that may be displayed and used by external tools.
  meta = let
    inherit (lib) licenses maintainers;
  in {
    # package information is useful to display by nix and by other search tools
    description = "A short description of the package";
    longDescription = ''
      A more detailed, arbitrarily long description of the package.
      It should be able to span multiple lines.
    '';
    homepage = "https://www.example.com";
    maintainers = [ maintainers.example ];
    # nixpkgs will restrict packages with unfree licenses if configured to do so
    license = licenses.mit;

    # there are several more defined attributes you can put in this set,
    # described in Standard meta-attributes (nixpkgs manual)
  };

  # `passthru` contains attributes that also aren't passed to the builder, but
  # appear on the derivation value as if they were. they can be used for
  # anything that you don't want to trigger a rebuild when changed.
  passthru = {
    # commonly, it may contain an attribute `tests`, containing a set of
    # derivations which may be built to see if they succeed.
    tests = {
      test = /*...*/ ; # Testers (nixpkgs manual)
    };
    # Tests (nixpkgs manual)

    # or an update script, which is used with the maintainer tool
    # `maintainers/scripts/update.nix`, and is given the env var
    # `UPDATE_NIX_ATTR_PATH` pointing to the attribute path to be updated.
    updateScript = ./update.sh;
    # or
    updateScript = [ ./update.sh "--with-args" ];
    # Special variables (nixpkgs manual)
  };
}

Additionally, derivations created with stdenv.mkDerivation supply a function that can override any of these attributes to produce a new, modified derivation. This can be useful for making ad hoc changes to derivations obtained from code that you don't control.

# overrideAttrs accepts a function with two arguments:
# `super` is the "old" set of attributes we're about to change,
# and `self` is the "final" set of attributes that will exist after all overrides
# are applied.
#
# the function should return a set of the attributes you want added/changed.
drv.overrideAttrs (self: super: {
  version = super.version + "-raccoon";
  src = ./extra-special-raccoon-fork;
})

# you can also pass a one-argument function to overrideAttrs, in which case
# it'll just receive `super`

For convenience, Nixpkgs also contains some other derivation wrappers for trivial actions like outputting plaintext files or running shell scripts, specialized build processes for specific languages or output formats, etc. See the Builders section in the Nixpkgs manual.

Packages

In Nixpkgs, a package refers to a function that accepts a set of dependencies and options and returns a customized derivation. This way the user can configure the way the derivation is built, and your derivation can depend on other derivations in the package collection without needing to specify them so exactly.

# in package functions, we specify dependencies as set attributes on our input
#
# when your package is called, they'll be filled in with derivations
# corresponding to other packages in Nixpkgs.
{
  # the Nixpkgs standard library. this isn't a derivation, but a set of useful
  # functions and constants inside `nixpkgs/lib/` that every package has access to
  lib,

  # this is our stdenv derivation, located at `nixpkgs/pkgs/stdenv/generic/`
  stdenv,

  # and these are some useful fetchers, discussed earlier
  fetchurl, fetchFromGitHub,

  # additionally, you can specify configuration options in this set that can be
  # modified to create different variants of the derivation
  enableFoo ? false
}:
stdenv.mkDerivation { /* ... */ }

Derivations produced from packages offer another override mechanism, so that you can modify the arguments passed to the package function. It works similarly to overrideAttrs, but just accepts a set of modified arguments:

drv.override {
  enableFoo = true;
}

nix develop and mkShell

nix develop is a Nix command that, rather than building the specified installable, launches an interactive bash subshell containing its build environment.

In that shell session, you have access to all the build dependencies in your PATH, and all the bash functions for each phase (configurePhase, buildPhase, ...) so you can build the package incrementally yourself.

This is useful for mimicking the environment used to build some particular package, but it's also useful in its own right to be able to create reproducible development environments for a variety of purposes. As such, Nixpkgs also provides mkShell, a specialized wrapper that doesn't produce any meaningful output by default, but provides some conveniences for creating specific build environments.

If you don't need to run the bash functions from setup.sh and just want e.g. the package dependencies in your PATH, you can choose to use your own shell with nix develop -c $SHELL [installable]. Alternatively, if you run a bash-compatible shell, you can source <(nix print-dev-env [installable]) to source the setup.sh environment into your own shell.

Distributing your Packages

TODO: flakes

TODO: buildFHSenv,