Nix Flakes: an Introduction

Published on 02/21/2022, 2677 words, 10 minutes to read

Nix is a package manager that lets you have a more deterministic view of your software dependencies and build processes. One if its biggest weaknesses out of the box is that there are very few conventions on how projects using Nix should work together. It's like having a build system but also having to configure systems to run software yourself. This could mean copying a NixOS module out of the project's git repo, writing your own or more. In contrast to this, Nix flakes define a set of conventions for how software can be build, run, integrated and deployed without having to rely on external tools such as Niv or Lorri to help you do basic tasks in a timely manner.

This is going to be a series of posts that will build on eachother. This post will be an introduction to Nix flakes and serve as a "why should I care?" style overview of what you can do with flakes without going into too much detail. Most of these will get separate posts (some more than one post).

In my opinion, here are some of the big reasons you should care about Nix flakes:

Flakes adds project templates to Nix
Flakes define a standard way to say "this is a program you can run"
Flakes consolidate development environments into project configuration
Flakes can pull in dependencies from outside git repos trivially
Flakes can work with people that don't use flakes too
Flakes supports using private git repos
Flakes let you define system configuration alongside your application code
Flakes let you embed the git hash of your configurations repository into machines you deploy

<Mara>

Something that may also help you understand why flakes matter is that Nix by itself is more akin to Dockerfiles. Dockerfiles help you build the software, but they don't really help you run or operate the software. Nix flakes is more akin to docker-compose, they help you compose packages written in Nix to run across machines.

Project Templates

One of the big annoying parts about getting into Nix is that setting up projects isn't totally a defined science. Nix configurations just tend to grow organically and can easily become weird or difficult to understand for people that didn't start the project. Nix flakes helps fix this by doing a few things:

Defining a flake.nix as the central "hub" for your project's dependencies, exposed packages, NixOS configuration modules and more.
Shipping a set of templates so that you can get projects started easily. Think something like Yeoman but built directly into Nix. You can also define your own templates in your flake.nix.

As an example that we will use for the rest of this post to help explain it, let's make a Go project with their Go template. First you will need to enable Nix flakes on your machine. If you are using NixOS, add this to your configuration.nix file:

nix = {
  package = pkgs.nixFlakes;
  extraOptions = ''
    experimental-features = nix-command flakes
  '';
};

Then rebuild your system and you can continue along with the article.

<Mara>

EDIT: You can use WSL for this. See here for more information.

If you are not on NixOS, you will need to either edit ~/.config/nix/nix.conf or /etc/nix/nix.conf and add the following line to it:

experimental-features = nix-command flakes

<Mara>

You may need to restart the Nix daemon here with sudo systemctl restart nix-daemon.service, but if you are unsure how Nix was set up on that non-NixOS machine feel free to totally restart your computer.

Now go to a temporary folder and run these commands to make a folder and create a new flake from a template:

mkdir ~/tmp/go-demo
cd ~/tmp/go-demo
nix flake new -t templates#go-hello .
git init && git add .

This will create a few files in the folder:

$ ls
flake.lock  flake.nix  go.mod  main.go

Then you can look at flake.nix to see what's up:

{
  description = "A simple Go package";

  # Nixpkgs / NixOS version to use.
  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

  outputs = { self, nixpkgs }:
    let

      # Generate a user-friendly version number.
      version = builtins.substring 0 8 self.lastModifiedDate;

      # System types to support.
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];

      # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Nixpkgs instantiated for supported system types.
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });

    in
    {

      # Provide some binary packages for selected system types.
      packages = forAllSystems (system:
        let
          pkgs = nixpkgsFor.${system};
        in
        {
          # The default package for 'nix build'. This makes sense if the
          # flake provides only one package or there is a clear "main"
          # package.
          default = pkgs.buildGoModule {
            pname = "go-hello";
            inherit version;
            # In 'nix develop', we don't need a copy of the source tree
            # in the Nix store.
            src = ./.;

            # This hash locks the dependencies of this package. It is
            # necessary because of how Go requires network access to resolve
            # VCS.  See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for
            # details. Normally one can build with a fake sha256 and rely on native Go
            # mechanisms to tell you what the hash should be or determine what
            # it should be "out-of-band" with other tooling (eg. gomod2nix).
            # To begin with it is recommended to set this, but one must
            # remeber to bump this hash when your dependencies change.
            #vendorSha256 = pkgs.lib.fakeSha256;

            vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
          };
        });
    };
}

This defines a single Go package that is supported on macOS and Linux for 64 bit x86 processors and 64 bit ARM processors.

<Mara>

In practice this spread should cover all of the main targets you'll need to care about for local development and cloud deployment.

You can then build the flake with nix build:

$ nix build

And then run it:

$ ./result/bin/go-hello
Hello Nix!

Standard Default Package

Let's take a closer look at the higher level things in the flake:

{
  description = "A simple Go package";

  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

  outputs = { self, nixpkgs }: {
    packages = { ... };
  };
}

<Cadey>

A note: in the rest of this article (and series of articles), when I refer to a "flake output", I am referring to an attribute in the outputs attribute of your flake.nix. Ditto with "flake input" referring to the inputs attribute of your flake.nix.

When you ran nix build earlier, it defaulted to building the default entry in packages. You can also build the default package by running this command:

$ nix build .#default

And if you want to build the copy I made for this post:

$ nix build github:Xe/gohello
$ ./result/bin/go-hello
Hello reader!

A standard default package means that you can more easily build software without having to read documentation on what file to build. nix build will Just Work™️.

Exposing Packages as Applications

Additionally, you can expose a package as an application. This allows you to simplify that above nix build and ./result/bin/go-hello cycle into a single nix run command. Open flake.nix in your favorite editor and let's configure go-hello to be the default app:

# below packages

apps = forAllSystems (system: {
  default = {
      type = "app";
      program = "${self.packages.${system}.default}/bin/go-hello";
  };
});

Then you can run it with nix run:

$ nix run
Hello Nix!

Or you can run my copy:

$ nix run github:Xe/gohello/main
Hello reader!

<Mara>

What is that extra part of the URL path for? Is that a git branch?

<Cadey>

Yes, you can use that syntax to set the git branch that Nix should build from. By default it will use the default branch (typically main), but sometimes you need to specify a branch or commit directly.

Development Environment Configuration

One of Nix's superpowers is the ability to declaratively manage the development environment for a project so that you can be sure that everyone working on the project is using the same tools.

<Cadey>

I use this with all of my projects to the point that when I am outside of a project folder I do not have any development tools available.

Flakes has the ability to specify this using the devShells flake output. You can add it to your flake.nix using this:

# after apps

devShells = forAllSystems (system:
  let pkgs = nixpkgsFor.${system};
  in {
    default = pkgs.mkShell {
      buildInputs = with pkgs; [ go gopls gotools go-tools ];
    };
  });

<Mara>

We consider this to be a basic Go development environment. It includes standard tools such as the language server, goimports for better formatting and tools like staticcheck. If you use staticcheck regularly at work, please consider throwing Dominik a couple bucks a month if you find it useful. It helps the project be more self-sustaining.

Then you can enter the development shell with nix develop:

$ nix develop

[cadey@pneuma:~/tmp/gohello]$ go version
go version go1.16.9 linux/amd64

And then hack at your project all you want. You can send this git repo to a friend and they will have the same setup.

External Dependencies

Now let's talk about inputs. Flake inputs let you add external dependencies to a project. As an example, let's look at the nixpkgs input:

# Nixpkgs / NixOS version to use.
inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

This defines the release of nixpkgs that should be used for the project. This template defaults to NixOS 21.11's version of nixpkgs, however we can upgrade it to nixos-unstable by changing it to this:

# Nixpkgs / NixOS version to use.
inputs.nixpkgs.url = "nixpkgs/nixos-unstable";

Then we can run nix flake update and then nix develop and see that we are running a newer version of Go:

$ nix flake update
warning: updating lock file '/home/cadey/tmp/gohello/flake.lock':
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/77aa71f66fd05d9e7b7d1c084865d703a8008ab7' (2022-01-19)
  → 'github:NixOS/nixpkgs/2128d0aa28edef51fd8fef38b132ffc0155595df' (2022-02-16)

$ nix develop

[cadey@pneuma:~/tmp/gohello]$ go version
go version go1.17.7 linux/amd64

This also lets you pull in other Nix flakes projects, such as my CSS framework Xess:

inputs.xess.url = "github:Xe/Xess";
inputs.xess.inputs.nixpkgs.follows = "nixpkgs";

<Mara>

Why is that second line needed?

<Cadey>

By default when you pull in another project with Nix flakes, it treats that project as an entirely separate universe and only interacts with the outputs of that flake. This means it pulls in its own version of nixpkgs, each dependency it has can pull in that own version of nixpkgs and vice versa ad infinitum. By making Xess' nixpkgs input follows our own one, we are saying "I understand this may be incompatible, but please use this version of nixpkgs instead". This can help larger projects with many inputs (such as a nixos configs repo made by someone with too many throwaway side projects) evaluate and build faster. Nix flakes does have a cached evaluator, but still it helps to avoid the problem in the first place.

Or anything you want! A useful library to pull in is flake-utils, that can help you simplify your flake.nix and get rid of those ugly forAllSystems and nixpkgsFor functions in the flake.nix that this post used by default. For an example of a flake that uses this library, see this flake.nix from the IRC bot that lives in #xeserv.

<Mara>

Adapting this trivial example to use flake-utils is an excellent exercise for the reader! This library should really be shipped with flakes by default.

Backwards Compatibility

Normally you need to enable Nix flakes in your Nix daemon to take advantage of them. This is great for when you can do that, but sometimes you'll need to make things work for people without flakes enabled. This could happen when needing to graft in a Nix flakes project to one without flakes enabled. There is a library called flake-compat that makes this easy.

Create default.nix with the following contents:

(import (
  fetchTarball {
    url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
    sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
) {
  src =  ./.;
}).defaultNix

And shell.nix with the following contents:

(import (
  fetchTarball {
    url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
    sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
) {
  src =  ./.;
}).shellNix

Then you can use nix-build and nix-shell like you have in other Nix projects.

Private Git Repos

Nix flakes has native support for private git repositories as inputs. This can be useful when trying to build software you don't want to release as open to the world. To use a private repo, your flake input URL should look something like this:

git+ssh://git@github.com/user/repo?ref=main

<Cadey>

I'm pretty sure you could use private git repos outside of flakes, however it was never really clear to me how you end up doing it.

<Mara>

I'm told you can bash Niv into shape enough to do this, but yeah it's never really been clear how you do this.

Embed NixOS Modules in Flakes

The biggest ticket item for me is that it lets you embed NixOS modules in flakes themselves. This lets you define the system configuration for software right next to where the software is defined, thus shipping it as a unit. Using this you can make installing software a matter of adding it to your system's flake, adding the module and then enabling the settings you want to enable.

As an example, here is the NixOS module for that IRC bot I mentioned:

nixosModules.bot = { config, lib, ... }: {
  options.within.services.mara-bot.enable =
    lib.mkEnableOption "enable Mara bot";

  config = lib.mkIf config.within.services.mara-bot.enable {
    users.groups.mara-bot = { };

    users.users.mara-bot = {
      createHome = true;
      isSystemUser = true;
      home = "/var/lib/mara-bot";
      group = "mara-bot";
    };

    systemd.services.mara-bot = {
      wantedBy = [ "multi-user.target" ];
      environment.RUST_LOG = "tower_http=debug,info";
      unitConfig.ConditionPathExists = "/var/lib/mara-bot/config.yaml";
      serviceConfig = {
        User = "mara-bot";
        Group = "mara-bot";
        Restart = "always";
        WorkingDirectory = "/var/lib/mara-bot";
        ExecStart = "${self.packages."${system}".default}/bin/mara";
      };
    };
  };
};

The key important part here is the ExecStart line. It points back to the flake's default package (which is hopefully where the bot's code is defined), and then has systemd manage that.

I plan to use this to radically simplify my nixos-configs repo. Right now it has a lot of code that is very project-specific and if I can move that into the projects in question, I can eliminate a lot of code out of the core of my configs repo.

Embedding Configuration Git Hash into Systems

Finally, Nix flakes lets you see the configuration version of a system by embedding it at the build step. Normally NixOS lets you see the following information with nixos-version --json:

{
  "nixosVersion": "22.05pre348581.c07b471b52b",
  "nixpkgsRevision": "c07b471b52be8fbc49a7dc194e9b37a6e19ee04d"
}

You have the NixOS version and the nixpkgs hash. That doesn't tell you what configuration you are running or anything about it though. However with flakes you can embed the git hash of your configuration into the system config:

{
  "configurationRevision": "f53891121ce4204f57409cbe9e6fcee3b030a350",
  "nixosVersion": "22.05.20220210.48d63e9",
  "nixpkgsRevision": "48d63e924a2666baf37f4f14a18f19347fbd54a2"
}

This can let you make a URL pointing to the commit in that output:

$ echo "https://tulpa.dev/cadey/nixos-configs/src/commit/$(ssh logos nixos-version --json | jq -r .configurationRevision)"

Which will spit out a link to cadey/nixos-configs@f53891121.

I'll cover more on how to do this in the NixOS deployment post.

There is a lot more to get into with each of these topics. I'm only really giving a very high level overview on them while I learn more and migrate over my NixOS configurations to flakes piecemeal. This has also given me the opportunity to clean things up and chew out a lot of the fat from my NixOS configurations. More to come when it is ready.

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: nix, nixos