Nixops Services on Your Home Network

My homelab has a few NixOS machines. Right now they mostly run services inside Docker, because that has been what I have done for years. This works fine, but persistent state gets annoying*. NixOS has a tool called Nixops that allows you to push configurations to remote machines. I use this for managing my fleet of machines, and today I'm going to show you how to create service deployments with Nixops and push them to your servers.

<Mara> Pedantically, Docker offers volumes to simplify this, but it is very easy to accidentally delete Docker volumes. Plain disk files like we are going to use today are a bit simpler than docker volumes, and thusly a bit harder to mess up.

Parts of a Service

For this example, let's deploy a chatbot. To make things easier, let's assume the following about this chatbot:

• The chatbot has a git repo somewhere
• The chatbot's git repo has a default.nix that builds the service and includes any supporting files it might need
• The chatbot reads its configuration from environment variables which may contain secret values (API keys, etc.)
• The chatbot stores any temporary files in its current working directory
• The chatbot is "well-behaved" (for some definition of "well-behaved")

I will also need to assume that you have a git repo (or at least a folder) with all of your configuration similar to mine.

For this example I'm going to use withinbot as the service we will deploy via Nixops. withinbot is a chatbot that I use on my own Discord guild that does a number of vital functions including supplying amusing facts about printers:

     <Cadey~> ~printerfact
<Within[BOT]> @Cadey~ Printers, especially older printers, do get cancer. Many
              times this disease can be treated successfully

<Mara> To get your own amusing facts about printers, see here or for using its API, call /fact. This API has no practical rate limits, but please don't test that.

Service Definition

We will need to do a few major things for defining this service:

1. Add the bot code as a package
2. Create a "services" folder for the service modules
3. Create a user account for the service
4. Set up a systemd unit for the service
5. Configure the secrets using Nixops keys

Add the Code as a Package

In order for the program to be installed to the remote system, you need to tell the system how to import it. There's many ways to do this, but the cheezy way is to add the packages to nixpkgs.config.packageOverrides like this:

nixpkgs.config = {
  packageOverrides = pkgs: {
    within = {
      withinbot = import (builtins.fetchTarball 
        "https://github.com/Xe/withinbot/archive/main.tar.gz") { };
    };
  };
};

And now we can access it as pkgs.within.withinbot in the rest of our config.

<Mara> In production circumstances you should probably use a fetcher that locks to a specific version using unique URLs and hashing, but this will work enough to get us off the ground in this example.

Create a "services" Folder

In your configuration folder, create a folder that you will use for these service definitions. I made mine in common/services. In that folder, create a default.nix with the following contents:

{ config, lib, ... }:

{
  imports = [ ./withinbot.nix ];

  users.groups.within = {};
}

The group listed here is optional, but I find that having a group like that can help you better share resources and files between services.

Now we need a folder for storing secrets. Let's create that under the services folder:

$ mkdir secrets

And let's also add a gitignore file so that we don't accidentally commit these secrets to the repo:

# common/services/secrets/.gitignore
*

Now we can put any secrets we want in the secrets folder without the risk of committing them to the git repo.

Service Manifest

Let's create withinbot.nix and set it up:

{ config, lib, pkgs, ... }:
with lib; {
  options.within.services.withinbot.enable =
    mkEnableOption "Activates Withinbot (the furryhole chatbot)";

  config = mkIf config.within.services.withinbot.enable {
    
  };
}

This sets up an option called within.services.withinbot.enable which will only add the service configuration if that option is set to true. This will allow us to define a lot of services that are available, but none of their config will be active unless they are explicitly enabled.

Now, let's create a user account for the service:

# ...
  config = ... {
    users.users.withinbot = {
      createHome = true;
      description = "github.com/Xe/withinbot";
      isSystemUser = true;
      group = "within";
      home = "/srv/within/withinbot";
      extraGroups = [ "keys" ];
    };
  };
# ...

This will create a user named withinbot with the home directory /srv/within/withinbot, the group within and also in the group keys so the withinbot user can read deployment secrets.

Now let's add the deployment secrets to the configuration:

# ...
  config = ... {
    users.users.withinbot = { ... };
    
    deployment.keys.withinbot = {
      text = builtins.readFile ./secrets/withinbot.env;
      user = "withinbot";
      group = "within";
      permissions = "0640";
    };
  };
# ...

Assuming you have the configuration at ./secrets/withinbot.env, this will register the secrets into /run/keys/withinbot and also create a systemd oneshot service named withinbot-key. This allows you to add the secret's existence as a condition for withinbot to run. However, Nixops puts these keys in /run, which by default is mounted using a temporary memory-only filesystem, meaning these keys will need to be re-added to machines when they are rebooted. Fortunately, nixops reboot will automatically add the keys back after the reboot succeeds.

Now that we have everything else we need, let's add the service configuration:

# ...
  config = ... {
    users.users.withinbot = { ... };
    deployment.keys.withinbot = { ... };
    
    systemd.services.withinbot = {
      wantedBy = [ "multi-user.target" ];
      after = [ "withinbot-key.service" ];
      wants = [ "withinbot-key.service" ];

      serviceConfig = {
        User = "withinbot";
        Group = "within";
        Restart = "on-failure"; # automatically restart the bot when it dies
        WorkingDirectory = "/srv/within/withinbot";
        RestartSec = "30s";
      };

      script = let withinbot = pkgs.within.withinbot;
      in ''
        # load the environment variables from /run/keys/withinbot
        export $(grep -v '^#' /run/keys/withinbot | xargs)
        # service-specific configuration
        export CAMPAIGN_FOLDER=${withinbot}/campaigns
        # kick off the chatbot
        exec ${withinbot}/bin/withinbot
      '';
    };
  };
# ...

This will create the systemd configuration for the service so that it starts on boot, waits to start until the secrets have been loaded into it, runs withinbot as its own user and in the within group, and throttles the service restart so that it doesn't incur Discord rate limits as easily. This will also put all withinbot logs in journald, meaning that you can manage and monitor this service like you would any other systemd service.

Deploying the Service

In your target server's configuration.nix file, add an import of your services directory:

{
  # ...
  imports = [
    # ...
    /home/cadey/code/nixos-configs/common/services
  ];
  # ...
}

And then enable the withinbot service:

{
  # ...
  within.services = {
    withinbot.enable = true;
  };
  # ...
}

<Mara> Make that a block so you can enable multiple services at once like this!

Now you are free to deploy it to your network with nixops deploy:

$ nixops deploy -d hexagone

And then you can verify the service is up with systemctl status:

$ nixops ssh -d hexagone chrysalis -- systemctl status withinbot
● withinbot.service
     Loaded: loaded (/nix/store/7ab7jzycpcci4f5wjwhjx3al7xy85ka7-unit-withinbot.service/withinbot.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-11-09 09:51:51 EST; 2h 29min ago
   Main PID: 12295 (withinbot)
         IP: 0B in, 0B out
      Tasks: 13 (limit: 4915)
     Memory: 7.9M
        CPU: 4.456s
     CGroup: /system.slice/withinbot.service
             └─12295 /nix/store/qpq281hcb1grh4k5fm6ksky6w0981arp-withinbot-0.1.0/bin/withinbot

Nov 09 09:51:51 chrysalis systemd[1]: Started withinbot.service.

This basic template is enough to expand out to anything you would need and is what I am using for my own network. This should be generic enough for most of your needs. Check out the NixOS manual for more examples and things you can do with this. The Nixops manual is also a good read. It can also set up deployments with VirtualBox, libvirtd, AWS, Digital Ocean, and even Google Cloud.

The cloud is the limit! Be well.