Cadey is coffee
<Cadey> Hello! Thank you for visiting my website. You seem to be using an ad-blocker. I understand why you do this, but I'd really appreciate if it you would turn it off for my website. These ads help pay for running the website and are done by Ethical Ads. I do not receive detailed analytics on the ads and from what I understand neither does Ethical Ads. If you don't want to disable your ad blocker, please consider donating on Patreon or sending some extra cash to xeiaso.eth or 0xeA223Ca8968Ca59e0Bc79Ba331c2F6f636A3fB82. It helps fund the website's hosting bills and pay for the expensive technical editor that I use for my longer articles. Thanks and be well!

Dhall for Kubernetes

Read time in minutes: 14

Kubernetes is a surprisingly complicated software package. Arguably, it has to be that complicated as a result of the problems it solves being complicated; but managing yaml configuration files for Kubernetes is a complicated task. YAML doesn't have support for variables or type metadata. This means that the validity (or sensibility) of a given Kubernetes configuration file (or files) isn't easy to figure out without using a Kubernetes server.

In my last post about Kubernetes, I mentioned I had developed a tool named dyson in order to help me manage Terraform as well as create Kubernetes manifests from a template. This works for the majority of my apps, but it is difficult to extend at this point for a few reasons:

  • It assumes that everything passed to it are already valid yaml terms
  • It doesn't assert the type of any values passed to it
  • It is difficult to add another container to a given deployment
  • Environment variables implicitly depend on the presence of a private git repo
  • It depends on the template being correct more than the output being correct

So, this won't scale. People in the community have created other solutions for this like Helm, but a lot of them have some of the same basic problems. Helm also assumes that your template is correct. Kustomize does help with a lot of the type-safe variable replacements, but it doesn't have the ability to ensure your manifest is valid.

I looked around for alternate solutions for a while and eventually found Dhall thanks to a friend. Dhall is a statically typed configuration language. This means that you can ensure that inputs are always the correct type or the configuration file won't load. There's also a built-in dhall-to-yaml tool that can be used with the Kubernetes package in order to declare Kubernetes manifests in a type-safe way.

Here's a small example of Dhall and the yaml it generates:

-- Mastodon usernames
[ { name = "Cadey", mastodon = "@cadey@mst3k.interlinked.me" }
, { name = "Nicole", mastodon = "@sharkgirl@mst3k.interlinked.me" }
]

Which produces:

- mastodon: "@cadey@mst3k.interlinked.me"
  name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
  name: Nicole

Which is fine, but we still have the type-safety problem that you would have in normal yaml. Dhall lets us define record types for this data like this:

let User =
      { Type = { name : Text, mastodon : Optional Text }
      , default = { name = "", mastodon = None }
      }

let users =
      [ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me" }
      , User::{
        , name = "Nicole"
        , mastodon = Some "@sharkgirl@mst3k.interlinked.me"
        }
      ]

in  users

Which produces:

- mastodon: "@cadey@mst3k.interlinked.me"
  name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
  name: Nicole

This is type-safe because you cannot add arbitrary fields to User instances without the compiler rejecting it. Let's add an invalid "preferred_language" field to Cadey's instance:

-- ...
let users =
      [ User::{
        , name = "Cadey"
        , mastodon = Some "@cadey@mst3k.interlinked.me"
        , preferred_language = "en-US"
        }
      -- ...
      ]

Which gives us:

$ dhall-to-yaml --file example.dhall
Error: Expression doesn't match annotation

{ + preferred_language : …
, …
}

4│         User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me",
5│       preferred_language = "en-US" }

example.dhall:4:9

Or this more detailed explanation if you add the --explain flag to the dhall-to-yaml call.

We tried to do something that violated the contract that the type specified. This means that it's an invalid configuration and is therefore rejected and no yaml file is created.

The Dhall Kubernetes package specifies record types for every object available by default in Kubernetes. This does mean that the package is incredibly large, but it also makes sure that everything you could possibly want to do in Kubernetes matches what it expects. In the package documentation, they give an example where a Deployment is created.

-- examples/deploymentSimple.dhall

-- Importing other files is done by specifying the HTTPS URL/disk location of
-- the file. Attaching a sha256 hash (obtained with `dhall freeze`) allows
-- the Dhall compiler to cache these files and speed up configuration loads
-- drastically.
let kubernetes =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/1.15/master/package.dhall
      sha256:4bd5939adb0a5fc83d76e0d69aa3c5a30bc1a5af8f9df515f44b6fc59a0a4815
      
let deployment =
      kubernetes.Deployment::{
      , metadata = kubernetes.ObjectMeta::{ name = "nginx" }
      , spec =
          Some
            kubernetes.DeploymentSpec::{
            , replicas = Some 2
            , template =
                kubernetes.PodTemplateSpec::{
                , metadata = kubernetes.ObjectMeta::{ name = "nginx" }
                , spec =
                    Some
                      kubernetes.PodSpec::{
                      , containers =
                          [ kubernetes.Container::{
                            , name = "nginx"
                            , image = Some "nginx:1.15.3"
                            , ports =
                                [ kubernetes.ContainerPort::{
                                  , containerPort = 80
                                  }
                                ]
                            }
                          ]
                      }
                }
            }
      }

in  deployment

Which creates the following yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  template:
    metadata:
      name: nginx
    spec:
      containers:
        - image: nginx:1.15.3
          name: nginx
          ports:
            - containerPort: 80

Dhall's lambda functions can help you break this into manageable chunks. For example, here's a Dhall function that helps create a docker image reference:

let formatImage
    : Text -> Text -> Text
    = \(repository : Text) -> \(tag : Text) ->
    "${repository}:${tag}"

in formatImage "xena/christinewebsite" "latest"

Which outputs xena/christinewebsite:latest when passed to dhall text.

All of this adds up into a powerful toolset that lets you express Kubernetes configuration in a way that does what you want without as many headaches.

Most of my apps on Kubernetes need only a few generic bits of configuration:

  • Their name
  • What port should be exposed
  • The domain that this service should be exposed on
  • How many replicas of the service are needed
  • Which Let's Encrypt Issuer to use (currently only "prod" or "staging")
  • The configuration variables of the service
  • Any other containers that may be needed for the service

From here, I defined all of the bits and pieces for the Kubernetes manifests that Dyson produces and then created a Config type that helps to template them out. Here's my Config type definition:

let kubernetes = ../kubernetes.dhall

in  { Type =
        { name : Text
        , appPort : Natural
        , image : Text
        , domain : Text
        , replicas : Natural
        , leIssuer : Text
        , envVars : List kubernetes.EnvVar.Type
        , otherContainers : List kubernetes.Container.Type
        }
    , default =
        { name = ""
        , appPort = 5000
        , image = ""
        , domain = ""
        , replicas = 1
        , leIssuer = "staging"
        , envVars = [] : List kubernetes.EnvVar.Type
        , otherContainers = [] : List kubernetes.Container.Type
        }
    }

Then I defined a makeApp function that creates everything I need to deploy my stuff on Kubernetes:

let Prelude = ../Prelude.dhall

let kubernetes = ../kubernetes.dhall

let typesUnion = ../typesUnion.dhall

let deployment = ../http/deployment.dhall

let ingress = ../http/ingress.dhall

let service = ../http/service.dhall

let Config = ../app/config.dhall

let K8sList = ../app/list.dhall

let buildService =
        \(config : Config.Type)
      -> let myService = service config

         let myDeployment = deployment config

         let myIngress = ingress config

         in  K8sList::{
             , items =
               [ typesUnion.Service myService
               , typesUnion.Deployment myDeployment
               , typesUnion.Ingress myIngress
               ]
             }

in  buildService

And used it to deploy the h language website:

let makeApp = ../app/make.dhall

let Config = ../app/config.dhall

let cfg =
      Config::{
      , name = "hlang"
      , appPort = 5000
      , image = "xena/hlang:latest"
      , domain = "h.christine.website"
      , leIssuer = "prod"
      }

in  makeApp cfg

Which produces the following Kubernetes config:

apiVersion: v1
items:
  - apiVersion: v1
    kind: Service
    metadata:
      annotations:
        external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
        external-dns.alpha.kubernetes.io/hostname: h.christine.website
        external-dns.alpha.kubernetes.io/ttl: "120"
      labels:
        app: hlang
      name: hlang
      namespace: apps
    spec:
      ports:
        - port: 5000
          targetPort: 5000
      selector:
        app: hlang
      type: ClusterIP
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hlang
      namespace: apps
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: hlang
      template:
        metadata:
          labels:
            app: hlang
          name: hlang
        spec:
          containers:
            - image: xena/hlang:latest
              imagePullPolicy: Always
              name: web
              ports:
                - containerPort: 5000
          imagePullSecrets:
            - name: regcred
  - apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        certmanager.k8s.io/cluster-issuer: letsencrypt-prod
        kubernetes.io/ingress.class: nginx
      labels:
        app: hlang
      name: hlang
      namespace: apps
    spec:
      rules:
        - host: h.christine.website
          http:
            paths:
              - backend:
                  serviceName: hlang
                  servicePort: 5000
      tls:
        - hosts:
            - h.christine.website
          secretName: prod-certs-hlang
kind: List

And when I applied it on my Kubernetes cluster, it worked the first time and had absolutely no effect on the existing configuration.

In the future, I hope to expand this to allow for multiple deployments (IE: a chatbot running in a separate deployment than a web API the chatbot depends on or non-web projects in general) as well as supporting multiple Kubernetes namespaces.

Dhall is probably the most viable replacement to Helm or other Kubernetes templating tools I have found in recent memory. I hope that it will be used by more people to help with configuration management, but I can understand that that may not happen. At least it works for me.

If you want to learn more about Dhall, I suggest checking out the following links:

I hope this was helpful and interesting. Be well.


This article was posted on M01 25 2020. Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: dhall kubernetes witchcraft

This post was not WebMentioned yet. You could be the first!

The art for Mara was drawn by Selicre.

The art for Cadey was drawn by ArtZora Studios.

Some of the art for Aoi was drawn by @Sandra_Thomas01.