Dhall for Kubernetes
Read time in minutes: 14Kubernetes is a surprisingly complicated software package. Arguably, it has to be that complicated as a result of the problems it solves being complicated; but managing yaml configuration files for Kubernetes is a complicated task. YAML doesn't have support for variables or type metadata. This means that the validity (or sensibility) of a given Kubernetes configuration file (or files) isn't easy to figure out without using a Kubernetes server.
In my last post about Kubernetes, I mentioned I had developed a tool named dyson in order to help me manage Terraform as well as create Kubernetes manifests from a template. This works for the majority of my apps, but it is difficult to extend at this point for a few reasons:
- It assumes that everything passed to it are already valid yaml terms
- It doesn't assert the type of any values passed to it
- It is difficult to add another container to a given deployment
- Environment variables implicitly depend on the presence of a private git repo
- It depends on the template being correct more than the output being correct
So, this won't scale. People in the community have created other solutions for this like Helm, but a lot of them have some of the same basic problems. Helm also assumes that your template is correct. Kustomize does help with a lot of the type-safe variable replacements, but it doesn't have the ability to ensure your manifest is valid.
I looked around for alternate solutions for a while and eventually found Dhall thanks to a friend. Dhall is a statically typed configuration language. This means that you can ensure that inputs are always the correct type or the configuration file won't load. There's also a built-in dhall-to-yaml tool that can be used with the Kubernetes package in order to declare Kubernetes manifests in a type-safe way.
Here's a small example of Dhall and the yaml it generates:
-- Mastodon usernames
[ { name = "Cadey", mastodon = "@cadey@mst3k.interlinked.me" }
, { name = "Nicole", mastodon = "@sharkgirl@mst3k.interlinked.me" }
]
Which produces:
- mastodon: "@cadey@mst3k.interlinked.me"
name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
name: Nicole
Which is fine, but we still have the type-safety problem that you would have in normal yaml. Dhall lets us define record types for this data like this:
let User =
{ Type = { name : Text, mastodon : Optional Text }
, default = { name = "", mastodon = None }
}
let users =
[ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me" }
, User::{
, name = "Nicole"
, mastodon = Some "@sharkgirl@mst3k.interlinked.me"
}
]
in users
Which produces:
- mastodon: "@cadey@mst3k.interlinked.me"
name: Cadey
- mastodon: "@sharkgirl@mst3k.interlinked.me"
name: Nicole
This is type-safe because you cannot add arbitrary fields to User instances without the compiler rejecting it. Let's add an invalid "preferred_language" field to Cadey's instance:
-- ...
let users =
[ User::{
, name = "Cadey"
, mastodon = Some "@cadey@mst3k.interlinked.me"
, preferred_language = "en-US"
}
-- ...
]
Which gives us:
$ dhall-to-yaml --file example.dhall
Error: Expression doesn't match annotation
{ + preferred_language : …
, …
}
4│ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me",
5│ preferred_language = "en-US" }
example.dhall:4:9
Or this more detailed explanation if you add the --explain flag
to the dhall-to-yaml call.
We tried to do something that violated the contract that the type specified. This means that it's an invalid configuration and is therefore rejected and no yaml file is created.
The Dhall Kubernetes package specifies record types for every object available by default in Kubernetes. This does mean that the package is incredibly large, but it also makes sure that everything you could possibly want to do in Kubernetes matches what it expects. In the package documentation, they give an example where a Deployment is created.
-- examples/deploymentSimple.dhall
-- Importing other files is done by specifying the HTTPS URL/disk location of
-- the file. Attaching a sha256 hash (obtained with `dhall freeze`) allows
-- the Dhall compiler to cache these files and speed up configuration loads
-- drastically.
let kubernetes =
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/1.15/master/package.dhall
sha256:4bd5939adb0a5fc83d76e0d69aa3c5a30bc1a5af8f9df515f44b6fc59a0a4815
let deployment =
kubernetes.Deployment::{
, metadata = kubernetes.ObjectMeta::{ name = "nginx" }
, spec =
Some
kubernetes.DeploymentSpec::{
, replicas = Some 2
, template =
kubernetes.PodTemplateSpec::{
, metadata = kubernetes.ObjectMeta::{ name = "nginx" }
, spec =
Some
kubernetes.PodSpec::{
, containers =
[ kubernetes.Container::{
, name = "nginx"
, image = Some "nginx:1.15.3"
, ports =
[ kubernetes.ContainerPort::{
, containerPort = 80
}
]
}
]
}
}
}
}
in deployment
Which creates the following yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
template:
metadata:
name: nginx
spec:
containers:
- image: nginx:1.15.3
name: nginx
ports:
- containerPort: 80
Dhall's lambda functions can help you break this into manageable chunks. For example, here's a Dhall function that helps create a docker image reference:
let formatImage
: Text -> Text -> Text
= \(repository : Text) -> \(tag : Text) ->
"${repository}:${tag}"
in formatImage "xena/christinewebsite" "latest"
Which outputs xena/christinewebsite:latest when passed to dhall text.
All of this adds up into a powerful toolset that lets you express Kubernetes configuration in a way that does what you want without as many headaches.
Most of my apps on Kubernetes need only a few generic bits of configuration:
- Their name
- What port should be exposed
- The domain that this service should be exposed on
- How many replicas of the service are needed
- Which Let's Encrypt Issuer to use (currently only
"prod"or"staging") - The configuration variables of the service
- Any other containers that may be needed for the service
From here, I defined all of the bits and pieces for the
Kubernetes manifests that Dyson produces and then created a Config type that
helps to template them out. Here's my Config type
definition:
let kubernetes = ../kubernetes.dhall
in { Type =
{ name : Text
, appPort : Natural
, image : Text
, domain : Text
, replicas : Natural
, leIssuer : Text
, envVars : List kubernetes.EnvVar.Type
, otherContainers : List kubernetes.Container.Type
}
, default =
{ name = ""
, appPort = 5000
, image = ""
, domain = ""
, replicas = 1
, leIssuer = "staging"
, envVars = [] : List kubernetes.EnvVar.Type
, otherContainers = [] : List kubernetes.Container.Type
}
}
Then I defined a makeApp function that creates everything I need to deploy my
stuff on Kubernetes:
let Prelude = ../Prelude.dhall
let kubernetes = ../kubernetes.dhall
let typesUnion = ../typesUnion.dhall
let deployment = ../http/deployment.dhall
let ingress = ../http/ingress.dhall
let service = ../http/service.dhall
let Config = ../app/config.dhall
let K8sList = ../app/list.dhall
let buildService =
\(config : Config.Type)
-> let myService = service config
let myDeployment = deployment config
let myIngress = ingress config
in K8sList::{
, items =
[ typesUnion.Service myService
, typesUnion.Deployment myDeployment
, typesUnion.Ingress myIngress
]
}
in buildService
And used it to deploy the h language website:
let makeApp = ../app/make.dhall
let Config = ../app/config.dhall
let cfg =
Config::{
, name = "hlang"
, appPort = 5000
, image = "xena/hlang:latest"
, domain = "h.christine.website"
, leIssuer = "prod"
}
in makeApp cfg
Which produces the following Kubernetes config:
apiVersion: v1
items:
- apiVersion: v1
kind: Service
metadata:
annotations:
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
external-dns.alpha.kubernetes.io/hostname: h.christine.website
external-dns.alpha.kubernetes.io/ttl: "120"
labels:
app: hlang
name: hlang
namespace: apps
spec:
ports:
- port: 5000
targetPort: 5000
selector:
app: hlang
type: ClusterIP
- apiVersion: apps/v1
kind: Deployment
metadata:
name: hlang
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: hlang
template:
metadata:
labels:
app: hlang
name: hlang
spec:
containers:
- image: xena/hlang:latest
imagePullPolicy: Always
name: web
ports:
- containerPort: 5000
imagePullSecrets:
- name: regcred
- apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
labels:
app: hlang
name: hlang
namespace: apps
spec:
rules:
- host: h.christine.website
http:
paths:
- backend:
serviceName: hlang
servicePort: 5000
tls:
- hosts:
- h.christine.website
secretName: prod-certs-hlang
kind: List
And when I applied it on my Kubernetes cluster, it worked the first time and had absolutely no effect on the existing configuration.
In the future, I hope to expand this to allow for multiple deployments (IE: a chatbot running in a separate deployment than a web API the chatbot depends on or non-web projects in general) as well as supporting multiple Kubernetes namespaces.
Dhall is probably the most viable replacement to Helm or other Kubernetes templating tools I have found in recent memory. I hope that it will be used by more people to help with configuration management, but I can understand that that may not happen. At least it works for me.
If you want to learn more about Dhall, I suggest checking out the following links:
I hope this was helpful and interesting. Be well.