Unix Domain Sockets for Serving HTTP in Production
Published on , 2016 words, 8 minutes to read
Securing production servers can be a chore. It is a seemingly endless game of balancing risks with convenience and not breaking what you want to do. Small, incremental gains are usually a very good idea however. Today we'll learn how to use Unix Domain Sockets to host your HTTP services. This allows you to run your services like normal on production machines without there being a risk of people being able to access the raw HTTP port.
Wait, what. You're having a service listen on a file? Why would you want to do this?
Mostly to prevent you from messing up and accidentally exposing your backend port to the internet. Firewall configuration is probably the most "correct" way to solve that concern, however this lets you also take advantage of filesystem permissions to fine-tune access down to the exact users and groups that should have access to the socket. In our case we only want ngnix to access this socket, so we can use filesystem permissions (and a unix group) to ensure this. Attackers can't connect to anything they aren't able to connect to.
At a high level every file in a unix filesystem has 3 kinds of permissions: user, group and "other". Every file has an owner and a UNIX group associated with it. Here's an example using the Cargo.toml of this website's app server:
$ stat ./Cargo.toml
File: ./Cargo.toml
Size: 1572 Blocks: 8 IO Block: 4096 regular file
Device: 10301h/66305d Inode: 20447261 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ cadey) Gid: ( 100/ users)
Access: 2021-04-01 19:48:44.791162535 -0400
Modify: 2021-04-01 19:48:44.786162545 -0400
Change: 2021-04-01 19:48:44.786162545 -0400
Birth: 2021-03-25 09:09:35.490311674 -0400
In this case the permissions of this file are 0644, which is a base-8 (octal)
number that describes the permissions for the user, group and others. It breaks
up something like this:
If we wanted to create a socket that only nginx can access, assuming we
share a group with nginx we would need a socket with something like 0770 (user
and group can read, write and "execute", everyone else gets denied) for its
permissions. Then we would need to chuck it somewhere that both the app backend
and nginx have access to and finally configure nginx to do this.
So let's do it! Let's take the venerable printer facts server server and make it listen on a Unix socket. Right now it uses something like this to listen for requests:
warp::serve(
fact_handler
.or(index_handler)
.or(files)
.or(not_found_handler)
.with(warp::log(APPLICATION_NAME)),
)
.run(([0, 0, 0, 0], port))
.await;
This configures warp (the HTTP framework
that I'm using for the printer facts server) to listen over TCP on some port.
This is hard-coded to listen on 0.0.0.0, which means that TCP sessions from
any network interface can connect to the service. This is very convenient
for development, so we are going to want to keep this behaviour in some way.
Fortunately warp has an
example
for listening on a unix socket. Let's make the service listen on
./printerfacts.sock so we can make sure that everything still works:
let server = warp::serve(
fact_handler
.or(index_handler)
.or(files)
.or(not_found_handler)
.with(warp::log(APPLICATION_NAME)),
);
if let Ok(sockpath) = std::env::var("SOCKPATH") {
use tokio::net::UnixListener;
use tokio_stream::wrappers::UnixListenerStream;
let listener = UnixListener::bind(sockpath).unwrap();
let incoming = UnixListenerStream::new(listener);
server.run_incoming(incoming).await;
} else {
server.run(([0, 0, 0, 0], port));
}
Then we can launch the service with a domain socket using a command like this:
$ env SOCKPATH=./printerfacts.sock cargo run
Let's see how the output of stat(1) changed compared to when we ran it on a
file:
$ stat ./printerfacts.sock
File: ./printerfacts.sock
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 10301h/66305d Inode: 23858442 Links: 1
Access: (0755/srwxr-xr-x) Uid: ( 1001/ cadey) Gid: ( 100/ users)
Access: 2021-04-01 21:00:51.558219253 -0400
Modify: 2021-04-01 21:00:51.558219253 -0400
Change: 2021-04-01 21:00:51.558219253 -0400
Birth: 2021-04-01 21:00:51.558219253 -0400
stat(1) reports that the file is a socket! Let's see if everything still works
by using curl --unix-socket to connect to the service and retrieve an amusing
fact about printers:
$ curl --unix-socket ./printerfacts.sock http://foo/fact
The strongest climber among the big printers, a leopard can carry prey twice its
weight up a tree.
It's important to note that the HTTP host doesn't matter here. I could have put anything there, but foo is fast for me
to type. The URL host information usually tells curl where to connect, but the
--unix-socket flag overrides this logic.
Wait, what the heck are printer facts?
Blame Foone and #infoforcefeed.
Anyways, let's make the TCP logic a bit more clean in the process. Right now it
only listens on IPv4 and it would be nice if it listened on IPv6 too. Let's
replace that last else body with this:
} else {
server
.run((std::net::IpAddr::from_str("::").unwrap(), port))
.await;
}
:: is the IPv6 version of 0.0.0.0, or the
unspecified address
. It tells most IP stacks to allow traffic from any network interface.
Now let's re-build the printer facts service and re-run it to make sure it still works:
$ env SOCKPATH=./printerfacts.sock cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.04s
Running `target/debug/printerfacts`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os {
code: 98, kind: AddrInUse, message: "Address already in use" }',
src/main.rs:73:53
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Wait, what. Isn't this serving HTTP from a file? Why would it be an address in use error?
Even though it looks like a file to us humans, it's still a socket under the hood. In this case it means the filename is already in use. Working around this is simple though, all we need to do is-
DELETE THIS!
But yes, we do need to delete the socket file if it doesn't already exist. Let's sneak this bit of code in before we listen on the Unix socket:
if let Ok(sockpath) = std::env::var("SOCKPATH") {
let _ = std::fs::remove_file(&sockpath); // nuke the socket
let listener = UnixListener::bind(sockpath).unwrap();
let incoming = UnixListenerStream::new(listener);
server.run_incoming(incoming).await;
} else {
server
.run((std::net::IpAddr::from_str("::").unwrap(), port))
.await;
}
Didn't you just say "if it doesn't already exist"? Why delete it unconditionally and throw away any errors?
Two reasons:
- Statistically if the file doesn't exist and the service can't create it when it binds to that path, you probably have bigger problems and it's probably better for the program to explode there.
- The filename is passed in as an environment variable. If your environment variable is wrong, we can treat this as a fundamental assertion error and blow up when the file fails to bind.
Let's define this in the NixOS module for the printerfacts service. First we will need to add a configuration option for the socket path:
let cfg = config.within.services.printerfacts;
in {
options.within.services.printerfacts = {
# ...
sockPath = mkOption rec {
type = types.str;
default = "/tmp/printerfacts.sock";
example = default;
description = "The unix domain socket that printerfacts should listen on";
};
};
# ...
}
This creates an option at cfg.sockPath that we can pipe through elsewhere,
such as the start script for the service:
# inside
script = let site = pkgs.tulpa.dev.cadey.printerfacts;
in ''
export SOCKPATH=${cfg.sockPath}
export DOMAIN=${toString cfg.domain}
export RUST_LOG=info
cd ${site}
exec ${site}/bin/printerfacts
'';
And then we can go on to setting up nginx. First, let's figure out how to
reverse proxy to a unix socket. In nginx configuration land,
proxy_pass
is the name of the configuration directive that lets you tell nginx to reverse
proxy to somewhere. There's an example with a unix socket! This would let us
reverse proxy a unix socket to a TCP port like this:
server {
listen 127.0.0.1:9000;
location / {
proxy_pass http://unix:/tmp/printerfacts.sock;
}
}
For comparison here's how you'd reverse proxy to a HTTP server running on port
42069:
server {
listen 127.0.0.1:9001;
location / {
proxy_pass http://127.0.0.1:42069;
}
}
So, we just need to change where nginx reverse proxies to in the NixOS config.
Let's look down at the nginx config for printerfacts:
# ...
services.nginx.virtualHosts."${cfg.domain}" = {
locations."/" = {
proxyPass = "http://127.0.0.1:${toString cfg.port}";
proxyWebsockets = true;
};
forceSSL = cfg.useACME;
useACMEHost = "cetacean.club";
extraConfig = ''
access_log /var/log/nginx/printerfacts.access.log;
'';
};
The proxyPass option directly translates to a proxy_pass directive, so we
can get away with something like this:
# ...
proxyPass = "http://unix:${cfg.sockPath}";
And now we can deploy the service and everything should work right? printerfacts provides a unix socket at the given path and then nginx is configured to use that socket to send back printer facts. Let's deploy it and see what happens:
Oh no. Let's see what journalctl -fu nginx has to say:
$ journalctl -fu nginx
Apr 01 23:29:58 lufta nginx[15396]: 2021/04/01 23:29:58 [crit] 15396#15396: *198
connect() to unix:/tmp/printerfacts.sock failed (13: Permission denied) while
connecting to upstream, client: lol.no.ip.here, server:
printerfacts.cetacean.club, request: "GET / HTTP/2.0", upstream:
"http://unix:/tmp/printerfacts.sock:/", host: "printerfacts.cetacean.club"
Wait, what. Isn't /tmp guaranteed by the filesystem hierarchy standards to
always be readable and writable by any user?
Normally, yes. However we are running nginx inside systemd, and one of the
things you can do with systemd is make /tmp isolated for given services.
This allows you to prevent a service from being able to exfiltrate data inside
/tmp. However, this is definitely NOT the behaviour we want in this case.
Let's change the systemd unit for nginx to disable this and also make nginx run as the same group as the printerfacts service:
systemd.services.nginx.serviceConfig = {
PrivateTmp = lib.mkForce "false";
SupplementaryGroups = "within";
};
In NixOS, most of the time if the same option is declared in multiple places
it will result in a build error. lib.mkForce disables this behaviour and
instead "forcibly" sets this value.
Now nginx has the same /tmp as the printerfacts service, everything will work
as we expect. Users are none the wiser that I'm using a domain socket here. I
get to have another service not bound to the network and I have moved towards
better security on my machine!
What about Prometheus? Doesn't it need a direct line of fire to the service to scrape metrics?
...Time for some percussive maintenance!
I'm experimenting with a new "smol" mode for the Mara interludes as well as introducing a few more characters to the xeiaso dot net cinematic universe. Please do let me know how this works out for you. I think I have the sizes optimized for mobile usage better, but contributions to fix my horrible CSS would really, really, really be appreciated.
I'm considering moving over all of the Mara interludes to use smol mode. If you have opinions about this please let me know them.
Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.
Tags: unix, nginx, devops, nixos, systemd