Anubis continues to expose new ways people configure webservers

One of the most annoying parts of writing web applications is that in general: you can't trust browsers. But, you have to trust browsers at some level because that's how users interact with your software. As browsers get more capable with APIs like WebUSB, Built-in AI, or other absurd things; administrators want to be able to turn off the features that their web applications don't use. This is the crux of why Content-Security-Policies (CSPs) exist.

Mara

Normally we avoid acronyms when writing posts like this, but for the purpose of this article when you see "CSP", think "Content-Security-Policy".

In general, a CSP disables all browser features and then selectively enables the features the website actually needs. For example (stolen from the Anubis docs):

default-src 'none';
script-src  'self' 'unsafe-inline';
style-src   'self' 'unsafe-inline';
img-src     'self';
font-src    'self' data:;
connect-src 'self';
worker-src  'self' blob:;
base-uri    'none';
form-action 'self';

This disables all browser features except loading scripts from the same origin, inline JavaScript in <script> tags, inline CSS, loading CSS from the same origin, loading images from the same origin, loading fonts from the same origin, loading fonts inline to CSS files (via data: URIs), making fetch() requests to the same origin, loading Worker scripts from the same origin, loading Worker scripts from blob: URIs, disallowing the use of the <base> element, and only allowing HTML <form> actions against the same origin.

Extra fun, when you have a CSP that forbids loading Worker scripts from blob: URIs, you don't get the error until after the Worker is constructed and the browser forks a background thread:

blobURL = URL.createObjectURL(
  new Blob([`console.log("Hello, world!");`], { type: "text/javascript" }),
);
const w = new Worker(blobURL);
// does not throw an error

You have to catch it in the async .onerror callback:

w.onerror = (event) => {
  console.error(`Got an error: ${event}`);
};

So if you (like me) implemented fallback logic that depends on this, you need to adapt your logic to account for this.

Let's face it, users don't like it when they get an Anubis challenge page. I've tried to make them show up less often, but this doesn't scale as the scrapers adapt to the changes I make. One of the ways Anubis mitigates the pain of seeing a challenge page is by making it go away as fast as possible by running its proof of work checks run in parallel. This works out pretty well as most CPU advancements in the past decade or so are around multi-core performance, not single-core performance.

By default, when you create a Worker pointed to a JavaScript program on your web server, browsers make requests to the server to load that program:

const w = new Worker("/static/js/worker/test1.mjs");

This results in the browser sending a GET /static/js/worker/test1.mjs request to the server which hopefully results in getting JavaScript source back. The browser then executes that JavaScript code in parallel and sets up the worker environment so that the program can do whatever it is that it needs to do.

One of the horrible parts of this is that when you spawn many workers in parallel, such as how Anubis does it:

const getHardwareConcurrency = () =>
  navigator.hardwareConcurrency !== undefined
    ? navigator.hardwareConcurrency
    : 1;

let workers: Worker[] = [];
const threads = Math.trunc(Math.max(getHardwareConcurrency() / 2, 1));

for (let i = 0; i < threads; i++) {
  let w: Worker;
  try {
    w = new Worker("/whatever/worker.mjs");
  } catch (err) {
    magic!(cleanup);
    magic!(throwError);
    return;
  }

  workers.push(w);

  // Draw the rest of the owl
}

This results in threads number of HTTP requests to the server. In circumstances where the server is already overloaded (such as when scrapers attack in droves from nearly every ISO country code on the planet), this means that a user getting through to the webpage can result in as many as 16 extra HTTP requests to the server. Even worse, there's not an easy way to do exponential backoff without adding fiddly logic to the parts surrounding the Worker constructor.

In order to work around this, Anubis loads the worker source once from the server with a standard fetch() request and then packs that into a blob: URI so clients don't need to make many parallel requests to the server.

The old logic that fans out requests is maintained in case admins have a CSP that forbids the use of blob: URIs. It's kinda sucky that it has to be there and mandates adding extra testing to ensure this works, but in this era of late stage capitalism we kinda need to make sure that things are reliable on the client even if this can cause increased request pressure on an already overloaded server.

Cadey

As a side note, this only really works because Anubis assumes that its worker code is inerrant unless something completely unrecoverable happens. Most of the proof of work code is "just math"*, so if the math fails then the user's CPU or ram is probably failing and the server will disagree anyways.

Ideally, you'd want the entire challenge solve attempt to get killed if any worker threads error after they start crunching solutions, but in practice it's "fine-ish" to lose a worker or two as long as there's at least one worker running.

One way to think about how the proof of work solver works is that each worker is a thread that gets its own "lane" of the nonce space to find solutions within. In general it's fair to assume that solutions are "dense" enough that losing any workers is "fine-ish" at the cost of skipping over solutions that may be in that "lane". Future improvements may involve trying to re-launch failed workers where they left off, but that is out of scope for now.

This is the kind of stuff I have to deal with when working on Anubis and why I end up writing essays in PR commit messages. Turns out most of this is edge cases. The joys of modern software know no bounds.

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: