Okay, fine, I'm using a static site generator now

Published on , 3352 words, 13 minutes to read

An image of A green-haired woman in a hoodie and jeans leaning on a shed that looks like the Hanzi character for "four"
A green-haired woman in a hoodie and jeans leaning on a shed that looks like the Hanzi character for "four" - SCMix

Hey all! Xesite v4 is now complete and has been rolled out. I'd've liked to have this post out sooner, but this is genuinely a lot of stuff that's changed and I'm still working on some of it. Here's a quick overview of what's changed in Xesite v4:

So for those of you that really did think my blog was a static site, you're right now. It is one.

Why did I do this?

At a high level, the architecture for Xesite v3 (the last version in Rust) was sufficient for my needs. I had extensibility via lol_html and defining my own custom HTML elements. Everything was compiled to native Rust code as much as possible, and I had exact control over the output.

Cadey is enby
<Cadey>

Arguably, I did have a static site generator, but it was just kinda halfassed and stored everything in memory.

However, there were a few problems with this approach:

I couldn't trigger updates to the website content without redeploying the entire server it was on, due to how it was implemented with NixOS. This is not a fault in how NixOS works, this was a fault in how I implemented it.

To be fair, I tried adding dynamic updates to the mix, but I was running into issues involving state contention with how I designed things in Rust. I could've fixed this, but it would've required a lot of work. It probably would have ended in me rendering every page to the disk and serving that disk folder, but that's not really what I wanted.

I wanted to adopt Tailwind so that I could style my posts a lot more freely, but I wasn't really able to find a way to fit it in because the Tailwind parser couldn't understand the HTML templates I was using.

I was using the proc macro Maud to write HTML, but the Tailwind parser can't handle reading class names out of Maud templates. Here's an example JSX component from my website that I wanted to port over:

export default function BlockQuote({ children }) {
  return (
    <div className="mx-auto mt-4 mb-2 rounded-lg bg-bg-2 p-4 dark:bg-bgDark-2 md:max-w-lg xe-dont-newline">
      &gt; {children}
    </div>
  );
}

In Maud, the template would look like this:

use maud::Markup;

pub fn blockquote(body: Markup) -> Markup {
  html! {
    ."mx-auto mt-4 mb-2 rounded-lg bg-bg-2 p-4 dark:bg-bgDark-2 md:max-w-lg xe-dont-newline" {
      "&gt; " (body)
    }
  }
}

This is all fine and dandy, but then the real trouble came in with passing this to lol_html. lol_html doesn't have the concept of getting the children of a component (because this is designed to do streaming replacement of HTML elements), so in order to make this work in lol_html I can't use that template function. I have to write it like this:

use lol_html::{element, RewriteStrSettings};

let mut html = magic_get_html_for_post!();

let html = rewrite_str(
    &html,
    RewriteStrSettings {
        element_content_handlers: vec![
          // ...
          element!("xeblog-blockquote", |el| {
            el.before("<div class=\"mx-auto mt-4 mb-2 rounded-lg bg-bg-2 p-4 dark:bg-bgDark-2 md:max-w-lg xe-dont-newline\">&gt; ");
            el.after("</div>");
            el.remove_and_keep_content();
          })
          // ..
        ],
        ..RewriteStrSettings::default()
    }
);

You can see how this would get fairly unmanintainable very quickly.

At work I was exposed to a new technology called MDX that looks like it could really solve all these problems. It's a bit of an unholy combination of React and JSX with Markdown, but it's really cool. Instead of defining my components in bespoke syntaxes or in Rust, I can just write them in React and use them in my blog posts. This is really cool, and I'm excited to see what I can do with it.

The biggest problem was the old format of these things:

Mara is hacker
<Mara>

These little conversation snippets were a huge pain to move over!

Previously they were done by hacking up the markdown parser in a way that is known to cause cancer in the state of California, which made them look like this:

[Wow this is text that I am saying!](conversation://Mara/hacker)

With the lol_html flow I had to explicitly namespace my HTML elements ad nauseum, so it looked like this:

<xeblog-conv name="Mara" mood="hacker">Wow this is text I am saying!</xeblog-conv>

But even this was annoying in practice because I could not use newlines in the conversation snippets without breaking the hell out of everything in ways that were difficult to diagnose. I ended up using <br />, <ul>, <li>, and other such elements everywhere in ways that were hard to read and write:

<xeblog-conv name="Mara" mood="hacker">Okay so when you use the
[rilkef method](/blog/experimental-rilkef-2018-11-30/) to dynamically
reparse the flux matricies, you need to follow these steps:<br /><ul>
<li>First, desalinate the yolo manifold</li>
<li>Then make sure you have Ubuntu up to date</li>
<li>Finally, watch <a href="https://youtu.be/MpJsYFZtQbw">this video</a>
to find out any missing steps</li></ul></xeblog-conv>

This sucked. Majorly. I hated it. I wanted to be able to write my conversations like this:

<XeblogConv name="Mara" mood="hacker">
  Okay so when you use the
  [rilkef method](/blog/experimental-rilkef-2018-11-30/) to dynamically
  reparse the flux matricies, you need to follow these steps:

  - First, desalinate the yolo manifold
  - Then make sure you have Ubuntu up to date
  - Finally, watch [this video](https://youtu.be/MpJsYFZtQbw) to find out any missing steps
  </XeblogConv>
Mara is hacker
<Mara>

Okay so when you use the rilkef method to dynamically reparse the flux matricies, you need to follow these steps:

  • First, desalinate the yolo manifold
  • Then make sure you have Ubuntu up to date
  • Finally, watch this video to find out any missing steps

This is the real strength of MDX. It combines React and Markdown to give you superpowers.

Migration

The main pain point was migration. For the most part the "new style" syntax transferred over without basically any editing. I chose to fix some minor spelling and grammar errors, but most of it was migrated over fully intact.

I probably missed something, and with the sheer number of articles I have (over 500 by the end of the year) I almost certainly missed something. Please let me know if I did! Sorry!

When it came to the CSS, I started with a blank HTML file and copied over rendered HTML from my website in production. Once I had the basic structure copied over, I started pouring over Tailwind UI to make a short list of the components I wanted to play with.

I had existing experience adding my Gruvbox inspired theme to Tailwind, so I copied over that Tailwind configuration file and went to town replicating the styles I had before, combining in parts from Tailwind UI and a few other places for inspiration. I had to make some minor changes to the colors, but for the most part it was a fairly straightforward process.

The part I was most worried about was the prose formatting in Tailwind. It didn't follow my old style of prose formatting, so I had to make a few minor changes. I'm not fully happy with this yet (it makes prose text a bit too dark for my tastes), but I'll get there in due time.

The light at the end of the tunnel

As an added bonus of using Tailwind, React, and all that startup goop, I can make satirical landing pages for fake products I make up. This is a huge win for me, because I absolutely love abstract methods and ways of making fun of my own industry.

Cadey is coffee
<Cadey>

Hilariously enough, when I published that landing page and shared it around, I expected people to click literally any of the links on it to see the associated blogpost. Instead, people just commented on how baity the page was. Some people thought it was serious. This was an even more hilarious result than I thought. I'd have hoped that having the Enterprise tier list a price on the page would be a dead giveaway that it's a joke, but I guess not. Same with the mention of artificial general intelligence. Oh well, lessons learned I guess!

Dynamic updating

The biggest change in Xesite v4 is that it can now update itself dynamically. This is a huge win for me, because it means I can update my blog posts, resume, and other content without having to redeploy the entire server. All I do is push things to GitHub and it updates itself within a minute.

This is thanks to me adopting a dystatic approach to my website. In essence, it boils down to this: the application itself serves a static site, but the static site is rebuilt every time something changes.

This is a bit of a weird concept, so let me explain it in a bit more detail. I made a diagram of all of this that you will need to click on to expand, because it's a bit dense:

The entire flow of my website's architecture (click on the image to expand it, the diagram is kinda regrettably dense)
The entire flow of my website's architecture (click on the image to expand it, the diagram is kinda regrettably dense)

When you think about it, a static site generator is really just a compiler. It takes input in the form of source files and outputs a folder with HTML in it. When I was evaluating static site generators, a feature of Lume kept standing out for me: shared data.

A lot of my site's content is actually stored in a series of increasingly large Dhall documents. This includes everything from my salary transparency history, the signalboost page, and even key parts of my resume. I wanted to be able to use this data in my blog posts, but I didn't want to have to copy and paste it everywhere.

I did make a draft of v4 that changed everything over to TypeScript that'd be parsed on the fly using tyson, but I didn't like the idea of having everything in kinda hard to read files. There's a certain surreal beauty to the way I'm using Dhall here and I want to keep that dream alive.

The way I hacked around this was by making the Go rebuild process dump a bunch of Dhall data into Lume shared data. Arguably this could be worked around if Lume supported loading Dhall data, but I just hacked it together using JSON in the meantime. This could probably be improved on in the future, but it has the advantage of working.

Amazingly enough, this means I could slap patron information into the right place with the same flow. I don't have to do anything special to make this work, it just works.

Combine this with dumping the right JSON file in the right place for Typst to pick up when building my resume and you have a pretty powerful system.

Once this all was working, I added in the dynamic updating system. This works like this:

This is how you get up to this point:

The entire flow of my website's architecture
The entire flow of my website's architecture

It makes a bit more sense now! I'm really happy with how this turned out, and I'm excited to see what I can do with it in the future.

I've looked around, and there doesn't seem to be a name for this concept. In order to trigger someone calling me wrong on the Internet, I'm calling this a dystatic approach. It's a dynamic website that rebuilds its static website when things change.

Fly

Cadey is enby
<Cadey>

I got some free credits from Fly a while back for writing about them. Please flavor your reading of this section with that in mind. Nothing about my setup has a hard requirement on Fly, but the fact that they have anycast routing out of the box really makes it convenient for XeDN and xesite.

My website has a few moving components now. Here's a quick overview of what's going on:

The entire flow of my website's architecture
The entire flow of my website's architecture

xesite is the binary that serves the website you are reading right now. It's what does all the rebuilds and stuff. It's written in Go, and it's what I'm most familiar with.

Aoi is wut
<Aoi>

Didn't you rewrite it in Rust from Go a while ago? Why go back?

Cadey is enby
<Cadey>

Go is my best language. It's not a perfect shining city on a hill, but I can write and maintain it without thinking. I can't say the same for Rust yet. Arguably there's nothing stopping it from being that way, but I wanted something easier to implement because this was already several months of work. Editing all of the articles took forever.

patreon-saasproxy and OAuth2 ""fun""

When it starts up, it reaches out to patreon-saasproxy fetch an authentication token for Patreon. Originally, I was going to make it a full reverse proxy for the Patreon API, but the Patreon API bindings I'm using didn't have support for this, so I just made it a token source.

The Go oauth2 library seems to very much not be designed with this kind of usecase in mind. In order to get things working, I had to write my own TokenSource like this:

type remoteTokenSource struct {
	curr       *oauth2.Token
	lock       sync.Mutex
	remoteURL  string
	httpClient *http.Client
}

func (r *remoteTokenSource) fetchToken() (*oauth2.Token, error) {
	resp, err := r.httpClient.Get(r.remoteURL)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return nil, web.NewError(http.StatusOK, resp)
	}

	var tok oauth2.Token
	if err := json.NewDecoder(resp.Body).Decode(&tok); err != nil {
		return nil, err
	}

	return &tok, nil
}

func (r *remoteTokenSource) Token() (*oauth2.Token, error) {
	r.lock.Lock()
	defer r.lock.Unlock()

	if r.curr == nil {
		tok, err := r.fetchToken()
		if err != nil {
			return nil, err
		}
		r.curr = tok
		return tok, nil
	}

	if r.curr.Expiry.Before(time.Now()) {
		tok, err := r.fetchToken()
		if err != nil {
			return nil, err
		}
		r.curr = tok
	}

	return r.curr, nil
}

It works, but it's kinda hacky. Ideally I'd like to make this a bit more generic in the future (so I can have it manage other tokens from different OAuth2 sources), but this has the advantage of working for now. I kinda hate how the Patreon API is abandonware, but I can vibe.

XeDN

There's not currently a direct dependency between xesite and XeDN, but in practice everything xesite serves depends on XeDN in some way or another. If you want to read more about XeDN, you can read these posts:

Mi

I haven't really mentioned mi in much detail on my blog (and I am probably going to wait until I've rewritten a good portion of it to go into much detail), but it's basically a personal API server that does a bunch of things I find convenient for myself.

One of those things is a bit of code that will grab my blog's JSONFeed, scrape it for new articles, and announce them in a few places.

Cadey is coffee
<Cadey>

I really wish this could include Patreon, but they seem to have no interest in maintaining their API. I'm not sure it I want to reverse-engineer their webapp to make this work, but I might have to. That's for another time though.

Conclusion

Xesite is here to stay. I hope this has given you an overview of everything that I've been up to with this. I'm really happy with how this turned out, and I'm excited to see what I can do with it in the future.

Oh, by the way, because MDX lets me embed React components in my blog posts, I can do this:

I can embed arbitrary HTML and React components in my blog posts now! This is the crucial part of how my recent story posts work. Just imagine what I can do with this!

Things I learned

Semantic import versioning isn't actually that bad in practice. I decided to use it when writing the code for this version of the site because I wanted to give it a fair assessment. It's fine. I don't agree with the design decisions, but it's fine in practice.

I have way more articles than I thought I did. I knew I had a lot, but having to touch every single file made me realize just how much I've written over the years. I'm really proud of myself for this.

React and Tailwind are stupidly powerful. Xeact isn't good enough for my needs anymore because I've outgrown it. Kinda sucks to be in this situation, but I am happy that I was able to use Xeact to help me learn what I needed to learn to make this work.

Bugs I need to fix

Here's to the next hundred articles. Stay safe out there!

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: lume, dhall, typst, go