Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

This is the whole of hu-tao.dev: a mail server, a Forgejo instance, a music library, a search engine, two Minecraft worlds, a Discord bot, and the CI that builds them — described by one flake, on two machines.

The guiding rule, stated once so the rest follows from it: there is no state on the server that this repo does not describe. One flake builds each machine; OpenTofu creates the machines and publishes their DNS. Anything a human would otherwise have to remember to run is a systemd unit instead.

The two machines

flowchart TB
    subgraph vps["hu-tao · CX43 · fsn1"]
        direction TB
        mail["mail · git · music<br/>search · status · minecraft"]
        pages_vol[("pages_data")]
    end

    subgraph runner["forgejo-runner · CX33 · nbg1"]
        direction TB
        jobs["job containers<br/>podman, one per job"]
        cache[("nix store<br/>Actions cache")]
    end

    runner -- "HTTPS 443 only<br/>fetch jobs, clone, upload artifacts" --> vps
    vps -- "ssh 22, for deploys<br/>initiated here, never there" --> runner
    vps -. "pulls published artifacts<br/>on each run, hourly as a safety net" .-> pages_vol

    classDef trusted fill:#1f6f43,stroke:#0d3a23,color:#fff
    classDef hostile fill:#8c2f2f,stroke:#4d1a1a,color:#fff
    class vps trusted
    class runner hostile

The green box holds everything. The red box is treated as hostile — it runs code from pull requests, so the design assumes a job will one day escape its container and get root there. What that costs is bounded by the split: a rooted runner holds a nix store, a job cache and its own runner token, and can reach exactly one thing on the VPS, over HTTPS, through an allow-list that permits four URL paths. See CI runner isolation.

Both directions matter and only one is symmetric-looking. The runner reaches git.hu-tao.dev the way any client on the internet does. The VPS reaches the runner over ssh because it is the deploy host. There is no private network between them, deliberately — there was one, it had a single member, and it was deleted.

Where to start

If you want toRead
understand how a packet reaches a serviceNetwork and trust boundaries
know what runs here and how it is reachedThe service stack
know what a tailnet peer may reachThe tailnet policy
ship a changeDeploying
fix something that is broken nowFailure modes and recovery
install a machine from nothingProvisioning a machine
know why a thing is the way it isthe module comment — every modules/**.nix opens with why it exists and which mistake it guards against

That last row is not a deflection. This book is the index to those comments, not a replacement for them: the reasoning lives next to the code it constrains, where it cannot rot independently of it.

Conventions used here

  • A “gotcha” has a date. Anything described as learned the hard way names when, so a reader can tell a live constraint from a superstition.
  • Numbers are measured, not estimated, and say where they came from.
  • Diagrams are mermaid, in the markdown. They render in this book, in the Forgejo web UI, and in a pull request that changes one — so a diagram cannot quietly go stale behind a build step.