The service stack

Containers are virtualisation.oci-containers (docker backend), one module per service under modules/containers/. Two conventions hold across all of them (containers/default.nix):

• Data is a named docker volume, never a bind-mounted host directory — so restic covers a new service the moment it declares a volume. See Data and backups.
• Config comes from the Nix store, read-only (0444). A store path changes with its content, so systemd recreates the container on a config-only change — which is what the old Ansible setup’s recreate: always was working around.

Restart policy is forced to always, and logs go to the journal — never json-file, which would put unbounded logs on the disk and break the forgejo jail that reads the journal.

What runs, and how it is reached

The Forgejo Actions runner is not on this list any more: it lives on its own machine. See CI runner isolation.

Why the Minecraft heap is two numbers

itzg’s MEMORY sets -Xms and -Xmx to the same value, so the JVM commits the whole heap at startup and never gives any of it back. That is why an idle world with nobody on it sat at 4.38 G of RSS at 0–1% CPU. Both worlds therefore set INIT_MEMORY and MAX_MEMORY separately — a low floor, the same ceiling as before — rather than MEMORY.

The floor alone is not enough. G1 only uncommits at the end of a GC cycle, and an idle server triggers no GCs at all, so the heap would stay at its high-water mark forever. -XX:G1PeriodicGCInterval=300000 (JEP 346) is the other half: one concurrent cycle per five idle minutes, which is what actually returns the pages. The two are a pair — either one on its own does nothing useful.

RSS is still not heap. Metaspace, the code cache, GC structures and direct buffers live outside -Xmx, so the ceiling is not a bound on what the container reports.

Forgejo owns port 22, so the host’s sshd is on 2222, and that is the way in — Tailscale SSH is off. Keeping 22 is what lets git remotes stay portless: ssh has no service discovery — it reads no SRV record — so anything else has to be spelled out in every clone URL or every client’s ssh config.

caddy is built here, not pulled

caddy is one of three locally built images (with pages-hook and the serenity bot, below), and the only one built to add a module. It is built locally with the caddy-ratelimit module compiled in (caddy.withPlugins, wrapped in a minimal dockerTools image), because stock caddy has no rate limiting — and rate limiting is load-bearing for every site, see Access control.

It is still caddy 2.11.4 — the version tracks nixpkgs, which matches the tag the official image used — and keeps the full container hardening: non-root uid from ids.nix, read-only rootfs, --cap-drop=ALL, and tmpfs for /data, /config and /tmp.

Images are pinned where the tag moves

Most images here are repo:tag on a release tag, which is both a version and a promise that the content behind it does not change. Three are not, and they carry a digest as well (forgejo carries one too, on a release tag; the table lists only the images whose tag itself moves):

A digest also makes the image archive’s restore sound: a pinned pull either returns those exact bytes or fails, so falling back to an archived copy cannot silently substitute a different image.

Renovate is configured to match this. Its regex manager captures currentDigest as an optional group — without it the tag group would swallow the digest and leave the version unparseable — and the rule that keeps itzg/minecraft-server off automatic version bumps is split so that matchUpdateTypes: ["digest"] stays enabled. Before the pin, disabling that package read as caution and meant the opposite: a rolling tag has no version to bump, so there was nothing to be deliberate with, and the content moved on the next pull with no PR ever saying so. A digest PR is that missing signal.

docker.autoPrune is weekly and its flags are empty, so it is a plain docker system prune — dangling layers only, never a tagged or digest- referenced image, and never a volume. The runner’s podman prune is the one that runs --all, and what that costs is covered in CI runner isolation.

The serenity bot is built during activation

Upstream publishes no image, so serenity-bot.nix builds one from upstream’s own Dockerfile off a full-SHA fetchgit pin. The build runs in the activation script, not in its unit: that is the only phase of a switch where the old container is still serving, so the ~3½-minute compile costs no outage. stopIfChanged = false on the containers and the builder keeps them up until then, and serenity-bot-image.service only does the build at boot, when docker is not up during activation. The tag embeds the rev plus a hash of the enabled features and rustflags, so a features-only change rebuilds too.

Sharding is two numbers, shards and instances, both 1. The per-instance ranges are computed and checked exhaustively at eval time up to 16 shards. Above one instance redis stops being optional: the AI locks and rate limits are per-process without it. Until then serenity-redis is a cache with a Discord fallback: in-memory only, no volume, 256 MB LRU.

Deliberately not containers

• postgres + pgbouncer (postgres.nix) — the first service whose data is not a docker volume, so its backup (pg_dumpall) is not optional; it is the only thing covering that data. On the host so a second service can share it without either owning the other’s volume. postgres is unix-socket and loopback only; pgbouncer (6432) is reachable from botnet and the tailnet.
• syncthing (syncthing.nix) — a user service with state in ~/.config/syncthing and folders under ~/syncthing, paths kept byte-identical to the old box because navidrome bind-mounts ~/syncthing/Music and the node’s device ID is derived from the TLS keypair in the config dir. Tailnet-only GUI on 8384.
• CoreDNS (caddy.nix) — answers split DNS for the half-public names, bound to the tailnet address on :53. See split DNS.
• Renovate (renovate.nix) — a daily host timer, on the VPS rather than the CI runner so its cross-org write token never reaches an untrusted machine.
• The Actions runner, which is now a different machine entirely.