The Oath manual
How the system works today: shipped behavior, not plans. It is built from docs/manual in the repository, and a change to the system changes these pages in the same commit. On a running machine, start with oath, which prints /oath/INDEX.md.
1Using oath§
oath is the only admin surface. Humans and agents use the same verbs.
Do not edit /etc or hunt random files.
1.1Start here§
oath the INDEX (generated, live; also /oath/INDEX.md)
--json prints the same facts as JSON. Failures include a hint
pointing at the next step (oath ls, oath schema <kind>, INDEX
safety).
The catalog root is /oath. --root / OATH_ROOT is for tests and
for build and install tooling on the build host.
1.2Verbs§
| Verb | Does |
|---|---|
oath ls | List object ids. --kind host (etc.) to filter |
oath schema [kind] | List kinds, or one kind's prose + JSON Schema |
oath get <id> | Desired + actual + status. --desired / --actual to clip |
oath set <id> k=v | Write desired only. Does not converge. --from-json '<obj>' sets several fields. The result must fit the kind's type. oath set plan:<name> --from-file <path> files a plan (canonical, oath fmt -w); --from-file - files the canonical form of a plan read from stdin |
oath new <id> k=v | Create a svc or pkg (--from-json too). Runs after apply |
oath rm <id> | Mark a svc or pkg for removal. Removed at apply |
oath diff [id] | Drift between desired and actual (or last applied, for svc) |
oath apply [id...] | Check, snapshot, then converge the listed ids or all drift |
oath keep | Keep an apply that is on probation. Run it from a new SSH session |
oath undo | Put back what the last apply changed (repeat to go further back) |
oath verify [pkg:…] | Hash linked package trees again against their pins; exit 1 if one changed on disk (-q: only damaged ones and the count) |
oath audit [pkg:…] | The daily security audit's findings for what is here, when a key this machine trusts signed them; --json is the report. From the next release |
oath reap | What holds each version in the store and what would go; sudo oath reap --confirm removes it, and each boot does. From the next release |
oath log | Apply log (JSON lines): ok, error, refused, undo:N, keep:N. -n N for the last N |
oath logs [id] | A service's output (svc:<name>), the DHCP client's (net:net0), the last build of a plan (plan:<name>), or PID 1's (no id). -n N lines, default 50; --json gives {id, path, lines} |
oath fmt <path> | Canonical plan.plan to stdout (-w in place) |
oath build plan:<name> | Isolated, netless build of a pinned plan. Does not link /bin. Creates the product's pkg object if it is missing; --no-pkg does not, for a product that is only a build input to other plans |
oath build --file <path> | Same, ad-hoc (does not store the plan) |
oath mcp | These verbs as an MCP server on stdin/stdout, for an agent (Agents) |
set, new and rm are staging. Nothing is live until apply, and
PID 1 never runs a spec that was not applied. set refuses a key the
kind does not have, or a value of the wrong type
(oath set svc:time restart=sometimes names the field and fails).
Writes to the catalog need root: over SSH, use sudo oath set,
sudo oath apply, and so on. Reads (ls, get, diff, log) do
not. Writers take a lock on /oath/.lock, so two oath commands never
interleave; the second waits up to 30 s.
Ids are kind:name — host:local, svc:sshd, snap:current,
pkg:curl, net:net0, ssh:local, dev:net0, plan:hello. Names
are letters, digits, ., _ and - (at most 64, not starting with
.).
1.3Safety§
apply checks every object it will touch before it changes anything.
A refused apply (bad value, wants cycle, broken package needs,
missing --confirm) changes nothing and takes no generation. Then it
takes a filesystem snapshot and converges.
- mutate — hostname, service enable, package present. Snapshot and go.
- confirm — halt, reboot, picking an old boot generation (not
undo), and anything that can cut this machine off from SSH: disabling, removing or changingsvc:sshd, dropping a key fromssh:local, changingnet:net0.oath applywithout--confirmexits 3 and explains. Do not pass--confirmunless the owner asked for that class of change.
oath set host:local hostname=atlas
oath diff
oath apply
oath set host:local power=reboot
oath apply # refused (exit 3)
oath apply --confirm # reboots
After a reboot apply, desired power is set back to run so the box
does not loop. The apply asks PID 1 for the reboot; PID 1 stops the
services, syncs and remounts / read-only first. A power change on its
own takes no generation (power is not configuration); plain
sudo reboot and sudo poweroff do the same.
A change that can cut off SSH goes on probation once applied:
sudo oath set net:net0 ipv4=192.168.1.20/24 gateway=192.168.1.1
sudo oath apply --confirm # on probation for 180 s
ssh home@192.168.1.20 sudo oath keep # a new session: kept
If no new session runs sudo oath keep in time (--probation <secs>,
10–3600, default 180), PID 1 undoes the apply and logs undo:N by
probation. A reboot before keep undoes it too. The session you
applied from cannot keep it: only a login made after the apply proves
the machine is still reachable. While a probation is open nothing else
applies; sudo oath undo takes it back at once.
oath undo puts back the spec of the objects the newest apply changed
(an apply that failed counts), converges them, and logs undo:N. Run
it again to undo the apply before that. It never rewinds the log, the
package store, or the SSH host keys.
1.4Disks§
Another disk mounts through host:local.mounts: a path under /mnt or
/home, the filesystem by label or UUID, and mount options.
sudo oath set host:local mounts='{"/mnt/warehouse": {"source": "LABEL=warehouse", "options": "noatime,compress=zstd:1"}}'
sudo oath apply host:local
Apply mounts it now if the disk is plugged in, or prints that it waits
for it. PID 1 mounts it at every boot and whenever the disk appears,
and unmounts it at shutdown. set replaces the whole map:
mounts='{}' and an apply unmount it, and oath undo takes a change
back. oath get host:local --actual lists what is mounted now. Make
the filesystem first (sfdisk, mkfs.btrfs -L <label>; blkid shows
labels and UUIDs). A device name such as /dev/sdb1 is refused: it
changes with the port and the boot. A machine's mounts belong in its
install overlay (hosts/<name>.json).
Home itself can live on another volume, a btrfs subvolume of another disk, say:
sudo oath set host:local mounts='{"/home": {"source": "UUID=580d861c-4b96-4017-856c-0b55ff4e58ce", "options": "subvol=@oath-home,noatime,compress=zstd:1"}}'
sudo oath apply host:local
The apply changes nothing yet: it prints /home is … from the next boot. At the next boot PID 1 mounts it before anything reads /home,
in place of the install's @home, and writes sshd's keys onto it. Copy
what /home holds into the new place first (cp -a): the old home stays
where it was, not mounted. If the disk is missing at boot, the machine
comes up on the old home, keys written, and logs it (oath logs | grep 'mount: /home').
1.5Kernel module options§
A machine's hardware workarounds are module parameters in
host:local.module_options, by module name. PID 1 passes them each
time it loads the module; one already loaded takes a change at the
next boot.
sudo oath set host:local module_options='{"nvidia": "NVreg_EnableGpuFirmware=0"}'
sudo oath apply host:local
They come after the parameters Oath always gives a module (amdgpu's
si_support=1, nvidia_drm's modeset=1) and before module.key=value
on the kernel command line, so a later one wins. Like mounts, they
belong in the machine's overlay, never in a profile: another machine
with the same driver gets none of them.
1.6A device's own protocol§
A program on the desk (as home) may need to talk to a device in its
own protocol: a wireless mouse's battery, say, which the kernel does
not report. host:local.seat_hid lists such devices by
vendor:product. grep HID_ID /sys/class/hidraw/*/device/uevent shows
them: 0003:00003554:0000F510 is 3554:f510.
sudo oath set host:local seat_hid='["3554:f510"]'
sudo oath apply host:local
PID 1 makes the device's vendor interfaces (/dev/hidraw*) root:home
0660, at once and whenever it is plugged in, whatever number it gets.
An interface that carries a keyboard, a mouse or touch stays root's, so
no program reads keys or motion; oath apply prints the nodes it gave.
The list belongs to the machine, never a profile.
1.7Who you are§
Serial login is root (break-glass) when svc:serial is enabled.
The screen and keyboard are home when svc:screen is enabled. SSH is home (uid 1, HOME=/home, login shell zsh). sudo (no
password) becomes root. The groups are root, home, work and void. The
apply log records uid and tty. host:local.env and TZ (from
host:local.timezone) are injected by PID 1 into services and written
to root-owned /etc/profile for logins, and to sshd's SetEnv in
/etc/ssh/sshd_config, so ssh host 'command' has them too, as of
sshd's last start. oath apply does not write $HOME/.profile.
work (uid 2, home /work, R55) is for servers and workloads: nobody logs in
as it, sudo refuses it, and /home is closed to it. sudo -u work <command>
runs a command as it from home, and it cannot type into home's terminal, since
the kernel's TIOCSTI needs root (R60); a service names it in user. /work is its
own subvolume, @work, which PID 1 makes on the first boot that lacks it, so no
generation or boot snapshot holds its data.
void (uid 65534, home /var/empty) is the fourth account: nobody logs
in as it; sshd, dhcpcd and chronyd drop to it.
A program can trace or read the memory of its own descendants only (Yama, R64, from the next
release), so strace -p or gdb -p on a program already running takes sudo.
1.7.1The password and the seat§
home has no password until you set one (R42, login.md):
oath password # asks the current one (if set), then the new one twice
oath password --status # password: set|none, autologin: on|off, seat: desk|console
printf 'old\nnew\n' | oath password # off a terminal: the current one (if set), then the new
An empty new password removes it. svc:login keeps it, hashed, in
/oath/login/password (root's alone); a wrong one is answered after a
delay that doubles each time, up to 30 s. It is asked at the console
when host:local.autologin is false, and by the desk's lock screen
(Sola's); SSH (keys only) and sudo never ask, so it guards the screen
and keyboard, not the account: login.md says
exactly what it does and does not protect. Auto-login is on by default;
Sola's Settings turn it off with the password, and root can with sudo oath set host:local autologin=false and sudo oath apply host:local.
On a desk, the screen and keyboard (tty1) are the seat, and either the
desk or the console has it. Exit Sola ends the desk on purpose
(oath desk stop does the same): its services are held (state: "held") and the console, a shell as home (or the password prompt), takes
tty1. oath desk, from that shell or over SSH, gives the seat back and
the desk's services start again. Every boot starts with the desk.
A machine installed before R42 gets the programs with oath update
(its pkg:oath moved first, install.md) but not the objects. Reboot into the update first: an older PID 1 knows no
seat and would start the console on tty1 under the running desk. Then
make them once:
sudo oath new svc:login --from-json '{"exec":["/lib/oath/login"],"wants":[],"restart":"always","enabled":true}'
sudo oath set svc:screen enabled=true seat=console
for s in rivulet sola-river sola-shell sola-session sola-kvm; do sudo oath set svc:$s seat=desk; done
sudo oath apply
1.7.2The shell§
home's login shell is zsh (pkg:zsh, not removable); /bin/sh is
bash, and root's shell. A login zsh reads /etc/profile through
/etc/zprofile. /etc/zshrc sets the prompt, history in
~/.zsh_history, emacs keys and completion:
14:02:07 home@canto Workspace/sola main §
The time, user@host, the last two directories of the working
directory, the git branch, and §. user@host and the branch take a
color from a hash of the hostname, so each machine keeps its own and a
new hostname gets a new one. oath writes /etc/zprofile and
/etc/zshrc with /etc/profile, when host:local converges and at
boot; put your own settings in ~/.zshrc. ssh home@host 'command'
runs the command through zsh, with host:local env from sshd.
1.8Agents§
An agent reaches the machine the way the owner does: over SSH as
home, with a key in ssh:local. oath mcp speaks the Model Context
Protocol on stdin and stdout, so an MCP client gets the verbs as tools
and the INDEX as its instructions. Nothing listens on a port; the
server lives as long as the SSH connection.
claude mcp add oath -- ssh -T -o BatchMode=yes home@<host> oath mcp
ssh -T home@<host> oath mcp --read-only # only the tools that look
Each tool runs one oath --json command and returns its JSON: index,
ls, schema, get, diff, log, logs and verify look;
set_host, set_svc, set_pkg, set_net, set_ssh, new_svc,
new_pkg, rm, apply, undo, keep, update (what the newest
output would move, or with confirm move to it), file_plan (a plan
from its text) and build run through sudo, as the owner's would. The same
reads are resources too: oath://index, oath://schema/<kind> and
oath://object/<kind>/<name>. A client that sends a progressToken
hears every 5 s that a long build or apply is still running. A tool's fields are the kind's schema,
so a field added to a kind reaches the tools with nothing else to do.
A failed command is a tool error carrying the CLI's hint; exit 3 says
the apply needs confirm, which an agent passes only when the owner
asked for that class of change.
oath log shows what came through MCP: such entries have
actor.via: "mcp:<client>", and an apply given confirm has
confirm: true. A change that can cut off SSH goes on probation as
usual, and the connection that applied it cannot keep it: a new
ssh … oath mcp (or ssh home@<host> sudo oath keep) can. The server
speaks both the current MCP revision (2026-07-28) and the
initialize-based ones before it.
2Catalog§
The catalog is the source of truth. It lives at /oath.
/oath/
INDEX.md generated. read this first
schema/<kind>.json JSON Schema 2020-12
schema/<kind>.md prompt-sized prose
objects/<kind>/<name>/ spec (snapshots hold it; undo reads it)
desired.json what you asked for
applied.json the spec as last applied (PID 1 runs a svc's)
meta.json id, safety; `fixed` on a seed-fixed package
new / remove markers for a pending `oath new` / `oath rm`
status/<kind>/<name>.json what the machine observed: tmpfs, rebuilt each boot
log/apply.jsonl every apply (ok, error, refused), undo and keep
probation.json an apply that can cut off SSH, until `oath keep`
log/init.log PID 1's messages
log/svc/<name>.log each service's output (`oath logs svc:<name>`)
log/net/net0.log the DHCP client
.lock held by whoever is writing the catalog
store/pkg/<name>/<hash>/ package trees (apply links /bin)
store/plan/<name>/<hash>/plan.plan
ssh/ host keys
run/ sockets and mounts, not desired state
init.sock `oath apply` asks PID 1 to converge here
fs/ btrfs top level (subvolid=0)
Desired and actual are separate files. You may cat any of these;
oath get is the supported way. If a kind is not under
/oath/schema, it does not exist. Every file is replaced atomically
(temp file, fsync, rename), so a reader never sees half of one.
2.1Kinds§
| Kind | Ids in the base profile | Role |
|---|---|---|
host | host:local | Hostname, power, env, timezone, hosts, origins, mounts |
svc | svc:serial, svc:sshd, svc:time | PID 1's only config |
snap | snap:current, then snap:N | Generations |
pkg | glibc, bash, zsh, coreutils, btrfs, oath, openssh, curl, … (39 in base) | Store + /bin and /lib symlink farms |
net | net:net0 | The link: dhcp or a static address |
ssh | ssh:local | Keys that may log in as home |
dev | dev:net0, dev:disk-pci-… (discovered) | Hardware inventory |
plan | none until you file one | Hashed plan.plan; oath build writes a pkg tree |
Drift is desired against applied.json, the spec as last applied, for
every kind; actual (/oath/status/…) is only what the machine observed (svc
actual is runtime: state, pid, restarts).
oath schema <kind> on the box gives fields, safety, and examples.
That text is the same as crates/oath-core/schema/ in the repo.
2.2Where objects come from§
On the box, oath new <id> creates a svc or pkg and oath rm <id>
removes one; both take effect at oath apply, and oath undo of that
apply reverses them. host, net, ssh, snap and dev objects,
and the base packages, cannot be created or removed. Before the box
first boots, objects come from:
- Seed profiles —
crates/oath-core/seed/<profile>.json, data that pkg:oath ships as/lib/oath/seed/and no binary carries (R72), so a profile's pins never change PID 1.baseis a headless machine you reach over SSH;deskextends it with the graphical session.oath seed --profile <name>reads/lib/oath/seed(OATH_SEED_DIRfor another directory;cargo makesets it to the tree's).cargo make build --profile <name>seeds the image from one; a profile lists the first desired document of each object, the packages that can never be removed (fixed), and any first actual that differs from the default. - Overlays —
{"objects": {"kind:name": {…}}}. Each object is a JSON merge patch (RFC 7386) on the current desired: objects merge key by key,nulldeletes a key, anything else replaces. A new id creates the object. Each patched document must pass the same checks asoath set. An overlay counts as applied: PID 1 boots with it.cargo make install --overlay <file.json>merges one into the installed catalog, after the hostname and the SSH keys the installer writes. - Plans —
oath set plan:<name> --from-filefiles a plan, andoath buildcreates the product'spkgobject if it is missing (--no-pkgdoes not, for a product only other plans build with).
oath seed is the build- and install-time tool behind the first two.
It is hidden from oath --help:
oath --root ./stage/oath seed --profile base
oath --root /mnt/oath seed --overlay overlay.json
A fresh root gets the profile (default base). An existing catalog
only takes --overlay; --profile on it is refused.
2.3Devices§
dev:* is the hardware inventory, discovered by PID 1 at boot and
again whenever a disk, NIC or serial port comes or goes: disks are
dev:disk-<bus path> (dev:disk-pci-0000.00.1f.2-ata-1), the NIC is
dev:net0 (net:net0 is its link) and other NICs dev:net1, …, a real
UART keeps its kernel name (dev:ttyS0). Actual has the node, bus
path, driver, and for a disk its size, model and removable.
A device that goes away loses its object. /dev is nodes from
devtmpfs, not the admin UI. There is no udevd; PID 1 loads the driver
a hot-plugged device asks for (devices spec).
PID 1 also mounts, outside the catalog: /proc, /sys, devtmpfs
/dev with /dev/pts, tmpfs /tmp, /dev/shm and /run, and
cgroup2 at /sys/fs/cgroup. /etc/hosts has localhost, the
hostname and host:local.hosts (Network).
3Services§
PID 1 is oath-init (/lib/oath/init, also /sbin/init). It is
not a catalog object. Every supervised process is a svc:* object.
There is no unit file, no systemd, no /etc/init.d.
3.1What ships§
The base profile:
| Id | Exec | Default | Notes |
|---|---|---|---|
svc:serial | /lib/oath/serial-login | enabled, restart=always | Root shell on the serial console (ttyS0 / hvc0). With no UART it idles and logs no serial console. Do not disable it unless you have another way in. |
svc:screen | /lib/oath/screen-login, user: home | enabled, restart=always | Login shell as home on the screen and keyboard (tty1); sudo for root. The screen shows it once the display driver loads, just before services start. A desk turns it off. |
svc:sshd | /bin/sshd -D -e (OpenSSH) | enabled, restart=always, keep_children | Keys in ssh:local; /etc/ssh/sshd_config comes from host:local. Passwords and root login off. Restarting it keeps open sessions. |
svc:time | /bin/chronyd -d against 0–3.pool.ntp.org | enabled, restart=always | Keeps the clock (see Network); runs as void after start-up. |
The desk profile adds the desk's own (rivulet, Sola's, PipeWire's)
and two for Bluetooth: svc:dbus-system, the system bus
(/run/dbus/system_bus_socket, as void), and svc:bluetooth,
BlueZ's bluetoothd. Pair a device with bluetoothctl: power on,
scan on, then pair, trust and connect with its address;
devices lists the known ones. Headphones then show in wpctl status;
a game controller, keyboard or mouse on Bluetooth LE is an input device.
Add a service on the box with oath new and oath apply (below), or
at install time with an overlay (see Catalog).
3.2Boot and converge§
At boot PID 1 mounts the floor, switch_roots into the root subvolume
@, mounts the btrfs top level at /oath/run/fs, sets the hostname from
host:local, loads the drivers its devices ask for, brings up net:net0,
refreshes dev:*, writes ssh:local keys, then starts every enabled
service in wants order (and the DHCP client, see
Network). Host, network and keys come from what was
last applied, not from pending oath set edits. Display and sound
drivers load after that, so svc:sshd is up before any GPU driver.
oath logs lists the modules each pass loaded.
PID 1 runs only what was applied: each service's applied.json, which
oath apply writes after the spec passes its checks. A set without
apply never reaches a running service, even across a reboot. A spec
PID 1 cannot read (a torn file) leaves the running process alone.
PID 1 reaps children and restarts them per restart
(never / always / on-failure). It stops services that are
disabled or whose applied spec changed. Each service runs in its own
cgroup (/sys/fs/cgroup/oath/svc-<name>): stopping it sends SIGTERM to
every process in it, waits up to 5 s, then kills what is left. With
keep_children (as on svc:sshd, and svc:sola-session on a desk) only
the main process is signalled, its exit leaves the rest running, and what
they write still reaches the service's log. Applying keep_children or
restart reaches a running service at once, without restarting it.
oath apply on a svc writes applied.json, notifies PID 1 on
/oath/run/init.sock, waits for PID 1 to answer that it has
converged, then checks that actual state is running or stopped.
oath undo puts back the previous applied.json and does the same.
wants is start order: enabled wanted services start first. Disabled
or unknown wants are skipped. A cycle refuses apply. If a cycle reaches
PID 1 anyway, it skips the services in the cycle (and those waiting on
them) and starts everything else, svc:sshd included.
oath ls --kind svc
oath get svc:time
oath set svc:time enabled=false
oath apply svc:time
oath get svc:time --actual
oath undo
3.3State, restarts and logs§
oath get svc:time --actual
oath logs svc:time # last 50 lines; -n 200 for more
oath logs # PID 1's own log
Actual has state, pid, restarts (restarts this boot),
last_exit ({"code": 3} or {"signal": 15}), since (UTC) and,
while it runs from a package, pkg (that package and hash).
state is:
running— the process is up.restarting— it exited and waits for its backoff.stopped— disabled, or a finishedrestart=neverrun.failed— it could not be started, it is in awantscycle (or waits on one), or it died within 10 s of starting five times in a row;reasonin actual says which. Restarts wait 1, 2, 4, then 8 s between those quick failures; a run of 10 s or more restarts at once. Afailedservice stays down until its applied spec or its program's package changes: fix it withoath setandoath apply.
A service's stdout and stderr go to /oath/log/svc/<name>.log, one
line each with a UTC time in front, rotated to <name>.log.1 past
256 KiB. PID 1 writes its own messages to /oath/log/init.log; the
DHCP client writes to /oath/log/net/net0.log (oath logs net:net0).
3.4Add and remove a service§
sudo oath new svc:tick --from-json '{"exec":["/bin/sleep","1000"],"wants":[],"restart":"always","enabled":true}'
sudo oath apply svc:tick # starts it
sudo oath rm svc:tick
sudo oath apply svc:tick # stops it and deletes the object
sudo oath undo # brings it back
oath new needs only exec; it writes the defaults into the object
(restart: always, enabled: true, wants: []), so oath get shows
what will run. Until oath apply, a new service has status new and
does not run; once applied it is not new, and if it cannot start (a
wrong exec, say) its status is error and its actual has the
reason;
oath rm before that deletes it at once.
svc:sshd changes that can cut off SSH (disabling or removing it, or
changing its exec, user or env) need oath apply --confirm and
go on probation: a new SSH session runs sudo oath keep, or PID 1
undoes the change after 180 s.
3.5Account, environment, low ports§
Oath has four Unix accounts: root, home (uid 1, gid 1), work (uid and
gid 2, home /work), for servers and workloads, which sudo refuses and which
cannot log in, and void (uid and gid 65534, home /var/empty), the unprivileged
one that owns nothing and cannot log in. user is root, home, work or
void; absent means root. A service that keeps data and needs no sudo runs
as work; one that needs no files of its own and no privilege runs as void.
Every service gets PATH=/bin, HOME, USER, LOGNAME,
SHELL=/bin/sh and XDG_RUNTIME_DIR for its account (the desk
profile's Sola services set SHELL=/bin/zsh, so terminals start the
login shell), then
host:local.env, then TZ from host:local.timezone (unless env
already sets TZ), then its own env. The service's env wins on
the same name.
bind_low_ports lets a service of any account but root bind ports below 1024
(CAP_NET_BIND_SERVICE). Root can bind them either way. On a
service an overlay added (here svc:web):
oath set svc:web user=home bind_low_ports=true
oath set svc:web --from-json '{"env":{"WEB_ROOT":"/home/site"}}'
oath apply svc:web
rtprio (1–99) lets a service give its threads realtime priority up
to that value and renice itself to -20 (RLIMIT_RTPRIO,
RLIMIT_NICE); the desk profile gives PipeWire 95.
Changing exec, user, env, bind_low_ports or rtprio restarts
the process. So does an apply that moves the package its program comes
from: PID 1 follows exec[0] through /bin into the store
(/oath/store/pkg/<name>/<hash>/…), and when that realization changes
it restarts the service. Actual pkg names it ({"id": "pkg:sola", "hash": "sha256-…"}). A service whose program is in another package
keeps running; one that was failed tries again with the new program.
Moving pkg:openssh restarts svc:sshd, so it needs --confirm and
goes on probation.
An update does not restart a service with a seat (the desk's, the
console's): oath update, and oath apply --keep-seat, leave it on
the program it started with, its actual reason says so, and it takes
the new one when it next starts (a reboot, Exit Sola and oath desk,
or its own exit). With --reboot, oath update restarts no service
before the reboot. A PID 1 from desk output fdecb1ec3d68 or older
restarts the seat's services anyway, unless the update reboots.
3.6Reboot and power off§
sudo reboot, sudo poweroff and sudo halt ask PID 1 to shut down,
and so does oath set host:local power=reboot (or halt) with
oath apply --confirm. PID 1 stops every service (SIGTERM, up to 5 s,
then SIGKILL), stops everything else, syncs, unmounts the btrfs top
level, remounts / read-only, then reboots or powers off. Ctrl-Alt-Del
on the console reboots the same way.
3.7Off-box backup§
The image ships two helpers, not a service:
/lib/oath/backup-send host:/export/path— snapshot@, mount the NFS export (version 4, else 3), write<hostname>.send(onebtrfs sendstream, overwritten each time) and<hostname>.json(generation, UTC time, size, sha256). It is not a second undo;oath undonever reads it./lib/oath/backup-daily host:/export/path— sleep until 04:00 local time (TZ, fromhost:local.timezone; UTC when unset), runbackup-send, repeat.
To run it daily, add a service in the machine's install overlay:
{"objects": {"svc:backup": {
"exec": ["/lib/oath/backup-daily", "nas:/export/backup/canto"],
"wants": [], "restart": "always", "enabled": true}}}
A present package may ship
/oath/store/pkg/<name>/live/libexec/oath-backup-quiesce and
oath-backup-thaw; backup-send runs them around the snapshot.
Without them the copy is crash-consistent (sync plus btrfs CoW).
3.8Scheduled services§
A schedule (R48) makes a oneshot start at times, as cron would: a
pattern (five fields, or six with seconds; @daily; croner's L, W
and #) in the machine's time zone, host:local.timezone, so 04:00 stays
04:00 across daylight saving.
sudo oath new svc:backup --from-json '{"exec":["/lib/oath/backup-send","nas:/export/backup/canto"],"wants":[],"restart":"never","enabled":true,"schedule":"0 4 * * *"}'
sudo oath apply svc:backup
oath get svc:backup # state: "scheduled", next: "…04:00:00-06:00", last_run
Apply does not start it, and refuses a schedule without restart: "never" or one croner cannot read. PID 1 starts it at each match; at
boot, once, when a match passed while the machine was down (the last run
is in /oath/log/svc/<name>.schedule.json); and never twice at once: a
match while the last run still goes is skipped, and init.log says so.
Its output goes to oath logs svc:<name> as any service's.
3.9If PID 1 panics§
A panic in PID 1's main loop is logged and survived: PID 1 hands its
state to a fresh copy of itself, and every service keeps running and
stays supervised (oath logs shows panic: re-executing PID 1 and
resumed after a panic). A third panic within a minute makes it shut
down cleanly and reboot.
3.10A new PID 1 without a reboot§
sudo oath update --confirm (no --reboot) has PID 1 re-execute into
the output's binary the same way, every service running (R73,
install.md). oath logs shows upgrade: … rehearsing,
upgrade: re-executing PID 1 into … and upgrade: this binary took over; /run/oath/init.json says which binary PID 1 runs (exe) and
how it started (how: boot, panic or upgrade). A binary with the
wrong hash, or one that cannot read PID 1's state, is refused
(upgrade: not into …), and PID 1 goes on as it was.
4Packages§
A package is a pkg:* catalog object. Bits live under
/oath/store/pkg/<name>/<hash>/. /bin is a symlink farm, not an
unpack target, and so is /lib: glibc's loader
(/lib/ld-linux-x86-64.so.2) and every linked pack's lib/ entries
(decision R17). There is no /lib64 and no RPATH; a program finds its
libraries through /lib. Name is a slot; hash is the bits (sha256- plus 64 hex
of oath-tree-v1). There is no apt and no oath install.
The slot also has live → <hash> and compatibility links (bin →
live/bin, …) so paths like /oath/store/pkg/<name>/bin resolve.
Those links are not part of the hash.
4.1What ships§
The base profile:
Not removable (present=false is refused):
| Id | What |
|---|---|
pkg:glibc | the C library and its loader |
pkg:bash | bash, also /bin/sh |
pkg:coreutils | ls, cp, cat, sort, hostname, … |
pkg:util-linux | mount, dmesg, sfdisk, wipefs, blkid, lsblk, findmnt |
pkg:iproute2 | ip, ss, bridge, tc |
pkg:dhcpcd | the DHCP client of net:net0 |
pkg:btrfs | btrfs, mkfs.btrfs |
pkg:oath | oath |
pkg:openssh | the SSH server sshd (R36), and ssh, scp, sftp, ssh-keygen, sftp-server |
pkg:zsh | zsh, home's login shell |
What the running oath's base profile fixes is what counts: a package a
machine was seeded with as fixed and the base no longer fixes
(pkg:dropbear, before R36) can be removed.
Present and removable: sed, grep, gawk, findutils, diffutils,
tar, gzip, xz, bzip2, procps (ps, pidof, free, top,
kill), less, vim (also vi), iputils (ping, tracepath,
arping), chrony (svc:time), dosfstools (mkfs.fat), curl
(Mbed TLS, CA bundle at /oath/store/pkg/curl/live/ssl/cert.pem),
bind (dig, host, nslookup; no named), openssl (openssl;
its default ssl/cert.pem links to curl's bundle), certs (Oath's CA roots, R50), and the libraries they load:
ncurses, gmp, mpfr, zlib, zstd, mbedtls, libuv,
liburcu, libcap, libxcrypt. A pack's needs lists the libraries it loads, so
apply refuses removing pkg:ncurses while pkg:less is present.
Every pack is built from pinned source on the build host: the Oath
binaries (pkg:oath) from this tree, the rest from pkgs/
(Build). There is no C compiler on the box yet: the
toolchain (pkg:gcc) exists on the build host but is not in the base
profile.
PID 1 stays at /lib/oath/init; it is not a package.
4.2Install / remove§
oath ls --kind pkg
oath get pkg:dosfstools
oath set pkg:dosfstools present=false
oath apply
ls /oath/store/pkg/dosfstools # the tree stays
oath set pkg:dosfstools present=true
oath apply
readlink /bin/mkfs.fat
oath undo
present=false removes this object's /bin links. The store tree
stays, so a re-install needs no network. A new hash relinks, and a
program the new tree no longer has leaves /bin. Apply refuses to clobber a
/bin name it does not own. present=false on a non-removable
package is refused (not --confirm).
4.3Fields in practice§
hashis the pin. Apply hashes the tree again before linking it and refuses one that changed on disk. Empty hash is discovery (actual.hashreports what is live). Switch withoath set pkg:foo hash=sha256-…, then apply. Other hashes stay on disk.url: ifpresentand the store is missing, apply fetches it, then links. A.taris a pack tree. An origin prefix plushashfetches{url}/pkg/{name}/{hash}.tar. A single file becomesbin/<name>. With nourl, a pinned pack the store lacks comes from the machine's origins (below).file: withurl, the URL is one file, a vendor's release, and the tree is that file alone at this path, not executable (a recipe's source; see Recipes).needslists other packs this one needs at runtime, each pinned by hash. Apply refuses a broken graph: a missing need, a need at the wrong hash, a cycle, or removing a need while a present pack still lists it. Apply does not change the other objects for you.descriptionandhomeare prose and a project URL.sourcerecords where the pinned tree was built from:{repo, commit, branch, dirty, built_by, built_at}(oath schema pkg). Set it with the hash it describes, in one apply. It is in no hash, and an apply that changes only it (or the prose) touches no store, link or service.
4.4A tree built here§
oath store puts a tree built on the machine (a directory, or a .tar
or .tar.gz of one) into the store and prints its hash. It changes no
object and no link; pinning the hash and applying does:
sudo oath store pkg:hello ./out # pkg:hello sha256-…
sudo oath --json store pkg:hello out.tar # {"hash":"sha256-…","id":"pkg:hello"}
sudo oath set pkg:hello hash=sha256-… needs='[{"id":"pkg:glibc","hash":"sha256-…"}]'
sudo oath apply pkg:hello
A tool that pins a tree it built records where from, with the hash:
sudo oath set pkg:hello --from-json '{"hash":"sha256-…","source":{"repo":"https://github.com/me/hello","commit":"<40 hex>","branch":"next","dirty":false,"built_by":"home@canto","built_at":"2026-11-01T12:00:00Z"}}'
A pin that is not the release's, or not the head this machine took
(below), is an overlay: oath get pkg:hello ends with overlay: pkg:hello is next@1a2b3c4d, not the release's sha256-… (output …); oath update replaces it (or not its head sha256-… (signed by …)), and
oath diff lists every overlay above its in-sync. sudo oath update --confirm replaces each; --keep pkg:hello keeps one.
A new name is oath new pkg:hello hash=… instead of set; a source
pack a plan builds with is oath new pkg:hello-src hash=… present=false
and goes in the plan's build_needs. Every file is stored 0755 or 0644
(the hash records only whether it is executable), and a pipe, socket or
device in the tree is refused, as a fetch refuses it.
A service follows its package: when an apply moves the pin of the
package a service's program comes from (exec[0], through /bin),
PID 1 restarts that service, and no other (an update leaves the
desk's running: Services). So a change to a program is
build, oath store, oath set … hash=, oath apply.
4.5Publishing a package on its own (R72)§
Whoever makes a package can publish it from the machine it was built
on, and every machine that trusts that key for the name takes it at its
next oath update: no profile, release or build host in between. After
the four steps above:
oath publish pkg:hello # what it would publish, and the head it replaces
oath publish --confirm pkg:hello # the pack (if the origin lacks it), then its head
oath publish --confirm pkg:hello --dest d # or write the same files into a directory
oath publish --confirm pkg:hello --committed 2026-11-06T12:00:00Z
The head is pkg/hello.json on the origin, beside pkg/hello/<hash>.tar:
the pin, its needs, source, version and description, signed beside
itself (.sig, namespace oath-pkg) by ~/.config/oath/sign_ed25519
(OATH_SIGN_KEY) and written with the origin's write key, as for cargo make publish (below). The pin must be applied and built from a clean
tree (source.dirty false). With --committed (the commit's time, any
offset; the head keeps it in UTC) it refuses to replace a head built
from a later commit. Publishing needs no
sudo.
A machine trusts a publisher for some names only by putting a list
before its key in host:local.publishers:
sudo oath set host:local publishers='["ssh-ed25519 AAAA… oath@buccina", "names=\"sola,sola-*\" ssh-ed25519 AAAA… oath@novus"]'
sudo oath apply host:local
Such a key signs heads of those names only: never a release or the
audit's report, so it cannot ship PID 1, the kernel or sshd. oath update reads the head of every package the machine has (and of what
those heads need) from each origin and moves each package to the newest
head a key may sign for it; oath update alone says what would move.
It records each head it took in /oath/heads.json and refuses an older
one after. A head wins over a release's pin of the same package, and a
package with a head stays when a release drops it. With
host:local.profile empty a machine takes heads only.
A need is a pin. When the moves leave one at another pin (a new
pkg:mesa under a package built against the old one), the update
re-points it if every library the package's ELF files load is still in
/lib after the moves, or in its own tree; otherwise it holds the moved
package back and says which package and library hold it:
pkg:mesa stays at sha256-…; sha256-… waits: pkg:sola needs pkg:mesa at sha256-…, and would lose libEGL.so.1
pkg:sola keeps its pin; its needs move: pkg:mesa sha256-… -> sha256-… (every library it loads is still in /lib)
A preview, whose packs are not fetched yet, says it checks the
libraries once they are; --confirm fetches them and checks before
anything changes. A library loaded with dlopen is not seen.
Upstream identity is components.json at the pack root (name,
version, purl); it is part of the tree hash, and missing means
unknown. Advisories for a pin live at
/oath/store/adv/<name>/<hash>.json, fetched from the origin when
url is one, or from the origin that just served the pack. oath get
shows both in actual. An empty list is not a safety proof.
4.6Origins§
A pack is its hash; it does not know where it is hosted. The machine
says where it looks: host:local.origins, a list of http:// or
https:// prefixes tried in order as {origin}/pkg/<name>/<hash>.tar
until one serves the pinned tree. Apply uses them for a pinned pkg with
no url; oath build uses them for a build pack the store lacks. The
base profile lists the default origin, the R2 bucket at
https://store.oath.moonlight.pm, which carries every locked pack
(build packs too) with its plan and source; its INDEX.md lists names
and hashes.
oath get host:local # origins
curl -fsS https://store.oath.moonlight.pm/INDEX.md # what it serves
sudo oath new pkg:m4 --from-json '{"present":true,"hash":"sha256-…"}'
sudo oath apply pkg:m4
sudo grep '"fetched"' /oath/log/oath.jsonl | tail -1 # where it came from
Some packs are on the origin and in no profile, and a machine adds them
this way: Node.js (pkg:node, which needs pkg:gcc), the GitHub CLI
(pkg:gh, which needs pkg:certs for its CA roots) and Cloudflare's
CLI (pkg:cf, which needs pkg:node). Each need is pinned at the hash
the pack's plan names (run_needs in {origin}/plan/<name>/<plan>.plan),
which must be the one the machine runs:
sudo oath new pkg:node --from-json '{"present":true,"hash":"sha256-…","needs":[{"id":"pkg:gcc","hash":"sha256-…"}]}'
sudo oath apply pkg:node
sudo oath new pkg:gh --from-json '{"present":true,"hash":"sha256-…","needs":[{"id":"pkg:curl","hash":"sha256-…"}]}'
sudo oath apply pkg:gh
sudo oath new pkg:cf --from-json '{"present":true,"hash":"sha256-…","needs":[{"id":"pkg:node","hash":"sha256-…"}]}'
sudo oath apply pkg:cf
node --version # v26.13.0
gh --version # gh version 2.103.3 (2026-10-26)
cf --version # 1.0.0-beta.26
4.6.1Servers: Blocky, restic, route, cell, PostgreSQL (R49)§
These are built from source here and are in no profile:
pkg:blocky, pkg:cloudflared, pkg:restic, pkg:rsync, pkg:route, pkg:cell (with pkg:crun,
pkg:passt), pkg:postgresql and pkg:postgis
(with pkg:geos, pkg:proj, pkg:libxml2, pkg:json-c, pkg:icu). A machine
takes them from the origin as above, with each need the plan names, once a
release has published them. pkg:go is what builds the Go ones; a machine needs
it only to build. cloudflared does not update itself: cloudflared update says so,
and a newer one is a new pin in pkgs/cloudflared.
Go programs read their CA roots from pkg:certs (R50), through Go's own
crypto/x509, unless SSL_CERT_FILE names others:
/oath/store/pkg/certs/live/ssl/cert.pem. A machine without pkg:certs makes it
first, at the hash the plan names. It is Mozilla's roots for servers, from
NSS's certdata.txt, less any with a distrust-after date and what
pkgs/certs/distrust.txt names. curl and OpenSSL still read pkg:curl's copy,
until the rebuild that takes CEF 157 (R77).
PostgreSQL finds PostGIS, a separate pack, through one line in
postgresql.conf:
extension_control_path = '$system:/oath/store/pkg/postgis/live/share'
The server refuses to run as root, so its svc names a user. A new major
version changes the data directory's format, so moving across one is
pg_upgrade, not a new pin alone.
pkg:cell runs a container image (R59), in apps/cell, with nothing but Go's standard
library over crun and pasta. There is no daemon: cell run stays in the foreground,
so a svc supervises it like any program, and its output is the service's log. Without
-t that output goes through pipes cell makes, so a program can open /dev/stderr.
sudo -u work cell run --name web -p 8080:8080 -v /work/web:/data --env-file /work/web.env nginxinc/nginx-unprivileged:alpine
sudo oath new svc:web --from-json '{"exec":["/bin/cell","run","--name","web","-p","8080:8080","nginxinc/nginx-unprivileged:alpine"],"wants":[],"restart":"always","enabled":true,"user":"work"}'
sudo oath apply svc:web
| Command | Does |
|---|---|
cell pull <image> | fetch an image, or move its tag; prints the digest for linux/amd64 |
cell run [flags] <image> [args...] | pull it if it is not on disk, and run it |
cell exec [-t] [-w dir] <name> <cmd...> | run a command in a running cell, under the same limits |
cell stop <name> | stop it: the image's STOPSIGNAL, then SIGKILL after --stop-after (10 s) |
cell ls, cell images | what is running, what is on disk |
cell rmi <image>, cell prune | forget a name; delete images no name holds, and what a killed cell left |
cell login <registry> <user> | keep a password, read from stdin, in auth.json |
run takes -p [addr:]host:port[/udp], --reach port[:hostport], -v /host:/path[:ro],
--tmpfs /path, -e NAME=value, --env-file file, -w dir, --entrypoint cmd,
--user uid[:gid], --read-only, --no-net, -t, --name.
Each run starts from a fresh copy of the image, a reflink on btrfs, and leaves nothing
behind: what is kept goes on a volume. Images live under ~/.local/share/cell. A name
with @sha256:... holds one image for good; a tag only moves on cell pull. Every
manifest, blob and layer is checked against its digest, and a layer's own urls are
never fetched.
A cell holds one uid (R53): the account that ran it, not root, which cell refuses.
It owns every file, and is root inside, or the number the image's USER or --user
names, so software that refuses to run as root is given --user. It cannot change to
another uid, so an image whose start script drops privileges (su-exec, gosu, stock
nginx's user nginx;) fails unless told to stay where it is. Setuid bits, devices and
xattrs in an image do not land.
A published port is on 127.0.0.1 unless it names an address. Nothing else comes in.
Going out, a cell reaches what its account reaches, the network and other machines
included, except this machine's loopback: --reach 5432 opens one of its ports at the
cell's own 127.0.0.1:5432, which is also how one cell talks to another. --no-net
leaves it only -p and --reach. New namespaces, mount, bpf, ptrace and
io_uring, among others, are refused by seccomp.
There is no way to build an image here, no bridge between cells, and no memory or CPU
limit: those need CONFIG_MEMCG and CONFIG_CFS_BANDWIDTH, which Oath's kernel leaves
off. PID 1 loads tun and opens /dev/net/tun at boot, from the release after
3e42879d38e5; on a machine that has not rebooted into it, do so by hand (modprobe tun, chmod 666).
pkg:route is Oath's own HTTP router (R59), in apps/route, with nothing but Go's
standard library: by host and path, to a local service, a redirect or a fixed answer.
It holds no TLS and belongs on a loopback address, behind a tunnel or other proxy:
{
"listen": "127.0.0.1:8080",
"routes": [
{"host": "a.example", "to": "127.0.0.1:9000"},
{"host": "a.example", "path": "/admin", "to": "127.0.0.1:9001", "strip": "/admin",
"request": {"X-Forwarded-Prefix": "/admin", "Cookie": ""},
"response": {"X-Frame-Options": "DENY", "Set-Cookie": ""}},
{"host": "a.example", "path": "/gone", "status": 404, "body": "not found\n"},
{"host": "old.example", "redirect": "https://new.example", "status": 301},
{"host": "*", "status": 404}
]
}
The longest path wins, at a segment's edge, and * stands in for a host no route
names. strip takes a prefix off, then prefix puts one on. An empty header value
removes that header. A service gets the path that matched: repeated slashes collapse,
and a path with a . or .. segment gets a 400. route <file> runs it, route -check <file> tries a file, SIGHUP reads it again and keeps the old routes if the new file
is bad, SIGTERM drains. Websockets and streams pass through. A forwarded X-Forwarded-*
is kept, so it trusts whatever reaches its address.
cf (also cloudflare) carries the npm packages it was pinned with in
its own tree and runs on pkg:node (R46); nothing goes in a global
node_modules, and npm install -g has nowhere to write, since the
store is read-only. cf still prints a notice when npm has a newer cf;
that is a new pin in pkgs/cf. gh runs with its update notice off and
gh auth login keeps its settings under ~/.config/gh. Both send
telemetry unless DO_NOT_TRACK=1 (or GH_TELEMETRY=false,
CF_SEND_TELEMETRY=false).
Put a LAN mirror first to prefer it (any directory cargo make publish --dest wrote, served over HTTP):
sudo oath set host:local origins='["http://10.0.0.5:8080", "https://store.oath.moonlight.pm"]'
sudo oath apply host:local
An empty list fetches nothing without a url. A failed origin is
skipped; apply fails only when none serves the pin, and says what each
answered.
4.6.2Minecraft: Prism Launcher (R74)§
pkg:prismlauncher is Prism Launcher, with pkg:jre-21 (Java 21) and Qt 6, in
no profile; each is on the origin as its own head. prismlauncher starts it in a box: it sees its own folder
(~/.local/share/PrismLauncher), the display, sound, the GPU, pads and the
network; not the rest of /home, and sudo does not work in it, so a mod
cannot reach them. Mods and modpacks install from Modrinth and CurseForge in
its window. Signing in shows a code and a QR Code to use on any device, at
microsoft.com/link; nothing plays until a Microsoft account that owns the game
is added. To bring a Modrinth App profile over, see
Minecraft.
4.7Plans§
A plan (plan:<name>) is a hashed plan.plan file that builds a
product tree for pkg:<name>: a JSON head (name, produces,
build_needs, run_needs, each need pinned by hash) and a /bin/sh
script that writes /out. oath build runs it in a sandbox with only
the pinned needs mounted and no network. Its /bin and /lib are link
farms of those packs, as on a machine, so a plan pins every pack its
tools load: apps/hello.plan pins bash, coreutils, gcc and glibc, and
gmp, mpfr and mpc, which they run with.
oath fmt -w apps/hello.plan
oath build --file apps/hello.plan
oath set plan:hello --from-file apps/hello.plan
oath build plan:hello
oath set pkg:hello present=true
oath apply
A missing build_needs pack is fetched first when its pkg object
has a url. The second build of an unchanged input set is a no-op.
The script's output goes to /oath/log/plan/<name>.log, not the
terminal: oath build prints the path (and a failed build the last
lines), and oath logs plan:hello reads it. The previous build's log
is hello.log.1.
The build cache is keyed on the plan hash and the input set, so an edited script always rebuilds.
Known gap: the base image does not carry pkg:gcc, so
apps/hello.plan builds on the build host (in ~/.local/share/oath), not on
the box. The plans in pkgs/ are the larger examples.
4.8Recipes§
A package whose licence does not allow redistribution is a recipe
(R23), and so is a coding agent (R31): the image holds its plan, not
its pack, and the machine builds the pack itself from the vendor's own
file. oath (INDEX.md) lists
the recipes under Recipes until their package is present. Claude
Code is one:
sudo oath build plan:claude-code # fetches ~240 MB from downloads.claude.ai
sudo oath set pkg:claude-code present=true
sudo oath apply pkg:claude-code
claude --version # 2.1.292 (Claude Code)
claude doctor then warns that ~/.local/bin/claude is missing: that
is where the vendor's own installer puts it, and Oath does not use it.
pkg:claude-code-src is the vendor's file: url is the file itself
and file is where it goes in the tree (src/claude); hash pins
that tree, so a changed download is refused. The plan builds with
packs every machine has (bash, coreutils, grep, glibc) and must make
the very pack the build host made: pkg:claude-code's hash is that
pin, and apply refuses another. claude runs with DISABLE_UPDATES=1;
a newer Claude Code is a new pin from the build host (cargo make update moves the plan and the source; a built pkg:claude-code is
left at its version, with the commands to rebuild it). Its settings,
login and history are under /home/.claude and /home/.claude.json.
Codex (OpenAI) and Grok Build (xAI) are recipes the same way (R31), though their licences would allow a pack:
sudo oath build plan:codex # fetches 117 MB from github.com
sudo oath set pkg:codex present=true
sudo oath apply pkg:codex
codex --version # codex-cli 0.164.0
sudo oath build plan:grok-build # fetches 57 MB from x.ai
sudo oath set pkg:grok-build present=true
sudo oath apply pkg:grok-build
grok --version # grok 1.0.70 (f50904eaee6c)
codex is OpenAI's package as it ships, with its own ripgrep and its
own bubblewrap, which sandboxes the commands it runs; it starts with its
update check off. codex login signs in; its settings and sessions are
under /home/.codex. OpenAI signs the package with Sigstore, and the
build host checks that signature. grok sets auto_update = false in
~/.grok/config.toml before Grok starts, since that is the only switch
its updater has. grok login signs in; its settings are under
/home/.grok. xAI publishes no checksum or signature, so the pin is
what the build host downloaded. Grok looks for bwrap only under
/usr, so on Oath it runs commands without its sandbox. A machine
installed before these recipes gets their objects the way the update
names them: oath new pkg:<name>-src --from-json and oath new pkg:<name> --from-json with the output's documents, and oath set plan:<name> --from-file pkgs/<name>/<name>.plan.
Steam is another, in the desk profile (R29): pkg:steam is Valve's
launcher, from repo.steampowered.com, checked against Valve's signed
.dsc. Its needs (bubblewrap, glibc for i386, libxcrypt) are present
on the desk. A machine whose catalog lacks the objects (an update
reports pkg:steam pkg:steam-src plan:steam) gets them as oath new pkg:steam-src --from-json … with the output's document and oath set plan:steam --from-file pkgs/steam/steam.plan. Then:
sudo oath build plan:steam # fetches 20 MB from repo.steampowered.com
sudo oath set pkg:steam present=true
sudo oath apply pkg:steam
steam # on the desk; DISPLAY defaults to :0
steam starts Valve's launcher in a sandbox that gives Steam the
root it assumes (/usr, /lib64, /lib32, /sbin/ldconfig, an
ld.so cache, /etc/ssl) and leaves the machine's root as it is. On a
first run it opts into Valve's 64-bit client (the beta channel), then
downloads the client (about 1 GB) into ~/.local/share/Steam, where
Steam updates itself; Oath pins only the launcher. Its logs are
~/.local/share/Steam/logs/console-linux.txt; the sandbox's tree is
$XDG_RUNTIME_DIR/oath-steam/.
oath update builds a recipe the owner built again, from the output's
plan, before it changes anything, and moves it in the same apply: Steam
when its plan or its needs move (the 32-bit Mesa), NVIDIA's driver when
pkg:linux moves.
NVIDIA's driver is a third, in desk (R34), for a machine with an
NVIDIA card. Its build compiles the kernel modules against
pkg:linux-build, so an update that moves pkg:linux builds it
again, to the new kernel:
sudo oath build plan:nvidia # fetches NVIDIA's 420 MB .run from download.nvidia.com
sudo oath set pkg:nvidia present=true
sudo oath apply pkg:nvidia
nvidia-smi # once its modules load (the pass after services, or the next boot)
libglvnd sends GL and EGL to it, the Vulkan loader and libgbm (32-bit
too) find it, and its 32-bit libraries serve Steam. Its DLSS DLLs are
in /lib/nvidia/wine, beside libGLX_nvidia, where Proton finds them;
a game also needs PROTON_ENABLE_NVAPI=1 %command% unless Proton turns
NVAPI on for it. The closed driver runs on the GSP firmware in its
package, which PID 1 puts where the kernel looks. A card's own settings (GSP off,
power caps) belong in the machine's overlay (Kernel module
options).
4.9Security audit§
cargo make audit (R48) matches every package pkgs/ builds, and the
Rust crates in pkg:oath, against cve.org's CVE List V5 (a local mirror
in ~/.local/share/oath/cvelist, fetched each run), OSV, NVD, CISA's
Known Exploited Vulnerabilities and EPSS, and writes
build/audit/latest.json. --offline keeps the mirror as it is and
skips NVD; --name <pkg> scans one. What it matches each package by is
pkgs/audit.toml: its NVD CPE, the names CVE records use, its upstream
repo and release tag (OSV's commit query), an OSV ecosystem name, or why
there is nothing to match; no plan reads that file, so an edit there
moves no hash. A CVE listed under [<name>.not_affected], or one a
pkgs/<name>/patches/*CVE-….patch fixes, is listed apart.
A CVE record can come days after the release that fixes it, so the scan
also reads each pin against upstream's newest stable release: from its
repo and tag by git ls-remote, else from the directory its source
comes from. Each package gets newer, or upstream_unread and why;
cargo make audit --upstream is that check alone (under a minute, and it
writes nothing). series holds a package to a release line, dev_odd and
dev_micro mark development releases, and unchecked says why a package
cannot be read. A newer release is a prompt to read its notes, not a
finding: only one that names a security fix moves the pin.
On buccina, svc:audit runs it daily at midnight through check/audit/run:
a headless Claude Code run of the oath-audit skill, in a box
(check/audit/box, R77), triages the hits, upgrades to upstream's fixed release
(or applies upstream's patch, or a backport marked oath-backport-CVE-…), commits
and checks. Outside the box the runner then lands those commits, pushes and
publishes, mails Joshua the report and moves buccina to the release. The box holds
no key but its own Claude token, ~/.config/oath/claude-token (mode 600), which
lasts a year. Without it the run mails that it did not run; to make one, run
claude setup-token and save what it prints there.
Every run, fix or no fix, it also runs cargo make pkgs --stale, which lists
each package built against something the lock has since replaced, and rebuilds
those (R75); a command may last 6 hours.
4.10Host store cache and publishing§
The build host keeps the same layout under ~/.local/share/oath/store (or
$OATH_STORE), outside the checkout:
cargo make store --name hello --from apps/hello --tar
cargo make advisories # OSV lookup from components.json
cargo make audit --publish # the audit's latest.json, signed, on the origin (R61)
cargo make publish # upload to the default origin what it lacks
cargo make publish --dest <dir> # or write the same tree into a directory
To the origin, publish goes only from a clean master that is
origin/master (it fetches first), and only with an output built from
a clean tree at a commit master has (R47): commit, push and cargo make build again otherwise; nothing overrides it. --dest is not checked.
publish carries every pack pkgs/lock.json records, build packs
included (gcc, rust, python, …), and every package a seed profile
names, each at the hash this tree records for it: the product in
pkgs/lock.json, or pkg:oath's last cargo make build. Each needs a
licence on record: lock.json's, or for pkg:oath the workspace
licence AND every crate's in its components.json. Each pack goes out
with its source: the plan that built it (plan/<name>/<plan hash>.plan) and the source pack the plan pins
(pkg/<name>-src/<hash>.tar). It refuses, before writing anything, a
package that is not built, has no licence, has no plan matching the
record, needs a package that is not published, or whose plan builds
with a pack not published at the hash it pins (the lock's, or one the
origin already has from an earlier publish, as CEF's older Mesa): an
origin alone rebuilds every plan on it. INDEX.md lists name, hash, licence, plan and source,
and goes up last. A recipe goes out as its plan only, listed with the
product hash it builds. Test packs and other hashes in the cache are
never published; the last line names what was left out.
To the default origin, publish lists the bucket, uploads only the
hashed files it lacks (pkg/ and plan/ never change) plus adv/ and
INDEX.md, and reads INDEX.md back from
https://store.oath.moonlight.pm. The write key is the build host's
alone: OATH_ORIGIN_S3 (https://<account>.r2.cloudflarestorage.com/oath-store),
OATH_ORIGIN_KEY_ID and OATH_ORIGIN_SECRET, from the environment or
~/.config/oath/origin.env (mode 600, never committed). They are an R2
S3 key: a Cloudflare account token allowed to read and write objects
in oath-store only; its id is the key id and the SHA-256 of its value
the secret.
Then publish puts out the newest output cargo make check passed
(--output <id> for another), which machines update to (R38,
Install): out/<id>/update.tar, out/<id>.json and
out/<profile>.json, each .json with a .sig from the build host's
signing key, ~/.config/oath/sign_ed25519 (OATH_SIGN_KEY; make one
with ssh-keygen -t ed25519 -N '' -C oath@<host> -f ~/.config/oath/sign_ed25519). Machines take it only if that key's
public half is in their host:local.publishers. It refuses an output
whose packs the origin lacks (a pack pinned from elsewhere is its
owner's to publish), and an output already published stays as it was,
so publishing an older one again never moves a profile back.
Optional backup hooks in a pack (not catalog fields):
libexec/oath-backup-quiesce and libexec/oath-backup-thaw. See
Services.
5Network and SSH§
5.1The link§
There is one object: net:net0. PID 1 renames the machine's only
non-loopback NIC to net0. Do not hunt eth0 or ens3. Do not admin
with ip / dhcpcd; apply converges with /bin/ip and
/bin/dhcpcd.
| Id | Default |
|---|---|
net:net0 | up: true, ipv4: dhcp |
oath get net:net0
sudo oath set net:net0 ipv4=192.168.1.20/24 gateway=192.168.1.1
sudo oath apply --confirm
ssh home@192.168.1.20 sudo oath keep
ping works as home: PID 1 opens ICMP echo sockets to every group
(net.ipv4.ping_group_range), so it needs no setuid bit.
ipv4 is dhcp or a.b.c.d/n; set refuses anything else. Any
change to up, ipv4 or gateway can cut your SSH session, so
oath apply refuses it (exit 3) without --confirm, and once applied
it is on probation: unless a new SSH session runs sudo oath keep
within 180 s (--probation <secs>), or if the machine reboots first,
PID 1 undoes it. A bad address rolls itself back. Serial still works.
On QEMU, user networking hands out 10.0.2.15 by DHCP and forwards
host 127.0.0.1:2222 to guest port 22 (OATH_SSH_PORT).
OATH_BRIDGE=br0 attaches the guest to a host bridge instead (needs
qemu-bridge-helper and /etc/qemu/bridge.conf; no port forward).
With ipv4: dhcp, converge takes the first lease with a one-shot
dhcpcd -1, and PID 1 keeps dhcpcd -B … net0 running so the lease
renews (IPv4 only, no config file; Oath's hook
/lib/oath/dhcpcd.hook writes /etc/resolv.conf; the processes that
read the network run as void, chrooted in /var/empty). ps shows it as
dhcpcd: net0 [ip4]; sudo dhcpcd -4 -n net0 makes it rebind now. That client is PID 1's, not a svc object; it restarts with
backoff like a service, and its log is oath logs net:net0. Applying a
static address stops it before the address changes; applying dhcp
again starts it. Each lease and renewal the client gets shows
in actual: lease is the address and dhcp has the router, DNS
servers, lease length and when it was renewed
(oath get net:net0 --actual).
5.2Names§
Names resolve through /etc/hosts, then the DNS servers the lease
names (or that /etc/resolv.conf has): /etc/nsswitch.conf says
hosts: files dns. A LAN machine the DNS does not know goes in
host:local.hosts, which /etc/hosts lists after localhost and this
hostname:
sudo oath set host:local hosts='{"novus": "10.0.0.5", "nas": "10.0.0.12"}'
sudo oath apply host:local
getent hosts novus
set replaces the whole map; oath undo takes it back. Do not edit
/etc/hosts: every apply and boot writes it again. When the network's
DNS can serve a name, add it there instead, for every machine at once.
A machine-specific map belongs in the host's install overlay
(hosts/<name>.json).
dig, host and nslookup (BIND 9, pkg:bind) ask the DNS servers
in /etc/resolv.conf directly and never read /etc/hosts; getent hosts <name> shows what programs get. dig @<server> <name> asks one
server, dig +trace <name> walks down from the root.
5.3Time§
svc:time is chronyd -d against 0–3.pool.ntp.org (decisions
R12, R18), configured on its command line, with no config file; after
start-up it runs as void, keeping only the right to set the clock. In its
first three updates it steps a clock that is off by more than a second
(makestep 1 3), then slews. Its log is chrony's own:
oath logs svc:time
… Selected source 192.0.2.10 (0.pool.ntp.org)
… System clock wrong by -102.361823 seconds
… System clock was stepped by -102.361823 seconds
To use other servers, change its exec (oath set svc:time --from-json '{"exec": […]}' and oath apply).
5.4SSH§
SSH login is home (uid 1, HOME=/home, login shell /bin/zsh). Root SSH
is denied and passwords are off. sudo has no password. Serial is
root when svc:serial is enabled. There is no private key in the
image.
Host keys are generated on first boot under /oath/ssh/. Login keys
are ssh:local authorized (OpenSSH public key lines); apply writes
/home/.ssh/authorized_keys.
oath get ssh:local
sudo oath set ssh:local --from-json '{"authorized":["ssh-ed25519 AAAA…"]}'
sudo oath apply
Each key must start ssh-, ecdsa- or sk-. Adding a key is a plain
apply. Dropping one (or emptying the list) can lock you out, so it
needs oath apply --confirm and goes on probation: log in with a key
that is still listed and run sudo oath keep, or PID 1 puts the old
list back. Undo restores the last keys; it never rolls back the host
keys under /oath/ssh/.
Disabling or removing svc:sshd, or changing its exec, user or
env, also needs --confirm and goes on probation.
Where the first keys come from:
cargo make installwrites the build host's public keys (~/.ssh/*.pub, default identities without a.pub,ssh-add -L, orOATH_SSH_PUBKEY) intossh:local.cargo make run/up/startpass the same keys to the QEMU guest, and PID 1 adds them tossh:local(desired and applied) at boot.cargo make checkpasses its own throwaway key the same way.
svc:sshd is OpenSSH's sshd (pkg:openssh, not removable, with
OpenSSL: RSA, ECDSA and ed25519 keys log in), which also gives /bin/ssh, /bin/scp,
/bin/sftp, /bin/ssh-keygen and /bin/sftp-server. Logins are
home's, by key; passwords and root are refused. Before a login, sshd
reads the network as void, chrooted in /var/empty. oath writes
/etc/ssh/sshd_config from host:local with /etc/profile, so a
command over SSH (ssh host 'cmd', no login shell) gets host:local
env and TZ too, as of sshd's last start. The host key is
/oath/ssh/ssh_host_ed25519_key; a machine that ran dropbear keeps its
key. Guest /etc/ssh/ssh_config is
StrictHostKeyChecking accept-new. scp and sftp as home work:
scp ./file home@host:/home/file
cargo make ssh # QEMU: ssh -p 2222 home@127.0.0.1
cargo make ssh -- ls / # extra ssh(1) args
6Generations§
Undo is a filesystem snapshot, not a journal.
6.1Layout§
The system disk is btrfs.
| Subvolume | Role |
|---|---|
@ | Live root, mounted at / |
@gen-N | Read-only snapshot of @ taken before apply N converges |
@boot-N | Snapshot of @ taken when cargo make esp archives a boot |
PID 1 mounts the btrfs top level (subvolid=0) at /oath/run/fs:
/oath/run/fs/@ same tree as /
/oath/run/fs/@gen-1 generation 1
/oath/run/fs/@gen-2 …
Generations are siblings of @, not nested under /.
A machine keeps the newest five generations and every one taken since it
booted; PID 1 removes the rest once a boot (oath reap, R62, from the next release), and undo
stops at the last one kept.
A generation is all of @, so every apply pins whatever is there until
that @gen-N is gone. Keep fast-changing
bulk, such as a cargo target/, on another disk or in a nested
subvolume, which a snapshot leaves out.
snap:current holds the last snapshot id. snap:N is a read-only
record (generation, parent, time, reason) in its desired
document. Do not set snap:N.
A generation holds spec only: status lives on the /oath/status
tmpfs, so no @gen-N or @boot-N carries it, and an archived boot
observes its own.
6.2Apply and undo§
oath apply first checks everything it can without touching the
machine: every selected object parses as its kind, confirm-class
changes have --confirm, there is no wants cycle, and the package
needs graph holds. A refusal takes no generation, changes nothing,
and is logged with "result":"refused". Then it snapshots @ to
@gen-N (holding what was applied before, not your pending edits) and
converges. N is never reused (it is max(existing)+1). The log line
says ok, or error if converging failed after the snapshot.
oath undo works like a stack. It takes the newest apply, ok or
error, that has not been undone yet, and puts back the spec of the
objects that apply touched from its @gen-N. An object that apply
created (oath new) is removed; one it removed (oath rm) comes back.
Then it converges those objects (hostname, pkg:* links, net:net0,
ssh:local, dev:*, and svc:* through PID 1), sets snap:current
back, and logs undo:N. Run it again to step back one more apply.
Undo does not touch the apply log, the package store, the SSH host
keys, or objects the apply did not touch. It keeps a package that the
image fixes even when that apply created it (cargo make update makes
one a machine lacks): the image's own files need it. A pending oath set on some
other object survives an undo.
An apply that can cut off SSH is on probation (/oath/probation.json)
until a new session runs oath keep. If its time runs out, or the
machine boots again first, PID 1 runs the same undo itself (logged
with tty probation).
A reboot or halt through the catalog (power=reboot) on its own takes
no generation: power is not configuration, so there is nothing for undo
to take back. It is logged ok without a generation, and PID 1 shuts
down cleanly. Applied together with other changes, it is part of their
generation. Plain sudo reboot reboots the same way.
6.3Boot menu§
On metal, EFI/BOOT/BOOTX64.EFI is oath-efi. It sets the native GOP
mode, paints the Oath mark, and shows a 5 s menu: the current kernel
plus the last five archived kernel+initrd pairs. Up/Down/Enter. An
archive boots oath.subvol=@boot-N, the userspace snapshot taken when
it was archived. oath-efi is the only loader; there is no fallback
loader on the ESP.
6.4What the boot screen says§
Under the mark, a line says what the machine is doing and a bar moves while it works. A machine that stops booting shows where:
Reading the kernelorReading the initrd, with the bar still: the firmware cannot read the disk; try another port.Starting Linux 7.3.1and nothing more: the kernel stopped before Oath's PID 1 ran.Loading drivers: <name>with the bar still: that driver hung.Waiting for the root disk … Disks: …: the disk Oath runs from has not appeared; the list is what the kernel sees.- Amber text: the boot failed, and says why. A shell waits on the serial console.
Oath is running, with no driver for this … display (vendor:device). Reach it with ssh home@<address>: the boot finished, but Oath has no driver for the graphics card.
Once a display driver loads, the screen is the desk's or tty1's.
cargo make esp --esp <vfat node> --confirm [--root /dev/sda2] [--output <id>] takes the kernel, initrd and oath-efi of a build
output (default: the newest one cargo make check passed;
--unchecked for another), archives the live ESP kernel as a new boot,
keeps five, snapshots @ to @boot-<id>, and copies the new bits. It
does not format anything. It mounts the node you name on the
machine where you run it, so it refuses to run on a machine that is
not Oath (no /oath/INDEX.md), such as the build host.
QEMU cargo make run / up / start boot with -kernel and have no
menu. cargo make install --qemu boots through OVMF with a 0 s
timeout.
qcow2 snapshots on the QEMU host are debugging aids, not this mechanism.
7Build and QEMU§
Everything is built on the build host with cargo make (crate
oath-make), from a plain shell.
Built here, from source: the kernel; the toolchain (GCC, binutils,
glibc); every package in the image (pkgs/); PID 1 (oath-init,
serial-login, sudo), oath, and the UEFI loader oath-efi
(pkg:oath, built by the vendor's Rust release, pkg:rust). On a
NixOS build host, Nix gives QEMU and its UEFI firmware for running
images (image/host.nix), and the host compilers that build stage 0 of
the toolchain and oath-make itself. None of it ships. An Oath machine
can be the build host too (R32): its own gcc builds stage 0, its
pkg:rust builds oath-make, and pkg:qemu (in no profile; add it to
the machine) runs images, with the edk2 firmware from QEMU's release as
OVMF. home has /dev/kvm there (R33). Without it, build, check
and the QEMU rehearsals stop and say so. QEMU loads libslirp, which no
profile has either, and a pack's needs must name the machine's packs
at the hashes it runs with: those are run_needs in its plan on the
origin (INDEX.md gives each pack's hash and plan).
curl -fsS https://store.oath.moonlight.pm/INDEX.md | grep -E '`(qemu|libslirp)`'
sudo oath new pkg:libslirp --from-json '{"present":true,"hash":"sha256-…"}'
curl -fsS https://store.oath.moonlight.pm/plan/qemu/sha256-….plan # its run_needs
sudo oath new pkg:qemu --from-json '{"present":true,"hash":"sha256-…","needs":[…]}'
sudo oath apply pkg:libslirp pkg:qemu
7.1The toolchain and packages§
cargo make bootstrap # once: stage 0 + stage 1 toolchain (an hour or more)
cargo make pkgs # every package: the base until it settles, then the rest
cargo make pkgs -j 4 # the same, four packages at a time after the base
cargo make pkgs --list # versions, licences, what pkgs/lock.json pins
cargo make pkgs curl # one package (and what it builds with)
bootstrap builds a cross binutils, GCC and glibc with the host's gcc
(stage 0, never shipped), then with stage 0 the base packs (stage 1):
pkg:gcc (GCC and binutils), pkg:glibc, and bash, coreutils, sed,
grep, gawk, findutils, diffutils, patch, tar, gzip, xz, bzip2 and make.
pkgs builds each pkgs/<name>/ as an oath build plan in the
sandbox (no network, only the declared packs, glibc's loader and every
pack's libraries under /lib), writes the exact plan to
pkgs/<name>/<name>.plan and the product to pkgs/lock.json. Every
plan builds with the base packs; the base packs rebuild themselves
this way too. Build-only packs (python, meson, ninja, m4, flex, bison,
bc, perl, elfutils) are not in the image. pkg:rust is the vendor's
Rust release (pinned by sha256, its binaries pointed at /lib's
loader); it builds pkg:oath, static against glibc. The image takes
every userspace package from the lock. How packages are written:
pkgs/README.md.
7.2The kernel§
There is one kernel, pkg:linux: vanilla kernel.org Linux, pinned by
sha256 in pkgs/linux/pkg.toml, plus pkgs/linux/patches/ and the
config fragment pkgs/linux/linux.fragment over defconfig. It is a
plan product like every other package, built by pkg:gcc with the
flex, bison, m4, bc, perl and elfutils (libelf, for
objtool) packs, which are build-only.
cargo make pkgs linux
The product holds boot/vmlinuz, boot/config and
lib/modules/<release>/ with the depmod index (modules.dep,
modules.alias, modules.softdep) written from each module's
.modinfo. cargo make takes it from pkgs/lock.json (or
OATH_KERNEL / OATH_MODULES). Firmware for the initrd comes from
pkg:firmware, not the kernel build. Packing refuses a bzImage whose
release does not match the module directory or the vermagic of
btrfs.ko. Bump the version and the sha256 together (pkgs/glibc
takes its UAPI headers from the same tarball).
7.3The image§
Needs: the packs in pkgs/lock.json (pkg:rust among them) and
mkfs.btrfs and unshare on the host (shell.nix has them); no root.
The build host's Rust compiles only oath-make.
cargo make build # profile base → build/out/<id>/
build builds pkg:oath from this tree: it packs the workspace and
every crates.io crate in Cargo.lock (fetched once into build/fetch,
checked against the lock's sha256) and runs cargo build with
pkg:rust in the sandbox, offline. A tree that has not changed is not
rebuilt; cargo make pkgs oath does this step alone. Then build
writes the boot initrd and the
installer initrd, stages the root tree, seeds /oath from the profile,
packs the profile's packages into the hashed store (also cached under
~/.local/share/oath/store), and writes a build output:
build/out/<id>/
manifest.json what the image ships; its hash is <id>
oath.qcow2 btrfs, subvolume @
bzImage initrd.gz BOOTX64.EFI (oath-efi)
initrd-install.gz initramfs-install/ the installer ramdisk
stage/ the root tree install copies
check-pass written by a clean `cargo make check`
build/out/latest -> <id>
A build packs into build/out/.<id>.part/ and renames it when it is
complete; after that only check adds check-pass. The id is the
first 12 hex digits of the manifest's SHA-256 (without the commit), so
a tree that has not changed reuses its output in seconds. The newest
five outputs are kept, and the newest checked one.
manifest.json lists every pack the image ships: its hash, version,
licence and plan hash; where it ships (rootfs for the profile's
store, initrd, installer for the install initramfs's own tools,
esp); and its sources, each with its sha256 and signing key
(pkgs/keys/…) or why it is unsigned. pkg:oath records the commit
of this tree instead, with -dirty when its sources had uncommitted
changes.
A profile can pin a package Oath does not build (R28): its entry has
hash and needs (and url if it is not on the profile's origins),
and there is no pkgs/<name>/:
"pkg:hello": {
"present": true,
"hash": "sha256-…",
"needs": [{ "id": "pkg:glibc", "hash": "sha256-…" }]
}
When the profile ships it (present), build fetches it into
~/.local/share/oath/store and the manifest marks its entry
"pinned": true; the manifest's pinned lists every package the
profile pins this way, shipped or not. When a need's hash is not the one the image
ships, the manifest lists it under stale_needs, the image takes the
shipped hash, and build, check and update print a warning: its
owner should build it again. publish leaves it out, and update
keeps whatever pin the machine has for it, moving only its needs.
7.4Test it§
cargo make check # every case in check/cases.toml
cargo make check --build # pack first
cargo make check --case index --case pkg-fetch
Each case boots its own VM from the latest output's oath.qcow2
(--output <id> for another; four at a time) and runs its steps over
SSH as home. The image is the one install ships; a case makes any fixture it needs with oath new. The summary marks each case PASS,
FAIL, GAP (a known failure; the gap line names the phase that fixes
it) or FIXED (a gap that now passes; remove its gap line). Logs, the
serial console of every boot and report.json are under
build/check/<time>/<case>/. A clean full run writes check-pass into
the output it booted, which cargo make install requires.
cargo make check --target home@10.0.0.108
--target runs the cases marked metal = true on an installed machine
instead: one at a time, as home, trusting only the host key that
install pinned in build/known_hosts. The machine must start
in-sync (oath diff). After each case the harness runs oath undo
back to the generation the case started from and requires oath diff
to say in-sync again; if it does not, the run stops there. A reboot
step waits up to five minutes for a new boot id. A case that needs
hardware (needs = { what = "a connected display", run = "…" }) runs
that test first and is SKIPped on a machine that fails it: the
desk's display and Sola cases on a headless machine, the RADV case
without an AMD GPU. QEMU always has what they need. Logs are under
build/check/<time>-<host>/. No stamp is written.
7.4.1A real model§
ANTHROPIC_API_KEY=… cargo make check --agent # every task, haiku
cargo make check --agent --case rename --model sonnet
cargo make check --agent --target home@<host> # on a machine
--agent gives a model that has never seen Oath one MCP server, the
machine's ssh -T home@… oath mcp, and each task in check/agent.toml
(rename the machine, run a service, undo a mistake, fix a failing
service, reboot), then checks the machine over SSH. The client is
Claude Code, headless, with no shell, files, memory or CLAUDE.md; it
is a template in check/agent.toml, so another client can run the same
tasks. A task fails if its checks fail, if a change in the apply log did
not come through MCP, or if the model passed confirm where the task
did not ask for it. It is opt-in and never gates install: it costs
money (the budget per task is in the file) and a model is not
deterministic. The key is ANTHROPIC_API_KEY from the environment; it
refuses to run without one and writes it nowhere. Results are in
build/out/<id>/agent-check.json (task, model, pass, cost, turns; a
run of some tasks replaces only their entries),
transcripts in build/out/<id>/agent/<task>.jsonl, and each VM's
serial console and steps under build/check/<time>-agent/<task>/; with
--target, all of it is under build/check/<time>-<host>-agent/.
7.5Run it in QEMU§
cargo make run # interactive serial on this terminal
cargo make run --build # pack first (same flag on up / start)
cargo make up # headless, foreground; Ctrl-C kills QEMU
cargo make start # headless, background
cargo make ssh # ssh -p 2222 home@127.0.0.1
cargo make stop
Each run gets build/runs/<id>/ with a qcow2 overlay (the built image
is never changed), serial.log, and meta.json. Every mode passes the
build host's SSH public keys to the guest. QEMU uses KVM when
/dev/kvm is there, 4 GiB of RAM, virtio disk, net, and rng.
The serial console is a root shell; SSH is home. Then:
oath
oath ls
When DISPLAY is set, run also opens a gtk window on the guest's
virtio-gpu: its screen, a shell as home (svc:screen).
OATH_DISPLAY=none hides the window.
7.6Leave the VM§
Ctrl-D leaves the shell; svc:serial is restart=always, so PID 1
starts a new one. There is no QEMU monitor, and Ctrl-C goes to the
guest under run. To stop the VM, power it off from inside:
sudo poweroff
(or oath set host:local power=halt and oath apply --confirm). PID 1
stops the services, syncs and remounts / read-only first.
run also exits when the guest reboots (-no-reboot). If the prompt
is dead, pkill -f qemu-system-x86_64 from another terminal.
7.7Telemetry§
PID 1 and oath write JSON lines prefixed oath-tel to stderr, and
to /oath/log/*.jsonl once the disk is mounted.
8Install§
Wipe a named disk and copy the packed Oath tree onto it.
cargo make build
cargo make install --target user@host --disk /dev/sda --confirm \
--hostname canto --overlay hosts/canto.json
--confirm is required. --disk is the whole disk (GPT). The live
OS on that host must be reachable over SSH as root, or as a user with
sudo -n. Every command is fed to the target's /bin/sh on stdin and
files are written with dd, so the user's login shell does not matter.
8.1What it does§
- Picks a build output:
--output <id>, else the newest one that passedcargo make check(itscheck-passmust match the kernel, initrd and image).--uncheckedtakes the latest output without that;--qemurehearsals take the latest and are not gated. Everything below comes from that one output; the build host's keys are baked into a copy of its installer ramdisk underbuild/runs/<time>-install/, never into the output. Reads and checks--overlayagainst a scratch catalog, before anything is wiped, and refuses a--targetthat is the machine it runs on (the sameboot_id). - Boots an installer ramdisk on the target (
oath.install=1, OpenSSH's sshd, the build host's keys for root). On EFI machines this is a one-shot loader entry; kexec is the fallback (Oath's own statickexec, copied to the target, so the live OS needs none). - Partitions GPT with the ramdisk's native tools (
wipefs,sfdisk): a 512 MiB ESP (mkfs.fat), then btrfs (mkfs.btrfs) with subvolume@and a new UUID, which the boot entry names the root by (oath.root=UUID=…): the disk's node changes with the order disks are found in. - Copies the packed root tree (GNU
tar). - Merges the catalog overlay with the ramdisk's
oath(oath --root /mnt/oath seed --overlay):host:local.hostname(defaultoath), the build host's public keys asssh:local, then your--overlayon top. Makes the new system's SSH host key (/oath/ssh/host_ed25519) and pins it inbuild/known_hosts. - Writes the ESP:
oath-efiasEFI/BOOT/BOOTX64.EFI(the only loader),vmlinuz,initrd.gz,loader/entries/oath.conf, andloader/oath-boots. The kernel and the initrd modules are the same checked build. - Reboots and waits for SSH as
home, accepting only the pinned host key, then runsoath ls.
net:net0 stays dhcp from the seed.
8.2The overlay§
Machine-specific facts live in an overlay, not in the seed. Each object is a JSON merge patch on the seeded desired; a new id creates the object. See Catalog.
{
"objects": {
"host:local": { "timezone": "MST7MDT,M3.2.0,M11.1.0" },
"svc:serial": { "enabled": false },
"svc:backup": {
"exec": ["/lib/oath/backup-daily", "nas:/export/backup/canto"],
"wants": [], "restart": "always", "enabled": true
}
}
}
An overlay that sets ssh:local.authorized replaces the build host's
keys rather than adding to them. Each patched object must pass the same
checks as oath set, and the overlay counts as applied: the machine
boots with it, no oath apply needed (except to link a package an
overlay turns on).
8.3Check the installed machine§
cargo make check --target home@10.0.0.108
Runs the harness cases marked metal on the machine, one at a time,
over SSH with the key pinned at install. See Build.
8.4Rehearse in QEMU§
cargo make install --qemu --disk /dev/vda --confirm
That runs the same steps against a fresh qcow2 in QEMU with OVMF, then
boots it through oath-efi.
8.5Onto a USB disk plugged into this host§
cargo make install --local --disk /dev/sdd --confirm \
--hostname novus --overlay hosts/novus.json
For a machine that boots Oath from a USB disk, the build host itself
included. The disk must be removable, USB and not mounted. The installer
ramdisk runs in QEMU with the raw disk as its own (/dev/vda; the disk
is lent to your user for the run), does steps 3 to 6, then OVMF boots
the disk once, oath ls must answer, and it is powered off cleanly. Its
loader waits 5 s, as on metal. Boot it from the firmware's boot menu;
nothing changes the firmware's boot order. Such a stick is also an
installer: oath install (below).
8.6From a stick, on the machine: oath install§
Boot an Oath stick (the one above) on the machine, open a terminal:
oath install # the disks: node, size, port, model, serial
sudo oath install /dev/sdb # what it would erase and write; writes nothing
sudo oath install /dev/sdb --confirm # wipe it and install
sudo poweroff # then remove the stick and start the machine
The disk is a node (/dev/sdb, sdb) or a serial; sudo oath install
also shows each filesystem's type and label. Node names follow the
order the kernel found the disks in and can differ from another
system's, so check the model and serial the preview repeats.
--confirm wipes that disk: a GPT with a 512 MiB ESP and btrfs for the
rest, with @ (the system) and @home (mounted at /home, outside
every generation and boot snapshot; the first boot adds @work, at /work,
likewise). It copies the system the stick
runs: its packages (NVIDIA's driver too, if the stick built it), its
settings (hostname, overlay, SSH keys it trusts) and its release, at
generation 0, with an empty home, a new SSH host key (printed at the
end), and none of the stick's history, password or pairings. It writes
the ESP, adds Oath to the firmware's boot list first and removes the
other systems' entries; the firmware's own (USB, network, setup) stay,
and nothing else in the firmware's variables is touched.
Remove the stick only once the machine is off: the system runs from it
until then, and reboot with the stick gone finds no programs. Some
firmware keeps its own order or makes entries again: AMI's (an ASUS
board) kept UEFI OS, its entry for another disk's fallback loader,
and did not start Oath first. If the machine starts something else,
move Oath first in the firmware's setup.
It checks no backups and reads no other disk: knowing what is on them
is yours. It refuses only the disk the stick runs from and a disk with
something mounted. Agents run the same over SSH (--json for each
step).
8.7USB installer stick§
cargo make install --usb --disk /dev/sdd --confirm
The disk must be a removable USB device. Boot the stick's EFI entry
(hold Option on a Mac). The ramdisk gets a shell on tty0 and DHCP on
the NIC with carrier. Then run the --target command from the build
host; it sees /run/oath-install/ready and skips step 2.
8.8Update an installed machine§
Each machine updates itself from its origins (R38). The build host
publishes every output it checked (cargo make publish,
Packages); on the machine:
oath update # the newest output for this machine, and what moves
sudo oath update --confirm # move to it; PID 1 re-executes into its binary
sudo oath update --confirm --reboot # move to it, then boot it
It takes the newest release for host:local.profile from the first of
host:local.origins that serves one a key in host:local.publishers
signed (the base profile trusts buccina's), and refuses one older
than the release the machine runs (/oath/output.json, written by the
install too), or one it cannot order against it, unless --output <id>
names it. It also refuses pending changes and an apply
on probation, and a set of moves that would leave a package needing
another at a hash it will not have (the check oath apply makes, made
first, so the preview says it too). The preview names each overlay,
a package the machine runs at a pin other than the release it runs set
(R47): pkg:sola is sessions-anywhere@0d99e340, not the release's: the update replaces it with sha256-… (master@…). The update replaces it;
--keep pkg:sola (repeat for more) keeps the machine's pin, and only
its needs move. With --confirm, it:
- fetches every pack it will pin that the store lacks, and builds each
recipe the owner built (Steam, NVIDIA's driver) from the output's
plan, before it changes anything: a fetch or build that fails (a
network error is tried twice more), or a build that is not the
output's pin, leaves the machine as it was, and running it again
fetches the rest. Then no two packages as the moves leave them may
ship one name in
/binor/lib(a package kept with--keepthat the output split in two), or it stops there; - without
--reboot, has PID 1 re-execute into the output's binary (below), then snapshots@as@boot-<n>; - writes the output's accounts (
/etc/passwd,group,shadow) and/var/empty, since services that restart next drop tovoid(R36); - sets each package the machine has to the output's pin, needs and
source; makes each package and service new to the profile since the release the machine runs (its record lists that release's objects,seed; a record from before R47 tells only for the packages pinned from elsewhere,pinned), each package the output fixes (the image's files need it:/etc/passwdnamespkg:zsh's shell) and each one a package it has present needs, down their needs (R41'spkg:libglvnd); files each plan new to the profile; and runsoath applyonce (a name in/binor/libthat moves from one package to another,libEGL.so.1there, moves in it whichever converges first).oath undotakes it back but keeps a fixed package. An object the release the machine runs had and the machine lacks was its owner's to remove and is not made again; one new to the profile that the machine has absent (oath buildleaves a product it made so) is made present, never the other way; any other the output's seed has is listed, foroath new. A package the machine's output had and this one does not is removed, unless--keepnames it or something that stays needs it (R62, from the next release); one its owner added keeps its pin. A recipe the owner built moves to the pin step 1 built. Each running service whose program moved restarts on it, except one with aseat(the desk's and the console's, R42), which keeps its program until it next starts: the desk takes its new programs at a reboot, or at Exit Sola andoath desk. With--rebootno service restarts, and the boot starts each on its new program. When sshd's program moved, the apply is on probation: the services restart then (the seat's still keep theirs), and the update waits until you open a new SSH session and runsudo oath keepthere; - replaces
/lib/oath, the image's static/etcfiles,/sbin/initand/bin/sudo; - archives the ESP's kernel and initrd as boot
<n>(with the snapshot from step 2) and writes the new kernel, initrd andoath-efi, keeping five archives (--esp, default partition 1 of the root disk); - with
--reboot, reboots; without it, says what waits for a boot (a new kernel, a PID 1 that could not re-execute, and what PID 1 does only at boot).oath updateafterwards says whether PID 1 is the output's. Anoathfrom before R47 that made the update wrote the record without the release's commit, pins and seed;oath updatesays so, andsudo oath update --confirm(the same release) writes the record as signed. Objects that release's seed has and the machine lacks are, from then on, the owner's to add (oath new).
A new PID 1 without a reboot (R73). When the output's PID 1 is not
the one running, an update without --reboot asks PID 1 to re-execute
into the output's binary before it applies anything, so the new PID 1
is the one that restarts what moved. PID 1 checks the binary's hash,
lets it read a copy of its state first, and refuses one that cannot;
every service keeps its process. The update prints PID 1 runs …, and
--json says "pid1": "re-executed". If the update stops after that,
PID 1 goes back to the binary it ran. A new kernel still takes a
reboot, and so does what PID 1 does only at boot (mounts, modules,
sysctls, device modes, /etc). A PID 1 from before R73 is not asked:
its first update to one takes a reboot, as before.
A machine that still runs dropbear as svc:sshd (from before R36)
moves to OpenSSH after it has booted the new PID 1, which writes
/etc/ssh/sshd_config and carries dropbear's host key over: run sudo oath update --confirm again, then sudo oath keep from a new SSH
session. If none gets in, PID 1 undoes it and dropbear serves again.
An oath from before step 1 (pkg:oath sha256-6e563f2d6c69… and
older) fetches during the apply instead; if a fetch fails there, the
machine is half moved and oath update refuses the pending changes.
sudo oath apply finishes them; the update run after it snapshots the
half-moved @ as @boot-<n>, so move the first one aside and back if
the archived boot should keep the old userspace. Two builds of step 1
(pkg:oath sha256-e3790fbd2f8b… and sha256-fa9f700146a9…) also
pinned each pack as they fetched it, so the apply skipped a package
that had been applied before and left it unlinked: on a machine with
either, take the newer oath by hand first, as above, then update.
An oath from before R42 (pkg:oath sha256-618db152642b… and older)
does not make the new packages the others need, nor hand /lib's names
from one package to another, so its apply refuses:
nothing converges, but the specs it set stay as drift, and the next
update sets them again. An oath from before 2026-10-30 (pkg:oath
sha256-92735e24e583… and older) does not build a recipe the owner
built, so a Steam pinned to the old 32-bit Mesa makes its apply refuse
the same way (canto, 2026-10-29), and it restarts the desk under the
person at it, --reboot or not. Take the newer oath (desk output
fed6fb141a5e's) first, then update:
sudo oath set pkg:oath hash=sha256-a937565c5f8a5a210f233f9ec58c56ec1ed844e93fc5847fb2edddedbdf890a5 && sudo oath apply pkg:oath
oath update && sudo oath update --confirm --reboot
A machine from before R38 has no oath update yet. Give it the newer
oath first (the pin is pkg:oath's hash in the output's seed,
oath get pkg:oath on any newer machine), then the profile it follows
and the key it trusts:
sudo oath set pkg:oath hash=sha256-… && sudo oath apply pkg:oath
sudo oath set host:local profile=desk \
publishers='["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH+YKD3yXYsGJMxwpBIWSYP7VuQvjhzGG+LWvYhKTESD oath@buccina"]'
sudo oath apply host:local
sudo oath update --confirm --reboot
To try an output first, cargo make update --qemu <older output>
installs the older one in QEMU and runs the same update from a local
origin on the build host.
If the new boot fails, pick the archived boot in the firmware menu: it is the old kernel with the old userspace.
cargo make esp does steps 2 and 6 on the Oath machine itself, for an
ESP you mount there. See Generations for why it must
not be pointed at the build host's own ESP.