Mokatai Mesh

Architecture

A walk-through of what actually happens when you run Mokatai Mesh. Starts from the outside (what the user sees) and drills into each layer. Written for a new engineer joining the project or a beta tester curious about the internals.

1. What Mokatai Mesh is, one paragraph

A self-hosted WireGuard overlay VPN with a central coordinator that tracks who’s allowed on the mesh, and per-peer agents that set up and maintain local tunnels. Peers get an overlay IP (e.g. 10.77.0.5) and can reach other peers by that IP regardless of what physical network they’re on. Designed to work over CGNAT and through tactical radio meshes (Silvus), which means it can’t assume any peer has a reachable public IP.

2. Topology

                 +---------------------+
                 |   Coordinator       |
                 |   wg0: 10.77.0.1    |
                 |   api: :8080        |
                 |   wg:  :51820 UDP   |
                 +---------+-----------+
                           |  (hub relay, every peer starts here)
            +--------------+--------------+
            |              |              |
       +----v----+   +-----v----+   +----v----+
       | Peer A  |   | Peer B   |   | Peer C  |
       | 10.77.  |   | 10.77.   |   | 10.77.  |
       | 0.2     |   | 0.3      |   | 0.4     |
       +---------+   +----------+   +---------+
            ^              ^              ^
            +---- direct p2p when possible ---+

The coord is a WireGuard peer like any other — its wg0 address is always 10.77.0.1. When peer A sends a packet to peer B, the default route is through the coord (AllowedIPs on the coord peer = the full /24). The coord IP-forwards the packet back out to peer B’s tunnel. Hub-and-spoke.

When two peers are both reachable to each other directly (both have good NAT, or they’re on the same LAN), the agent detects it and adds a direct peer entry with AllowedIPs pinned to just that peer’s /32. Traffic bypasses the coord. We still keep the /24 coord route as a fallback if the direct handshake goes stale.

3. Components

cmd/
├── coordinator/     Linux-only daemon. Owns the mesh state (sqlite DB),
│                    runs its own wg0, enforces ACLs via nftables.
│                    Boots into setup mode (first-run wizard) until a
│                    public endpoint or a join target is configured, else
│                    main mode (where it auto-elects into the cluster).
├── agent/           Runs on every peer. Brings up wg0, talks to the coord,
│                    reconciles peer config. Linux + Windows.
├── mokatai-mesh-client/      Native GUI for peers (Status hero, Connect/
│                    Disconnect, in-app Join). Soft Slate dark-only,
│                    icon-rail nav. WebKitGTK on Linux, WebView2 on
│                    Windows. amd64 only.
├── mokatai-mesh-admin/       Native GUI for the admin (same technology, points at
│                    the coord's /ui — Soft Slate dark-only, Dashboard
│                    landing). amd64 only.
└── mesh/            CLI for the admin — scriptable, plus `mesh tui` for
   └── tui/          a full-screen bubbletea TUI usable on headless boxes.

internal/
├── api/             Shared request/response types. One source of truth
│                    for what coord and agent agree to send each other.
├── db/              SQLite layer (peers, rules, tokens, coordinators,
│                    join_codes, invites).
├── wg/              Kernel-WireGuard wrapper (coord only — needs root).
├── wgagent/         Userspace WireGuard wrapper (agents, cross-platform).
├── acl/             Compiles role-based rules into nftables rulesets.
├── cluster/         Hand-rolled Raft-*style* leader election state machine
│                    (terms / RequestVote / majority / heartbeats) — pure
│                    decision logic, no I/O. Leadership only, not a log.
├── replication/     Leader → follower replicate-all push for the cluster
│                    (peers/rules/coords + invites/tokens/join-codes).
└── update/          Self-update pipeline (manifest fetch with arch+flavor
                     resolution + legacy fallback, SHA verify, install.sh
                     --upgrade, rollback, cluster-wide push).

The shared internal/api/types.go matters a lot — it’s the contract. Any field change there affects both coord and agent.

4. First-launch wizard (coord boot modes)

A freshly installed coord boots into one of two modes. The trigger is not the role — under the flat cluster an empty MOKATAI_MESH_ROLE is a normal auto-elect deployment (see §11). Instead the coord drops into setup mode only when it has neither a public endpoint (create-a-network) nor a join target (join-a-mesh) configured — i.e. the operator hasn’t chosen a path yet (wantsSetupWizard):

  • Setup mode (no public endpoint AND no join target). The coord runs a minimal HTTP server exposing only /setup/status, /setup/primary, /setup/join, and /healthz. None of the /admin/* routes exist yet. The web UI’s first-launch chooser hits /setup/primary or /setup/join, which rewrites /etc/mokatai-mesh/coordinator.env with the public endpoint (and, for a join, the primary URL + internal token), then exits 0 so systemd restarts into main mode.

  • Main mode (a public endpoint or a join target is set). Full admin API plus /coord/* cluster RPCs, replication, the leader election, ACL enforcement, manifest polling. /setup/status is still exposed here too — it returns {"configured":true, "role":"...", "primary_url":"..."} so the web UI knows to skip the wizard. --role is no longer required: an empty role auto-elects; --role primary / --prefer-leader only bias the election (see §11).

The first-launch chooser offers two paths:

  1. Create new network — this is the first coord. Operator provides a public endpoint (host:port peers will dial). On commit, MOKATAI_MESH_PUBLIC_ENDPOINT is set and role becomes primary. (primary is now a leader preference, not a hard role — the coord still auto-elects and needs quorum to write; a lone coord is trivially its own majority and leads immediately. See §11.)

  2. Join existing network — this coord joins the cluster as another equal hub. Operator provides:

    • Join URL — any current coord (e.g. https://coord1.example.com:8080); the join target it dumps initial state from.
    • Coordinator invite code — a one-shot, 30-min secret. A coord mints it from the top-bar Invite → Coordinator Add slide-over (no longer from Settings); it lands in the invites table with the cluster {internal_token, primary_url} in its payload. Important: this is NOT the same as a peer invite or a peer-enrollment token; the redeem differs by type=coordinator.
    • This coord’s public endpoint — host:port peers will dial when they hit this coord directly.

    On commit, the setup handler POSTs to the join target’s /coord/join-codes/redeem with the code (the redeem URL is unchanged so the setup wizard isn’t touched). The redeem handler reads the invites table first, falling back to the legacy join_codes table for any in-flight old codes, and returns the cluster’s internal_token (the shared secret used to auth all coord-to-coord RPCs). The joining coord writes that token + the join URL + its public endpoint into its env file (role left empty — auto-elect, no leader bias) and self-restarts. On reboot it boots straight into main mode, dumps an initial state snapshot, registers into the coordinators membership table, and participates in the election from then on.

mesh tui detects setup mode by parsing /setup/status’s body (configured: false) and refuses to launch with a friendly “finish the web wizard first” message — admin actions require main mode.

5. Enrollment: from “Invite” click to working tunnel

The primary onboarding path is the unified invite flow: one “Invite” button in the admin top bar mints one shareable artifact (link + QR + short code) that works for both peers and coordinators, served by the coord itself through three friendly join paths. The legacy per-peer enrollment token (mesh add-peer → URL + token) still works and is documented at the end of this section — it’s the CLI/back-compat path, no longer the headline.

The invites table

Invites live in an additive invites table in the coord DB, alongside the legacy enroll_tokens and join_codes tables (which stay). Each row is keyed by a URL-safe code (also the /join/{code} slug) and carries:

  • type — peer or coordinator.
  • name / role — label and (for peers) the role to assign on redeem.
  • reusable — single-use by default; reusable is opt-in.
  • expires_at — TTL: 24h for peer invites, 30 min for coordinator invites (the cluster trust model wants a tight window).
  • revoked_at / redeemed_at / redeem_count — revocation + the single-use gate + audit for reusable invites.
  • payload — JSON; for coordinator invites this holds the cluster {internal_token, primary_url} (the same secret the legacy join-code minted).

DB methods mirror the join-code patterns: CreateInvite, GetInvite, ListInvites, RevokeInvite, and RedeemInvite. RedeemInvite is the atomic gate — it checks not-expired, not-revoked, and (for single-use) redeemed_at IS NULL in one transaction, then stamps redeemed_at / bumps redeem_count — so two racing redeems can’t double-mint a single-use invite.

Admin side — minting an invite

  1. Admin clicks Invite in the top bar → the Add slide-over opens. Choose Person/device or Coordinator, set name / role / expiry / reusable. (Adding another coordinator happens here too now, not in Settings.)
  2. The UI sends POST /admin/invites with {type, name?, role?, reusable?, expiry_mins?}. For type=coordinator, the coord mints the cluster payload ({internal_token, primary_url}, via the same internals the legacy handleCreateJoinCode used) into the row’s payload.
  3. Coord writes the invites row and replies with {code, link, qr_svg, expires_at, …}, where link = <coord>/join/<code>.
  4. The slide-over renders the invite: a copyable link, a QR, the short code, and an expiry chip. Admin hands it off (link, QR, or code) however they like.

Two more admin endpoints round it out — GET /admin/invites lists invites with a derived status (active / expired / revoked / redeemed); DELETE /admin/invites/{code} sets revoked_at. All three are admin-authenticated and primary-only for writes (replicas defer writes to the primary, same as peers).

Invitee surfaces — the coordinator serves onboarding itself

The coord already serves /ui/ and /agent-binary; it serves the whole join experience too, so there’s no website dependency:

  • GET /join/{code} — a public, unauthenticated, Soft-Slate HTML page. It looks up the invite (valid / expired / revoked), sniffs the OS from the User-Agent, and renders the right primary path:
    • Desktop — a one-liner: curl -fsSL <coord>/join/<code>/install.sh | sudo bash. It downloads the agent, installs it, and auto-enrolls — no UI.
    • Mobile — (coming soon) a placeholder for a WireGuard QR; the wg-quick + QR generation currently exists only behind the admin POST /admin/peers/{id}/wg-conf endpoint, not the public /join path.
    • Already-installed peers — a secondary mokatai-mesh://join/{code} deep link that hands the code to the client’s in-app Join a mesh screen.
  • GET /join/{code}/install.sh — the per-invite bootstrap script the desktop one-liner pipes into sudo bash. It downloads the agent from the coord’s GET /agent-binary and installs it with a self-contained minimal installer (binary + systemd unit; it does not re-run pkg/linux-agent-install.sh), writing MOKATAI_MESH_INVITE + MOKATAI_MESH_ENROLL_URL into the agent’s systemd env so the agent auto-enrolls on first boot via its --invite / --enroll-url flag defaults. (A Windows install.ps1 is not yet served — the join page marks Windows “coming soon”. macOS desktop is similarly deferred; there is no packaged macOS installer yet.)

The code is the credential: /join/{code}, /join/{code}/install.sh, and the redeem paths are otherwise unauthenticated, so they’re rate-limited per code and per source IP.

Peer side — create-on-redeem

The invite path flips when the peer row is created. The legacy flow pre-creates the peer (blank pubkey) at “add peer” time; the invite flow is create-on-redeem — the peer doesn’t exist until someone burns the invite.

  1. The agent has the invite (from the bootstrap installer’s --invite, the client’s Join screen, or a mobile QR). It generates a WireGuard keypair locally (private key never leaves the machine).
  2. Agent POSTs {invite_code, pubkey, arch, flavor} to the coord’s /enroll. (The same endpoint accepts the legacy {token, pubkey} — invite_code is an additive field on EnrollRequest.)
  3. Coord validates the invite, then creates the peer now: AllocateNextIP
    • CreatePeer with the invite’s name/role, then consumes the invite atomically via RedeemInvite last (rolling back the new peer if the redeem loses a race or the invite was revoked/expired), then the unchanged tail — SetPeerPubkey, push into wg0 with allowed_ips = <peer_ip>/32, syncACL, replicate — and replies with the standard EnrollResponse:
    {
  "assigned_ip": "10.77.0.5",
  "server_pubkey": "...",
  "server_endpoint": "73.140.176.8:51820",
  "server_endpoints": ["73.140.176.8:51820", "10.0.0.61:51820"],
  "network_cidr": "10.77.0.0/24",
  "coordinator_ip": "10.77.0.1"
}
    A reusable invite creates a fresh peer on every redeem (name suffixed to avoid collisions); a single-use invite is now spent.
  4. Agent writes this to state.json, then exits cleanly (status 0). systemd on Linux / SCM with failure actions on Windows restarts the service. On the next startup the agent sees state.json exists and boots in “enrolled mode”.
  5. Enrolled mode: ip link add wg0 ..., add the coord as a peer with allowed-ips = the network CIDR (catch-all route through the coord), then start polling /config on a 30s ticker.

Agent --invite auto-enroll

The agent takes a --invite CODE flag (or MOKATAI_MESH_INVITE env) plus an --enroll-url (the coord URL). On startup, if there’s no state file and an invite is present, the agent auto-enrolls — generate keypair, POST /enroll {invite_code, …}, save state, proceed — with no UI. The bootstrap installer drops these into the agent’s systemd env so the very first boot enrolls itself.

Legacy: one-shot enrollment token (CLI / back-compat)

The original flow is preserved for scripts and back-compat. mesh add-peer alice (or POST /admin/peers) pre-creates the peer with a blank pubkey and issues a one-shot, 24h enrollment token linked to that row; the admin hands off the URL + token. The agent (or the client’s old Join form) POSTs {token, pubkey} to /enroll; the coord atomically consumes the token, writes the pubkey, and returns the same EnrollResponse. Everything after that — state file, restart, enrolled mode — is identical. This is the CLI/back-compat path now, not the primary one.

6. Endpoint probing

The coord advertises every address it’s reachable on — public first, then each RFC1918 LAN IP on its non-loopback interfaces. This is in the enrollment response (server_endpoints) and in every /config response (on the coord’s own entry in the coordinators[] list).

The agent’s runHandshakeProbeLoop ticks every 5 seconds:

every 5s:
  look up the current coord peer on wg0
  if handshake age < 15s:
    mark the current endpoint sticky (save as active_wg_endpoint)
  else if we've been stale for 15s+ and cooldown has elapsed:
    rotate to next endpoint in the ring
    push the new endpoint to wg0 via AddPeer (upsert)
    save state

Why this matters: if your laptop is on the same LAN as the coord, your router probably doesn’t support NAT hairpin — the public IP isn’t reachable from inside your own network. Without probing, you’d be stuck. With it: first endpoint fails for 15s, rotate to the LAN IP, handshake in ~1s, connected.

The sticky part (active_wg_endpoint in state.json) means a reboot doesn’t re-probe from scratch — we remember the one that worked.

7. Direct peer-to-peer path

Hub-and-spoke is the default because it always works, but it’s inefficient for peer-to-peer traffic (every packet goes through the coord and back). The coord tries to help peers find each other directly:

  1. Coord’s runEndpointTracker samples wg show wg0 every 10s and records each peer’s current endpoint (the ip:port their UDP packets are arriving from, as observed by the kernel).
  2. In /config responses, the coord includes every other enrolled peer’s pubkey + observed endpoint.
  3. Agent’s applyConfig adds those peers as direct WG peers with allowed_ips = peer_ip/32. The more-specific /32 wins against the coord’s /24 catch-all.
  4. If a direct peer’s handshake goes stale (>90s), the agent drops it from the peer set — traffic falls back through the coord via the /24 route.

This is the “try direct, fall back to hub” pattern — simpler than ICE or STUN and works because we don’t need symmetric-NAT traversal, just good enough NAT that UDP state survives.

8. ACL enforcement

Roles are strings attached to each peer: user, operator, admin by default (extensible). Rules are role→role pairs: “users can reach operators”. A 3×3 grid in the admin UI.

The enforcement happens on the coord via nftables. When any peer tries to route a packet through the coord (hub path), the coord’s nftables forward chain checks:

  • source IP’s role
  • destination IP’s role
  • is there a rule allowing src_role → dst_role?

No rule = packet dropped. Return traffic for an established flow is auto-allowed via conntrack.

syncACL() runs after every peer create/delete/role change or rule change. It rebuilds the entire ruleset from the DB and atomically swaps it in. Rebuilding is O(peers × rules) but the DB is small (100 nodes tops) so it’s microseconds.

Direct peer-to-peer traffic is gated, not bypassed (P3, v0.19.7). The coordinator hub nftables remains the single authoritative enforcement boundary; the direct path is steered back to it via AllowedIPs gating + hub-demotion. For each requesting agent the coord computes a per-peer verdict — direct-ok / hub-only / deny — from the DB rules and the per-network default_policy:

  • direct-ok: the pair is allowed for all traffic; the peer is sent with direct_ok: true and the agent may form a direct /32.
  • hub-only: the pair is allowed but L4-restricted (a proto/port rule); the peer is sent with direct_ok: false, so the agent forms NO direct /32 and all of that pair’s traffic hairpins through the hub, where the real L4 filter runs. (Accepted cost: an L4-restricted pair loses its direct fast-path.)
  • deny: the pair may not talk; the peer is OMITTED from that agent’s /config peer list entirely (no info leak, no direct path) and the hub drops it too.

The agent signal is an optimization; the hub nftables is authoritative, so even a misbehaving agent cannot reach a forbidden or restricted pair unfiltered. The gate is applied at the top of the agent’s applyConfig, before the direct-peer liveness/backoff machinery — deny is permanent, a liveness drop is temporary — and the hub catch-all peer is never gated.

All coordinators in a cluster must be v0.19.7+ for the gating to hold on every hub: a pre-0.19.7 coord omits direct_ok and an agent then keeps the legacy direct path (the hub nftables still backstops it, but the hub-only demotion optimization is lost until every coord is upgraded).

8b. Subnet routing (P3c, v0.19.8)

A peer can route traffic to a subnet behind it (Tailscale-style) — e.g. a gateway node advertising 192.168.50.0/24 so other peers reach that LAN through it. A subnet route is modeled as just another destination the coordinator computes an ACL verdict for, reusing the P3 ACL spine.

Advertise + approve. A router declares routed CIDRs (an --advertise-routes agent flag and/or an admin in the Routes UI page). Declaration creates rows in the routes table in pending state — never auto-distributed. An admin approves each route (pending → approved); only approved routes ever distribute or compile into nftables.

Anti-hijack validation (the single most important control). AllowedIPs is cryptographic routing + authorization authority, so approval validates the CIDR and rejects:

  • a CIDR overlapping the overlay network (networkCIDR) — else a router could steal the hub catch-all;
  • 0.0.0.0/0 (and ::/0) unless an explicit exit-node opt-in is set on the approval (the Routes-page checkbox);
  • a CIDR overlapping an already-approved route on another peer — WireGuard forbids the same AllowedIP on two peers of one interface, so it is a hard conflict rejected at approval time. (An approved 0.0.0.0/0/::/0 is exempt from the containment half of this check — a default route legitimately coexists with more-specific routes, longest-prefix wins — but genuine same-width/partial overlaps are still rejected.)

Exit-node consumption is DEFERRED. A router may advertise and forward a 0.0.0.0/0/::/0 route (it is approvable under the exit-node opt-in, and the router enables forwarding/SNAT for it), but clients do not route their default through it yet. The agent gates /0 out of the consumption paths (it never enters a consuming peer’s AllowedIPs nor an ip route ... dev wgN), because client default-routing needs its own dedicated feature: split-default routing, underlay (WG-handshake) protection, and a kill-switch. Installing a bare default dev wgN without those would blackhole the host’s real default route and the handshake packets. Until that feature lands, an approved /0 is a no-op for consumers (harmless on the router).

Hub-enforced, route-gated. For each (requesting peer, approved route) the coord runs the same acl.Verdict used for peer pairs, extended so a rule whose destination is a CIDR literal matches a route whose CIDR it contains (tag dev → 192.168.50.0/24 : tcp 5432 is valid; source stays tag-only). The verdict steers the data plane:

  • direct-ok (allowed, no L4 restriction): the agent adds the CIDR to that router peer’s direct /32 AllowedIPs → direct subnet path.
  • hub-only (L4-restricted): the CIDR is NOT added to the agent’s direct peer; traffic hairpins through the hub, where the coord nftables enforces src-tag → CIDR [L4]. (Accepted cost: L4-restricted subnet traffic hubs.)
  • deny: the CIDR is not distributed to that agent and the hub drops it.

Router demotion closes the direct-peering bypass (the hub stays authoritative). A router peer is otherwise reachable directly (its /32 AllowedIPs), and once an agent has a direct tunnel to the router it could send to a hub-only CIDR over that direct path — bypassing the coord nftables where src-tag → CIDR [L4] is enforced. To close this, a router is demoted to hub-only for any agent that is not direct-ok on at least one of the router’s routes: that agent gets no direct /32 to the router, so all of its router-bound traffic (including hub-only CIDRs) hairpins through the hub where the CIDR/L4 rule is applied. An agent that is direct-ok on some route keeps the direct /32 for its direct-ok CIDRs; its hub-only CIDRs still hairpin (they were never in its direct AllowedIPs). This restores the “hub nftables is the single authoritative enforcement point” guarantee for subnet traffic. (A stronger future hardening is the router compiling its own nft FORWARD chain so per-source enforcement happens at the router itself, not only at the hub.)

AllowedIPs assembly: an agent’s router peer = {router /32} ∪ {direct-ok CIDRs} when the agent is direct-ok on ≥1 route, else omitted entirely (demoted to hub-only — see above); the coord’s kernel-WG peer for the router = {router /32} ∪ {ALL approved CIDRs of that router} so the hub can forward any allowed/hub-routed subnet traffic. Liveness is unaffected — the agent still treats only the /32 as the probe target and ignores wider CIDRs.

Client + coord kernel routes (the P3c-e finding). WireGuard AllowedIPs is crypto-routing inside the tunnel — it is NOT a kernel route. So a CIDR landing in AllowedIPs is necessary but not sufficient: both ends also need a kernel route for the subnet pointing at the WG interface. The agent installs a client kernel-route for each direct-ok CIDR (so the host actually sends that subnet’s packets into the tunnel), and the coordinator installs a coord kernel-route for every approved CIDR of each router (so the hub can forward hub-only/exit traffic out the WG interface toward the owning router). Both are torn down on route removal / agent stop.

Router datapath + NAT model. A router enables IP forwarding and, by default, SNAT/masquerades overlay→LAN egress (out-of-box: LAN hosts need no overlay return routes — the Tailscale default). The masquerade rule is ip daddr <CIDR> ... masquerade so only traffic destined for the routed subnet is NATed. ACLs still hold because the coord nftables checks src-tag → CIDR on the hub path before the router NATs — the overlay source is intact on the overlay side; NAT only rewrites at LAN egress. --snat=false opts out for source-preserving pure routing (the operator adds LAN-side return routes). Forwarding/NAT is torn down cleanly on agent stop / route removal.

Linux vs Windows. A per-OS router abstraction mirrors the setaddr_linux.go / setaddr_windows.go split. Linux uses sysctl net.ipv4.ip_forward=1 + an nft masquerade rule and gets full netns integration coverage. Windows uses Set-NetIPInterface -Forwarding Enabled

  • New-NetNat via PowerShell; the netns harness is Linux-only, so the Windows router path is covered by command-generation unit tests and is documented as manual-verify (see docs/superpowers/plans/2026-06-16-p3c-d-windows-router-manual-verify.md) — the Linux path is the proven one.

Replication. New route_upsert / route_delete kinds (each fires syncACL); routes are carried in localDump → syncFromPrimary, in mergeDump, and re-pushed in resyncCluster phase-2, so a follower enforces a replicated route and the policy survives a leader kill. All coordinators in a cluster must be v0.19.8+ for subnet routing to hold on every hub.

8c. MagicDNS v1 — hosts-file sync (P4, v0.19.9)

Reach an overlay peer by name (alice.mesh) instead of only by overlay IP. Entirely agent-side, derived from data already on the wire — no coordinator change, no DNS server, no :53 listener, no new dependency.

The managed block. The agent owns a delimited block in the OS hosts file:

# BEGIN mokatai-mesh (managed - do not edit)
10.77.0.2  alice.mesh
10.77.0.3  bob.mesh
10.77.0.1  coord-Ifw_yT8iFc8.mesh
# END mokatai-mesh
  • Paths. Linux /etc/hosts; Windows %SystemRoot%\System32\drivers\etc\hosts. Per-OS via a build-tagged split (internal/wgagent/hosts_linux.go / hosts_windows.go), mirroring setaddr_*. MOKATAI_MESH_HOSTS_PATH overrides the path (operator/test affordance).
  • Reconcile (pure + I/O split). reconcileHostsBlock(current, entries) (internal/wgagent/hosts.go) replaces ONLY the bytes between the BEGIN/END sentinels, preserving every byte outside it; inserts the block at EOF when absent, removes it when empty, and leaves a malformed/partial block (only one sentinel) untouched while appending a fresh one — never corrupting the system-shared file. The I/O layer reads, reconciles, and writes atomically (temp file in the same dir + rename, mode preserved). Idempotent: the same desired set re-renders byte-identically and skips the write.
  • What’s resolvable. Overlay peers + coords only (name → /32 overlay IP), assembled each /config poll from cfg.Peers (Name + AssignedIP) and cfg.Coordinators (NodeID + overlay IP). It does NOT emit subnet-route CIDRs or LAN hosts behind routers (no conditional forwarding — that’s the deferred responder’s domain).
  • Naming. Flat <label>.mesh (magicDNSSuffix). Names are normalized to a valid hostname label (cmd/agent/magicdns.go normalizeDNSLabel): lowercase; non-[a-z0-9-] → -; collapse repeats; strip leading/trailing -; truncate to 63. On a post-normalization collision (two distinct names → the same label, e.g. My Laptop and my-laptop), each entry gets its assigned-IP last octet appended (my-laptop-7.mesh) — deterministic and logged.
  • ACL scoping for free. The table derives from the per-requesting-agent PeerEndpointView set, which already omits ACL-denied (and disabled/expired) peers. So a peer you cannot reach is absent from your hosts file — name resolution inherits ACL scoping with no extra code, and a mid-session deny removes the record on the next poll.
  • Flag + teardown. --magicdns (default true). On graceful stop/leave (and on --magicdns=false) the agent removes the managed block (a defer wgagent.TeardownHosts(), mirroring router.Disable). A hard kill leaves a stale block; it’s self-healing — the entries still point at valid overlay IPs (harmless), and the next start reconciles it. Surfaced per-peer as mesh_name in the loopback /status so the GUI can show the resolvable name.
  • Best-effort. A hosts read/write failure is logged and swallowed — MagicDNS must never break the tunnel/data plane.

Not split-DNS. hosts-file entries are global to the host, not scoped to a mesh-only resolver; .mesh is not a real TLD so a public-DNS collision is unlikely. The agent-local DNS responder + split-DNS (the extensible “real” MagicDNS, conditional forwarding for LAN-behind-route names) is a review-gated follow-on deferred from this spec.

Installer note. No installer change ships with v1: MagicDNS is on by default in the agent and needs no extra packaging, service unit, NSS config, or :53 port. (The deferred responder will need installer/NSS/systemd-resolved wiring; that lands with it.)

8d. RBAC + audit (P4, v0.19.10)

Role-based access control for the admin API plus a per-coord-local audit trail. Before this, every /admin call carried the one flat admin token (full power or nothing). Now the API recognises role-bearing tokens (a least-privilege gradient) and records every mutating call — including denials and forwarded writes — to a queryable log, attributed to the credential that made it.

Role-bearing API tokens

  • Roles (a strict ladder). viewer < operator < admin (internal/db.Role, AtLeast is the >= test). viewer = read-only (every GET /admin/*); operator = the day-to-day mutations (peers/rules/routes/invites/tags); admin = the privileged surface on top (token mint/revoke, network policy, settings, env-token rotate, restart, self-update). The set is closed — minting rejects anything outside viewer|operator|admin (db.ParseRole), so a typo can never create an unresolvable or accidentally over-privileged token.
  • Mint / list / revoke (cmd/coordinator/tokens.go):
    • POST /admin/tokens {label, role} (admin-only) → generates a high-entropy 32-byte base64url secret, stores only sha256(secret) at rest (hashToken, never the plaintext), and returns the plaintext exactly once (MintTokenResponse.Token). Lose it ⇒ revoke + re-mint.
    • GET /admin/tokens (viewer+) → a purely descriptive list (label/role/created_by/created_at/revoked_at). It carries neither the secret nor the at-rest hash.
    • DELETE /admin/tokens/{id} (admin-only) → revoke.
  • Revoke is set-once, never a delete. RevokeAdminToken stamps revoked_at via COALESCE(revoked_at, now) — it can only transition null→set, never re-open. The row is never hard-deleted: a delete would let an older active copy resurrect the dead admin on a failover resync. Because the merge is COALESCE-set-once and the row persists, a revocation survives a leader kill — the new leader’s mergeDump/resync can never clear an already-revoked token. (Proven live: test/integration/rbac_multicoord_test.go TestRBACRevokeSurvivesLeaderKill.)
  • Replicated to every coord (3 sync paths). Auth resolves against the local admin_tokens table, so a token must reach every coord to work cluster-wide. The admin_token_upsert op rides all three replication paths: the live fan-out (replicator.Send on mint and on revoke), the initial /internal/dump a joiner pulls (DumpResponse.AdminTokens), and the post-election re-sync. A token minted on the leader becomes usable on a follower within the replication window (TestRBACTokenReplicatesToFollower); omitempty keeps a pre-v0.19.10 follower structurally compatible.

requireRole + the endpoint → role map

requireRole(min, h) (cmd/coordinator/main.go) is the edge gate. It resolves the presented bearer via resolveBearerRole to (role, actor, actorKind), then: missing/unknown/revoked bearer ⇒ 401; resolved but role < min ⇒ 403 (“forbidden: insufficient role”). On allow it attaches the actor to the request context and calls the handler. RBAC is enforced at the edge — the coord the request first hits — before any leaderOnly forward, so a denial never crosses the wire as an internal-token write.

Leader-forwardable writes wrap requireRoleOrInternal(min, leaderOnly(h)): the admin-bearer edge path is RBAC-checked as above; a cluster-internal forwarded request (X-Mokatai-Forwarded:1 + the internal token) is trusted as already-authorised at the edge (its actor is recovered from the signed header, below). The endpoint → minimum-role map (route table in main.go):

Min roleEndpoints
viewer (read)every GET /admin/*: peers, rules, routes, invites, cluster, logs, audit, tokens (list), settings, peer tags, update status
operator (write)POST/DELETE/PATCH /admin/peers, …/peers/{id}/tags, …/peers/{id}/wg-conf, POST/DELETE /admin/rules, …/routes (+ approve), POST/DELETE /admin/invites
adminPOST/DELETE /admin/tokens, PUT /admin/network (default policy), POST /admin/settings, POST /admin/rotate-admin-token, POST /admin/restart, all POST /admin/update/*

The env admin token (bootstrap / break-glass)

The flat --admin-token / MOKATAI_MESH_ADMIN_TOKEN is unchanged in form but is now the built-in admin / break-glass identity. resolveBearerRole const-time-compares it first and resolves it to (RoleAdmin, "env-admin", "env-admin"). It is what bootstraps RBAC: a fresh cluster has no scoped tokens, so the env token is the only credential that can mint the first ones — and it remains the recovery path if every minted admin token is lost or revoked. (TestRBACEnvTokenFullAdmin proves it does an admin-only mint and audits as env-admin.)

Actor propagation across the leader forward (X-Mokatai-Actor)

A write that lands on a follower is forwarded to the leader by forwardToLeader, so the audit row must still name the originating caller, not the cluster. The follower signs the resolved actor into X-Mokatai-Actor with X-Mokatai-Actor-Sig = base64(HMAC-SHA256(actor, internal_token)). On the leader, actorFromForward recovers the actor only if the request also passed the internal-token + X-Mokatai-Forwarded check and the HMAC verifies (verifyActorSig, constant-time) — so an external caller cannot forge an actor claim, and a tampered/missing header fails closed to a bare ("forwarded", "forwarded") (the write already cleared edge RBAC, so it is recorded as cluster-internal-but-unattributed rather than dropped). The result: the leader’s audit row reads actor=token:<label>, actor_kind=forwarded — the real originating token, never the internal token or cluster. (Proven: TestRBACForwardedActorAudited.)

Caveat — plaintext bearers over http://. Both the admin/role bearers and the internal token + X-Mokatai-Actor headers are sent as plaintext over the coord’s http:// API listener (§16). The HMAC stops an external actor forging an actor, but an on-path attacker between two coords can still capture a bearer (or the internal token) off the wire. Run coordinators over the WireGuard overlay or behind a TLS-terminated front — never expose the coord API in the clear on an untrusted network.

Audit log (per-coord-local, queryable)

  • What’s recorded. The audit(h) middleware wraps every mutating admin chain and appends one audit_log row after it runs: result=ok on a 2xx, result=denied on a 403. Reads (GET/HEAD) are not audited. A row is {ts, actor, actor_kind, action, target, result, coord_node_id}; action is a stable verb (peer.create, rule.create, token.mint, token.revoke, …) from the route map, target is the path resource (peer:7, token:3, invite:<code>) — never a credential. A denied attempt is still attributed: requireRole stashes the resolved actor into a side-channel holder before writing the 403 (the handler never runs), so the audit row names the originating token even on a rejection (TestRBACViewerForbiddenAudited).
  • actor_kind is one of token (a minted role-bearing token, actor token:<label>), env-admin (the env token), or forwarded (recovered across a leader hop).
  • Per-coord-LOCAL, never replicated. Each coord records only the actions it served, tagged with its own coord_node_id. The log is not part of replicate-all, so GET /admin/audit returns that coord’s slice — a forwarded write is audited on the leader (where the handler ran), not the follower that forwarded it. The admin UI aggregates the per-coord slices into one timeline.
  • Query + retention. GET /admin/audit (viewer+) is paged and filterable by actor, action, result, since/until, limit/offset (handleGetAudit). Each coord prunes its own log hourly, deleting rows older than --audit-retention-days (default 90; 0 = keep forever).

Mixed-version note. Scoped role-bearing tokens require all coords on v0.19.10+ — admin_token_upsert is omitempty, so a pre-v0.19.10 coord silently ignores replicated tokens and would 401 them, leaving a hole in the cluster’s auth. Until every coord is upgraded, use the env admin token, which every version honours, as the bridge.

8e. Peer lifecycle (P4, v0.19.11)

Peers gain a soft, reversible lifecycle distinct from hard deletion: disabled, expires_at, and a lifecycle_updated_at LWW clock (DB columns on peers). Three operator primitives sit on PATCH /admin/peers/{id} {disabled?,expires_at?} (operator-tier, audited peer.disable / peer.enable) and POST /rotate-key, plus a leader-only reaper.

  • Disable (soft, reversible) vs delete (hard). Disable flips disabled=1 but retains the row, the overlay IP, the role, and the tags — it is the reversible middle ground. Delete (DELETE /admin/peers/{id}) removes the row and frees the IP for reallocation. A disabled peer can be re-enabled to the same IP via the peer_upsert -> AddPeer restore path; a deleted peer cannot (a new enrollment gets a fresh IP).

  • Neither delete nor disable is fail-closed-certain across an abrupt failover. It is tempting to treat hard-DELETE as the “certain” revocation primitive, but it is not: the re-sync merge never tombstones a delete (mergeDump is additive — see the §11 delete-resurrection caveat), so a peer_delete that hasn’t reached a follower can be resurrected onto a newly-elected leader, transiently re-granting access. A disable lost on abrupt leader death can likewise briefly resurrect as enabled (the reversible-LWW caveat below). So across the failover window delete is no more certain than disable — both can fail open until reconvergence. The most robust revocation available today is a confirmed-propagated DISABLE: it is a re-converging, re-applicable LWW row (lifecycle-updated_at-keyed) that the next admin action or re-sync can re-assert, whereas a delete leaves no row to re-assert. (Future hardening: a real deleted_at tombstone merged set-once — like the admin-token revoked_at = COALESCE(...) revoke — would make delete monotonic and genuinely fail-closed; tracked, not shipped this pass.)

  • The omit mechanism (one off-switch for three planes). handleConfig omits any disabled OR expired peer from the PeerEndpointView set — the same omission already noted in §8c (the MagicDNS hosts sync builds off /config). Because every other agent’s WireGuard peer set, its ACL view, AND its MagicDNS table are all derived from /config, a single omit code path is the off-switch for the data plane, name resolution, and the ACL at once. On the boundary side, disable/expiry also drives applyPeerLifecycleWG (iface.RemovePeer) + syncACL on the coord hub and replicates, so the authoritative nftables boundary blocks the peer’s overlay IP too.

  • Expiry + the leader-only reaper (A+C enforcement). expires_at (unix seconds; NULL = never) gives a peer a TTL. Enforcement is A+C: lazy — handleConfig omits an already-expired peer the moment it is read (the same omit above, now >= expires_at); and deterministic — a leader-only reaper periodically ListExpiredPeers and flips each expired peer to disabled (so the teardown — RemovePeer + syncACL + replicate peer_upsert) happens once, authoritatively, independent of any poll traffic.

  • Key rotation overlap (no tunnel gap). POST /rotate-key (peerAuth on the agent’s current pubkey, audited peer.rotate_key) does AddPeer(new) BEFORE RemovePeer(old) on the hub, and peerAuth accepts both the new and the prior pubkey during a bounded grace window — so the tunnel never drops across the swap. This is the only place the “already-enrolled” rejection is lifted, and only for the authenticated self-rotate. The agent swaps its wg private key in place on the live userspace device (wgagent.SetPrivateKey, not a TUN recreate) and re-handshakes, keeping the same peer row + IP.

  • The reversible-LWW failover caveat. Lifecycle fields replicate via the 3-path carry and merge last-writer-wins by lifecycle_updated_at (InsertPeerWithID’s correlated-subquery guard: a stale write cannot resurrect a newer state). But a disable made on the old leader that has not yet replicated before an abrupt failover can briefly resurrect as enabled on the new leader, until the next admin action re-applies it — the same narrow window as the §11 delete-resurrection caveat (a lost last write on abrupt leader death). Hard DELETE does not escape this window — it can resurrect too (no tombstone; see the bullet above) — so for a removal that must hold across an abrupt failover, a confirmed-propagated disable (a re-assertable LWW row) is the more robust choice; disable is also the convenient, reversible default.

  • The agent removed-state (surface-only, never self-evict). When an agent’s pubkey is no longer a valid peer (deleted / rotated-out), every coord answers its /config with a hard 401/403. The agent classifies a sustained all-known-coords-401 over a window (removedConfirmWindow, ~90s) as removed and surfaces it in its loopback /status — but it never self-leaves, deletes state.json, or tears the interface; re-enrollment is an operator decision. The bar is deliberately strict: a transient outage (a coord down, an election) or a single non-401 result resets the streak, so a healthy peer is never self-evicted. Critically, a disabled peer still authenticates (its pubkey is valid) and gets a recoverable 200 peer-less /config, which is not removed and clears any prior removed flag — so a re-enable auto-recovers. Only a hard 401/403 (unknown pubkey) counts.

  • KNOWN LIMITATION — peer auth proves a pubkey, not its holder (decision C). Peer endpoints (/config, /rotate-key, file-transfer, …) authenticate by the WG pubkey the caller presents, matched against the peers table. A WG public key is not a secret — it ships in every peer’s /config (and in the topology view), and there is no private-key challenge or transport-level identity proof on the control API. So an already-enrolled insider can present another enrolled peer’s pubkey and act as that peer (read its config, send/receive its transfers, rotate against it within the gate). The shipped mitigation is the lifecycle gate: peerAuth still resolves to the row, so a disabled/expired (or deleted) target peer is rejected — a revoked peer can’t be impersonated into access. The residual risk is therefore an authenticated-insider horizontal issue (any current member can act as any other current member), consistent with the existing §16 plaintext control-plane transport caveat: the agent API is reached over the underlay and is not confidential or identity-bound by itself. The planned fix (0.20.x fast-follow) is to route the agent control API over the WG tunnel, so the transport cryptographically proves identity AND encrypts — which must not disturb the P1 local-vs-public endpoint self-healing (the agent’s ability to fall back between a coord’s local and public endpoints).

Installer-skip (intentional). The peer-lifecycle feature adds no new install paths or systemd units: disable/expiry/rotation are coord-DB + agent-runtime only, written to the existing coord state dir and the agent’s existing state.json. The build-dist.sh install-path lint (tools/lint-install-paths.sh) therefore needs no change, and Layer-3 test-system (install -> podman) is unchanged for this phase — the installer omission is intentional, not an oversight.

8f. Topology + live observability (P5, v0.19.12)

The admin UI gains a packet-tracer-style live topology view — coordinators, agents, and subnets as nodes; cluster, hub, direct, and subnet-route links as edges, each styled by path type + health — assembled by a single GET /admin/topology and drawn as hand-rolled SVG (the zero-dependency SPA).

  • The graph model (api.Topology{Nodes,Edges,GeneratedAt,Partial}). TopoNode is one of three kinds: coord (Leader/Term/OverlayIP/ Endpoint/Health), agent (Name/OverlayIP/Role/OS/Disabled/ Expired/HomeCoordID/Health), subnet (CIDR/RouterPeerID). Node ID is the stable cross-coord key, prefixed by kind: coord:<node_id>, agent:<peer_id>, subnet:<route_id> (the peer/route ids are the replicated DB row ids, so every coord mints the same id for the same entity). RouterPeerID is the int64 peer-row id of the router (the subnet-route edge carries the agent:<peer_id> node-id form). TopoEdge is one of four kinds — coord-coord (the replication mesh, the leader’s edges emphasized), agent-hub (agent → its home coord coord:<node_id>, carrying LastHandshake

    • bytes), agent-direct (agent ↔ agent, undirected, only when a live report shows a /32), subnet-route (subnet → its router peer) — with PathType (direct|hub) and Health. Health is fresh|stale|down, derived from handshake/heartbeat recency. The agent node’s HomeCoordID is the bare coord node id (no prefix), stamped only by the coord that holds the agent’s live report.

  • The /report live-path telemetry (the part the coord can’t otherwise see). The coord reads kernel-WG handshake/bytes for the hub links of its homed agents, but it cannot see an agent’s direct peer-to-peer /32s (the agent runs userspace-WG; §7). So each agent’s existing POST /report is extended (api.PeerReport.HomeCoord + DirectPeers []DirectPeerReport{Pubkey, LastHandshakeTS}, both omitempty) with its home-coord id + the direct peers it currently holds, read from its own wg stats. The home coord keeps the latest report per agent in-memory, keyed by peer id, mutex-guarded, TTL’d at ~3× the report interval, lazily pruned on read plus a reaper — never persisted, never replicated (a vanished agent’s entry simply expires; no failover/merge hazard). The reported pubkeys are resolved to peer ids via the DB so the agent-direct edge keys are stable across coords. Back-compat: an old agent omits the fields and is shown hub-only.

  • Assembly + the scatter-gather (Partial degradation). GET /admin/topology (requireRole(viewer), a read, not audited) builds the base graph from the local replicated DB + local wg stats + the local live-path store (complete on any coord), then scatter-gathers every sibling’s GET /internal/topology-fragment (internal-token-authed, concurrent, a short per-coord timeout, best-effort) and merges by node id. A slow/down/ pre-0.19.12 sibling (a 404 on the new route) is simply omitted and the result is flagged Partial=true — the call never hangs or errors on a dead coord (the kill-a-sibling integration test pins the bound: Partial flips well within seconds, never the test timeout). The merge is read-time only; no telemetry is replicated. A fully-alive cluster returns Partial=false.

  • The visualization (hand-rolled SVG, deterministic). A new Topology nav item + renderTopology() lay coords out in a central band (the leader emphasized + badged), agents clustered beneath their home coord, subnets behind their routers; positions are computed deterministically from the graph so a poll-refresh re-renders in place (no jitter). Edges are colored by PathType + Health; a Partial graph shows a subtle “a coordinator is unreachable” banner; a disabled/expired agent is greyed/badged. Interactivity is read-only v1 — hover an edge for link detail, click a node to navigate to its existing peers/cluster page. No control actions ride the map.

  • Telemetry trust (advisory). An agent self-reports its direct paths, so a lying agent could mis-draw a direct edge it doesn’t hold — but this is a visualization (no enforcement rides it), and the coord cross-checks the hub-link handshake it observes directly; the direct edges are advisory. (Independently: an agent that re-homes from a follower to the leader can be left with a stale observed endpoint, so its agent-direct /32 may not establish a live datapath — a known agent/coord datapath limitation orthogonal to the topology view, which simply reflects whatever live /32s the report carries.)

  • Deferred (not in v0.19.12). Throughput time-series / a metrics store / Prometheus, SSE/websocket push (the view poll-refreshes on the SPA’s existing interval), a control surface (disable/edit from the map), and historical playback are explicitly out of scope — this ships the live snapshot only.

Installer-skip (intentional). Topology adds no new install paths or systemd units: the live-path store is in-memory coord state, the /report extension rides the existing agent report, and the viz is static SPA assets already served from cmd/coordinator/web. The build-dist.sh install-path lint (tools/lint-install-paths.sh) needs no change, and Layer-3 test-system (install → podman) is unchanged for this phase — the omission is intentional, consistent with the v0.19.11 peer-lifecycle precedent (§8e installer-skip).

8g. File transfer — coordinator-relayed store-and-forward (v0.19.13)

mesh send <file> <peer> moves a file node→node across the mesh by relaying it through a coordinator: the sender uploads the blob to its home coord, which holds it until the recipient explicitly accepts and pulls it (checksum- verified) into a receive dir. ≤1GB per file; CLI-primary; ACL-governed.

  • The record replicates; the blob does not. A replicated transfers table {id, from_peer_id, to_peer_id, filename, size, sha256, holder_coord_id, state, created_at, updated_at, expires_at} rides a transfer_upsert replication kind through handleReplicate + all three sync paths (localDump/syncFromPrimary/ mergeDump/resyncCluster), so every coord knows a transfer is pending and which coord holds the blob. The blob lives on disk only on the holder (<dataDir>/transfers/<id>, 0600) — it is never replicated, never buffered in memory (io.Copy + a sha256 hash writer + a capped reader). State is monotonic toward a terminal (delivered/rejected/expired never revert), so a stale re-dump after failover can never un-deliver a file (SetTransferState merges by state-rank; holder_coord_id is set once at create).

  • The proxy (a coord that doesn’t hold the blob). Download is recipient-only (to_peer_id == the authed peer). If the recipient’s coord isn’t the holder, it proxies the bytes from the holder over an internal, internal-token-authed GET /internal/transfers/{id}/blob — streamed, never buffered. A holder that is down before accept yields a fast connection-refused on the proxy dial, so the download returns 503 (degraded), never hangs — the sender can re-send once the holder returns. Blob replication for HA is a deferred enhancement (the durability caveat).

  • The flow. POST /transfers (peer-authed) gates on acl.Verdict(sender, recipient) (a denied pair → 403 — file-send is not an ACL hole), enforces the ≤1GB cap via a limited reader (a partial past the cap is removed; the cap is --transfer-max-bytes, default 1GiB), a free-disk guard (refuse with 507 if free < size + margin), computes the sha256 while streaming to the holder’s disk, then creates the record (pending, holder_coord_id=self) via a leader-forwarded record-create (the small metadata forwards; the blob stays on the receiving coord). The recipient polls GET /transfers (only rows where it is from/to), then mesh accept <id> → the agent downloads, verifies the sha256 into a temp file, renames into the receive dir, then marks the record delivered; the holder deletes the blob. mesh reject <id> → rejected + blob deleted. A leader-only retention reaper expires pending past expires_at (default 7 days, --transfer-retention) and a per-coord blob GC drops the bytes for any terminal transfer it holds; both tick on --transfer-reap-interval (default 60s).

  • Interface + scoping. The peer-side CLI (mesh send/transfers/accept/ reject) drives the agent loopback (127.0.0.1:51821 — POST /send with ?to=&filename= + the raw body, GET /transfers, POST /accept/{id}, POST /reject/{id}); the agent owns the peer identity and does the peer-authed coord I/O. The receive dir is --receive-dir/$MOKATAI_MESH_RECEIVE_DIR/a default; /status gains a pending-transfers count. You can only send as yourself; only the recipient can download/accept/reject.

Installer-skip (intentional). File transfer adds no new install paths or systemd units: the blob store is a subdirectory of the existing coord data dir, the transfer endpoints ride the existing coord HTTP server + replication channel, and the CLI/loopback reuse the existing agent control server. The build-dist.sh install-path lint (tools/lint-install-paths.sh) needs no change, and Layer-3 test-system is unchanged — consistent with the §8e/§8f installer-skip precedent.

9. State persistence

Nothing is in-memory-only. Reboots always bring back exactly the state that was last persisted:

  • Coord: /var/lib/mokatai-mesh/mokatai-mesh.db (SQLite: peers, rules, routes, tokens, coordinators, join_codes, invites). Config in /etc/mokatai-mesh/coordinator.env (tokens, endpoints, WG port). Server keypair in /var/lib/mokatai-mesh/server-keys.json.
  • Agent (Linux): /etc/mokatai-mesh-agent/state.json (pubkey, assigned IP, current coord, known coords list, sticky endpoint).
  • Agent (Windows): C:\ProgramData\MokataiMesh\state.json (same shape).

The state files are plain JSON, intentionally — humans can read them and we can hand-edit in an emergency without needing any Mokatai Mesh tooling. They’re also small (~1KB for a typical agent).

10. Self-updates

Manifest schema and per-arch components

The coord fetches a manifest JSON every 6 hours from a configurable URL (MOKATAI_MESH_UPDATE_MANIFEST_URL). The manifest is keyed by (role, os, arch, flavor):

coord_linux_amd64               coord_linux_amd64_headless
coord_linux_arm64               coord_linux_armv7
agent_linux_amd64               agent_linux_amd64_headless
agent_linux_arm64               agent_linux_armv7
agent_windows_amd64

Plus three legacy aliases (coord, agent_linux, agent_windows) pointing at the amd64-full tarballs. The aliases exist so pre-v0.17 clients — which didn’t know about arch/flavor — still resolve a valid component when they poll a v0.17+ manifest.

internal/update/update.go::componentKey(role, os, arch, flavor) builds the precise key (e.g. coord_linux_arm64) and falls back to the legacy alias if the precise key isn’t present. A v0.17 binary tries coord_linux_arm64; misses on a v0.16-era manifest; falls back to coord and updates. A v0.16 binary doesn’t know to compose the precise key; hits coord directly. Both work.

Each binary’s main.flavor is stamped at build time via -X main.flavor=amd64-full|amd64-headless|arm64|armv7. The coord mirrors this into update.processFlavor at boot so the resolver knows which key to compose for its own self-update.

Coord-side: how an update applies

When admin clicks “Apply” in Settings (or the primary fires it via “Push to mesh”):

  1. Coord downloads the tarball to /opt/mokatai-mesh/staging/.
  2. Verifies SHA-256 matches the manifest.
  3. Extracts to /opt/mokatai-mesh/<version>/.
  4. Runs that version’s install.sh --upgrade — because the installer ships alongside the binaries, it always knows the current version’s prerequisites. If a release needs a new apt package, the new install.sh installs it automatically.
  5. install.sh --upgrade atomically swaps binaries in /usr/local/bin/ keeping the old ones as .prev. The caller (coord) then os.Exit(0) so systemd restarts the service onto the new binary.
  6. If the new coord fails to come back up (health check), there’s a rollback button that moves .prev back into place.

The non-upgrade path (curl one-liner installer, or sudo ./install.sh without --upgrade on an extracted tarball) also swaps the binary, but since the caller isn’t the coord exiting, install.sh detects an already-active systemd unit and runs systemctl restart itself. Without this, the new binary sits on disk while the running process keeps executing the old one — a real bug that hit v0.17.1–v0.17.3.

The key design decision: every tarball contains its own installer, so we don’t have to maintain forward-compat in an older installer. The running coord downloads the new tarball and runs the new script, which knows everything the new version needs.

Agents notice the coord’s new version on their next /config poll and self-update by fetching the binary from the coord, atomic-renaming, and re-exec-ing in place — TUN device + enrollment state survive the swap, no re-enrollment required.

Cluster-wide push (“Push to mesh”)

“Push to mesh” runs on the leader (it’s gated by leaderOnly, so a follower forwards the request). When the admin clicks it:

  1. StartPushToPeers seeds pushAgentComponents with every agent_* key from the manifest. Agents pulling /config see an advert and self-update on next poll.
  2. In a goroutine, the leader enumerates db.ListCoordinators() and for each entry with role=replica and api_endpoint set, POSTs /coord/update/check then /coord/update/apply against that coord. The RPC is authed by the cluster’s internal_token (same shared secret used for replication and join-code redemption). (Note: the update fan-out still keys on the legacy role=replica tag, so an operator who wants a no-role auto-elect coord to receive pushed updates can bias/tag it, or update it directly via the curl installer — replicate-all state sync, by contrast, reaches every coord.)
  3. The receiving coord’s handler triggers its own local StartApply("coord"), which resolves the right tarball for its (arch, flavor) and runs through steps 1-6 above.
  4. The leader logs each coord’s outcome; one failing coord doesn’t abort the overall push.

The cluster-wide push is async with no per-coord progress streamed back to the UI — each coord’s converge state lives in journalctl on each box. Tracked on the roadmap for a future SSE/polling upgrade.

Operator-side: publishing a release

The other half of self-updates is the operator pipeline that produces the manifest + tarballs that coords poll. The full release flow is three scripts:

./build-dist.sh --archive
./tools/publish-manifest.sh vX.Y.Z https://updates.mesh.mokatai.io
source ~/.mokatai-mesh-r2.env
./tools/publish-release.sh vX.Y.Z

What each step does:

  • build-dist.sh --archive — compiles all four binaries (coordinator, agent, mesh, mokatai-mesh-admin + mokatai-mesh-client) with the version from build-dist.sh:26 linked in, then bundles them into the release tarballs in dist/. dist/ is gitignored — regenerated every build.
  • publish-manifest.sh vX.Y.Z <base-url> — generates dist/manifest.json with latest, per-component SHA-256s, sizes, released-at timestamp, and download URLs rooted at <base-url>.
  • publish-release.sh vX.Y.Z — uploads to the R2 bucket whose credentials are in ~/.aws/credentials (profile mokatai-mesh-r2) and endpoint in ~/.mokatai-mesh-r2.env. Specifically: tarballs go to <bucket>/<version>/<file> (archival); the manifest goes to both the versioned path and the root (which is the always-latest pointer every coord on Earth polls forever). The script curls the public URL before declaring success — so a successful exit means the new release is genuinely live.

Setting the manifest URL on a coord (one time, then forever):

  • Via UI: Settings → Update manifest URL → paste the manifest URL (e.g. https://updates.mesh.mokatai.io/manifest.json) → Save.
  • Via env: set MOKATAI_MESH_UPDATE_MANIFEST_URL= in /etc/mokatai-mesh/coordinator.env and restart the service.

Rollback is a single button in Settings → Rollback to vX.Y.Z. The previous binary is retained as /usr/local/bin/mokatai-mesh-coordinator.prev for 24h after an update; the rollback just moves it back.

11. Multi-coord clusters (flat, auto-electing)

A single coord is a single point of failure. Mokatai Mesh runs a flat, auto-managed cluster — every coord is an equal member that can lead, serve reads, and act as a tunnel hub. There is no statically configured primary:

  • Auto-elected leader. Coords run a hand-rolled, Raft-style leader election (terms / RequestVote / majority / heartbeats) in internal/cluster — this is leadership election only, not a full Raft replicated log. The elected leader handles the (infrequent) writes. A coord with no --role auto-elects; --role primary and the new --prefer-leader only bias the election so a preferred node tends to win when healthy — they never override quorum. A single-coord install is trivially its own majority and leads immediately.
  • Quorum-safe writes. To lead, a coord needs the votes of a majority of the membership (the replicated coordinators table), so a partition produces at most one leader and AllocateNextIP never double-allocates. While an election is in flight, a write returns a retryable 503 (“no leader elected yet; retry shortly”).
  • Write routing (CP). Cluster-mutating handlers run only on the current leader (leaderOnly). A non-leader that receives a write transparently forwards it to the leader’s API endpoint over the internal channel (or returns the 503 above if no leader is known yet).
  • Replicate-all. The leader pushes every kind of state to all coords via POST /internal/replicate — peers, rules, coords and invites, enroll-tokens, and join-codes (idempotent, last-writer-wins) — closing the failover data-loss gap. A newly-elected leader re-syncs: it pulls each follower’s recent state and merges it (recovering a write the prior leader didn’t finish propagating), then re-dumps its own state to the followers. The merge only ever advances monotonic fields, so it’s order-independent and safe to re-run.
    • Delete resurrection caveat. Because the re-sync merge is monotonic LWW and does not tombstone deletes (a row missing on the pulled follower is not treated as evidence to delete), a peer_delete that hasn’t reached a follower before an abrupt leader failover can be resurrected onto the new leader during the Phase-1 pull/merge — the follower’s stale peers row re-inserts via InsertPeerWithID, and Phase-2 re-pushes it as a peer_upsert, re-adding the WireGuard peer and transiently re-granting access until the delete reconverges (an operator re-issuing the removal, or the original peer_delete finally propagating). The window is narrow (a follower must miss the delete and a failover must occur before reconvergence) and is consistent with the “a last write may be lost on abrupt leader death” model — but for a peer removal the lost write transiently fails open. Note this affects only hard row deletes: revocations are safe — an invite revoke (and the invite_delete kind, which is itself a revoke, not a hard delete), an invite redeem, and an enroll-token consume are applied set-once via COALESCE-from-self (invites.revoked_at/redeemed_at, enroll_tokens.used_at) on both the pull/merge and the replicate PUSH arms (handleReplicate + resyncCluster Phase-2), not just the pull side — so a stale snapshot pushed after a failover can never un-revoke an invite or un-consume an enroll token; they are monotonic and can never be resurrected by the merge. The same caveat extends to ACL rules and peer tags: the re-sync merge applies d.Rules/d.PeerTags additively (never deletes), so a rule or tag deleted on the leader but still held by a lagging follower can transiently resurrect after a failover. A resurrected deny rule fails closed (over-restrictive until reconvergence); a resurrected allow rule (or tag that matches one) transiently re-grants access until the delete reconverges — same fail-open window as a resurrected peer_delete. Full authoritative delete-reconciliation is out of scope (matches the existing monotonic-LWW semantics).
  • Reads + data plane from any coord (AP). Any reachable coord serves /config, the public /join/{code} surfaces, /agent-binary, and the WireGuard tunnel — from its replicated local state, regardless of quorum. The minority side of a partition keeps peers connected and configs flowing (eventually consistent); only writes pause.
  • Heartbeats + membership. The leader heartbeats all known coords; a follower that misses heartbeats past its randomized election timeout starts an election. Membership is the coordinators table, updated via the join flow + heartbeat-register. A dead coord is reaped so it doesn’t inflate the quorum denominator forever — but eviction is leader-only and replicated (a partitioned minority must never shrink its own membership view, or it could manufacture a false majority and split-brain).
  • Agents use all coords. Peers learn every coord from /config’s coordinators[] list (each with its advertised overlay/WG endpoints), rank them by least-resistance reachability, and fail over to another coord when their current one stops answering an active overlay probe. Every coord already holds the agent’s pubkey/IP (replicate-all), so any coord accepts the tunnel — agents read from whatever coord they land on.

Real failover: kill the leader → a new one auto-elects from the majority within the election window and writes resume; the old leader rejoins as a follower when it comes back. A coord that joins (or reboots) mid-term adopts the cluster’s current term on its initial sync, so two coords can never both win the same term. This replaces the previous static primary + manual promote — promotion is now automatic via the election; there is no command to promote a replica.

2-coord caveat: write-HA needs ≥3 coords. A 2-coord cluster keeps serving reads/tunnels if one node is down, but pauses writes until the second returns (a majority of two is two). Run three coords for write-HA.

This is deliberately scoped: a real Raft election for correct quorum leadership, paired with best-effort leader→follower push replication — not a linearizable replicated log. Writes are rare admin actions, so a brief failover window (a last write may be lost on abrupt leader death, mitigated by the re-sync-on-election pull/merge) is an accepted trade.

Coord-to-coord RPCs at a glance

EndpointDirectionAuthPurpose
POST /coord/join-codes/redeemjoiner → any coord (during setup)invite/join code in bodyNew coord turns a coordinator invite (or legacy join code) into the cluster internal_token; redeem reads the invites table first, then falls back to legacy join_codes. Gated by leaderOnly (a non-leader forwards it)
POST /internal/request-votecandidate → coordsinternal_tokenRaft-style leader election: {term, candidate_id} → grant/deny
POST /internal/replicateleader → followersinternal_tokenState sync — peer/rule/coord and invite/token/join-code upsert + delete (replicate-all)
POST /internal/heartbeatleader → followersinternal_tokenLeader liveness (carries term + leader ID); also the follower’s endpoint advertise / membership register
GET /internal/dumpany coord → any coordinternal_tokenFull state snapshot — pulled by a joiner on first connect and by a new leader from each follower during re-sync; carries the cluster term so a joiner adopts it
POST /coord/update/checkleader → followersinternal_token”Refresh your manifest cache before I tell you to apply”
POST /coord/update/applyleader → followersinternal_tokenTrigger local self-update

12. Why certain choices

Userspace WireGuard (wireguard-go) on agents, not kernel wg. Windows doesn’t have kernel WG without Wintun. Kernel WG also requires CAP_NET_ADMIN which is awkward for user-mode tooling. Userspace works everywhere (Linux, Windows, later macOS) with zero OS-level config. Coord uses kernel wg because it’s server-class and performance matters.

Two separate processes on each peer (agent + mokatai-mesh-client). The agent is a system service that needs to survive user logins/logouts. The mokatai-mesh-client is a per-user GUI app that opens a webview. Keeping them separate means the agent doesn’t need any GUI dependencies (webkit2gtk on Linux, WebView2 on Windows) — just a plain Go binary. They talk via the loopback HTTP API on 127.0.0.1:51821.

Embed the web UI in the Go binary. //go:embed all:web bundles HTML/CSS/JS into the executable, served over a local HTTP listener on a random port. Single binary, no “oops forgot to copy the assets” bug.

Apple-style CSS in vanilla HTML. No React/Vue/framework. The UI is small enough (5 pages) that a framework would be more code than the pages themselves. Reduces build complexity and binary size.

Folder-per-version on disk. ~/Documents/Projects/UTM-vX.Y.Z/ is a full copy, not a git branch. We don’t need branching for a small project with one maintainer; we need to be able to rm -rf a version and know nothing else got touched. build-dist.sh vendors all Go deps so builds are reproducible even offline. (Historical snapshot folders use the original UTM-vX.Y.Z naming — these are immutable history and are not renamed as part of the Mokatai Mesh rebrand.)

13. Failure modes + debugging

  • “agent not responding” in the app → agent service is stopped or crashed. Check sudo systemctl status mokatai-mesh-agent (Linux) or Get-Service MokataiMeshAgent (Windows).
  • “connecting — no handshake yet” → network layer reaching the coord but WG handshake not completing. Check sudo wg show wg0 on the coord for a peer with the agent’s pubkey; if it’s missing, enrollment didn’t land. If it’s there but latest handshake is never, UDP 51820 isn’t reaching the coord (router not forwarding, or peer on same LAN as coord without hairpin — v0.15 probe should rotate to LAN endpoint within 20s).
  • “enrolled peer but coord has no record” → token was consumed on a different coord, or peer was deleted after enrollment. /admin/peers is the source of truth; Leave mesh on the peer and rejoin with a fresh token.
  • Settings Save → “Load failed” → coord exited for self-restart but systemd didn’t bring it back (must be Restart=always not Restart=on-failure because the coord exits cleanly).
  • Tunnel shows up, peer can reach the coord, but can’t reach another peer → ACL default-deny. As of v0.15.1 the mesh is open until you add even one rule, at which point the matrix flips to default-deny. Open ACL Rules and verify the source-role → destination-role cell is enabled. Reply traffic for established flows is conntrack-allowed automatically; you only need the one direction.
  • Same-machine “tunnel up but ping fails” → multiple WG interfaces on the same host hit routing weirdness; ping picks the wrong source IP. Run the test in a netns or from a separate host to isolate.
  • Windows agent: wintun.dll failed to load → wintun.dll must sit next to agent.exe. The installer puts both in C:\Program Files\MokataiMesh\; if files have been moved out of that directory, agent.exe can’t find the driver. Reinstall to restore.
  • Update check shows “Check failed” → Settings → Updates surfaces the underlying error in the UI. Common causes: HTTP URL where HTTPS is required, manifest URL unreachable from the coord’s network, invalid JSON, or TLS error. The manifest URL must be reachable from the coord specifically — not just from your laptop.
  • Update installed but the coordinator didn’t come back → the rollback button in Settings is unreachable. Use the manual fallback: 
    sudo mv /usr/local/bin/mokatai-mesh-coordinator.prev /usr/local/bin/mokatai-mesh-coordinator
sudo systemctl restart mokatai-mesh-coordinator
    Then journalctl -u mokatai-mesh-coordinator -n 100 for the underlying error.
  • v0.15.x → v0.16.0 rebrand migration — the v0.16.0 release renames binaries, service units, config paths, and env vars (UTM → Mokatai Mesh). The migration runs in two stages (Stage 1: install new binaries alongside old; Stage 2: cut over service units + config paths). If a stage fails, roll back by reinstating the .prev binaries and re-enabling the old service unit name. Full migration procedure and rollback steps are in REBRAND.md and the design spec at docs/superpowers/specs/rebrand-mokatai-mesh.md.
  • mesh tui refuses with “this coordinator hasn’t finished first-run setup” even though setup is done — fixed in v0.17.3. If still hit on v0.17.0–v0.17.2, the TUI’s detection only checks /setup/status status code, not the body. Workaround: bring the host to v0.17.3+.
  • “Push to mesh” updates agents but not replica coords — fixed in v0.17.2. The primary needs v0.17.2+ for the goroutine that fires coord-to-coord update RPCs; the replica needs v0.17.2+ for the /coord/update/{check,apply} routes. Bring a stale replica forward once via the curl installer; subsequent pushes work.
  • Coord installer swapped binary but old process still running — fixed in v0.17.4. install.sh (non---upgrade path) didn’t restart the unit if it was already active. Symptom: --version reports the new version but HTTP routes return 404. Workaround on stale installs: sudo systemctl restart mokatai-mesh-coordinator.
  • “Join existing network” wizard rejects the join code — the field expects a coordinator invite code minted from the primary’s top-bar Invite → Coordinator Add slide-over (in the invites table, with the cluster payload). A peer invite or a peer-enrollment token is a different artifact and won’t match. The redeem reads invites first and falls back to legacy join_codes, so an old Settings → Add another coordinator → Generate join code value still redeems too. Wording on the wizard was improved in v0.17.3 to make this clearer.

14. mesh CLI + mesh tui

The admin UI is the primary surface — everything routine is one click away. mesh is the same surface for scripts, CI, headless boxes, and “I don’t want to leave the terminal” workflows. It talks to the coord’s HTTP API directly, so it works equally well over the mesh from a remote peer.

Auth

mesh reads two env vars:

export MOKATAI_MESH_URL=http://localhost:8080
export MOKATAI_MESH_ADMIN_TOKEN=$(sudo awk -F= '/MOKATAI_MESH_ADMIN_TOKEN/ {print $2}' /etc/mokatai-mesh/coordinator.env)

MOKATAI_MESH_ADMIN_TOKEN is in the same /etc/mokatai-mesh/coordinator.env file the coord reads at boot — root-owned (0600), so sudo is needed to read it. The token is 32 chars of base64 random; rotate it from the admin UI Settings page when you suspect leakage.

For remote use, set MOKATAI_MESH_URL to the coord’s mesh IP (e.g. http://10.77.0.1:8080) and copy the token across via a separate secure channel.

Common commands

mesh --version

mesh list                              # show all peers
mesh status                            # show coord + cluster summary

mesh add-peer alice                    # add peer (default role: user)
mesh add-peer bob --role operator      # flags BEFORE the name
mesh remove alice                      # remove a peer

mesh rule-add --from user --to operator
mesh rule-list
mesh rule-remove <id>

Argument-order gotcha: flags must come before the positional peer name. mesh add-peer bob --role operator works; mesh add-peer --role operator bob does not. This is a quirk of the flag-parser the project uses; the man-page-style discipline is what the parser expects.

mesh tui — interactive terminal UI

sudo mesh tui                   # auto-detect coord vs agent on this host
mesh tui --coord                # force coord mode
mesh tui --agent                # force agent mode
mesh tui --url <url> --token <t>  # connect to a remote coord

Built on Charm’s bubbletea. Auto-detects by probing 127.0.0.1:8080 (coord) then 127.0.0.1:51821 (agent), and parses /setup/status to refuse if the coord is still in first-run setup mode.

Coord screens (Tab cycles, q quits, ? help overlay):

  • Status — role, current version, latest in manifest, update state, push target, last manifest check.
  • Peers — table with full CRUD. a add, e edit, d delete (confirm modal), r refresh, ↑/↓ navigate, Enter detail.
  • Enroll — onboarding shortcut: n opens an Add-peer form; on success, the URL + token render in big text ready to hand off.
  • Logs — tails journalctl -fu mokatai-mesh-coordinator. ↑/↓ scroll, g/G top/bottom, f toggle follow.

Agent screens (same global bindings):

  • Status — enrolled? coord URL, my mesh IP, my pubkey, active WG endpoint, last handshake age.
  • Peers — read-only table of WG peers from the agent’s /status.
  • Logs — tails journalctl -fu mokatai-mesh-agent.
  • Enroll/Leave — j join (paste URL + token), L leave (confirm). Calls the agent’s loopback /enroll and /leave.

The TUI reads the coord’s admin token from /etc/mokatai-mesh/coordinator.env (needs root or membership in the file’s read group). The agent’s loopback API has no auth (localhost is the boundary).

Output shape

mesh writes human-readable tables to stdout by default. For scripts, every subcommand also accepts --json, which dumps the same data as line-delimited JSON suitable for jq. Errors go to stderr; exit codes follow the usual convention (0 = success, 1 = command failed, 2 = bad arguments).

15. Ports used

  • 8080/tcp — coord HTTP API (configurable via --listen). Admin UI, agent polls, enrollment. Needs internet reachability for remote peers.
  • 51820/udp — coord WireGuard. Needs internet reachability for remote peers to handshake.
  • 51821/tcp — agent loopback-only status API. Never exposed externally.
  • Random high port — mokatai-mesh-client’s local web server (embedded UI). Loopback-only.

16. Cryptography

  • WireGuard: Curve25519 + ChaCha20-Poly1305 + BLAKE2s. Standard WG, no custom crypto.
  • Admin token: 32-char base64 random, rotated via UI, stored in /etc/mokatai-mesh/coordinator.env. Bearer token on every admin API call.
  • Enrollment token: 32-char base64 random (24 random bytes), one-shot, 24h expiry, stored in coord DB. Consumed atomically at /enroll. Legacy/CLI path.
  • Invite code: URL-safe random token (the /join/{code} slug and the credential for the public join surfaces). Single-use by default (reusable opt-in), TTL (peer 24h, coordinator 30 min), revocable. Redeemed atomically (RedeemInvite) so a single-use invite can’t be double-minted; the public /join + redeem surfaces are rate-limited per code and per source IP.
  • Tarball integrity: SHA-256 in the manifest, verified before extraction. Prevents tampered downloads even over plain HTTP (we use HTTPS anyway).
  • Server identity: coord’s WireGuard public key is in the enrollment response; the agent pins it. A MITM on the HTTP channel can’t replace the coord without the pubkey mismatch being immediate.
  • Coord-to-coord transport (trust the wire). Every coord↔coord RPC — heartbeat, request-vote, replicate-all, /internal/dump, and the forwarded admin write (X-Mokatai-Forwarded) — authenticates with the shared internal_token sent as a plaintext bearer over the http:// API listener. The wire itself is unprotected by default, so coordinators must communicate over a trusted network (the WireGuard overlay) or behind a TLS-terminated front. An on-path attacker between two coords can capture the internal token off the wire and then forge admin-equivalent writes: the X-Mokatai-Forwarded + internal-token path authorizes the same cluster mutations as the admin token. We do mitigate the token itself — strength is validated at boot (internal_token must be ≥16 chars), internal != admin is enforced at boot (a constant- time compare refuses to start if they’re equal, so a captured internal token never doubles as the admin token), and the forward path carries a hop-guard so a forwarded write isn’t re-forwarded into a loop — but none of that protects the bytes on the wire. Don’t expose the coord API listener on an untrusted network in the clear.

17. Code reading order

If you want to understand the whole system, read in this order:

  1. internal/api/types.go — the contract.
  2. cmd/coordinator/main.go handlers — handleEnroll, handleConfig, handleCreatePeer, handlePushToPeers (cluster-wide push), handleHeartbeat (replica auto-register).
  3. cmd/coordinator/setup.go — first-run wizard mode, runSetupMode, handleSetupJoin (join-code redemption against primary).
  4. cmd/coordinator/invites.go — the unified invite flow: /admin/invites (create/list/revoke), the public /join/{code} page + /join/{code}/install.sh bootstrap, and redeemPeerInvite (the create-on-redeem wrapper; the atomic RedeemInvite itself lives in internal/db/db.go). Plus cmd/coordinator/joincodes.go — the legacy coord join codes and the /admin/join-codes endpoint, kept for back-compat.
  5. cmd/agent/main.go — main, doEnroll, runPollLoop, applyConfig, runHandshakeProbeLoop.
  6. cmd/client/main.go + cmd/client/web/app.js — native window → webview → polls the loopback API.
  7. internal/update/update.go — self-update pipeline, componentKey, resolveComponent, StartPushToPeers, PushTargetFor.
  8. cmd/mesh/tui/ — bubbletea TUI. Start at tui.go::Run → coord.go::coordModel → screens/coord_peers.go for the meatiest single screen.
  9. internal/cluster/ — the Raft-style leader election state machine (pure decision logic), then cmd/coordinator/main.go’s election driver (onBecomeLeader, resyncCluster, handleRequestVote, leaderOnly/forwardToLeader) for how the I/O is wired.
  10. internal/replication/replication.go — leader→follower /internal/replicate replicate-all fan-out + heartbeat client.

Total code is ~4500 lines of Go + ~1800 lines of JS/CSS/HTML + a bubbletea TUI. Small enough to read cover-to-cover in an afternoon.