Backend auto-deploy (Hetzner)¶

Production runs on a single Hetzner host (mailserver in the owner's ~/.ssh/config), under /opt/jarvis, started with:

docker compose -f compose.yaml -f compose.prod.yaml up -d

Exposure is a Cloudflare tunnel → https://jarvis.richarddepierre.com, whose health endpoint is https://jarvis.richarddepierre.com/api/health.

When a change lands on main that touches backend paths, GitHub Actions rebuilds and restarts the stack on the host automatically, once CI has passed. UI-only merges never deploy. The workflow runs on the self-hosted runner ([self-hosted, deroches-ci]), like CI.

Architecture¶

The design keeps the CI credential powerless beyond deploying: the SSH key GitHub holds can do exactly one thing on the server — run the deploy script with a payload on stdin. It cannot open a shell, forward ports, or read arbitrary files.

sequenceDiagram
    participant Dev as PR merged to main
    participant GH as GitHub Actions
    participant CI as CI workflow
    participant Host as Hetzner host
    Dev->>CI: push to main
    CI-->>GH: workflow_run completed (success)
    GH->>GH: gate: backend paths changed since the last deploy?
    GH->>Host: ssh (forced command) < payload.tgz
    Host->>Host: backup volumes → extract → rsync --delete → build → up -d
    Host->>Host: wait for health 200 + containers
    Host-->>GH: one-line summary
    GH->>Host: curl public /api/health (200?)

Three pieces make it work:

ops/jarvis-deploy.sh — the server-side script. The canonical copy is version-controlled here in the repo; the active copy is installed at /usr/local/bin/jarvis-deploy.sh (root:root, 0755), deliberately outside /opt/jarvis so a deploy payload can never overwrite it. It reads a gzipped tar of the repo checkout from stdin and: validates the payload; snapshots the jarvis_pb_data + jarvis_vault volumes to /root/backups (newest 7 kept); extracts to a staging dir and sanity-checks it; rsync -a --deletes into /opt/jarvis (preserving server-local files, see below); rebuilds and restarts the stack; waits for health 200 plus the long-running containers; and prints a one-line summary.
A forced-command SSH key. A dedicated ed25519 keypair is generated for deploys. Its public half is added to /root/.ssh/authorized_keys as a restricted entry:

command="/usr/local/bin/jarvis-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty <key>

Any remote command string the client sends is ignored — the forced command always runs instead. So ssh -i deploy_key root@host 'echo pwned' runs the deploy script (which then refuses the empty stdin), never the echo.

.github/workflows/deploy-backend.yml: triggered by CI completing on main (workflow_run). It used to trigger on push and poll CI from a job, but the self-hosted runner runs one job at a time, so a polling job would hold the runner that CI needs. Then:
job gate refuses to deploy if CI failed (a cancelled CI run was superseded by a newer push, whose own CI run deploys). It finds the last commit that actually deployed (the newest run whose deploy job succeeded; the run name carries the commit) and diffs it against the tested commit. It only deploys when a backend path changed. A failed deploy is retried by the next push, because its commit never becomes the baseline.
job deploy checks out the tested commit, assembles the module migrations, builds the payload with the same exclusions as the manual recipe, writes the two secrets to a job-scoped folder under $RUNNER_TEMP (never ~/.ssh; the runner is a persistent machine), pipes the payload to the forced command over SSH with a pinned host key, curls the public health endpoint, and deletes the key whatever happened.

What deploys, and what doesn't¶

The gate job deploys only when one of these paths changed since the last deploy (a merge that touches none of them does not deploy):

Deploys	Never deploys (UI-only)
`backend/**`	`app/**`
`agent-runtime/**`	`core/**`
`docker/**`	`modules//lib/`
`compose.yaml`, `compose.prod.yaml`	`modules//test/`
`modules.yaml`	`docs/**`, `mkdocs.yml`
`modules//agent/`
`modules//schema/`
`modules/**/module.yaml`

workflow_dispatch is also available for manual runs (Actions → deploy-backend → Run workflow). It deploys the branch head whatever paths changed, after checking once that CI passed for that commit. If CI is still running, it stops and asks you to run it again, since CI needs the same runner.

Server-local files that survive a deploy¶

rsync --delete would otherwise remove anything not in the payload. Two files are explicitly excluded and therefore preserved across every deploy:

/opt/jarvis/.env — rotated secrets + TUNNEL_TOKEN. Never in the repo, never overwritten, excluded forever.
/opt/jarvis/compose.override.yaml.dev-only — the renamed dev override, kept so a bare docker compose up can't merge it.

compose.prod.yaml used to be server-local too; it is now committed to the repo (it references the tunnel token as ${TUNNEL_TOKEN} from .env, so it carries no secret) and is therefore no longer excluded — the repo copy is the source of truth. The Docker named volumes (jarvis_pb_data, jarvis_vault, jarvis_chat_data, jarvis_model_cache) live outside /opt/jarvis, so rsync never touches them.

Module migrations must be assembled before packaging

A module's schema/pb_migrations/*.js only reach the PocketBase image if they are staged into backend/generated/module_pb_migrations/ first (that dir is git-ignored except a .gitkeep). The deploy workflow runs the assembly step before building the payload; the manual recipe below runs ./jarvis deploy (or up), which does the same. Skip it and module collections silently bake as core-only. The server script aborts if the payload contains zero assembled migrations.

Manual fallback recipe¶

If Actions is unavailable, deploy by hand from a clean checkout of main. This is exactly what the workflow does, minus the CI gate:

# 0. from a clean checkout of main, with the deploy key available
DEPLOY_KEY=~/.ssh/jarvis_deploy_ed25519      # the private half of the deploy key
HOST=178.105.145.39

# 1. assemble enabled-module migrations into the image-baked dir
PYTHONPATH=cli/src python3 -c \
  "from pathlib import Path; from jarvis_cli.deploy import assemble_module_migrations; assemble_module_migrations(Path('.'))"

# 2. build the payload (exclusions mirror the server-side rsync). .env is never
#    in a checkout; compose.prod.yaml IS shipped (repo is source of truth).
#    COPYFILE_DISABLE=1 stops macOS tar from adding AppleDouble (._*) sidecars,
#    which PocketBase would otherwise try to load as migrations and panic on.
mkdir -p /tmp/jarvis-payload-src
rsync -a \
  --exclude='.git/' --exclude='temp/' --exclude='dev-vault/' \
  --exclude='build/' --exclude='.dart_tool/' --exclude='__pycache__/' \
  --exclude='*.pyc' --exclude='.fvm/' --exclude='.DS_Store' --exclude='._*' \
  --exclude='compose.override.yaml' --exclude='.env' \
  ./ /tmp/jarvis-payload-src/
COPYFILE_DISABLE=1 tar czf /tmp/jarvis-payload.tgz -C /tmp/jarvis-payload-src .

# 3. pipe it to the forced command (the 'deploy' arg is ignored; stdin is the
#    interface). The script backs up, syncs, rebuilds, restarts, health-checks.
ssh -i "$DEPLOY_KEY" -o StrictHostKeyChecking=yes "root@${HOST}" deploy < /tmp/jarvis-payload.tgz

The owner's ordinary admin key (mailserver) still has full shell access for everything else — the forced-command key is only for deploys. See also The jarvis CLI for what ./jarvis deploy stages.

Rotating the deploy key¶

The deploy key is dedicated and disposable. To rotate it:

# 1. generate a fresh keypair locally (no passphrase; never commit it)
tmp=$(mktemp -d)
ssh-keygen -t ed25519 -N '' -C 'jarvis-deploy' -f "$tmp/jarvis_deploy_ed25519"

# 2. on the server, replace the forced-command line in authorized_keys. Edit
#    /root/.ssh/authorized_keys and swap the old key for:
#    command="/usr/local/bin/jarvis-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty <new-pubkey>

# 3. update the GitHub secret with the new private key
gh secret set DEPLOY_SSH_KEY < "$tmp/jarvis_deploy_ed25519" --repo Richard-DEPIERRE/jarvis-dev

# 4. verify, then shred the local copy
rm -rf "$tmp"

The host key is pinned separately in the DEPLOY_KNOWN_HOSTS secret. If the server's host key changes, refresh it:

ssh-keyscan -t ed25519 178.105.145.39 | gh secret set DEPLOY_KNOWN_HOSTS --repo Richard-DEPIERRE/jarvis-dev

Restoring a volume snapshot¶

Every deploy tars jarvis_pb_data + jarvis_vault into /root/backups/jarvis-<UTC-timestamp>.tgz before touching anything (newest 7 kept). To restore one, stop the stack, wipe the target volumes, and untar back:

cd /opt/jarvis
docker compose -f compose.yaml -f compose.prod.yaml down     # stop; keep volumes
SNAP=/root/backups/jarvis-20260828T130000Z.tgz                # pick the snapshot

# restore both volumes from the snapshot (the archive holds pb_data/ and vault/)
docker run --rm \
  -v jarvis_pb_data:/snap/jarvis_pb_data \
  -v jarvis_vault:/snap/jarvis_vault \
  -v /root/backups:/backups:ro \
  alpine:3.21 sh -c 'cd /snap && rm -rf jarvis_pb_data/* jarvis_vault/* && tar xzf /backups/'"$(basename "$SNAP")"

docker compose -f compose.yaml -f compose.prod.yaml up -d     # bring it back

PocketBase migrations auto-apply

Migrations are baked into the image and PocketBase applies any pending ones when its container restarts. After restoring an older pb_data, the current image may re-apply migrations forward — restore to a snapshot whose schema matches the image you intend to run.

Updating the server script¶

The active script lives at /usr/local/bin/jarvis-deploy.sh. To update it after editing the repo copy:

scp ops/jarvis-deploy.sh mailserver:/usr/local/bin/jarvis-deploy.sh
ssh mailserver 'chown root:root /usr/local/bin/jarvis-deploy.sh && chmod 0755 /usr/local/bin/jarvis-deploy.sh'

It is intentionally not part of the deploy payload's active path, so bumping the repo copy does not change the running deployer until you re-install it.