Homelab

A different shape of connector: instead of pulling messages into cap_items, the homelab agent watches your self-hosted infrastructure via Grafana, proposes fixes, and — only once you approve — applies them over SSH. It is its own module (modules/homelab), not part of capture, and every part of it that touches the LAN or your SSH keys is host-only: it runs under launchd on the owner's Mac, never in the container, and there is deliberately no module.yaml schedule for the executor or the updates loop.

What it does

1. Gather — reads firing Grafana alerts plus per-alert PromQL evidence from VictoriaMetrics into a source-agnostic health snapshot (homelab_agent/gather.py, GrafanaHealthcheck).
2. Propose — an LLM turns a degraded snapshot into a remediation proposal (commands to run, on which host), gated at plan time so it never proposes an unsafe command.
3. Approve — nothing runs until you review the proposal and approve it.
4. Execute — on approval, the executor re-checks every stored command against the allowlist immediately before running it, applies the commands over SSH in order (stopping on first failure), then re-gathers from Grafana to verify the fix actually took.

Everything is recorded in four collections: hl_runs (gather runs), hl_proposals (remediation proposals), hl_llm_log (audited LLM calls), and hl_updates (monthly image-update proposals). A proposal (in either hl_proposals or hl_updates) moves through the states proposed → approved → applied (or skipped if you decline it).

Grafana access

The gather step needs read access to your Grafana instance:

Env var	Required	Purpose
JARVIS_GRAFANA_URL	yes	Base URL of your Grafana instance
JARVIS_GRAFANA_TOKEN	yes	Bearer token for the Grafana/Prometheus API
JARVIS_GRAFANA_DS_UID	no	Datasource UID used to fetch per-alert PromQL evidence; without it, alerts are reported without evidence

SSH targets

Proposals carry a logical host label (e.g. vm101), not a raw IP. The executor resolves that label to an ssh alias through homelab_agent/hosts.py's HOST_ALIASES map. Edit that map to match your own hosts — each key is a Grafana instance label, each value is the ssh alias to reach it (they can be identical, or the value can be a real hostname):

# Illustrative — replace with your own hosts (values are placeholders).
HOST_ALIASES = {
    "proxmox": "proxmox",
    "vm101": "<your-ssh-alias-or-hostname>",
    "vm103": "vm103",
    "servarrct": "servarrct",
    "gssh": "gssh",
}

Each alias resolves to a real host and key through your own ~/.ssh/config — no IPs or keys are committed anywhere, and none are ever written to an hl_* row. SSH runs non-interactively: BatchMode=yes, ConnectTimeout=10, StrictHostKeyChecking=accept-new.

Approve → execute flow

Review the pending card in the Home inbox (or the hl_proposals row directly), then approve or skip it:

python3 -m homelab_agent.approve <proposal_id>

python3 -m homelab_agent.approve <proposal_id> --skip

This flips hl_proposals.state proposed → approved (or skipped) and marks the inbox card done. Applying is a separate step, run by the com.jarvis.homelab.execute.plist LaunchAgent every 5 minutes:

python3 -m homelab_agent.execute --host

--host is a required marker — the executor refuses to run without it, so it can never accidentally run container-side. Full install and shakedown steps live in modules/homelab/agent/host/HOST_EXECUTE.md.

Monthly updates flow

A parallel loop scans your docker stacks for available image updates (moved registry digests on floating tags) and, on approval, redeploys just that service:

python3 -m homelab_agent.update_scan --host

python3 -m homelab_agent.approve <update_id> --collection hl_updates

python3 -m homelab_agent.update_execute --host

Two LaunchAgents drive this: com.jarvis.homelab.scan-updates.plist runs the scan on the 1st of each month at 04:00 (under caffeinate, so a sleeping Mac still wakes for it), and com.jarvis.homelab.update-execute.plist polls for approved updates every 10 minutes. Full install and rollback steps live in modules/homelab/agent/host/HOST_UPDATES.md.

Gotchas

Host-label alignment

Each proposal's host value comes from the LLM, which reads it off the Grafana alert's instance/host label (port stripped). The keys in HOST_ALIASES must match your real Grafana instance labels exactly. If they don't, resolve_alias fail-closes on every proposal — safe, but the executor silently "blocks" everything instead of applying anything. After your first live run, confirm the proposal was actually applied over SSH, not left approved with a "blocked at apply: no ssh alias for host label …" card; if you see that, edit HOST_ALIASES to match your labels.

Double-apply race

Neither executor has an atomic claim on approved rows. Hand-running execute --host or update_execute --host while its LaunchAgent is loaded risks the same proposal being applied twice. Run launchctl unload ~/Library/LaunchAgents/com.jarvis.homelab.execute.plist (or the update-execute equivalent) before hand-running, or only hand-run before installing the LaunchAgent in the first place.

Rollback is not automated

If an applied update verifies unhealthy, nothing rolls it back for you. The outcome card carries the recorded previous_image (<repo>@<olddigest>); pin that digest in the service's compose file and redeploy by hand:

docker compose -f <stack_path> up -d <service>