Fresh start runbook¶

ops/capture/fresh_start.py is a one-off, owner-run script (planner spec §4.1) that clears the stale capture backlog in a single pass: every cap_items row still stuck in raw/triage older than 30 days, and every open/snoozed reply_owed/commitment follow-up whose source is older than 30 days, except the ~20 most important follow-ups, which stay open for review. Nothing is deleted — items and follow-ups move to state='archived', which stays searchable and excluded from the active views.

It is a single stdlib file (like ops/brain-routine/apply_backlog_triage.py) so it can run from the host with no repo checkout: scp it to the box, or pipe it over ssh ... python3 - < fresh_start.py from a shell that has it locally.

Prerequisite¶

The archived-state migrations must already be deployed before running this script: modules/capture/schema/pb_migrations/16_cap_items_archived.js (adds archived to cap_items.state) and modules/conversation/schema/pb_migrations/16_followups_archived.js (adds archived to followups.state), plus the follow-up cron change that makes it treat archived like done — skip it, never re-open it (conversation_agent/followups.py, the state in ("done", "archived") / state in ("skipped", "archived") checks). Running this script against a box still on the old cron will have the very next hourly pass flip archived follow-ups back to open/done, undoing the cleanup silently.

Rebuild and install the iOS/macOS app from this branch first, too. An older app build filters follow-ups only on state != 'done' and maps any state it does not know to open, so every follow-up this script archives would reappear in the app as open work.

Post-merge order¶

Box deploy — automatic when the branch merges to main (migrations and the follow-up cron land together).
App rebuild — build and install the iOS and macOS apps from main.
Set "Your names in group chats" in Settings › Follow-up rules, so the group-thread rule knows when you are addressed.
Fresh start dry run, review the counts.
Fresh start --apply.
Entity-name cleanup dry run (see the last section below), review the counts.
Entity-name cleanup --apply, then copy its undo file out of the container.

Commands¶

Copy the script to a durable directory on the box first — /tmp can be cleared by the OS or a reboot, which would strand an undo file mid-review:

ssh mailserver 'mkdir -p /root/fresh-start'
scp ops/capture/fresh_start.py mailserver:/root/fresh-start/

# dry run — no writes, prints counts only
ssh mailserver 'cd /root/fresh-start && python3 fresh_start.py --env-file /opt/jarvis/.env'

# apply — writes the undo file FIRST, then archives (default undo path is
# ./fresh-start-undo-<timestamp>.jsonl in the cwd above; pass --undo-file
# only to name it explicitly, and never point it at a path that already exists)
ssh mailserver 'cd /root/fresh-start && python3 fresh_start.py --apply --env-file /opt/jarvis/.env'

# undo — restores every row the apply touched (only rows still archived with
# a matching tag; anything the pipeline already touched again is skipped)
ssh mailserver 'cd /root/fresh-start && python3 fresh_start.py --undo fresh-start-undo-<timestamp>.jsonl --env-file /opt/jarvis/.env'

--days (default 30) and --keep (default 20) override the cutoff and the kept-follow-up count if a different pass is ever needed. --env-file reads only DEV_USER_EMAIL / DEV_USER_PASSWORD out of /opt/jarvis/.env — never shell-source that file directly (see brain-routine.md for why).

If --apply refuses with "undo file already exists" (exit code 2), it means a previous pass's undo file is still sitting at that path — nothing was written or patched. Either restore/clear the old one first (once you are sure it's no longer needed) or pass a different --undo-file; never delete an undo file you have not yet reviewed.

Expected dry-run output¶

Counts only — never names, handles, titles or bodies:

{"mode": "dry-run", "items": 42, "items_by_state": {"raw": 30, "triage": 12},
 "followups_archive": 18, "followups_keep": 20}
dry-run: nothing written.

An --apply run prints the same shape with "mode": "apply", followed by the tally line (items_archived, followups_archived, failed, and the undo_file path it wrote to).

The fresh start does not apply the keep-list. It archives every raw/triage cap_items row older than the cutoff, including invoices, receipts and bank-sender mail that a routing pass never picked up — those rows are not exempted. They stay searchable in archived state. --undo replays a whole undo file, so to bring back a single item, copy its line (find it by id) from the undo JSONL into a new file and run --undo on that new file.

The undo file¶

--apply writes the undo JSONL in full before the first PATCH (using "create, don't overwrite" semantics — it refuses to touch a path that already exists, so a re-run after a partial failure can never clobber the only undo for the first pass). It defaults to ./fresh-start-undo-<timestamp>.jsonl unless --undo-file names a path. Keep it until the owner is confident the pass was correct — --undo needs the exact file it wrote, and it only restores a row that is still exactly where the fresh start left it (state archived, and — for cap_items — the same archived_by tag); anything the pipeline already re-routed or a later pass already touched is left alone and counted as skipped.

After a retry (an --apply that refused an existing path and was re-run with a different --undo-file, or a second pass) there can be several undo files. Each covers only the rows its own run touched: replay each one you want reverted, one --undo per file.

Each line is one JSON object holding the fields needed to restore the row: collection, id, state, and — for cap_items — the pre-patch links dict plus the run's tag. Treat this file as sensitive. The links dict is whatever the row already carried — project/identity slugs (which can be client or contact names), a person entity id, and a free-text rationale from the original routing decision can all appear in it. It never holds a message's title or body (those aren't part of links), but it is not just harmless ids — delete it once the owner has confirmed the pass was correct and no longer needs the undo.

After the fresh start: entity name cleanup¶

Once the backlog is archived, the owner runs the companion one-off inside the running scheduler container (jarvis-scheduler, see compose.yaml) to normalize any entity display names left over from before the 2026-08-27 RFC 2047 fix:

ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.clean_entity_names'                 # dry run
ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.clean_entity_names --apply'          # writes ./entity-names-undo-<ts>.jsonl inside the container
ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.clean_entity_names --undo PATH'      # restore from an undo file

Run the dry run first, review the counts, then --apply. The scheduler container only mounts the vault volume — there is no separate persistent data directory to point --undo-file at — so immediately after --apply, copy the undo file out to the host before it can be lost to a container restart or rebuild:

ssh mailserver 'docker cp jarvis-scheduler:/app/entity-names-undo-<ts>.jsonl /root/fresh-start/'

Restore with --undo PATH the same way as fresh_start.py, after copying the file back into the container (docker cp /root/fresh-start/<file> jarvis-scheduler:/app/).

Noise backfill¶

The deterministic noise gate (capture_agent.noise.classify, planner slice 1) only runs at ingest, on newly polled email. It never swept the raw/triage backlog that predates it. Run this companion one-off, also inside jarvis-scheduler, to apply the same rules to that backlog and archive whatever matches. Run it right after a brain-routine pass, ideally in a quiet window: the backfill re-checks each row right before writing it and skips anything the brain routine already routed or skipped out from under it, but a quiet window keeps the skipped count low and the sweep more useful:

ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.noise_backfill'                 # dry run
ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.noise_backfill --apply'          # writes ./noise-backfill-undo-<ts>.jsonl inside the container
ssh mailserver 'docker exec jarvis-scheduler python -m capture_agent.noise_backfill --undo PATH'      # restore from an undo file

Run the dry run first and review by_rule before --apply. As with the entity name cleanup, copy the undo file out of the container immediately after --apply:

ssh mailserver 'docker cp jarvis-scheduler:/app/noise-backfill-undo-<ts>.jsonl /root/fresh-start/'

Only sender, text, and attachment rules can fire here. signal rules (newsletter/auto-submitted/bulk) key off email headers, and stored cap_items never carry those headers — the dry-run and --apply summaries both say so via signal_rules_skipped: true, so a lower archive count than expected here is expected, not a bug.

Treat this undo file as sensitive, too. Like the entity-name-cleanup undo, each line holds the row's previous links dict — project/identity slugs and routing rationale that can name a client or contact — not just a bare id. Delete it once the pass is confirmed correct and no longer needed.

Restore with --undo PATH the same way as fresh_start.py, after copying the file back into the container (docker cp /root/fresh-start/<file> jarvis-scheduler:/app/).