Chat & RAG

Once the semantic index is built, Jarvis can answer natural-language questions over your vault and structured records — cited, with sensitive data withheld before anything reaches the cloud. Three surfaces sit on top of the same answer loop: the CLI, the MCP server, and the in-app chat tab.

How AgenticRagClient.ask works

Asking a question makes exactly two cloud calls, both propose_audited hops, with one local retrieval round in between:

1. Plan hop (purpose="rag_plan") — the LLM is given the question and a tool schema, and proposes a short list of retrieval steps (e.g. a search_vault call with a query, or a query_records call against an allowlisted collection). Any step that fails validation is dropped before execution — the plan hop can only select what to retrieve, never see or return data itself.
2. Local execution — no cloud call. Each valid step runs against the local store: search_vault (semantic search over the vector index), query_records, person_timeline, or list_inbox. Results from every step are merged by Reciprocal Rank Fusion into one ranked list, and each item's citation is resolved to a [[note]] wikilink where a matching vault note is found — vault-note chunks already carry one, and cap_items / entities resolve when their frontmatter matches; anything else (and any unmatched record) keeps its collection:id citation. The list is then split by sensitivity: sensitive items are held back, and the remaining items are capped to the top 8 (configurable) by fused rank.
3. Synthesize hop (purpose="rag_answer") — only the retrieved, non-sensitive, top-N items are sent to the LLM, which returns an answer plus citations.

Invariant: the citations ask returns are always a subset of the items that were actually retrieved and sent to the synthesize hop — a citation the model invents that isn't in that set is silently dropped. Citations are never fabricated, even if the model tries.

If any items were withheld for being sensitive, the answer text gets a trailing note. It's emitted with Markdown italics (the surrounding _), so it renders emphasized:

_(2 sensitive sources withheld from the cloud answer.)_

Both hops are logged to conv_llm_log (via propose_audited), the same redacted, fail-closed, audited path every other cloud call in Jarvis uses.

CLI

Run from the repo root, with the agent runtime on PYTHONPATH:

PYTHONPATH="agent-runtime/src:modules/conversation/agent" \
  python -m conversation_agent.agent ask "What did we agree with the landlord about the deposit?"

ask accepts --limit (default 8) to change the top-N sent to synthesis:

PYTHONPATH="agent-runtime/src:modules/conversation/agent" \
  python -m conversation_agent.agent ask "..." --limit 4

Related subcommands of the same agent.py entrypoint, for direct/structured access without going through the RAG loop:

Command	Purpose
ask "<question>" [--limit N]	Cited natural-language answer via the agentic RAG loop above.
query <collection> [--filter JSON] [--sort FIELD] [--limit N]	Structured read over one allowlisted collection.
timeline <entity_id> [--limit N]	Cross-module contact timeline for one entity.
inbox	List pending platform inbox items.
index	(Re)build the semantic index.
serve-mcp	Run the MCP server over stdio.
serve-chat	Run the chat-tab worker described below.

All of ask, query, timeline, and inbox need PocketBase reachable and credentials set (POCKETBASE_URL, DEV_USER_EMAIL, DEV_USER_PASSWORD); ask additionally needs DEEPSEEK_API_KEY (or LLM_PROVIDER=stub for an offline canned response) and a built vector index.

The chat tab

The chat tab (Jarvis app, jarvis_conversation Flutter package) is a multi-thread conversational surface backed entirely by PocketBase — the app never talks to Python directly. All the work happens through two collections defined by the 3_conv_chat.js migration:

• conv_threads — one row per conversation: title, identity.
• conv_messages — one row per turn: thread (relation, cascade-delete with its thread), role (user or assistant), text, citations (JSON), state, error.

User turns carry a lifecycle: pending → thinking → answered or error. Assistant turns are written already-terminal, carrying the answer's text and its citations (wikilinks where resolvable, otherwise collection:id).

The round trip

1. The tab creates a conv_messages row for the user's question with state="pending".
2. The serve-chat worker's poll loop claims the oldest pending user turn, flipping it to state="thinking" (a single-writer claim — only one worker instance should run against a given store).
3. It mechanically composes the query context from the thread's prior turns: newest-first, accumulating lines until a character cap is hit, then replaying the kept turns chronologically ahead of the live question as Earlier in this conversation: ...\n\nNow: <question>. This step is pure string assembly — it never calls an LLM. A thread with no prior turns (or one entirely over the cap) just passes the question through unchanged.
4. It calls AgenticRagClient.ask with the composed query (the two-hop loop described above).
5. On success, it creates a new conv_messages row for the assistant's answer (role="assistant", text, citations) and marks the original user turn state="answered".
6. On any failure — composing the context, the ask call itself, or writing the terminal records — the user turn is marked state="error" with the exception message (truncated) in error. A single bad turn can't crash the worker's poll loop.

The tab watches both collections and renders turns as their state changes.

Run the worker

PYTHONPATH="agent-runtime/src:modules/conversation/agent" \
  python -m conversation_agent.agent serve-chat

On startup the worker first requeues any turn orphaned in state="thinking" back to pending — that happens when a previous worker process died mid-answer — so a downed worker doesn't leave the tab showing "Thinking…" forever. It then loops: claim and answer one pending turn if there is one, otherwise sleep for the poll interval. A downed worker simply stops making progress — turns pile up as pending — and the backlog drains automatically once a worker is running again.

Env vars

Var	Default	Purpose
POCKETBASE_URL, DEV_USER_EMAIL, DEV_USER_PASSWORD	— (required)	PocketBase connection and auth.
DEEPSEEK_API_KEY	—	Needed for real answers (LLM_PROVIDER=deepseek); otherwise falls back to the stub provider.
JARVIS_VECTOR_DB / JARVIS_PB_DATA	see Semantic index	Sidecar vector store the retrieval step reads.
JARVIS_VAULT_DIR	— (optional)	Vault root. When set, cap_items / entities citations resolve to [[note]] wikilinks via note frontmatter; unset, they degrade to collection:id (chunk citations are wikilinks regardless).
JARVIS_CHAT_POLL_SECS	2.0	Poll interval between checks for pending turns.
JARVIS_CHAT_CONTEXT_CHARS	4000	Character cap on the whole-thread context assembled per turn.

The worker also requires the 3_conv_chat.js PocketBase migration applied (it creates conv_threads and conv_messages). See Configuration for where each of these vars is set, and MCP server for the ask tool exposed to MCP clients like Claude Code — the same agentic loop, a different surface.