Paste what the member said. The agent searches the regulations, opens the official interpretations through the citation graph, computes the deadlines, drafts a reply with verified citations and opens a handoff ticket.
Recorded real runs
Pipeline
Lights up live as the agent works (server-sent events).
- Guardmask PII
- Agentplan + tools
- Toolssearch · graph · dates
- Verifyquotes · dates · $
- Finalizesend · withhold
Agent trace
Each model turn and tool call appears here as it happens, then what the
deterministic finalize step accepted or rejected.
Result
Verdict
Semantic search
Search, then pick a result to open its neighborhood.
Citation graph
official interpretationrule it explains
citescited by
Edges are parsed from the text itself (“§ 1005.6(b)”, “paragraph (c)(1) of
this section”, Supplement I comments), never by a model. In Kubernetes the same graph is served from Neo4j with identical
neighbors for every node.
The fixed pipeline: guard → retrieve → generate → verify → one retry with verifier feedback → finalize. Configuration C of the hallucination evaluation.
Pipeline trace
Stages, tokens, cost and verifier results.
Retrieval: recall@5 on 33 answerable golden questions
Means with 95% bootstrap intervals over questions. Every run records corpus hash, golden-set hash and git commit.
Answer quality
Stores and graph
Architecture
Controls
| Risk | Control | Where |
|---|---|---|
| Invented rule, deadline or amount | Quote must be a verbatim substring of a passage a tool returned this session; one retry with feedback | verify.py · copilot.py |
| Model date or money arithmetic | Dates and dollar limits come from the calculator; any other date or amount in a reply withholds it | timeline.py · copilot.py |
| Live member problem | Disputes and unauthorized use always open a handoff ticket; unsupported replies go to a specialist | copilot.py |
| PII in member text | SSN, card, account, email, phone masked before retrieval, model and traces; logs carry paths only | guard.py · server |
| Cost and abuse | Worst-case admission against a USD ledger; per-IP rate limit and daily cap in Redis; fails closed | guard.py · limits.py |
| Web attack surface | Strict CSP (no inline script), no framing, request ids, JSON errors without tracebacks | server/app.py |
| The law changes | Weekly job compares eCFR amendment dates with the pinned snapshot and fails if a covered section changed | corpus/freshness.py |
| Regression | Hashes and commit in every report; CI runs tests against real Qdrant and Neo4j, the offline eval and a Kubernetes deploy | eval/ · .github/ |