Start here: a local evidence assistant
Build your first evidence assistant: a complete local baseline
Turn a small document collection into a deterministic, source-linked response system. Understand the contract, implement retrieval and abstention, and establish a baseline before introducing a language model or agent.
Sources & version details
In this article
1. The problem and the promise
Imagine an on-call engineer searching a runbook for “timeout retry”. Before building a conversational agent, make the simpler system correct: find a public passage, preserve its identifier and return no passage when there is no lexical evidence. Our assistant returns a verbatim document, not generated prose. It is an information-retrieval baseline for a future AI application, not an LLM, a semantic search engine or a secure production service.
The outcome is a runnable Python file and a separate test file. You will explain the response schema, calculate one ranking by hand, reproduce a known failure and identify the point where model inference could be added. Budget one or two focused sessions; this is a planning estimate, not a measured completion time.
2. Prerequisites with meaning
A Python function transforms an input into an output; a dictionary names the output fields. A set removes duplicate words, which means repeated query terms do not increase our score. A document ID is a stable reference, whereas a list position changes when documents are reordered. A frozen dataclass makes accidental field reassignment fail, but it does not authenticate the source or prevent a caller from replacing the collection.
A baseline is a deliberately simple comparator with fixed behavior. Abstention means refusing to provide evidence below a declared rule; it does not mean proving no answer exists. A trace records how candidates were ranked. These definitions matter because a fluent generated answer can hide poor retrieval, and a high score can still select an irrelevant passage.
3. Architecture and data contracts
The collection contains D1, a retry instruction; D2, a key-rotation instruction; and D3, a restricted incident record. The CLI passes a query to answer(). Validation checks type and length before tokenization. The public-only filter runs before scoring and trace construction so a restricted ID does not leak through a diagnostic result. This filter is a teaching policy, not identity-based authorization; there is no login or tenant system.
The output has status, text, source and trace. status is evidence or abstain; abstain sets text and source to null. Scores are dimensionless and source is an ID, not a URL. Invalid inputs raise ValueError instead of masquerading as successful empty answers. The program has no network, persistence or side effects; repeated calls need no retry or transaction mechanism. If you add an HTTP service, implement authentication, quotas, deadlines and redaction at that boundary.
4. Derive the score and work an example
Let Q be the set of normalized query terms and D the set of terms in a candidate document. Our score is the fraction of query terms present in the document. It is asymmetric: extra document words are not penalized. For “timeout retry”, Q has two terms and D1 contains both, giving 2/2 = 1. D2 contains neither, giving 0/2 = 0. D3 is filtered out, not assigned a zero score. The threshold is 0.5, so an exactly equal score is accepted.
With “timeout missing”, D1 scores 1/2 and returns evidence although half the question is unsupported. With “connection stalled”, every score is zero although a human might connect it to timeouts. These are counterexamples, not bugs in the arithmetic. Lowering the threshold trades more returns for more weak matches. Increasing it cannot solve synonyms. We deliberately avoid presenting coverage as confidence or answer correctness.
| ID | Q ∩ D | s(Q,D) |
|---|---|---|
| D1 | timeout, retry | 2/2 = 1 |
| D2 | ∅ | 0/2 = 0 |
5. Implementation choices and limits
terms() lowercases and extracts ASCII letters and digits; it is a lexical tokenizer for this English fixture, not a model tokenizer. It does not implement Chinese or Japanese segmentation, stemming, Unicode normalization or stop-word removal. The five translated lessons share the same English test data so expected outputs remain identical. A real multilingual system needs language-appropriate analysis and new evaluation sets.
For N eligible documents with average L characters, this implementation re-tokenizes about N×L characters per query and sorts N scores in O(N log N). That is acceptable for three documents, not a large corpus. Precompute document terms before introducing an inverted index; version that index together with source text. A stale index can point to evidence that no longer exists. Sorting ties by document ID makes behavior stable across input order.
6. Extend only after the baseline is understandable
BM25 introduces term-frequency, rarity and length effects; dense retrieval adds representation-based similarity. Neither automatically establishes that evidence answers a question. If adding a generator, pass only eligible retrieved text and preserve source IDs in a separate structured channel. Treat document instructions as untrusted data. Verify generated citations rather than accepting a model-invented identifier. A full RAG system also needs ingestion, freshness, access checks and evidence sufficiency.
The next project is not “add every framework”. Read the RAG survey to choose one retrieval improvement, the SDE tutorial to understand the model-call boundary, and the SRE agent tutorial only when you need tools and stateful actions. Keep the same held-out task when comparing versions. Otherwise a changed dataset can look like progress in your architecture.
7. Practice and verification
Create an empty folder and save the two files below as assistant.py and test_assistant.py. Use Python 3.11 or later; there are no third-party dependencies, downloads or API keys. No random sampling is used, so a random seed is unnecessary. Run the commands from that folder. The full source and tests are included: no hidden helper or remote dataset is required.
assistant.py
Download assistant.py"""Offline, deterministic retrieval baseline. Python 3.11+, standard library only."""
import json
import re
import sys
from dataclasses import dataclass
@dataclass(frozen=True)
class Document:
id: str
text: str
audience: str = "public"
DOCS = (
Document("D1", "Retry timeout requests with backoff."),
Document("D2", "Rotate access keys every month."),
Document("D3", "Private incident recovery password.", "restricted"),
)
def terms(text):
return set(re.findall(r"[a-z0-9]+", text.lower()))
def answer(query, docs=DOCS, threshold=0.5):
if not isinstance(query, str) or len(query) > 2000:
raise ValueError("query must be a string of at most 2000 characters")
if not 0 < threshold <= 1:
raise ValueError("threshold must be in (0, 1]")
if len({d.id for d in docs}) != len(docs):
raise ValueError("duplicate document ID")
q = terms(query)
candidates = []
for doc in docs:
if doc.audience != "public":
continue # Filter before ranking or constructing the trace.
score = len(q & terms(doc.text)) / len(q) if q else 0.0
candidates.append((score, doc.id, doc.text))
candidates.sort(key=lambda row: (-row[0], row[1]))
trace = [{"id": d, "score": s} for s, d, _ in candidates]
if not candidates or candidates[0][0] < threshold:
return {"status": "abstain", "text": None, "source": None, "trace": trace}
_, doc_id, text = candidates[0]
return {"status": "evidence", "text": text, "source": doc_id, "trace": trace}
if __name__ == "__main__":
query = " ".join(sys.argv[1:]) or "timeout retry"
print(json.dumps(answer(query), indent=2))
test_assistant.py
Download test_assistant.pyimport unittest
from assistant import answer, Document
class ContractTests(unittest.TestCase):
def test_known(self):
r = answer("timeout retry")
self.assertEqual((r["status"], r["source"]), ("evidence", "D1"))
self.assertEqual(r["trace"][0]["score"], 1.0)
def test_unknown_and_empty(self):
for query in ("", "quasar orbit", "日本語"):
self.assertEqual(answer(query)["status"], "abstain")
def test_no_private_trace(self):
r = answer("private incident recovery password")
self.assertEqual(r["status"], "abstain")
self.assertNotIn("D3", str(r))
def test_stable_tie(self):
docs = (Document("B", "retry"), Document("A", "retry"))
self.assertEqual(answer("retry", docs)["source"], "A")
def test_boundaries(self):
self.assertEqual(answer("timeout missing")["status"], "evidence")
self.assertEqual(answer("timeout missing extra")["status"], "abstain")
self.assertEqual(answer("retry", ())["status"], "abstain")
def test_invalid(self):
for query in (None, "x" * 2001):
with self.assertRaises(ValueError): answer(query)
with self.assertRaises(ValueError): answer("retry", threshold=0)
with self.assertRaises(ValueError): answer("retry", (Document("A", "x"), Document("A", "y")))
def test_lexical_failure_is_visible(self):
# Semantically similar wording is intentionally NOT solved by this baseline.
self.assertEqual(answer("connection stalled")["status"], "abstain")
if __name__ == "__main__":
unittest.main()
python3 assistant.py "timeout retry"
python3 assistant.py "quasar orbit"
python3 -m unittest -v test_assistant.pyExpected: the default query returns
Compare fairly: keep corpus, query list and permissions fixed. First change only the threshold (0.5 to 0.75); then revert it and precompute document terms. The first changes behavior; the second should preserve behavior. Record returned evidence, unsupported returns and abstentions separately. Use a development set to choose thresholds and a new, untouched set for final reporting; the tiny fixtures here are unit tests, not evidence of deployment quality.
Original interview exercises — not company interview questions
Why is a score of 1 not proof of an answer?
Hint: compare word coverage with meaning. Answer: “do not retry timeout requests” contains every query term but changes the instruction. The method ignores negation and entailment; add a counterexample and an evidence-support check.
Why filter before constructing the trace?
Hint: diagnostics are outputs too. Answer: returning no private text is insufficient if a private ID or score is exposed. Filtering first removes those candidates from every result channel. This still does not provide authenticated permissions.
How would you improve speed without changing answers?
Hint: identify repeated work. Answer: cache token sets for an immutable corpus version, then verify identical ranked scores and tie handling across queries. Invalidate on document or tokenizer changes. Measure only after proving equivalence.
Scoring: 1 — repeats definitions; 2 — computes the worked example; 3 — runs and explains all tests; 4 — adds a new failure case and fair comparison; 5 — independently implements an extension, preserves its contract and explains remaining failure modes. These are learning criteria, not professional certification.