This is a reference architecture, not a case study. It describes a pattern and the boundaries that make it trustworthy. The retrieval behaviour it depends on can be inspected in the Stripe API docs retrieval demo, and the confidence-threshold fallback in the ScoutLocal case study.
The problem this addresses. Operational knowledge ends up spread across documents, tickets, and chat threads. Answering "where is the procedure for X" becomes a search task that someone repeats every day, and the answer quality depends on who is asked.
Another wiki page rarely fixes this. The useful pattern is a retrieval layer that finds the relevant source material, drafts an answer from it, and shows exactly what it used.
1. Scope the corpus first
Search is only as good as the set of documents it is allowed to read. The first decision is which sources are in scope, who may see them, and how they get updated.
- Sources: procedures, resolved tickets, and product documentation are the usual candidates. Each should have an owner and an update path.
- Access: if different people may see different documents, the index needs to carry that boundary. Retrieval should filter by permission before ranking, not after.
- Storage: a relational database with a vector extension is enough for most internal corpora. Hosted vector search becomes worth its cost when data size or latency demands it, not before.
2. The retrieval loop
The core of the system is a bounded loop: take the question, retrieve candidate passages from the corpus, generate an answer grounded in those passages, and return the answer together with its sources.
- Question in. From whichever surface the team already works in.
- Retrieve. Hybrid retrieval, combining keyword and embedding matches, so exact terms such as product names and error codes are not lost to semantic similarity.
- Ground. The model answers only from the retrieved passages and cites them.
- Answer out. The response carries links back to the original documents so the reader can verify it.
The retrieval demo runs this loop over a fixed documentation corpus: every answer is bounded by the source documents and lists the passages it drew on.
3. The "not in corpus" boundary
The most important behaviour is the refusal. When retrieval returns nothing relevant, or the match confidence is below a threshold agreed with the team, the system should say that the answer is not in the corpus rather than generate one.
This does two things. It prevents confident answers that have no source, and it produces a list of questions the documentation does not cover, which is the maintenance backlog.
The same principle appears in the ScoutLocal case study, where semantic search falls back to structured SQL matching when the confidence score is below a tuned threshold, so a weak semantic match never replaces a deterministic result.
4. What to measure
The metrics worth tracking are practical, not theatrical.
- Time to answer: how long it takes a team member to reach a verified answer, compared with the manual search it replaces.
- Refusal rate: how often the system reports "not in corpus". A rising rate points at documentation gaps, not at the search.
- Citation checks: a sample of answers reviewed against their cited sources, so grounding is verified rather than assumed.
When this is the wrong build
- Keyword search over the same documents already returns the right page quickly.
- The documents are stale or contradictory. Retrieval will surface the contradiction faithfully; it will not resolve it.
- Nobody owns the corpus. A search layer over an unmaintained source set degrades within months.