Stop Searching: Build a Knowledge Assistant on Your Documents

The concept: ask instead of search, with proof

Stop treating your knowledge base as a filing cabinet where everyone has to dig for themselves. Think of it instead as a colleague you can ask, one who answers and tells you where the answer came from. A knowledge assistant does not search folders for you. It answers a question from exactly your own documents and states the source (document and location). A person can then check every claim. The technology behind this is called RAG (Retrieval-Augmented Generation): the model answers only from retrieved text passages, not from memory. But the technology is not what matters most. Discipline is: only backed answers, and a clear 'I don't know' when something is not in the documents, instead of a guess.

The essentials in 60 seconds

• A knowledge assistant is not a better search box. It is a colleague you ask, one who answers with a source.
• Start with one clean, clearly bounded area (for example your current manuals), not your entire document store.
• Sourcing is non-negotiable: every answer names the document and the location, otherwise treat it as unverified.
• If something is not in the documents, the assistant says so openly instead of inventing a plausible-sounding answer.
• Garbage in, garbage out: outdated or contradictory documents do more harm than the assistant does good.
• Build a small test set of real questions with known answers before you let the team near it.
• Measure one number before (search time, questions to colleagues) and the same number after two weeks. No measurement, no verdict.

Three levels: search, assistant on one area, connected assistant

Before you build anything, work out what you actually need. The levels differ in how broad the knowledge base is and how much upkeep sits behind it. Many SMEs need level 2, and are better served by it than by an overambitious level 3 project that stalls on data maintenance.

• Level 1, better search: full text or semantic search across your files. Good when people mainly want to find the right document and then read it themselves.
• Level 2, assistant on one area: the assistant answers questions from a maintained, bounded set of documents (manuals, policies, processes) with a source attached. The right first step for most companies.
• Level 3, connected assistant: it draws on several sources (file storage, wiki, ticketing system, email threads) and respects access rights per user. Powerful, but far more maintenance and permission work.
• Rule of thumb: the more uniform and well maintained your knowledge is, the more likely level 2 is enough. The more scattered and sensitive it is, the more carefully level 3 needs to be planned.

Phase 1: Choose the right knowledge area

Goal: know within ten minutes which area you will start with. Do not pick the biggest one. Pick the most frequently asked about and best maintained knowledge. A clean small area beats a large mess.

• Ask: what do employees search for most often, and where does the answer actually live in documents (not just in people's heads)?
• Estimate roughly: how many times per week is this searched or asked about, and how many minutes per case? That is your baseline measurement.
• Check the maintenance state: are the documents current, unambiguous, and free of contradictions? If not, clean up first.
• Examples: current product and service manuals, the quality management handbook, travel expense and HR policies, technical specifications.

We are considering setting up a knowledge assistant on our own documents. Here are our knowledge areas with a short description of each: [list your areas, e.g. service manuals, HR policies, contract templates, project documentation].
Rate each area on four criteria:
1) How often is it searched?
2) Does the answer live clearly in documents, or mostly in people's heads?
3) How current and contradiction-free is the document set?
4) How sensitive is the content (access rights)?
Give a short assessment for each area and an overall recommendation for which area we should start with, small and low risk.

Phase 2: Clean up the area (the most important phase)

A knowledge assistant is only as good as the documents it reads. Outdated, duplicated, or contradictory content leads to answers that sound confident but are wrong, and that is exactly what destroys trust. So run an inventory before you connect anything.

• List every document in the area, with title, date of last update, and owner.
• Mark each one: keep as current, revise, or archive (remove from the knowledge base).
• Resolve contradictions: if two documents disagree, decide which one is authoritative.
• Set, per document, who maintains it and how often it gets reviewed.

I will give you a list of internal documents with titles and last-modified dates. Help me run an inventory for a knowledge assistant.
For each document, suggest a category: 'keep as current', 'revise', or 'archive', and give your reasoning (for example age, likely duplicates, unclear title).
Flag documents that likely contradict or overlap with each other so I check those first.
List any open questions at the end that only a person can resolve.
Documents: [insert list].

Phase 3: Connect the documents and enforce sourcing

Now connect the maintained area to an AI assistant. The decisive step is not the upload. It is the instruction: the assistant may only answer from the supplied documents, must cite its source, and must pass when there is no coverage. This instruction is your most important safeguard against hallucination.

You are a knowledge assistant for our company. Rules you always follow:
1) Answer questions only based on the documents provided. Do not use general knowledge.
2) For every statement, cite the source: document name and, where possible, section or page.
3) If the answer is not in the documents, or only partly covered, say so clearly (for example 'Not covered in the available documents') and do not guess.
4) If documents contradict each other, point this out and cite both sources instead of picking one.
5) Answer formally, concisely, and without embellishment.
Confirm these rules briefly, then wait for the first question.

Phase 4: Test, measure, sign off

Before the whole team gets access, check with real questions whether the assistant answers with sources and honestly. Build a small test set for this: questions whose correct answer you already know, plus a few questions whose answer deliberately is NOT in the documents. That way you see both things at once: whether it finds what is there, and whether it honestly passes when nothing is there.

• Collect 15 to 25 real questions, including several with no answer in the knowledge base (trap questions).
• Have the assistant answer all of them and check each answer: is it correct, is the source right, and for trap questions, does it honestly say 'I don't know'?
• Record the hit rate and, above all, the hallucination rate (invented or wrongly sourced answers).
• Refine from there: add missing documents, clarify confusing passages, sharpen the instruction.
• Only sign off once the hallucination rate on the test set is close to zero.

Help me build a test set for our knowledge assistant.
Here is the knowledge area: [describe the area].
Suggest 20 realistic test questions, mixed from:
a) typical employee questions,
b) rarer detail questions,
c) three to four trap questions whose answer is probably NOT covered in a normal manual.
For each question, mark the expected type of correct response ('sourced answer' or 'honest pass').
Output this as a table I can use as a test log.

A worked example: service manuals at a machine builder

A machine builder with 80 employees has field technicians and phone support constantly digging through manuals, wiring diagrams, and maintenance instructions for details. Until now, when in doubt, the technician calls an experienced colleague. Here is what the same need looks like with a level 2 knowledge assistant:

• Area: only the current service and maintenance manuals for the three best-selling machine lines, cleaned up properly.
• Technician's question: 'What is the tightening torque for the main spindle on the X series?'
• Assistant's answer: the specific value plus a source ('Maintenance manual X, section 4.2').
• Trap case: if someone asks about a machine whose manual is not connected, the assistant answers 'not covered' instead of guessing.
• Sign-off: during the pilot, an experienced technician checks a sample of answers before they count as reliable.
• Before, roughly 12 minutes of searching or asking per case, after, roughly 2 minutes. These are example numbers, measure your own real values in the pilot.

How to set it up (two routes, no developers needed)

You do not need developers for the first version. Choose based on how sensitive the data is and whether the document set is stable or constantly changing.

• Route A (no technical setup, about an hour): create a project in ChatGPT (Projects) or Claude (Projects), upload the cleaned-up documents for the area, and save the sourcing instruction as the project instruction. Ideal for proving the concept works on one small, stable area.
• Route B (team-wide, with permissions): Microsoft Copilot Studio (if you already use Microsoft 365) or a specialized RAG solution with EU hosting. Here you connect a document source, can mirror access rights, and roll the assistant out to the whole team. More setup work, but current and usable by everyone.
• Recommendation: prove the concept with route A on one area first, then invest in route B. Switching providers later is far easier than fixing a poorly maintained knowledge base.

For the technically inclined: building RAG properly (how we run it ourselves)

Once the first version proves itself, a few techniques separate a usable assistant from one the team truly trusts. These are exactly the techniques we use in our own document-based systems. If you have technical staff, this is where they should focus.

• Chunking: documents are split into sections before they become searchable. Cut along meaning (headings, paragraphs), not by a fixed character count, and let sections overlap slightly so no sentence is cut off mid-thought. Rough guide: a few hundred words per chunk, tune this in the pilot.
• Embeddings and vector search: each chunk is translated into a numeric vector that represents its meaning. A question is translated the same way, and the system retrieves the chunks with the closest meaning. This way it still finds the right passage even when the user's wording differs from the document's.
• Retrieval with metadata and re-ranking: give every chunk metadata (source, date, area, approval status) and filter on it. A second step, re-ranking, sorts the retrieved chunks by actual relevance before the model answers. This noticeably improves quality.
• Sourcing enforcement and source scoping: enforce technically that only retrieved chunks serve as the basis for an answer, and that every answer carries its source. Scope the searchable set per request (for example to the area or to the user's permissions).
• An eval set against hallucination: maintain a fixed set of questions with known answers, plus trap questions with no coverage. Run it automatically after every significant change, and measure hit rate, source accuracy, and hallucination rate. Without an eval set, you only notice regressions once the team has already lost trust.
• An update pipeline: define how new or changed documents enter the knowledge base and how outdated ones leave it. A clean approach is a recurring update (for example nightly or on change) that only ingests approved documents and reliably removes archived ones. Stale content left in the index is the most common source of wrong answers.
• Access rights (permissions per user): mirror your existing permissions inside the assistant. Anyone who cannot see a document in the original should not be able to get an answer derived from it either. Implement this through per-chunk metadata and a permission filter before retrieval. Otherwise the assistant becomes a data leak.

Tools and costs (rough guide, as of 2026, check current pricing)

First the area and the data maintenance, then the tool. The bigger effort almost always sits in maintenance, not in the license.

• Microsoft Copilot Studio: a natural choice if you already use Microsoft 365 and want to mirror permissions from SharePoint. Cost depends on your Microsoft 365 and Copilot plan, check the licensing model.
• Claude (Team) or ChatGPT Team with Projects: fast for route A on one bounded area. Roughly 25 to 30 euros per user per month.
• A specialized RAG solution with EU hosting: for sensitive content and finer control over chunking, permissions, and updates. Price depends heavily on volume and hosting, request quotes.
• Building it yourself adds costs for embeddings and model queries, usually small relative to the time saved. Measure this in the pilot with real volume.

A worked cost example

Example numbers, replace with your own real values in the pilot.

• 10 employees each spend roughly 15 minutes a day searching for information in documents or asking about it.
• That is 150 minutes a day, roughly 12.5 hours a week across the team.
• If the assistant halves that search time, you save roughly 6 hours a week. That alone is clearly noticeable.
• Weigh the tool and maintenance effort against those saved hours at your internal hourly rate.
• Important: the real value is often not just the time saved, but fewer errors from outdated or misremembered knowledge. Hard to put a number on, but real.

Most common mistakes

• Connecting the entire document store at once. Start small and clean, then expand.
• Leaving outdated or contradictory documents in place. Wrong answers that sound confident destroy trust faster than good answers build it.
• Accepting answers without a source. Sourcing is non-negotiable.
• No eval set. Without a fixed test, you only notice hallucinations once the team already has.
• Forgetting access rights. An assistant that answers everything for everyone can leak confidential information.
• Not settling data maintenance. Without an owner and a review rhythm, the knowledge base decays, and so does the quality of every answer.

Frequently asked questions

• Do I need developers? For a first version on one area, usually no (route A). For a team-wide solution with permissions and updates, technical support helps.
• Does the AI make up answers? That is the main risk. Sourcing enforcement, source scoping, and an eval set limit it heavily. It can never be fully ruled out, which is why citing sources stays mandatory.
• What about outdated documents? They are the most common source of errors. A clear update pipeline that only ingests approved documents and removes old ones is mandatory, not optional.
• Can every employee see everything? Only if you allow it. Mirror your existing access rights inside the assistant, otherwise it becomes a data leak.
• Are my documents secure? With EU-hosted tools, a data processing agreement, and a commitment that your data is not used for training: yes, with the usual due diligence.

GDPR and the EU AI Act in one paragraph

Internal documents often contain personal and confidential information. Before you start, check for EU hosting, a data processing agreement, and an explicit commitment that your content will not be used to train the provider's models. Mirror access rights carefully so nobody can use the assistant to reach documents they could not see in the original. Document the purpose and the data sources. Further EU AI Act obligations take effect from December 2027. In German-speaking markets, thinking this through from the start is an advantage over bolting it on later.

When you should skip this

If your knowledge is disorganized, outdated, or full of contradictions, clean up first, otherwise garbage in, garbage out applies. If the relevant knowledge lives mainly in people's heads rather than in documents, an assistant on documents will not help much. Writing that knowledge down is the first step instead. And wherever a wrong answer would be costly or dangerous (for example safety-critical or legally binding information), human review stays mandatory. The assistant should at most provide a sourced draft there.

Your 30 day start

• Week 1: choose an area (area selection prompt), run the inventory, clean up, and resolve contradictions.
• Week 2: set up route A (a ChatGPT or Claude project), upload the documents, save the sourcing instruction, test the first questions.
• Week 3: build a test set from real questions plus trap questions, run it, measure hit rate and hallucination rate, refine.
• Week 4: release to a small pilot group, measure time saved, decide whether to expand or stop. If successful, plan route B with permissions and updates.

Where this comes from

This playbook is not built on theory. It comes from systems we run ourselves. We work daily with document-based assistants and know the pitfalls from practice, above all data maintenance, sourcing enforcement, and the eval set against hallucination. That is exactly why these points sit so far up front here. What is written here, we built, measured, and refined on our own knowledge bases. We do not just talk about AI, we build it ourselves.