How to test an AI knowledge assistant before you trust it with more

An AI knowledge assistant answers questions from your own documents. Most are built with retrieval-augmented generation, usually shortened to RAG. The system first finds the passages that look relevant, then a language model writes an answer from them. On ten friendly questions it can look excellent. What decides whether you can rely on it is the awkward questions, and those only show up if you go looking for them.

This guide describes a pilot you can run before giving an assistant more users, more documents or the power to do anything beyond answering.

Give it one narrow job

Start with one audience and one body of information. For example: the support team asks questions about our approved returns and warranty policies. Write down which questions it should answer, which documents it may use, and when it should send someone to a person.

Keep answering separate from acting. Finding the right policy is a different responsibility from changing a customer record or approving a refund. Pilot the answering first and watch how people really use it. If actions become worth adding, treat them as a new project with their own permissions, a confirmation step and a way to undo mistakes.

Clean the knowledge before you test the model

Inventory the documents you plan to load. For each one, record an owner and check whether it is current, duplicated, incomplete or contradicted by another document. If two files give different approval rules, the assistant cannot choose for you. Somebody has to decide which one is right before either goes in.

Keep the context that lets a reader check an answer: document titles, section headings and last-updated dates. Agree how updates reach the assistant, too. Policies change, move and get withdrawn, so someone must own the job of keeping the index in step with the files. Keep the pilot collection small enough that one person can read all of it. When an answer is wrong, you can then tell whether the cause was a missing document, poor retrieval or the way the answer was written.

Test permissions with real test accounts

The assistant should see exactly what the person asking is allowed to see, and nothing more. A shared search index does not keep those boundaries on its own.

Create test users with different roles and a few documents whose access differs. Then check three things for each role:

1. Can this user get an answer built from a document they cannot open directly?
2. Can they discover that a restricted document exists, for example from its title?
3. Can they infer its contents from a summary?

Include messy conversations: switching topics, pasting in a block of text, or telling the assistant to ignore its earlier instructions. OWASP lists this class of problem as prompt injection, including indirect injection hidden in a document or web page the assistant reads. OWASP is candid that it is unclear whether any prevention is fool-proof. Its practical advice is to limit the damage: give the application only the access it needs, handle privileged actions in code instead of trusting the model, require human approval for risky ones, and keep untrusted content clearly separated. Keep personal or sensitive data out of the pilot set unless including it is a deliberate, controlled decision. Log access failures as their own category so a fluent answer cannot hide a leak.

Build a question set from real work

Ask the people who do the work what they are really asked. A good pilot set mixes several kinds of question, and each needs a different definition of "good":

For each question, write down the information a good answer contains and the document that supports it. Where several answers would be acceptable, describe the limits of a useful answer rather than demanding exact wording. Put the hard cases in on purpose: an exception buried far from the main rule, a process retired last year, a request for a prediction the documents cannot support.

Check retrieval and answers separately

If you only read the final answer, you cannot tell why it went wrong. Evaluate the two stages on their own. Retrieval is the question of whether the right passages were found. Generation is the question of whether the answer is supported by the passages it was given. An engineering write-up from Tweag on RAG evaluation makes the same point: keep them apart because end-to-end scores do not reveal which part failed. It also recommends adding bad-retrieval cases, where the system finds distracting or irrelevant documents, so you can see whether it degrades gracefully instead of confidently inventing something.

For the generation side, the usual measure is faithfulness. The Ragas documentation defines it as the share of the claims in an answer that can be inferred from the retrieved context. You can do this by hand on a small set: split each answer into its claims and tick the ones the retrieved passages support. Tools automate it with another language model acting as judge. Treat that judge as fallible, spot-check its verdicts against your own reading, and never let it be the only reviewer.

Score support and usefulness separately

Use a short rubric with separate scores:

• Relevant: does it answer the question that was asked?
• Supported: is every claim backed by the retrieved passages?
• Cited: do its references open the right document?
• Honest: does it flag uncertainty when it should?
• Useful: does it give the person a next step?

Keep these apart because the failures look different. An answer can be fluent and wrong, or accurate and useless. Look at results by kind of question too. An average score can hide a whole category, such as every question that needs two documents, that fails consistently.

Have subject experts and everyday users review the same examples. Experts catch wrong interpretations. Users catch confusing wording and extra steps. Record each failure with enough detail to reproduce it: the question, the document version, the user's role and the response. Rerun the whole set whenever the documents or the system change, so you notice a regression before your customers do.

Mistakes that waste a pilot

• Testing only the happy path. If every question has an answer in the documents, you have tested search, not judgement.
• Reading answers without the sources. A good-sounding answer may rest on the wrong passage.
• Letting the vendor or builder choose the questions. Use questions from the people who will live with the assistant.
• Adding actions too soon. Sending an email or editing a record raises the stakes of every wrong answer.
• Skipping an owner. Without a named person to review flagged answers and update documents, quality decays quietly.

When not to build one

An assistant is the wrong tool in a few cases. If you have a handful of short documents, good search may be enough. If the answer lives in live data, such as an order status or a stock level, a database query returns the exact figure and a language model adds risk without adding value. If the documents contradict each other and nobody will own the fix, the assistant will reproduce the confusion fluently. And if a wrong answer could cause real harm, the design needs a person approving the result, which changes the economics. It is better to learn this in a pilot than after launch.

Plan the handoff and the review

Decide what happens when the assistant cannot help. A good handoff states the unresolved question, keeps the useful context and routes it to the right person or queue, so nobody has to repeat the whole conversation. Give users a simple way to flag a wrong answer, and name who reviews those flags.

At the end of the pilot, compare what happened with the job you wrote down at the start:

Any of these is a legitimate result. The point of a pilot is to find out cheaply.

Our INTELLIGENCE service covers AI assistants, knowledge bases and AI search built around your own data. If the assistant will eventually trigger actions, read how to plan workflow automation first, and if you want help designing a pilot, tell us about it or write to contact@wisvera.com.