As of November 2026 · CODLAB editorial team
Short answer: You test an AI document search with four cases: a question with a sourced answer, a question without a source, an updated document and a case where access is denied. Use approved or synthetic documents, define the criteria in advance and let your team judge whether the cited passage and the answer actually match.
1. What an AI document search should do
An assistant for your own documents has two tasks: show the matching passages and respect the existing access rights. Technically, this is usually retrieval-augmented generation (RAG): the system first searches for passages in your documents and then formulates an answer on that basis. This leads to the most important testing rule: search and answer are assessed separately. A well-worded answer based on the wrong passage is not a success.
2. The four cases of the trial
- Sourced answer: a typical question whose answer is in a specific document. Expected: the answer and the correct passage.
- Question without a source: a question the documents say nothing about, such as an undocumented exception. Expected: a note that the basis is missing, not a plausible invention.
- Updated document: a rule has changed. Expected: the answer follows the valid version, not an outdated one.
- Denied access: a person without permission asks about content from a restricted area. Expected: neither the answer nor the cited passage reveals that content.
The trial also includes the counter-check: if the assistant still delivers an answer to the question without a source, that is a clear result – and a valuable one.
3. Access rights belong in the search, not only in the answer
One design mistake to avoid is checking permissions only after the search. Restricted content can then already enter the selection of passages and resurface in the wording of the answer. It is safer to apply the rights during the search itself: only documents the person asking is allowed to see are considered at all. So before the trial, name the areas that must stay closed to certain roles, as well as outdated versions that should no longer be used.
4. A cited passage makes an answer checkable, not correct
Citations are valuable because they make an answer verifiable. A citation alone, however, does not confirm that an answer is correct: the cited passage may be misunderstood, incomplete or not relevant to the specific case. The professional review stays with your team. In the trial, therefore, assess two things separately: does the passage fit the question, and does the answer really follow from the passage?
5. Making results comparable
Language models do not answer the same way in every run. A single good run therefore proves little. Ask the same questions several times, write down the expected passages and note how much correction effort remains. How to build such assessments methodically is described in our articles on evaluating stochastic AI and on RAG architecture, and in the engineering note on RAG evaluation.
6. What you need for a first conversation
For a first conversation, the document types, a typical search question and a description of the access rights are enough. Confidential documents are not needed for this. For an agreed trial we choose approved or synthetic documents and record who judges the expected passages. Klaus, our document assistant, relies on Mistral; data paths and processing locations are set out in the specific offer. Custom assistants with open models run on your hardware or on chosen European infrastructure. More on our page Document assistants.
Frequently Asked Questions (FAQ)
With a trial of four cases: a sourced answer, a question without a source, an updated document and a case where access is denied. Define the criteria in advance, use approved or synthetic documents and let your team judge the cited passages.
No. A citation makes an answer checkable but does not confirm that it is correct. The cited passage may be misunderstood or not relevant to the case; the professional review stays with the team.
No. For a first conversation, document types, a typical search question and a description of the access rights are enough. Approved or synthetic documents are suitable for the trial.