CODLAB / Blog · AI & integration

Testing AI Document Search: Sources, Missing Sources and Denied Access

18 Nov 2026 CODLAB Team 4 min

As of November 2026 · CODLAB editorial team

Short answer: You test an AI document search with four cases: a question with a sourced answer, a question without a source, an updated document and a case where access is denied. Use approved or synthetic documents, define the criteria in advance and let your team judge whether the cited passage and the answer actually match.

1. What an AI document search should do

An assistant for your own documents has two tasks: show the matching passages and respect the existing access rights. Technically, this is usually retrieval-augmented generation (RAG): the system first searches for passages in your documents and then formulates an answer on that basis. This leads to the most important testing rule: search and answer are assessed separately. A well-worded answer based on the wrong passage is not a success.

2. The four cases of the trial

  1. Sourced answer: a typical question whose answer is in a specific document. Expected: the answer and the correct passage.
  2. Question without a source: a question the documents say nothing about, such as an undocumented exception. Expected: a note that the basis is missing, not a plausible invention.
  3. Updated document: a rule has changed. Expected: the answer follows the valid version, not an outdated one.
  4. Denied access: a person without permission asks about content from a restricted area. Expected: neither the answer nor the cited passage reveals that content.

The trial also includes the counter-check: if the assistant still delivers an answer to the question without a source, that is a clear result – and a valuable one.

3. Access rights belong in the search, not only in the answer

One design mistake to avoid is checking permissions only after the search. Restricted content can then already enter the selection of passages and resurface in the wording of the answer. It is safer to apply the rights during the search itself: only documents the person asking is allowed to see are considered at all. So before the trial, name the areas that must stay closed to certain roles, as well as outdated versions that should no longer be used.

4. A cited passage makes an answer checkable, not correct

Citations are valuable because they make an answer verifiable. A citation alone, however, does not confirm that an answer is correct: the cited passage may be misunderstood, incomplete or not relevant to the specific case. The professional review stays with your team. In the trial, therefore, assess two things separately: does the passage fit the question, and does the answer really follow from the passage?

5. Making results comparable

Language models do not answer the same way in every run. A single good run therefore proves little. Ask the same questions several times, write down the expected passages and note how much correction effort remains. How to build such assessments methodically is described in our articles on evaluating stochastic AI and on RAG architecture, and in the engineering note on RAG evaluation.

6. What you need for a first conversation

For a first conversation, the document types, a typical search question and a description of the access rights are enough. Confidential documents are not needed for this. For an agreed trial we choose approved or synthetic documents and record who judges the expected passages. Klaus, our document assistant, relies on Mistral; data paths and processing locations are set out in the specific offer. Custom assistants with open models run on your hardware or on chosen European infrastructure. More on our page Document assistants.

Frequently Asked Questions (FAQ)

How do I check whether an AI document search is reliable?

With a trial of four cases: a sourced answer, a question without a source, an updated document and a case where access is denied. Define the criteria in advance, use approved or synthetic documents and let your team judge the cited passages.

Is a citation proof that an answer is correct?

No. A citation makes an answer checkable but does not confirm that it is correct. The cited passage may be misunderstood or not relevant to the case; the professional review stays with the team.

Do I need confidential documents for the trial?

No. For a first conversation, document types, a typical search question and a description of the access rights are enough. Approved or synthetic documents are suitable for the trial.

Share this article

Comments

No comments yet. Be the first to comment!

Leave a comment

Will not be published

Related Articles

Project on your mind?

We develop tailored software and AI solutions. Let's start your next project together.

Free initial consultation