Measuring Stochastic AI Systems: Why a Single Demo Proves Nothing and How Multi-Run Evaluation Works

As of November 2026 · CODLAB Team

Direct answer: Because large language models sample from probability distributions, one impressive live demo proves nothing about operational reliability. Rigorous AI evaluation requires multi-run testing per query (at least three independent executions to expose variance), separate metrics for retrieval recall, grounding faithfulness, and abstention accuracy, plus automated regression gates on every update.

When Deterministic Software Habits Meet Probabilistic Models

In conventional business software engineering, verification rests on determinism: if an invoicing function computes the expected VAT amount for a given input payload once, calling that function a thousand times with the same state produces the exact same number. A single passing unit test establishes behavior along that code path.

Large language models (LLMs) and RAG assistants invalidate that assumption. At every decoding step, a generative transformer estimates a probability distribution over its vocabulary. Even at low sampling temperatures (such as temperature = 0.2 in CODLAB's reference assistant ragbot in the codlab.de codebase — public chat demo switched off since October 2026 — or nominally 0.0), tiny variations in retrieved passage ordering, minor user phrasing shifts, or floating-point non-determinism across batched GPU kernels can cause an assistant to include a mandatory safety condition on run one and silently omit it on run two.

Across our applied engineering and research work at CODLAB on Sovereign AI & Data, we therefore separate live demonstrations strictly from verification: a prototype can shine during a ten-minute walkthrough and still fail unpredictably on edge cases. Both Germany's Federal Office for Information Security (BSI, Generative AI Models, v2.0, 2025) and the NIST Generative AI Profile (NIST AI 600-1, 2024) warn that ad-hoc spot checks cannot substitute for structured measurement. What does an honest evaluation methodology look like in practice?

1. Glossary and Statistics of Multi-Run Testing: pass@k vs. pass^k

Running a probabilistic pipeline once draws a sample of size one. Evaluation literature distinguishes two fundamentally different multi-run metrics that must never be conflated: pass@k (popularized in code generation with automated unit-test filtering, Chen et al., 2021, arXiv:2107.03374) measures the probability that at least one of k attempts succeeds (any) — meaningful only when an external verifier discards bad candidates before a user sees them. When an end user receives a single unscreened response, the operational metric that matters is strict all-run consistency pass^k (Yao et al., τ-bench, 2024, arXiv:2406.12045), requiring every single one of k independent runs to satisfy all domain criteria (all).

Small-sample probability limits must also be stated plainly: if a pipeline preserves a mandatory contractual exception with per-run probability p = 2/3, three independent runs still all succeed with probability (2/3)³ ≈ 29.6%. Three runs per test case therefore serve at CODLAB as a pragmatic development floor to surface gross flakiness early — not a statistical guarantee. Production sign-off requires larger sample sizes with explicit confidence awareness, separating two distinct experimental protocols:

• Identical-input repeats under frozen context (sampling variance): Running the exact same query against the same frozen retrieval snapshot tests whether mandatory caveats (deadlines, limits, approval roles) and citations hold across all k executions.
• Controlled paraphrase perturbations (input sensitivity): Tracked separately from pure repeats, realistic user paraphrases (abbreviations, colloquial phrasing, minor typos) measure joint retrieval and generation sensitivity.

# Methodological pattern (check_domain_criteria is an illustrative stub for domain validators):
# Distinguish at-least-one success (pass@k) from strict all-run consistency (pass^k)
def evaluate_case_stability(pipeline_fn, test_case, k=3):
    outcomes = []
    for run_idx in range(k):
        result = pipeline_fn(test_case.query)
        outcomes.append({
            "run": run_idx + 1,
            "retrieved_urls": [s["url"] for s in result["sources"]],
            "grounded": result["grounded"],
            "passes_rubric": check_domain_criteria(result["answer"], test_case.criteria),
        })
    flags = [item["passes_rubric"] for item in outcomes]
    return {
        "any_passed_at_k": any(flags),   # pass@k logic: at least 1 success
        "all_passed_hat_k": all(flags),  # pass^k logic: all k runs succeeded
        "pass_rate": sum(flags) / k,
        "runs": outcomes,
    }

2. Four Distinct Diagnostic Dimensions Instead of One Vague Accuracy Score

Reporting a single headline number such as "our assistant is X percent accurate" obscures more than it reveals, because it collapses different failure modes into one average. Drawing on published evaluation research such as RAGAS (Es et al., 2023/2024, arXiv:2309.15217), reflective retrieval architectures (Asai et al., Self-RAG, arXiv:2310.11511), and our note Evaluating RAG: Separating Retrieval and Answers, we separate diagnosis into four distinct dimensions (causally coupled in operation, yet measured separately):

1. Retrieval Recall@k: Did the top-k retrieved chunks actually include the authoritative passage required to answer the question? If not, engineering effort belongs in chunking, indexing, or query expansion — never in prompt tweaking.
2. Context Precision (Signal-to-Noise): What fraction of retrieved passages is genuinely relevant? Flooding the prompt with near-duplicate or tangential sections raises the probability of cross-contaminating rules from neighboring products.
3. Answer Faithfulness (Grounding): Is every atomic claim in the generated answer strictly entailed by the retrieved passages, without fabricating external details or dropping documented caveats?
4. Refusal & Boundary Accuracy: When queried about unindexed topics, restricted files, or unresolved contradictions, does the system deterministically abstain rather than guessing?

Decomposing evaluation this way turns optimization from guesswork into engineering: if a new embedding model lifts Recall@5 while pulling in superseded policy drafts that depress Faithfulness, the diagnostic table pinpoints the trade-off immediately.

3. Using "LLM-as-a-Judge" Without Falling for Its Biases

Because human specialists cannot manually grade hundreds of multi-run outputs after every configuration commit, engineering teams frequently use a strong language model as an automated grader ("LLM-as-a-Judge"). Lianmin Zheng et al. analyzed this paradigm thoroughly, documenting both its practical utility and its systematic biases (Zheng et al., NeurIPS 2023, arXiv:2306.05685):

• Verbosity bias: Automated judges systematically favor longer, polished-sounding responses even when a concise answer is more accurate or when the longer text smuggles in unsupported claims.
• Position bias: When comparing Candidate A against Candidate B, automated preference shifts noticeably depending on which answer appears first in the prompt.
• Self-enhancement bias: Judge models tend to score outputs generated by their own model family more generously.

At CODLAB, automated judges are therefore constrained by three rules. First, anything verifiable by code is checked deterministically (presence of expected source URLs, refusal flag triggers, absence of prohibited commitments or pricing numbers). Second, judge prompts evaluate explicit binary rubric items ("Does the response state that written supervisor sign-off is required: Yes/No?") rather than subjective 1-to-10 ratings. Third, automated rubrics are calibrated against domain-expert annotations, and pairwise comparisons always swap A/B positions across runs.

4. Combining Deterministic Gates with Statistical Gates

A production AI application is far more than a prompt: it comprises web views, authentication boundaries, rate limiters, citation links, and accessible user interfaces. In our Engineering Notes, Playwright and axe: A Test Capable of Turning Red illustrates a core CODLAB engineering rule: a verification gate is trustworthy only if we have proven that it turns red when a defect is introduced.

We apply this exact principle by pairing deterministic gates with statistical evaluation suites:

• Deterministic gates (zero tolerance): Tenant isolation queries, same-origin API protections, hard refusal when cosine distance exceeds cutoff, valid structured metadata, keyboard navigation, and WCAG contrast across mobile and desktop viewports. A single failure halts deployment.
• Statistical gates (multi-run rubric thresholds): Before any change to chunking logic, retrieval parameters, prompts, or model weights goes live, a frozen benchmark suite of domain questions, edge cases, and unanswerable counter-examples runs at least three times per item. Any regression on safety caveats or refusal behavior blocks release, and every newly discovered production edge case is permanently added to the regression set.

5. Synthetic Walkthrough: Swapping Models Without Flying Blind

Methodological note: The scenario below is purely synthetic and illustrates our evaluation workflow; it does not represent a client engagement or claimed customer performance figures.

Imagine a mid-sized testing laboratory running an internal assistant over standard operating procedures (SOPs). Six months into operation, the team considers replacing its current model with a smaller, faster open-weights model to cut local GPU latency. Without a structured benchmark, someone might type ten casual questions, admire the snappier replies, and deploy.

Under a multi-run evaluation protocol, the decision becomes evidence-based:

1. The frozen benchmark holds 60 curated prompts: 35 standard SOP lookups, 10 edge cases involving recently amended exception clauses, and 15 out-of-scope questions that require abstention.
2. Both candidate models are evaluated against the identical retrieval snapshot across three independent runs per prompt (180 runs per model).
3. In this synthetic test, the smaller model matches the baseline on standard queries on run one, yet across repeated runs it intermittently overlooks "superseded draft" headers when two document versions appear side by side.
4. Rather than discovering that regression through an operational mistake, the team resolves the root cause beforehand — either filtering draft statuses strictly in PostgreSQL or retaining the stronger model.

Continuous post-deployment verification across document updates and model upgrades is an integral part of AI Operations & Maintenance. If you want to establish a reproducible evaluation baseline for a planned or existing AI system, get in touch via Contact & Initial Consultation.

Continue in this research series: To see how upstream indexing and ranking drive retrieval metrics, read our architectural study on hybrid retrieval, structure-aware chunking, and cross-encoder reranking; to see how pass^k reliability extends to multi-step tool execution, explore bounded AI agents and deterministic guardrails.

Verified Primary Sources

• Chen, M., Tworek, J., Jun, H. et al. (2021): Evaluating Large Language Models Trained on Code (arXiv:2107.03374).
• Yao, S., Shinn, N., Razavi, P. & Narasimhan, K. (2024): τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045).
• Es, S., James, J., Espinosa-Anke, L. & Schockaert, S. (2023/2024): RAGAS: Automated Evaluation of Retrieval Augmented Generation (EACL 2024, arXiv:2309.15217).
• Zheng, L. et al. (2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023, arXiv:2306.05685).
• Asai, A. et al. (2023/2024): Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024, arXiv:2310.11511).
• National Institute of Standards and Technology (2024): Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1).
• Federal Office for Information Security / BSI (2025): Generative KI-Modelle — Chancen und Risiken für Industrie und Behörden, Version 2.0, 17.01.2025.

Related operational guide (German edition): KI im Unternehmen einführen: Vom Arbeitsauftrag zur überprüfbaren Freigabe.