As of November 2026 · CODLAB Team
Direct answer: Production LLM and RAG pipelines do not fail solely through loud server crashes; they also degrade silently via KV-cache stalls, stale document indexes, and shifting query topics. Resilient AI observability separates stage-level OpenTelemetry tracing and statistical drift signals strictly from cleartext payloads so teams isolate root causes without stockpiling confidential prompts.
Why a Green HTTP 200 Status Masks Silent Failures in Language Model Systems
In conventional web services, returning HTTP 200 within an acceptable response budget generally confirms healthy execution. Across a multi-stage Retrieval-Augmented Generation (RAG) pipeline, that exact same status code can conceal a broken user experience: a vector index returning marginal outdated chunks after a document import, long prompt contexts stalling GPU prefill queues, or a post-schema guardrail quietly routing responses to fallback states. Across our engineering practice at CODLAB on Sovereign AI & Data and ongoing Operations, Maintenance & Evolution, we design observability as a dual discipline uniting stage-level systems telemetry with data-minimised quality monitoring.
At the same time, adopting external cloud LLM tracing tools uncritically carries a substantial data-protection risk: whereas OpenTelemetry GenAI treats message content capture as opt-in, some third-party wrappers or carelessly enabled full-payload options record the system prompt, retrieved passages, and user inputs in cleartext and forward them to external platforms. That anti-pattern creates an unguarded shadow copy of confidential corporate records outside primary tenant boundaries (see Tenant Boundaries in Django).
This study examines how standards-based distributed tracing, phase-split inference latency metrics, statistical shift detection, and obligations under GDPR and the EU AI Act converge inside a two-tier telemetry architecture — extending our analyses of end-to-end data paths across on-premise and EU APIs and structured LLM outputs and domain guardrails.
1. Stage-Level Tracing: Why End-to-End Latency Hides Prefill vs. Decode Bottlenecks
Every request handled by a document assistant traverses input guards, query embedding, hybrid PostgreSQL retrieval, optional reranking, language model inference, and output verification. Attributing latency spikes accurately across service boundaries requires propagating standard trace context headers (traceparent and tracestate) defined by the World Wide Web Consortium (W3C Recommendation, Trace Context, 23.11.2021) while mapping model spans onto the OpenTelemetry semantic conventions maintained in the dedicated GenAI repository (OpenTelemetry, GenAI Semantic Conventions — Spans).
Inside the LLM serving layer itself, a single total-duration timer hides fundamental hardware dynamics. As Amey Agrawal et al. demonstrate in their OSDI analysis of Sarathi-Serve, every inference request splits into two distinct execution regimes: a compute-bound prefill phase that processes all prompt tokens in parallel to produce the first output token (governing Time-to-First-Token, TTFT), followed by a memory-bandwidth-bound decode phase that emits subsequent tokens sequentially (governing Time-Between-Tokens, TBT) (Agrawal et al., OSDI 2024, arXiv:2403.02310). When long RAG contexts enter continuous batches unthrottled, compute-heavy prefill iterations stall concurrent decode iterations and introduce visible generation pauses.
Actionable operational telemetry therefore records at least four distinct timing and volume signals per request trace:
- Retrieval and reranking span durations: Separate latency measurements for
pgvectorcosine search, lexical predicates, and cross-encoder scoring alongside candidate counts. - Server TTFT vs. first stream chunk (
gen_ai.response.time_to_first_chunk): Whereas engine-side TTFT measures queueing and prefill up to the first generated token, the OpenTelemetry attribute records seconds from the start of the logical inference operation until the client receives the first stream event carrying non-empty output (metadata-only or keep-alive events do not count as chunks, whereas authentication, retries, and network transit are included) — an early-warning signal pointing to candidate causes such as long prompts, queue contention, cold prefix caches, or transport latency. - Decode throughput and termination reason: Ratio of
gen_ai.usage.output_tokensto pure decode duration paired withgen_ai.response.finish_reasonsto spot truncated payloads immediately. - Guardrail and abstention counters: Deterministic boolean flags recording whether a response passed verification, abstained due to low retrieval proximity, or escalated to review.
2. Two-Tier Telemetry Without Cleartext Leaks: Codebase Reality vs. Enterprise Target
The foundational rule of privacy-preserving AI observability is straightforward: structural metrics stay enabled by default; cleartext payloads stay disabled by default. Across OpenTelemetry's GenAI semantic conventions, operational metadata such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens sit alongside content-bearing attributes such as gen_ai.input.messages, gen_ai.output.messages, and gen_ai.retrieval.query.text: the OpenTelemetry GenAI attribute registry flags the latter as potentially sensitive, while the “Content capture configuration” section of the GenAI span conventions gates their capture behind an explicit user opt-in.
Here too, we state transparently how our enterprise multi-tenant observability target architecture differs from the compact reference assistant (ragbot) in the codlab.de codebase (whose public chat demo has been switched off since October 2026, while code and data remain retained): in ragbot/models.py and ragbot/views.py, the ChatLog model hashes client IP addresses via salted SHA-256 (ip_hash), accumulates total_input_tokens, total_output_tokens, and error_count, and caps transcripts at MAX_STORED_TURNS = 60; however, it also persists plaintext questions, answers, and cited URLs inside the messages JSON column alongside a 255-character user_agent string — without stage-level latency spans, cosine-distance distributions, or automated retention purge.
For confidential enterprise deployments, our target architecture replaces routine cleartext transcript logging with a strict two-tier model: Tier one emits exclusively content-free spans carrying hashed chunk identifiers, distance metrics, and token counters in continuous operation; Tier two permits short-lived, pre-redacted diagnostic sampling solely under explicit authorization for bounded troubleshooting:
import hashlib
def build_redacted_rag_span(trace_id: str, model: str, distances: list[float],
chunk_ids: list[str], ttft_ms: float, tbt_ms: float,
tokens_in: int, tokens_out: int, abstained: bool) -> dict:
"""Emit operational GenAI telemetry without storing prompt or document cleartext."""
hashed_refs = [hashlib.sha256(cid.encode("utf-8")).hexdigest()[:16] for cid in chunk_ids]
return {
"trace_id": trace_id,
"gen_ai.request.model": model,
"gen_ai.usage.input_tokens": int(tokens_in),
"gen_ai.usage.output_tokens": int(tokens_out),
"rag.chunk_count": len(chunk_ids),
"rag.top_cosine_distance": round(min(distances), 4) if distances else None,
"rag.chunk_ref_hashes": hashed_refs,
"latency.ttft_ms": round(ttft_ms, 2),
"latency.tbt_ms": round(tbt_ms, 2),
"policy.abstained": bool(abstained),
}
3. Detecting Silent Drift Before Domain Teams Lose Trust
Even when GPU hardware and response latencies remain steady, the input and knowledge distributions surrounding an assistant evolve continuously: new product lines introduce unfamiliar nomenclature, updated manuals shift chunk density, or an external provider silently updates model weights behind a version alias. As Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton demonstrate in Failing Loudly, machine learning systems confronting dataset shift typically fail without raising runtime exceptions unless paired with dedicated statistical shift detectors combining dimensionality reduction and two-sample hypothesis testing (Rabanser et al., NeurIPS 2019, arXiv:1810.11953).
Moreover, RAG quality cannot be collapsed into a single scalar: as Shahul Es et al. formalize in RAGAs, retrieval context relevance, answer faithfulness to retrieved passages, and answer relevance to the user query can degrade along separate axes (Es et al., 2023, arXiv:2309.15217). Because live production traffic lacks human ground-truth labels, we monitor three privacy-preserving proxy distributions across our content-free Tier-one spans:
- Cosine-distance distribution shift: A statistically significant upward shift in median top-hit cosine distance across a rolling window can point to incoming query topics missing from the current index or terminology shifts.
- Abstention and guardrail-rejection rate jumps: A spike in abstentions can point to index gaps, whereas a drop in abstentions paired with worse retrieval distances serves as a warning signal for weakened refusal discipline after a prompt or checkpoint update.
- Skew in retrieved chunk-hash entropy: When a handful of chunk hashes dominates retrieval results or newly ingested revisions never appear among top candidates, this provides a cue for possible normalization or embedding mismatches to verify offline.
Whenever any proxy crosses its alert threshold, teams trigger our curated offline benchmark suite (see RAG Evaluation as a Regression Test and our study on statistical multi-run evaluation) rather than browsing live employee conversations.
4. Regulatory Alignment (November 2026): GDPR Data Minimisation vs. EU AI Act Logging Duties
In technical governance debates, a common assumption holds that European AI regulation forces every company to archive full cleartext prompts and completions for years. Examining the binding statutory texts reveals a sharp legal distinction:
- GDPR — Data minimisation, privacy by design, and security of processing: Whenever prompts or retrieved passages touch personal data, Article 5 (data minimisation and storage limitation) and Article 25 GDPR (data protection by design and by default) require limiting processing to what is necessary for the specified purpose, whereas Article 32 GDPR mandates risk-appropriate technical and organisational security measures including pseudonymisation and encryption (Regulation (EU) 2016/679). Indiscriminate cleartext prompt warehousing in external observability tools conflicts with these requirements.
- EU AI Act (consolidated text of 27.07.2026) — strict scoping by risk category: Automatic event-logging capabilities under Article 12 and deployer log-keeping duties under Article 26 belong to Chapter III, Sections 2 and 3 governing high-risk AI systems (Regulation (EU) 2024/1689, consolidated version of 27.07.2026 incorporating Amending Regulation (EU) 2026/1744 of 08.07.2026). Under amended Article 113, those high-risk sections take effect on staggered timelines from 02.12.2027 and 02.08.2028, whereas general duties such as staff AI literacy (Article 4, since 02.02.2025) and transparency obligations (Article 50, since 02.08.2026; for systems placed on the market before 02.08.2026, Article 50(2) applies from 02.12.2026 under Article 111(4)) apply. Standard internal knowledge assistants outside high-risk categories therefore face no statutory mandate for blanket cleartext retention; structured operational spans recording model version, timestamps, index hashes, and verification flags provide auditability alongside data minimisation. To structure alignment with your legal counsel, see our EU AI Act Guide (an engineering worksheet, not legal advice).
5. Reproducible Operations and Canary Protocol for Mid-Market AI Deployments
Methodological note: The operational scenario below is entirely synthetic and illustrates our applied research methodology; it does not describe a specific client engagement or claimed customer measurements.
Suppose a mid-sized engineering services provider operates an internal assistant over quality and maintenance procedures. To catch both inference slowdowns and silent retrieval drift without logging confidential cleartext queries, we combine three reproducible operational loops:
- Rolling distribution comparison (7-day window against 30-day baseline): Automated daily comparison of top-hit cosine distances, abstention ratios, and TTFT/TBT percentiles computed purely from content-free Tier-one spans.
- Synthetic canary runs on every change: Before and after every document re-indexing batch or model update, a fixed set of approved synthetic canary queries executes across 5 repeated runs through the identical pipeline to surface latency or retrieval regressions immediately.
- Strict access and expiry controls on diagnostic escalations: When a specific faulty answer is explicitly flagged by a user for investigation, inspection stays tenant-scoped and time-bounded rather than relying on permanent cleartext archives.
Additional architectural verification patterns are documented across our CODLAB Engineering Notes. If your team is preparing to transition AI prototypes into reliable, privacy-aligned production operations, contact us via Contact & Project Inquiry.
Continue in this research series: For statistical methods quantifying run-to-run output consistency and abstention precision prior to release, read our study on evaluating stochastic AI systems; for auditing all seven hops of the infrastructure data path between local GPUs and European APIs, see on-premise LLMs vs European cloud APIs.
Verified Primary Sources
- World Wide Web Consortium / W3C (2021): Trace Context — W3C Recommendation (23.11.2021), Eds. Kanzhelev, S., McLean, M., Reitbauer, A., Drutu, B., Molnar, N. & Shkuro, Y.
- OpenTelemetry Project: Semantic Conventions for Generative Client AI Spans and Gen AI Attribute Registry (
open-telemetry/semantic-conventions-genai, Cloud Native Computing Foundation). - Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A. & Ramjee, R. (2024): Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024, arXiv:2403.02310).
- Rabanser, S., Günnemann, S. & Lipton, Z. C. (2019): Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019, arXiv:1810.11953).
- Es, S., James, J., Espinosa-Anke, L. & Schockaert, S. (2023): RAGAs: Automated Evaluation of Retrieval Augmented Generation (arXiv:2309.15217).
- Publications Office of the EU (Cellar): Regulation (EU) 2016/679 (General Data Protection Regulation / GDPR), notably Articles 5, 25, and 32.
- Publications Office of the EU (Cellar): Regulation (EU) 2024/1689 (AI Act), consolidated version of 27.07.2026 incorporating Amending Regulation (EU) 2026/1744 of 08.07.2026, notably Articles 4, 12, 26, 50, 111, and 113.
Frequently Asked Questions (FAQ)
No. Latency diagnostics, capacity planning, and drift early warnings rely on content-free structural telemetry such as stage durations, token counts, cosine distances, hashed chunk references, and guardrail flags, keeping confidential prose out of central log stores.
Because the two metrics expose distinct hardware bottlenecks: elevated TTFT points to long prompt contexts or queueing delays during compute-bound prefill, whereas rising TBT indicates memory-bandwidth saturation or batching interference during sequential token decoding.
By tracking statistical shifts across content-free proxy signals: rising median top-hit cosine distances across a rolling window or sudden jumps in abstention rates indicate topic drift or stale indexes and trigger a curated offline regression benchmark.
No. Specific automatic event-logging duties under Articles 12 and 26 target high-risk AI systems on staggered Article 113 timelines. For standard internal knowledge assistants, GDPR data minimisation governs, satisfied cleanly through structured operational metadata.
Related operational guide (German edition): KI im Unternehmen einführen: Vom Arbeitsauftrag zur überprüfbaren Freigabe.