CODLAB / Blog · AI & integration

Document Ingestion for RAG: PDF Layout Recovery, OCR Error Classes and Table Structure Preservation

14 Nov 2026 CODLAB Team 13 min

As of November 2026 · CODLAB Team

Direct answer: Reliable enterprise Retrieval-Augmented Generation (RAG) is won or lost before chunking, inside document ingestion: only when layout analysis reconstructs multi-column reading order, OCR routing activates selectively on scanned pages, tables keep column headers bound to every cell, and page provenance stays versioned via SHA-256 can hybrid retrieval surface trustworthy engineering facts.

Why RAG Accuracy Is Decided Inside the PDF Parser Before the First Embedding

Architectural discussions around internal knowledge assistants typically revolve around vector indexes, embedding dimensions, and prompt templates. Yet as soon as a system must index real operational repositories — maintenance manuals, inspection certificates, safety datasheets, bills of quantities, or scanned legacy archives — an often underestimated failure mode sits upstream of retrieval itself: inside document ingestion. In our applied research and engineering work at CODLAB on Sovereign AI & Data, we treat heterogeneous document parsing as a first-class engineering stage governed by measurable acceptance criteria.

The root cause is structural. Unlike HTML or Markdown, which encode headings, paragraphs, and table cells explicitly in markup, the Portable Document Format (PDF) was designed for fixed-layout visual reproduction. As Lukas Blecher et al. note in their work on neural optical understanding for academic documents, the PDF format strips away semantic structural information, particularly for mathematical expressions (Blecher et al., 2023, Nougat, arXiv:2308.13418). Extracting raw PDF text streams line by line and feeding them straight into an embedding model indexes interleaved columns, orphaned footnotes, and headerless numeric rows — structural corruption that no downstream reranker can repair.

This study extends our architectural analysis of structure-aware chunking, hybrid retrieval, and cross-encoder reranking to the upstream ingestion pipeline: how to engineer and test layout segmentation, optical character recognition (OCR), table topology preservation, and untrusted-file security boundaries before indexing begins.

1. Coordinates Instead of Reading Order: Layout Segmentation on Heterogeneous Pages

Programmatic PDFs rarely store logical paragraphs; instead, they place glyph runs and vector lines at two-dimensional coordinates on fixed pages. Naive stream extractors emit characters in whatever internal order the authoring software wrote them to the file. Across technical and legal documents, that shortcut produces three recurring structural defects:

  • Cross-column interleaving: On two-column datasheets or regulatory standards, text from the left and right columns at the same vertical coordinate merges into a single garbled sentence.
  • Running header and footer contamination: Repeating document codes, approval stamps, and page numbers land straight inside sentences that wrap across a page break.
  • Detached callouts and footnotes: Safety warnings in side boxes or qualifying footnotes beneath specification tables lose their structural link to the clause they restrict.

Production document pipelines therefore place explicit visual-geometric layout segmentation ahead of text extraction. As Birgit Pfitzmann et al. demonstrate with DocLayNet across 80863 human-annotated pages spanning 11 distinct layout classes, layout detectors trained solely on uniform academic repositories suffer sharp accuracy drops when confronted with financial reports, manuals, patents, laws and regulations, or tenders (Pfitzmann et al., KDD 2022, arXiv:2206.01062). Multimodal transformers such as LayoutLMv3 address this variance by pre-training jointly over text tokens, bounding boxes, and image patches through unified masking and word-patch alignment (Huang et al., ACM Multimedia 2022, arXiv:2204.08387).

In enterprise deployments, this means every page region is classified first (heading, body prose, table, figure caption, running header/footer, footnote), non-substantive page chrome is stripped, and a deterministic reading sequence is reconstructed across columns and page boundaries before any chunk boundary is computed.

2. Scanned Archives and Asymmetric OCR Error Propagation in Hybrid Retrieval

Long-lived industrial and legal archives frequently mix born-digital PDFs with rasterized scans — signed addenda, stamped supplier test reports, or legacy drawings — alongside older PDFs carrying corrupted OCR text layers from decade-old scanner workflows. A resilient ingestion pipeline must neither trust embedded PDF text blindly nor rasterize every born-digital page through OCR unnecessarily.

In our target architecture, the pipeline routes pages adaptively: whenever a page contains a valid programmatic character stream with clean glyph mappings and expected character density, native extraction preserves exact symbols; when character streams are missing or exhibit broken font encodings (such as unmapped replacement glyphs), that specific page routes to an OCR engine. From an applied RAG perspective, residual recognition errors — quantified via Character Error Rate (CER) and Word Error Rate (WER) — degrade the two channels of hybrid retrieval asymmetrically:

  1. Fragility of exact lexical matching: In PostgreSQL Full Text Search using tsvector and GIN indexes (PostgreSQL Documentation, Chapter 12: Full Text Search), a single character substitution inside an identifier — such as confusing 0 with O or 1 with I in a part code — causes a complete lexical miss unless backed by trigram similarity or domain normalization.
  2. Silent vector drift on technical notation: Dense bi-encoder models (Karpukhin et al., 2020, arXiv:2004.04906) map queries and passages into continuous vector space; even keyword typos in queries can significantly reduce dense retrieval and ranking effectiveness unless typo-aware training is used (Zhuang & Zuccon, EMNLP 2021, arXiv:2108.12139). How strongly end-of-line split compounds or OCR-corrupted engineering units degrade passage recall on a target corpus is treated as an empirical hypothesis measured by the protocol in Section 5.

Consequently, our target ingestion schema records the page-level extraction path (native PDF stream vs. OCR fallback) alongside deterministic de-hyphenation and Unicode unit normalization. How relational metadata predicates subsequently interact with vector indexing is detailed in our note pgvector with Django: Checking Filters and Recall.

3. Preserving Table Topology: Binding Row and Column Headers to Every Cell

Across engineering datasheets, maintenance schedules, fee tables, and chemical compatibility matrices, authoritative answers reside in two-dimensional tables rather than narrative paragraphs. Flattening a table into whitespace-separated tokens leaves numbers inside the chunk while severing their link to column headers, units of measure, and row conditions. When a technician queries the allowable test pressure for a specific nominal diameter, the language model receives an unlabeled sequence of numbers and risks citing an adjacent column.

Document AI research therefore treats Table Structure Recognition as a dedicated structural inference task distinct from plain OCR. Brandon Smock et al. introduced PubTables-1M with a canonicalization procedure to eliminate oversegmentation in the ground-truth annotations of prior datasets, including multi-line and spanning headers (Smock et al., CVPR 2022, arXiv:2110.00061). Ahmed Nassar et al. designed TableFormer with a transformer structure decoder paired with a table-cell object detector, enabling programmatic PDFs to pull exact cell text directly from the PDF source stream rather than re-running error-prone OCR (Nassar et al., CVPR 2022, arXiv:2203.01017). Open-source conversion toolkits such as Docling combine DocLayNet layout segmentation and TableFormer structure recovery within a self-contained commodity-hardware pipeline (Auer et al., 2024, arXiv:2408.09869).

Here again, we distinguish transparently between our enterprise document-ingestion target architecture and the intentionally lean reference assistant (ragbot) in the codlab.de codebase (whose public chat demo has been switched off since October 2026, while code and data remain retained): its rag_ingest command ingests curated HTML pages from our own website, strips navigation chrome via BeautifulSoup, flattens main content into space-separated prose, and slices windows up to 900 characters with 150 characters of overlap — without PDF parsing, OCR routing, or table graph recovery. What ragbot implements directly in code is deterministic SHA-256 change detection via content_hash.

In our enterprise target architecture, we extend that SHA-256 idempotency pattern to reconstructed table row groups: every extracted table is partitioned into self-contained blocks repeating document identifier, column headers, and page number, hashing the canonical JSON cell array while escaping pipes and line breaks in Markdown:

import hashlib, json

def _md_cell(val):
    txt = str(val).strip().replace("\\", "\\\\").replace("|", "\\|")
    return txt.replace("\r\n", "\\n").replace("\r", "\\n").replace("\n", "\\n")

def serialize_table_blocks(doc_id, rev, page_no, section_path, headers, rows, group_size=12):
    """Hash canonical cell structure and emit pipe-safe Markdown row groups."""
    hdrs = [str(h).strip() for h in headers]
    header_line = " | ".join(_md_cell(h) for h in hdrs)
    sep_line = " | ".join("---" for _ in hdrs)
    blocks = []
    for start in range(0, len(rows), group_size):
        batch = [[str(c).strip() for c in r] for r in rows[start:start + group_size]]
        if any(len(r) != len(hdrs) for r in batch):
            raise ValueError("Row width does not match table headers")
        canon = {"doc_id": doc_id, "rev": rev, "page": page_no,
                 "section": section_path, "headers": hdrs, "rows": batch}
        digest = hashlib.sha256(
            json.dumps(canon, ensure_ascii=False, sort_keys=True).encode("utf-8")
        ).hexdigest()
        body_lines = [" | ".join(_md_cell(c) for c in r) for r in batch]
        markdown_table = "\n".join([
            f"Source: {doc_id} (Rev. {rev}, p. {page_no}) — {section_path}",
            header_line, sep_line, *body_lines,
        ])
        blocks.append({"page": page_no, "content": markdown_table, "sha256": digest})
    return blocks

4. Security and Revision Boundaries When Ingesting Untrusted Files

Document ingestion is simultaneously a data-quality stage and a security boundary. Whenever an organization automatically indexes external supplier certificates, tender specifications, application dossiers, or email attachments, untrusted third-party inputs enter the retrieval corpus. Germany's Federal Office for Information Security (BSI) explicitly catalogs this threat vector as Indirect Prompt Injection (risk R28) originating from external information sources (BSI, Generative KI-Modelle, Version 2.0, 17.01.2025).

PDF internals create subtle hiding places that pass casual visual review unnoticed: white text rendered on a white background, zero-point font runs, occluded text layers behind vector graphics, or hidden form and metadata strings. A naive stream extractor reads those invisible instructions alongside visible prose and injects them straight into downstream prompts. Downstream isolation rules in retrieval and prompt assembly are covered in our note RAG: Sources as Untrusted Data and our study on Indirect Prompt Injection and source boundaries.

In our target architecture, the ingestion layer therefore specifies three structural safeguards:

  • Stripping non-visible text objects: Glyph runs positioned outside the crop box, rendered at near-zero font size, or carrying zero opacity are discarded or flagged during parsing rather than silently indexed.
  • Relational revision and validity metadata: Every chunk stores document identifier, revision code, approval state, and page number as mandatory relational columns in PostgreSQL so superseded manuals can be deterministically retired when a new revision arrives.
  • Quarantine on parser anomalies: Encrypted containers, embedded scripts, or extreme discrepancies between rendered page visuals and raw character streams halt automatic indexing into an isolated review queue.

5. Reproducible Pre-Indexing Verification Protocol for Technical Decision-Makers

Methodological note: The evaluation protocol below is entirely synthetic and illustrates our applied research methodology; it does not describe a specific client engagement or claimed customer measurements.

Suppose a mid-sized industrial equipment manufacturer wants to verify whether a mixed PDF archive — engineering standards (German), scanned supplier mill certificates (Italian), and electronic component datasheets (English) — can be ingested reliably before building a RAG assistant. Before connecting any generative model, we evaluate ingestion fidelity in a four-step experiment:

  1. Stratified page sample (30 to 50 pages): Select challenging real pages across all 3 document classes, intentionally including two-column layouts, multi-row table headers, footnotes, and skewed stamped scans.
  2. Establish ground-truth reference transcriptions: Manually verify the exact target output for that sample — clean reading-order prose free of running headers plus exact row-and-column key-value pairs for every table.
  3. Measure parser candidates independently: Evaluate Character Error Rate (CER) on scanned pages, running-header leakage rate, and the percentage of table cells preserving both column header and measurement unit.
  4. Quantify downstream retrieval impact: Execute identical engineering lookup queries in PostgreSQL against naive text extraction versus structure-preserving table and layout blocks to measure passage recall differences directly.

You can explore how we scope structured document pilots on our Document Assistants page and across our CODLAB Engineering Notes. To assess which ingestion and indexing architecture fits your own document archive, contact us via Contact & Project Inquiry.

Continue in this research series: To see how hybrid PostgreSQL search, cross-encoder reranking, and cosine distance cutoffs build upon clean document blocks, read our foundational study on applied RAG architecture; for architectural defenses against hostile instructions hidden inside retrieved files, see RAG security and Indirect Prompt Injection.

Verified Primary Sources

Frequently Asked Questions (FAQ)

Why is basic PDF text extraction usually insufficient for technical RAG systems?

PDF files position glyphs by page coordinates rather than storing logical paragraphs or relational tables. Without layout and table-structure recovery, multi-column text interleaves, running headers split sentences, and numeric values lose their column headers and units.

Should an enterprise run OCR indiscriminately across every page of every PDF?

No. When a page already carries a clean programmatic text stream, reading native characters is faster and avoids OCR substitutions on part numbers and engineering units. OCR should trigger selectively on scanned pages or corrupted font encodings.

How should wide or multi-page specification tables be chunked for vector search?

Rather than flattening tables into plain text, table-structure recognition reconstructs rows and columns and serializes bounded row groups. Each group repeats the document identifier, page number, and full column headers so every cell retains its exact meaning.

What security risks arise already during PDF ingestion?

External PDFs can hide white-on-white text, zero-point fonts, or occluded text layers carrying Indirect Prompt Injection payloads (BSI R28) invisible to human reviewers. Ingestion pipelines must strip or quarantine non-visible text objects prior to indexing.

Related operational guide (German edition): KI Dokumentenextraktion mit OCR: strukturierte Daten aus Rechnungen, Verträgen und Formularen — lokal statt Cloud.

Share this article

Comments

No comments yet. Be the first to comment!

Leave a comment

Will not be published

Related Articles

Project on your mind?

We develop tailored software and AI solutions. Let's start your next project together.

Free initial consultation