Your corpus is scanned PDFs, spreadsheets and slide decks. How do you ingest it?

There is no single text extractor

The instinct is to find one library that turns any file into a string, and that instinct is why enterprise RAG projects stall in week three. These three formats fail differently and need different handling, so ingestion is a router with a parser per format rather than a pipeline with an extractor at the front.

A PDF is not one format in practice. Some PDFs carry a text layer and extraction is close to lossless, though reading order is still a guess: a two-column page or a header running down the margin comes out interleaved unless the parser reasons about layout. Some PDFs are images of paper with no text layer at all, and those need optical character recognition. The first thing your pipeline must do with a PDF, therefore, is detect which kind it is holding, because running OCR over a digital PDF wastes money and degrades quality, while assuming a text layer that is absent yields an empty document that is silently indexed as empty.

A spreadsheet is a grid of cells whose meaning lives in the relationship between a value and its row and column headers, plus formulas whose results matter and whose text usually does not. A slide deck is a set of loosely ordered text boxes with speaker notes attached, where the deck's argument is often carried by the diagram rather than the words.

flowchart TD
    A[Source file] --> B{Format and text layer}
    B -->|Digital PDF| C[Layout-aware text plus tables]
    B -->|Scanned PDF| D[OCR with per-word confidence]
    B -->|Spreadsheet| E[Sheet and header-aware row extraction]
    B -->|Slide deck| F[Per-slide text plus speaker notes]
    C --> G[Normalised document with section path and anchors]
    D --> G
    E --> G
    F --> G
    G --> H[Chunking and embedding]

The interesting part of that shape is the single normalised representation in the middle. Everything upstream is format-specific and everything downstream is not, which is what lets you add a format later without touching chunking, and lets you re-chunk without re-parsing.

OCR output is text with a confidence attached

The mistake with OCR is treating its output as text. It is a hypothesis, and the engine will tell you how confident it is per word or per line if you ask. Keep that number. It is the only cheap signal you have for the page that came out as noise, and noise is the worst possible input to an embedding model, which will happily produce a vector for gibberish and place it somewhere in the space where it can be retrieved.

So gate on it. A page whose mean confidence falls below a threshold you set by looking at your own corpus goes to a review queue rather than to the index, and the threshold is calibrated by sampling pages and reading them, not by copying a number from somewhere. The same applies to whole documents: a scan that produces almost no extracted characters for a page that plainly has print on it has failed, and it should be reported as a failure rather than contributing an empty chunk.

Image quality upstream matters more than the choice of engine. Deskewing, resolving to a sensible DPI and correcting page rotation move accuracy more than swapping OCR engines does, and rotation in particular is the one that produces total garbage rather than degraded text. Language matters too: an engine configured for English on a document containing accented names will substitute characters confidently, which corrupts exactly the proper nouns that keyword search depends on.

Tables lose their meaning when flattened

The default behaviour of a text extractor on a table is to emit the cells in reading order separated by whitespace, which destroys the thing that made the table useful. Standard 14 4.99 Priority 2 12.99 is not retrievable and not answerable, because no cell knows which column it belongs to.

The handling that works is to extract the table as a structure and then serialise each row as a self-contained statement that repeats the headers. A row rendered as Tier: Standard | Delivery days: 14 | Price: 4.99 embeds meaningfully, matches a question about standard delivery, and hands the model a fact it cannot misattribute. For a wide table, one chunk per row is usually right; for a narrow lookup table, the whole table in one chunk with its caption is better than fragments. Either way the caption and the surrounding sentence that introduced the table belong in the chunk, because that is where the table's subject is stated and it is almost never inside the table.

Spreadsheets add their own traps. Merged cells produce values whose header is two rows up, multiple tables on one sheet mean a single header detection is wrong for most of it, and hidden sheets and columns often contain the workings rather than the answer. A pragmatic rule is to treat each sheet as a document, detect a header row rather than assuming row one, and record the sheet name and cell range on every chunk so a retrieved row can be pointed at.

Structure and metadata have to survive the parse

The reason to keep a normalised intermediate representation is that a retrieved chunk is useless to a reader unless it can be traced back. Every chunk should leave ingestion carrying the source document identifier and version, the section path as the author would name it — document title, heading, sub-heading, or sheet name, or slide number — and an anchor precise enough to open: a page number for a PDF, a cell range for a spreadsheet, a slide index for a deck. That is what turns a citation from a filename into a link somebody can check, and the difference decides whether users trust the system.

This has to happen during the parse, because it cannot be reconstructed afterwards. Once a PDF has become a flat string, the page a sentence came from is gone, and recovering it means parsing again and rebuilding the index. The same is true of anything else you did not capture: the document's owner, its effective date, the permission set on the file. Ingestion is the only moment those are cheaply available.

What to do with the documents that do not parse

Every real corpus contains files that defeat the pipeline: password-protected PDFs, a scan of a handwritten annotation, a deck that is one exported image per slide, a spreadsheet that is a pivot of a data source you do not have. The wrong response is a try/except that logs and continues, because the result is an index that is quietly missing material while reporting success, and the first person to notice is a user who was told the answer is not in the documents.

Make the failure a first-class output. Count parsed, partially parsed and failed documents per run, expose the failed list to whoever owns the content, and treat coverage as a number you report next to retrieval quality. A system that has indexed 82 per cent of the corpus and says so is usable. One that has indexed 82 per cent and claims completeness is not.

Everything you fail to capture while the original file is open — the page number, the header row above a value, the OCR confidence, the owner — cannot be recovered later without parsing and indexing the whole corpus again.