Loading & parsing real documents

What parsing has to produce

Documents don't store meaning — they store layout instructions. A PDF is a sequence of content streams that position glyphs on a canvas. An HTML page is a DOM tree optimized for rendering, not for semantic extraction. Before any of these can enter a retrieval pipeline, they must be reduced to text blocks — each carrying:

• The extracted text content
• A structural type (paragraph, heading, list item, table cell, code block)
• Source metadata (filename, page number, section path)

Metadata travels with the text through chunking and indexing. At retrieval time, it enables filtered search ("only search the 2024 annual report") and source attribution ("page 14, section 3.2"). Lose it here and you cannot recover it later.

Parsing quality sets an absolute ceiling on retrieval quality. If a table flattens into a meaningless string, no embedding model can recover the relationships. If a section heading is lost, a chunker cannot respect document structure. If an OCR pass misreads a decimal, the downstream answer is wrong.

PDF extraction

PDFs are the hardest common format because the format encodes appearance, not content. A sentence that looks contiguous on the page may be stored as three separate text-drawing operations at different coordinates. Tables have no <table> element — they are positioned text fragments.

import pymupdf  # PyMuPDF

def extract_pdf(path: str) -> list[dict]:
    doc = pymupdf.open(path)
    blocks = []
    for page_num, page in enumerate(doc):
        # get_text("blocks") returns (x0, y0, x1, y1, text, block_no, type)
        for block in page.get_text("blocks"):
            text = block[4].strip()
            if not text or block[6] != 0:  # type 0 = text, 1 = image
                continue
            blocks.append({
                "text": text,
                "page": page_num + 1,
                "source": path,
            })
    return blocks

get_text("blocks") preserves reading order for single-column documents. Multi-column layouts, academic papers, and complex tables need more sophisticated tools.

Table extraction is where most PDF parsers break. pdfplumber reconstructs grid geometry from ruled lines or whitespace alignment:

import pdfplumber

def extract_tables(path: str) -> list[dict]:
    results = []
    with pdfplumber.open(path) as pdf:
        for page_num, page in enumerate(pdf.pages):
            for table in page.extract_tables():
                headers = table[0]
                rows = table[1:]
                md = "| " + " | ".join(str(h) for h in headers) + " |\n"
                md += "| " + " | ".join("---" for _ in headers) + " |\n"
                for row in rows:
                    md += "| " + " | ".join(str(c or "") for c in row) + " |\n"
                results.append({
                    "text": md,
                    "type": "table",
                    "page": page_num + 1,
                    "source": path,
                })
    return results

Converting tables to markdown preserves column relationships. A flattened string — "Name Revenue Growth Acme 4.2M 12%" — destroys them.

HTML extraction

HTML is structurally richer than PDF but noisier: navigation bars, footers, ads, cookie banners, and boilerplate surround the content. The extraction problem is content isolation.

import trafilatura

def extract_html(url: str) -> dict | None:
    downloaded = trafilatura.fetch_url(url)
    if not downloaded:
        return None
    text = trafilatura.extract(
        downloaded, include_tables=True, include_links=True
    )
    if not text:
        return None
    return {"text": text, "source": url, "type": "html"}

trafilatura uses heuristics and statistical models to identify the main content region. For known site structures, a targeted CSS selector with BeautifulSoup gives more control:

from bs4 import BeautifulSoup
import httpx

def extract_docs_page(url: str) -> list[dict]:
    resp = httpx.get(url)
    soup = BeautifulSoup(resp.text, "html.parser")
    main = soup.select_one("article, main, .content")
    if not main:
        return []
    sections = []
    for elem in main.find_all(["h1", "h2", "h3", "p", "pre", "li"]):
        text = elem.get_text(separator=" ", strip=True)
        if text:
            sections.append({"tag": elem.name, "text": text})
    return [{"sections": sections, "source": url}]

Preserving heading hierarchy in the output matters for structure-aware chunking (next lesson).

Scanned documents and OCR

Scanned PDFs contain raster images, not text streams — get_text() returns nothing. OCR converts the image back to text, with lossy accuracy.

• Tesseract — open-source, batch-capable, adequate for clean scans of printed text.
• Google Document AI / AWS Textract — cloud APIs with higher accuracy on complex layouts, table detection, and handwriting.
• Docling (IBM) — end-to-end document understanding: layout detection, table recognition, reading-order reconstruction.
• marker — PDF-to-markdown with layout preservation, GPU-accelerated.

OCR accuracy degrades on poor scans, handwriting, and non-Latin scripts. For high-stakes retrieval (legal, medical, financial), validate a sample of OCR output before trusting the pipeline.

The parser landscape

No single parser handles all formats well. Production pipelines route by file type:

• PyMuPDF — fast text extraction, good reading order. Weak on tables.
• pdfplumber — strong table extraction from ruled/whitespace tables. Slower.
• Unstructured — multi-format (PDF, DOCX, HTML, images), layout detection. Heavy dependencies, variable quality.
• Docling — end-to-end document understanding, table+figure detection. Newer, less battle-tested.
• LlamaParse — LLM-assisted parsing for complex layouts. API dependency, cost per page.
• trafilatura — web content extraction with boilerplate removal. HTML only.

from pathlib import Path

def parse_document(path: str) -> list[dict]:
    ext = Path(path).suffix.lower()
    if ext == ".pdf":
        blocks = extract_pdf(path)
        blocks.extend(extract_tables(path))
        return blocks
    elif ext in (".html", ".htm"):
        result = extract_html(path)
        return [result] if result else []
    elif ext in (".md", ".txt"):
        return [{"text": Path(path).read_text(), "source": path}]
    else:
        raise ValueError(f"Unsupported format: {ext}")

Add logging to this router: track parse success rates and mean text-length-per-page. Anomalously low values suggest OCR is needed.

Failure modes that silently poison retrieval

These don't raise exceptions — they produce plausible but wrong text:

• Header/footer repetition. Every page appends "Acme Corp — Confidential" and a page number. These fragments end up in chunks and pollute similarity scores.
• Column merging. A two-column layout extracts as alternating fragments from each column, interleaving unrelated sentences.
• Table flattening. Column relationships are destroyed when cells are concatenated without structure.
• Hyphenation and line breaks. Words split across lines produce "imple-\nmentation" instead of "implementation".
• Ligature and encoding errors. Characters like fi (fi ligature) or — (em dash) may extract as garbage depending on font encoding.
• Image-only content. Diagrams, charts, and screenshots are missed by text extraction entirely.

When retrieval quality drops for a specific document set, the parser is the first place to look.

Key takeaways

• Parsing converts layout-encoded documents into text blocks with structural type and source metadata — this step sets the ceiling on everything downstream.
• PDFs encode appearance, not content; tables, multi-column layouts, and scanned pages each need specialized handling.
• No single parser covers all formats — route by file type and validate output quality before trusting the pipeline.
• Silent parsing failures (column merging, table flattening, header repetition) are the most common root cause of retrieval bugs.