Graph RAG & multimodal retrieval

Graph RAG and multimodal retrieval

Vector similarity retrieves documents that are semantically close to a query in embedding space. It fails predictably on three categories of queries: relational ("which companies acquired subsidiaries of Acme Corp"), multi-hop ("what did the CEO say about the product mentioned in the Q3 earnings call"), and aggregation ("summarize all risk factors across these 50 filings"). These require traversing relationships between entities — a graph structure, not a flat vector space.

When vector-only retrieval fails

The failure pattern: a query requires connecting information from two or more chunks that are not semantically similar to each other or to the query. Consider a corpus containing "Acme Corp acquired Beta Inc in 2023" in one chunk and "Beta Inc developed the Quantum processor" in another. A query asking "what processor technology did Acme gain through acquisitions" requires joining across the acquisition relationship. Neither chunk individually is a strong semantic match for the query, and the two chunks are not semantically similar to each other.

Dense retrieval finds documents near the query in embedding space. It cannot traverse edges between entities stored in different chunks. This is not a chunking problem — no chunking strategy fixes it because the relationship itself spans document boundaries.

GraphRAG: entity extraction and community summaries

Microsoft's GraphRAG (2024) constructs a knowledge graph from the corpus, then uses graph structure to answer queries at multiple levels of abstraction.

The indexing pipeline has five stages:

• Entity extraction: for each chunk, an LLM extracts entities (people, organizations, locations, concepts, events) and relationships between them (causal, temporal, hierarchical, associative). Typical yield: 5–15 entities per 512-token chunk.
• Entity resolution: deduplicate entities across chunks using fuzzy string matching and LLM-based coreference. "Microsoft Corp", "MSFT", and "Microsoft" merge into one node.
• Graph construction: nodes are entities, edges are typed relationships. Each edge carries the source chunk ID for provenance.
• Community detection: apply Leiden clustering to identify groups of densely-connected entities. These communities represent coherent subtopics.
• Community summarization: generate LLM summaries for each community at multiple hierarchy levels (leaf communities, mid-level, global). These summaries enable answering broad questions without traversing the full graph.

At query time, two retrieval modes:

• Local search: identify query entities via embedding similarity, retrieve their 1–2 hop neighborhood in the graph, synthesize an answer from the subgraph context. Best for specific questions about known entities.
• Global search: map the query to relevant community summaries, generate a high-level answer from those summaries. Best for broad, theme-level questions ("what are the main risks discussed in these documents").

GraphRAG reported significant improvements on global sensemaking queries: +30–40% on comprehensiveness metrics compared to naive RAG when tested on podcast transcripts and news corpora. On local factoid queries, the advantage is smaller — standard vector retrieval is often sufficient for single-hop questions.

Hybrid retrieval: dense + sparse + graph

Combine all three signal types using weighted Reciprocal Rank Fusion:

RRF_score(d) = sum(weight_i / (k + rank_i(d))) for each retrieval system i, with k = 60 (standard constant).

A typical starting weight configuration:

• Dense (embedding similarity): weight 0.4
• Sparse (BM25 keyword match): weight 0.3
• Graph (entity-relationship traversal): weight 0.3

Tune weights on your eval set. Increase graph weight for corpora with dense entity relationships (legal contracts, biomedical literature, corporate filings). Increase sparse weight for corpora with heavy jargon or acronyms. The weights are not fixed — they represent your belief about which signal is most informative for your specific domain.

Implementation: entity extraction and graph query

import anthropic
from neo4j import GraphDatabase
import json
from typing import Any

client = anthropic.Anthropic()

EXTRACTION_PROMPT = """Extract entities and relationships from this text.

Return valid JSON with two keys:
- "entities": list of {{"name": str, "type": str}} where type is one of: person, organization, concept, event, location, product
- "relationships": list of {{"source": str, "target": str, "relation": str}}

Text: {text}

Return ONLY the JSON, no other text."""


def extract_entities(text: str) -> dict[str, Any]:
    """Extract entities and relationships from a text chunk."""
    response = client.messages.create(
        model="claude-haiku-4-20250414",
        max_tokens=1000,
        messages=[{
            "role": "user",
            "content": EXTRACTION_PROMPT.format(text=text)
        }]
    )
    return json.loads(response.content[0].text)


def build_knowledge_graph(chunks: list[str], neo4j_uri: str,
                          neo4j_auth: tuple[str, str]):
    """Extract entities from all chunks and build a Neo4j graph."""
    driver = GraphDatabase.driver(neo4j_uri, auth=neo4j_auth)

    with driver.session() as session:
        for chunk_idx, chunk in enumerate(chunks):
            extracted = extract_entities(chunk)

            for entity in extracted["entities"]:
                session.run(
                    "MERGE (e:Entity {name: $name}) "
                    "ON CREATE SET e.type = $type, e.sources = [$source] "
                    "ON MATCH SET e.sources = e.sources + $source",
                    name=entity["name"],
                    type=entity["type"],
                    source=f"chunk_{chunk_idx}"
                )

            for rel in extracted["relationships"]:
                session.run(
                    "MATCH (a:Entity {name: $source}) "
                    "MATCH (b:Entity {name: $target}) "
                    "MERGE (a)-[r:RELATES {type: $relation}]->(b) "
                    "ON CREATE SET r.chunk_source = $chunk",
                    source=rel["source"],
                    target=rel["target"],
                    relation=rel["relation"],
                    chunk=f"chunk_{chunk_idx}"
                )

    driver.close()


def graph_retrieve(query: str, neo4j_uri: str, neo4j_auth: tuple[str, str],
                   hops: int = 2, limit: int = 20) -> list[dict]:
    """Retrieve subgraph relevant to a query."""
    extracted = extract_entities(query)
    entity_names = [e["name"] for e in extracted.get("entities", [])]

    if not entity_names:
        return []

    driver = GraphDatabase.driver(neo4j_uri, auth=neo4j_auth)
    with driver.session() as session:
        result = session.run(
            f"MATCH path = (e:Entity)-[r*1..{hops}]-(connected:Entity) "
            "WHERE e.name IN $names "
            "UNWIND relationships(path) AS rel "
            "RETURN startNode(rel).name AS source, "
            "       endNode(rel).name AS target, "
            "       rel.type AS relation, "
            "       rel.chunk_source AS chunk "
            "LIMIT $limit",
            names=entity_names, limit=limit
        )
        triples = [record.data() for record in result]

    driver.close()
    return triples

When graph retrieval is worth the complexity

Graph retrieval adds substantial overhead: LLM entity extraction at index time (one call per chunk, same cost profile as contextual retrieval), a separate graph database to operate, and query-time entity extraction + traversal adding 200–500ms. Justify it when:

• Your eval set shows retrieval failures specifically on multi-hop or relational queries that vector search cannot resolve
• The corpus has dense entity relationships (>5 entities per chunk on average)
• Users ask comparison or aggregation questions across entities
• You need explainable retrieval paths (the graph provides a traceable chain: entity A → relationship → entity B → relationship → entity C)

Do not add graph retrieval because it sounds sophisticated. Audit your actual retrieval failures first. If 80% of failures are caused by poor chunking, missing context, or keyword mismatch, fix those with contextual retrieval, BM25 hybrid, or reranking — cheaper and faster to implement, cheaper to operate.

Multimodal RAG

Documents contain more than text: architecture diagrams, data flow charts, tables of specifications, product photographs. Multimodal RAG indexes non-text elements alongside text so that a text query can surface relevant images, and image content contributes to text-query retrieval.

Approaches by modality:

• Images: embed using CLIP (ViT-L/14, 768 dimensions) or SigLIP into a shared vector space with text. Cross-modal retrieval lets a text query like "network architecture diagram" surface the relevant figure. Retrieve the image directly — pass it to the generation LLM as a vision input alongside text context.
• Tables: extract to structured text (markdown format or JSON rows) using a specialized model (table-transformer) or a vision-capable LLM. Embed the structured text as a regular chunk. At generation time, pass both the structured text and the original table image for maximum fidelity.
• Charts and diagrams: generate detailed text descriptions via a vision-language model (Claude with vision, GPT-4o), then embed those descriptions. Store the original image URL/path for retrieval-time inclusion in the generation prompt.

The generation step requires a vision-capable model. Pass retrieved images as base64-encoded content or URLs in the message alongside text context. Models like Claude handle interleaved text and image inputs natively.

Key takeaways

• Vector similarity fails on relational, multi-hop, and aggregation queries that require traversing entity relationships between separate chunks
• GraphRAG (Microsoft, 2024) builds a knowledge graph and uses community summaries for global sensemaking queries (+30–40% comprehensiveness over flat RAG)
• Hybrid retrieval (dense + sparse + graph) with weighted RRF combines all three signal types; tune weights per domain
• Entity extraction at index time costs one LLM call per chunk — same cost profile as contextual retrieval
• Multimodal RAG indexes images (CLIP/SigLIP embeddings), tables (structured text extraction), and charts (vision-model descriptions) alongside text
• Add graph retrieval only after auditing retrieval failures — if the dominant failure mode is not relational, cheaper solutions exist