RAG with a framework

RAG with a framework

LlamaIndex and LangChain wrap the chunk-embed-search-generate loop into higher-level abstractions: document loaders, node parsers, retrievers, and query engines. This lesson rebuilds the same RAG application from the previous two lessons using LlamaIndex, notes where LangChain diverges, and examines what frameworks hide.

What the framework provides

A framework turns the five-function pipeline (load, chunk, embed, search, generate) into configuration:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./documents").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

response = query_engine.query("How does authentication work?")
print(response)

Five lines. Behind them, LlamaIndex:

• Parsed each file in ./documents using a format-appropriate loader (plain text, PDF via pypdf, HTML via beautifulsoup4)
• Split each document into nodes using SentenceSplitter (default chunk_size=1024 tokens, chunk_overlap=200)
• Embedded each node with OpenAI text-embedding-ada-002 (the default — you should override this)
• Stored embeddings in an in-memory SimpleVectorStore
• At query time: embedded the query, found the top-2 most similar nodes (default similarity_top_k=2), built a prompt with those nodes, sent it to gpt-3.5-turbo (the default — override this too)

Those defaults are worth knowing because they're what you get if you don't configure anything, and several of them (ada-002, gpt-3.5-turbo, top_k=2) are outdated or insufficient for production use.

Configuring the components

Override the defaults to match production requirements:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.anthropic import Anthropic

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    dimensions=1536,
)
Settings.llm = Anthropic(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
)
Settings.node_parser = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50,
)

documents = SimpleDirectoryReader("./documents").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("How does authentication work?")

Settings is a global configuration object. Every component that needs an embedding model or LLM reads from it unless you pass one explicitly. This is convenient for prototyping and dangerous in production — a stray Settings.llm = ... in one module silently changes behavior everywhere.

Plugging in pgvector

Replace the in-memory store with pgvector to persist embeddings across restarts:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.postgres import PGVectorStore

vector_store = PGVectorStore.from_params(
    database="ragdb",
    host="localhost",
    password="rag",
    port=5432,
    user="rag",
    table_name="llama_chunks",
    embed_dim=1536,
    hnsw_kwargs={
        "hnsw_m": 16,
        "hnsw_ef_construction": 200,
        "hnsw_ef_search": 100,
        "hnsw_dist_method": "vector_cosine_ops",
    },
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
documents = SimpleDirectoryReader("./documents").load_data()
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

LlamaIndex creates the table and HNSW index automatically. Subsequent runs can load the existing index without re-ingesting:

index = VectorStoreIndex.from_vector_store(vector_store)
query_engine = index.as_query_engine(similarity_top_k=5)

Compare this to the manual pgvector setup from L15 — the framework handles table creation, embedding insertion, and index management. The tradeoff: you don't control the schema. LlamaIndex's PGVectorStore creates its own table structure; your metadata JSONB column is there but follows LlamaIndex's naming conventions, not yours.

Hybrid retrieval with reranking

Wire in BM25 sparse retrieval alongside vector search, then rerank the merged results. This combines keyword precision with semantic recall:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.retrievers import QueryFusionRetriever

documents = SimpleDirectoryReader("./documents").load_data()
index = VectorStoreIndex.from_documents(documents)

vector_retriever = index.as_retriever(similarity_top_k=10)

bm25_retriever = BM25Retriever.from_defaults(
    nodes=index.docstore.docs.values(),
    similarity_top_k=10,
)

hybrid_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    num_queries=1,
    similarity_top_k=10,
)

reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_n=5,
)

from llama_index.core.query_engine import RetrieverQueryEngine

query_engine = RetrieverQueryEngine.from_args(
    retriever=hybrid_retriever,
    node_postprocessors=[reranker],
)

response = query_engine.query("How does HNSW indexing work?")
print(response)

Without a framework, hybrid retrieval + reranking is ~80 lines of manual code (merge result sets, normalize scores, load the cross-encoder, re-score). With the framework, it's configuration. The same pipeline in LangChain uses EnsembleRetriever and CrossEncoderReranker from langchain_community:

from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

loader = DirectoryLoader("./documents")
docs = loader.load()

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
faiss_store = FAISS.from_documents(docs, embeddings)
vector_retriever = faiss_store.as_retriever(search_kwargs={"k": 10})

bm25_retriever = BM25Retriever.from_documents(docs, k=10)

ensemble = EnsembleRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    weights=[0.5, 0.5],
)

cross_encoder = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
reranker = CrossEncoderReranker(model=cross_encoder, top_n=5)

retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=ensemble,
)

results = retriever.invoke("How does HNSW indexing work?")

LangChain's approach differs: it uses EnsembleRetriever with explicit weight parameters (0.5/0.5 gives equal weight to vector and BM25 results) and wraps reranking in a ContextualCompressionRetriever. Both achieve the same result — the API surface diverges but the underlying operations are identical.

Metadata filtering through the framework

Both frameworks support metadata filtering, but the syntax wraps the SQL you'd write directly:

from llama_index.core.vector_stores.types import MetadataFilter, MetadataFilters

filters = MetadataFilters(
    filters=[
        MetadataFilter(key="source_type", value="api_docs"),
        MetadataFilter(key="version", value="3.0", operator=">="),
    ]
)

query_engine = index.as_query_engine(
    similarity_top_k=5,
    filters=filters,
)

response = query_engine.query("How do I configure OAuth?")

This generates the same WHERE metadata->>'source_type' = 'api_docs' clause you'd write in raw SQL. The framework abstraction is useful when targeting multiple vector stores — the same MetadataFilters object works with pgvector, Pinecone, Weaviate, and Qdrant.

When NOT to use a framework

Frameworks solve real problems — loader compatibility, chunking strategies, retriever composition. They also create real problems:

• Debugging opacity. When retrieval quality drops, you need to inspect the chunks, embeddings, and similarity scores at each stage. Frameworks make this harder. LlamaIndex's query_engine.query() is a single call that hides 6+ internal steps. Finding which step failed means digging into callback handlers or response synthesis internals.
• Version churn. LlamaIndex made a breaking change to its core abstractions in v0.10 (the llama-index-core split). LangChain has broken backward compatibility across major versions multiple times. Pinning versions locks you out of bug fixes; upgrading breaks your pipeline. Production code that wraps these frameworks needs its own abstraction layer on top — at which point the framework's value proposition weakens.
• Implicit behavior. SentenceSplitter silently truncates chunks that exceed the token limit. VectorStoreIndex.from_documents embeds everything on construction — there's no lazy or incremental path without switching to a lower-level API. The defaults are reasonable for demos but rarely match production requirements.
• Dependency weight. pip install llama-index pulls in 50+ packages. langchain is similar. In containerized deployments, this increases image size and build time. In serverless (Lambda, Cloud Run), cold-start latency grows measurably.

The decision framework

Use a framework when:

• You need multiple loaders. If your corpus spans PDFs, HTML, Confluence, Notion, and Slack, writing loaders from scratch is wasted effort. LlamaIndex has 100+ loaders via LlamaHub.
• You're prototyping. Going from idea to working demo in an afternoon matters. The framework's defaults are fine for evaluation.
• You want retriever composition. Hybrid retrieval + reranking + metadata filtering with hot-swappable vector stores is genuinely complex to build from scratch.

Skip the framework when:

• You have one document format and one vector store. The manual pipeline from L14 and L15 is simpler, faster, and easier to debug.
• You need fine-grained control over ingestion. Content hashing, incremental updates, orphan pruning — the ingestion pipeline from L15 is purpose-built for corpus management. Frameworks treat ingestion as a bulk operation.
• You're deploying to production. Pin your dependencies, own your abstractions, and avoid coupling your pipeline to a framework's release cadence.

A common pattern in production: use the framework during prototyping to discover which components you need (sentence-window retrieval? reranking? query decomposition?), then reimplement the winning configuration with direct library calls (openai, psycopg2, sentence-transformers) for the deployed system.

Key takeaways

• LlamaIndex and LangChain reduce the RAG pipeline to configuration: loaders, splitters, retrievers, and query engines compose with minimal code.
• Override the defaults — both frameworks ship with outdated models (ada-002, gpt-3.5-turbo) and conservative retrieval settings (top_k=2).
• Hybrid retrieval (vector + BM25) with cross-encoder reranking is the strongest general retrieval pattern. Frameworks make this a configuration problem rather than an implementation problem.
• Framework overhead matters: dependency weight, version churn, and debugging opacity are real costs in production.
• A practical pattern: prototype with the framework, then reimplement the winning configuration with direct library calls for the deployed system.