Building a Production RAG Pipeline with LangChain

What Makes RAG Hard in Production

Retrieval Augmented Generation is straightforward as a demo: embed documents, store in a vector database, retrieve top-k chunks at query time, pass to an LLM. Production is different. Documents change, embeddings drift, retrieval quality varies by query type, and hallucinations are harder to catch when they reference real-but-incorrect document content. Evaluation is the hardest part.

The Pipeline Architecture

A production-grade pipeline has three distinct phases: ingestion, retrieval, and generation. Ingestion runs as a background job triggered by document changes — it chunks documents intelligently (respecting semantic boundaries, not arbitrary character counts), generates embeddings with text-embedding-3-large, and upserts into Pinecone or pgvector with metadata. Retrieval uses hybrid search: dense vector similarity combined with BM25 keyword matching, then re-ranked with a cross-encoder.

• Chunk size matters: 512 tokens with 64-token overlap works for most prose
• Metadata filters reduce retrieval space and improve precision dramatically
• Hybrid search + reranking beats pure vector search on most benchmarks
• Cache frequent queries — most production RAG systems have a long tail of repeated questions

Evaluation You Can Actually Use

Evaluation is where most RAG projects stall. Use RAGAS: it measures faithfulness (does the answer stick to the retrieved context?), answer relevance (does it actually answer the question?), and context precision (were the retrieved chunks the right ones?). Set up a golden dataset of 50–100 question/answer pairs and run RAGAS on every pipeline change as a CI check. Regressions are caught before they reach users.

Deployment and Observability

Deploy the retrieval service separately from the generation service so they can scale independently. Log every query, the retrieved chunks, and the final answer — this dataset is invaluable for debugging and fine-tuning. Set up LangSmith or a custom tracing layer so you can trace any production answer back to exactly which chunks influenced it. When a user reports a bad answer, you can diagnose it in minutes.