RAG Architecture

Retrieval-Augmented Generation (RAG) is the dominant architecture pattern for deploying LLMs in enterprise settings. The core insight is that LLM parametric memory — knowledge baked into model weights during training — is static, opaque, and prone to confident confabulation. RAG replaces or supplements parametric memory with dynamic retrieval from a vector store, making the knowledge base auditable, updatable, and domain-specific.

The RAG Pipeline has four phases: Ingestion, Retrieval, Augmentation, and Generation. Ingestion: documents are chunked, embedded into vector representations, and stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant). Retrieval: the user query is embedded using the same model, and nearest-neighbor search finds the most semantically similar document chunks. Augmentation: the retrieved chunks are injected into the LLM prompt as context. Generation: the LLM generates a response grounded in the retrieved context, ideally citing its sources.

Chunking Strategy is the most underestimated design decision in RAG. Naive fixed-size chunking (split every 512 tokens) breaks semantic units — a function definition may be split from its explanation, a numbered list may be severed mid-step. Semantic chunking uses sentence boundary detection and paragraph structure. Hierarchical chunking stores multiple granularities: fine-grained chunks for retrieval, parent sections for context window injection. The choice of chunking strategy can swing end-to-end answer quality by 30–40% independently of the LLM choice.

Embedding Model Selection requires domain-specific evaluation. General-purpose models (OpenAI text-embedding-ada-002, Cohere embed, Google gecko) perform well on broad English text but degrade on technical, legal, or Tagalog/Filipino content. Always evaluate candidate embedding models on a golden question-answer dataset drawn from your actual domain before committing. For Philippine banking deployments, evaluate multilingual embedding models that handle Filipino financial terminology.

Hybrid Search consistently outperforms pure vector search in production. Dense vector search (semantic similarity) excels at conceptual queries: "What is our policy on data retention?" Sparse keyword search (BM25, TF-IDF) excels at specific term lookup: "BSP Circular 982 section 4.3." Reciprocal Rank Fusion (RRF) combines the ranked results from both methods. Cohere's benchmark data shows hybrid search improves precision@5 by 15–25% over pure vector search across enterprise document corpora.

Reranking is a second retrieval stage that significantly improves precision. After retrieving the top-20 candidates via hybrid search, a cross-encoder reranker (Cohere Rerank, BGE Reranker, MS MARCO) rescores all 20 by actual relevance to the query — not just vector similarity. The top 3–5 reranked results go into the prompt. This two-stage approach (fast retrieval → precise reranking) achieves near-gold-standard precision at manageable latency.

Evaluation is mandatory and often skipped. Build a golden evaluation dataset of 200–500 question-answer pairs from your knowledge base. Evaluate three metrics independently: Context Recall (did retrieval surface the right documents?), Answer Faithfulness (does the answer only use retrieved content, no hallucination?), and Answer Relevance (does the answer address the question?). The RAGAS framework automates all three. Track these metrics on every pipeline change — a better embedding model may help Context Recall while hurting Answer Faithfulness.

Production Considerations: metadata filtering prevents date-expired documents from being retrieved (use document date and status fields as filter parameters). Streaming responses improve perceived latency for long answers. Caching of frequent query embeddings and retrieved contexts reduces cost significantly. And critically: log every query, every retrieved chunk, and every response for audit and quality improvement.