Production RAG architecture explained

Two paths: ingestion and answering

Retrieval-augmented generation retrieves relevant material and supplies it as context to a language model. It can ground answers in a changing knowledge collection without putting that entire collection into each request. Retrieval does not guarantee correctness: a model can misread context, and the search layer can return irrelevant or outdated evidence.

The starter separates document ingestion from the online question path. Approved documents pass through chunking and embedding into a vector index. A user question reaches an API, authorized retrieval selects context, and the API sends a bounded prompt to the model. The API returns an answer with sources and records quality and usage signals.

Build an ingestion contract

Choose which repositories are eligible, who approves them and how updates are detected. Preserve document identifiers, versions, source locations and access metadata alongside chunks. Chunk size and overlap are retrieval choices: evaluate them on representative questions rather than assuming one setting works for every document type.

Embedding and indexing jobs should be repeatable. Track successful versions and define how a deleted or access-restricted document is removed from retrieval. A fresh source file with an old index can produce misleading answers even when the model is working correctly. Include ingestion lag and failed-document handling in the operational view.

Enforce permissions before building context

Authenticate the user and enforce document or tenant permissions during retrieval. Filtering only the final answer is insufficient if unauthorized material has already entered model context. Apply equivalent access checks to caches, source links and conversation history, and ensure a user cannot retrieve another tenant's content by changing a request field.

Retrieved text is untrusted input. It can contain instructions intended to override the application's policy. Keep tool execution and sensitive operations behind independent authorization checks; a persuasive paragraph in a document must not grant access. Minimize sensitive content sent to external model providers and select retention settings appropriate to your data.

• Carry access metadata through ingestion and retrieval.
• Test cross-tenant queries and permission changes.
• Keep document text out of general analytics and redact operational logs.

Retrieve, rank and generate with a budget

A production retrieval pipeline may combine keyword and vector search, then rerank candidates before selecting context. Treat these as experiments: compare answer quality and latency before adding complexity. Limit context size and leave room for the user's question and the response within the chosen model's budget.

Give the response a clear contract: cite the supporting sources and admit when evidence is insufficient. Validate source identifiers against retrieved documents instead of accepting invented links. Define timeouts, bounded retries and a clear unavailable state when retrieval or generation fails. Streaming can improve perceived responsiveness, but does not make the underlying answer more reliable.

Evaluate the whole system

Create a versioned set of realistic questions covering answerable, unanswerable and access-restricted cases. Check retrieval relevance, groundedness, citation validity and task completion. Include adversarial documents, stale content and conflicting sources so the benchmark reflects more than easy demonstrations.

Measure latency by stage, token usage, cost per successful answer and retrieval failures. Record model, prompt and index versions so regressions can be traced without logging raw private questions. This diagram is a discussion starter: provider controls, threat modelling and workload-specific evaluation remain implementation work.