AI/LLM · Editable reference architecture

RAG LLM Production System Architecture

Separate knowledge ingestion from online question answering in a retrieval-augmented generation system. Trace how documents become retrievable chunks and how a query uses selected context to produce a response with sources.

Open & Edit in DrawArc (Free, No Login Required)

Your changes stay in a separate draft on this browser. Download an editable project to keep a portable copy.

Read the step-by-step architecture tutorial →

LinkedIn X (Twitter)WhatsApp

7 components · 8 connections · Open the editor to zoom, move nodes and edit source.

Architectural overview

Retrieval-augmented generation combines a language model with a searchable knowledge corpus. This starter separates the offline ingestion path from the online query path. Documents pass through chunking and embedding before entering a vector store. At query time, an application retrieves relevant chunks, assembles a bounded context and calls an LLM gateway before returning a response to the user.

Retrieval is not an authorization boundary. Apply document and tenant access controls before selected content reaches the model, and preserve source identifiers so responses can be checked against their evidence. Decide how deleted or superseded documents are removed from both chunk storage and the retrieval index. Keep prompts, model configuration and context limits explicit so a model change can be evaluated rather than silently altering application behavior.

A production discussion should cover latency budgets, model availability, request limits and per-query cost. Add evaluation datasets for retrieval relevance, unsupported answers and adversarial content. This diagram includes an evaluation and tracing component to show where quality feedback and operational measurements belong. Customize it with your chosen model provider, embedding strategy and data residency constraints before treating it as an implementation plan.

The real-world problem

An internal knowledge assistant must answer from approved, changing documents while enforcing each user's access and providing inspectable sources instead of unsupported confident answers.

Data flow

Connections from this diagram. Branches and control signals are not a single chronological sequence.

Knowledge documents → Chunk and embed: Ingest approved content.
Chunk and embed → Vector store: Index chunks.
User → RAG application: Question.
RAG application → Vector store: Authorized retrieval.
RAG application → LLM gateway: Question and selected context.
LLM gateway → RAG application: Generated response.
RAG application → User: Answer with sources.
RAG application → Evaluation and traces: Quality and usage signals.

Security boundaries

Enforce tenant/document permissions during retrieval before context reaches the model; apply the same rules to caches and source links.
Treat retrieved documents as untrusted input. Model text must not authorize tools or sensitive actions.
Minimize sensitive provider payloads, choose suitable retention/residency settings and keep raw questions/documents out of general analytics.

Failure handling

Retrieval returns weak or unauthorized context

Response: Reject unauthorized material before generation and provide an insufficient-evidence response when relevant sources are absent.

Verify: Test unanswerable questions and cross-tenant requests against a versioned evaluation set.

Model provider times out or throttles

Response: Use per-stage budgets and bounded retries; fail clearly or use an evaluated fallback with compatible data handling.

Verify: Inject timeout and rate-limit responses and check latency and cost ceilings.

Deleted documents remain indexed

Response: Propagate deletion and permission changes to chunks, index and caches with a measurable freshness target.

Verify: Revoke a document and verify it cannot appear in retrieval or citations afterward.

Trade-offs and alternatives

Add retrieval and source attribution

Benefit: Provides inspectable evidence from a changing corpus.
Cost: Index maintenance, retrieval errors and authorization complexity; grounding is not guaranteed.
Alternative: Use deterministic search when users mainly need document discovery.

Increase context or add reranking

Benefit: May improve evidence selection on evaluated tasks.
Cost: More latency and token/compute cost; extra context can add noise.
Alternative: Measure a smaller retrieval baseline before adding stages.

When to use this architecture

Design a knowledge assistant that needs document retrieval, source attribution and controlled model access.
Separate ingestion failures and index freshness concerns from the online response latency budget.
Review tenant authorization, model costs, prompt changes and answer quality with engineering and product teams.

Key components

Knowledge documents: Approved source documents with ownership, versions and access metadata.
Chunk and embed: Splits content and computes embeddings while preserving source references.
Vector store: Stores embeddings and metadata for access-filtered similarity retrieval.
User: Submits an authenticated question to the assistant.
RAG application: Enforces authorization, retrieves context and constructs the model request.
LLM gateway: Applies provider routing, request limits and bounded model configuration.
Evaluation and traces: Tracks retrieval relevance, response quality, latency and usage costs.

Questions about this template

Does RAG eliminate hallucinations?

No. Retrieval can improve grounding, but the application still needs evaluation, source attribution and handling for unsupported answers.

Can I use a self-hosted model?

Yes. Replace the gateway component and document model hosting, capacity, latency and data handling requirements.

Related architecture templates

AWS