- Use a RAG pipeline to retrieve relevant evidence at query time and place it in the model’s context before generation.
- Review source quality, parsing, chunking, retrieval, ranking, prompting, and monitoring as connected parts of the pipeline.
- Treat security as a design concern across ingestion, retrieval, context assembly, generation, and output handling.
- Plan to evaluate retrieval, answers, operations, safety, and access-control behavior.
- Pair document evidence with governed business context when questions depend on definitions, calculations, relationships, or permissions.
A retrieval-augmented generation (RAG) pipeline gives a large language model access to relevant source material when it answers a question. That sounds simple, but production reliability depends on every step between raw content and the final response.
For enterprise teams, the real test is whether an answer is grounded, permitted, current, and reviewable. How do you build a pipeline that meets those expectations without treating retrieval as a cure for every model limitation?
What Is a RAG Pipeline?
A RAG pipeline is a system that finds relevant information and supplies it to an LLM as context for a response. Instead of relying only on patterns learned during training, the model can use private documents, recent material, or domain-specific sources at query time.
Lewis et al.’s original RAG paper describes language generation augmented with retrieved external passages through non-parametric memory. Modern systems may use different retrieval, indexing, and generation implementations.
The basic flow can start with a user question. A system may search an index, select useful passages, add them to a prompt, and ask the model to answer from that evidence.
Why does this matter? A standalone model may lack your latest policy, product documentation, support record, or internal procedure, even when its answer sounds confident.
Grounding can reduce unsupported statements, but grounding is not a guarantee of correctness. A source may be wrong, retrieval may miss a useful passage, or the model may misread the evidence.
How a RAG Pipeline Works From Data to Answer
One practical RAG design separates offline source preparation from online retrieval and generation. The stages below are common design choices and a testing map, not a mandatory architecture.
Kong et al.’s 2024 RAG practices study describes implementations that may involve document splitting, embeddings and vector stores, retrieval, reranking, repacking, and summarization. Teams can select and test the steps that fit their content and questions.
| Stage | Purpose | Practical Test Question |
|---|---|---|
| Inventory | Select owned, permitted sources | Is this source current and approved? |
| Parse | Extract usable text and structure | Were tables, headings, and fields preserved? |
| Chunk | Divide content into meaningful units | Does each chunk retain enough context? |
| Embed and index | Represent and store searchable content | Can relevant material be found reliably? |
| Retrieve and rank | Select the strongest evidence | Did the best passages reach the top? |
| Assemble context | Package evidence with instructions | Is the context clear, permitted, and within limits? |
| Generate | Produce a grounded response | Does the answer follow and cite the evidence? |
| Evaluate and monitor | Track quality, safety, and drift | Do failures trigger review and correction? |
Teams also need repeatable data preparation around these stages. ThoughtSpot Analyst Studio helps technical teams prepare data for AI with SQL, Python, R, data preparation, and modeling workflows.
Prepare, Parse, and Index Your Source Content
A practical approach is to start with a source inventory that records owners, update schedules, sensitivity, and permitted audiences. If a document has no clear owner or retention policy, should it become model context at all?
Consider retaining structure during parsing, including headings, lists, tables, field names, and document relationships. You can then review extracted content before indexing to see whether its meaning remains clear.
Choose chunk boundaries based on document structure, question type, embedding model, retrieval method, and available context window. Rather than prescribing one size, test whether each chunk is focused enough for retrieval and complete enough for interpretation.
Consider retaining metadata such as source, title, section, owner, date, sensitivity, and access policy. These fields can support filtering, review, and source identification later in the workflow.
Embeddings can represent chunks numerically for similarity search, while an index can store those representations and associated metadata. Some teams may also test keyword search or structured filtering alongside vector similarity.
This preparation can follow the same disciplined practices used for an AI-ready data foundation. Consider documenting sources, ownership, definitions, and maintenance processes before an AI experience depends on the data.
Retrieve, Rank, and Ground the Response
At query time, some pipelines may rewrite the question, identify filters, or create multiple search queries. Retrieval can find candidate passages through vector search, keyword search, structured filters, or a hybrid approach.
Some pipelines use a reranker to reorder those candidates before context assembly. Does the selected set address the user’s actual question, or does it only share similar words?
Context assembly can order evidence, remove duplication, and account for the model’s context limit. As a conservative design choice, prompts can ask the model to identify sources and decline when evidence is insufficient. They can also direct the model to treat retrieved text as data rather than instructions.
Source attribution can give reviewers a path back to the evidence. A citation alone is not proof, since the cited passage may be irrelevant, outdated, or interpreted incorrectly.
The OWASP RAG Security Cheat Sheet recommends storing access-control metadata with vector chunks and checking access during retrieval, before content reaches the model. OWASP also recommends scanning retrieved chunks for prompt-injection patterns before context assembly.
Build Security and Governance Into Every Stage
Treat a RAG pipeline as a data access system with an LLM in the response path. That framing invites a practical question: should this user receive this evidence and response?
Apply access controls before retrieved content is assembled into model context. Authorization can use the requesting user’s identity and the source system’s access policy rather than relying on model instructions.
Keep authorization logic outside prompts and other model instructions. Consider testing those controls with cases in which access should be denied.
Teams can pair retrieval-time checks with provenance records, monitoring, and output validation as layered practices. These controls reduce exposure but do not guarantee that every unsafe instruction or response will be caught.
The NIST Generative AI Profile identifies indirect prompt injection through data likely to be retrieved. NIST also advises organizations to review and verify sources and citations in generative AI outputs.
For higher-stakes workflows, consider human review before the system acts or publishes an answer. Logging can capture the question, retrieved sources, model and prompt versions, permissions applied, response, and reviewer action for later analysis.
Evaluate the Pipeline, Not Just the Final Answer
An internal evaluation approach can begin with a representative test set drawn from real user needs. Each case may include the question, expected source evidence, acceptable answer characteristics, access conditions, and known failure modes.
You can then examine the pipeline in layers. If the final answer is wrong, can you identify the likely layer? The cause may be missing content, weak retrieval, misleading context, generation behavior, or a permission failure.
Retrieval quality: Consider checking whether relevant passages appear, where they rank, and which unrelated passages are also returned.
Answer quality: You can review relevance, accuracy, faithfulness to context, completeness, clarity, abstention, and citation support.
Operational quality: Teams may monitor latency, cost, index freshness, failure rates, observability, and recovery behavior.
Safety and governance: Consider testing prompt injection, poisoned content, sensitive-data exposure, denied access, source traceability, and review routing.
Yu et al.’s 2024 survey of retrieval and generation metrics compares measures such as relevance, accuracy, and faithfulness. The survey offers a metric inventory, not a standard or the only valid evaluation approach.
Consider rerunning relevant evaluations after material changes to source content, parsing, chunking, embeddings, index settings, retrieval logic, reranking, prompts, models, or permissions. Production monitoring may also help teams identify drift as documents, questions, and user behavior change.
Use RAG for Evidence and a Context Layer for Business Meaning
RAG, prompt engineering, fine-tuning, and governed context can support different needs, and teams may combine them. Treat the table below as a planning heuristic rather than a universal selection rule.
Kermani, Perez-Rosas, and Metsis compare RAG, prompting, and fine-tuning in a specific mental-health text-analysis setting in their 2025 comparative study. Because that scope is task-specific, selection should be tested against your target task, data, operating needs, and evaluation set.
| Planning Heuristic | May Fit When | Question to Test |
|---|---|---|
| RAG | Answers may need changing, private, or cited source material | Which sources and retrieval checks fit the use case? |
| Prompt engineering | The task needs clearer instructions or response structure | Which instructions can be tested consistently? |
| Fine-tuning | Repeated behavior, style, or task patterns need adjustment | Which examples represent the desired behavior? |
| Governed context layer | Business questions depend on shared definitions, relationships, calculations, or access rules | Which governed meanings should agents apply? |
For business-data questions, governed context can add definitions, relationships, calculations, and access rules beyond document retrieval, including within governed enterprise BI workflows.
For example, “What was net revenue by region last quarter?” may require more than semantically similar text. Which revenue definition applies, which tables should join, which fiscal calendar is valid, and which rows may the user see?
ThoughtSpot Context Layer gives AI agents business context for business-data questions. See ThoughtSpot’s AI agents for business data.
ThoughtSpot is the Agentic Analytics Platform, not a generic RAG or document-retrieval product. This distinction keeps the focus on governed business meaning rather than general document search.
Make Your RAG Pipeline Ready for Real Decisions
Before launch, consider documenting the use case and the decisions the experience may support. You may also assign an owner to coordinate source, quality, security, and user-feedback work across the pipeline.
A practical launch checklist can include:
Consider defining the users, questions, decisions, and acceptable failure behavior.
Identify maintained sources with named owners and update schedules.
Plan explicit access checks before retrieval and generation.
Consider building a test set with expected evidence and denied-access cases.
Set internal quality goals for retrieval, answers, safety, latency, and cost.
Consider logging source, retrieval, permission, prompt, model, and response details for review.
Offer users a way to inspect sources, flag errors, and request review.
Adoption also depends on fitting governed answers into real work. Matillion’s analytics adoption story offers a first-party example of expanding access to analytics, without serving as a claim about RAG performance.
The goal is not a chatbot that presents uncertain information with confidence. Build an experience that can show where an answer came from, respect access rules, and identify when human review is needed.
For governed, contextual answers on business data, explore ThoughtSpot AI agents.
