What finally made RAG click for me_

RAG stands for Retrieval-Augmented Generation: retrieving information and including it in the context a model uses to generate an answer. I understood that definition long before I understood why our product needed it.

The question for me was why it was worth the engineering effort. What did retrieval help us do that simply giving the model our documents couldn’t?

I’m a frontend engineer, and I tend to understand things by working through concrete examples. Watching how product context changed the output gave me a more useful way to think about RAG.

The question I now come back to is: what does the model actually need for this task? This article follows the examples and the mental model that helped me start answering it.

“Why don’t we just extract the PDF and put it in the prompt?”

In our product discussions, a question kept coming up: why couldn’t we just extract the text from the PDF and put it in the prompt?

If the information is small, relevant, stable, and fits comfortably in the model’s context window, supplying it directly can be a reasonable solution.

When does that stop being enough?

One question, a lot of story

The analogy that helped me was much simpler than any architecture diagram.

Imagine I want to answer one question about The Lord of the Rings: what is the Ring, and why does everyone want it?

One option is to hand over the whole story. The answer is definitely in there somewhere. Another option is to first find the parts that actually matter for this question, and then use those. And if I change the question, the relevant parts change too.

That was the first distinction that really helped me: having access to all of the information and selecting the information that matters for this task are not the same problem.

Have → Find → Give

I started separating the problem into three parts: Have, Find, and Give. This is my own explanatory model, not standard RAG terminology.

Have. Does your application actually have access to the information? A PDF, internal documentation, customer data, product information, or whatever else your application is allowed to use.

Find. Out of everything available, what matters for this particular task? Retrieval can help select relevant knowledge from that collection.

Give. What actually becomes part of the context the model receives for this request? Information the application can access, or that a retrieval system finds, still has to be included in that context.

I used to mentally bundle all of this together as “the AI has the data”. They’re different problems.

Retrieved knowledge is only one part of that context. The model may also receive instructions, conversation history, user or task state, examples, or tool results. RAG combines retrieval with generation; the broader context problem includes deciding how all these inputs support the task.

Same task. Different context.

In an early prototype, we tested a hypothetical sales scenario.

The task was to create a technical sales presentation for an Australian hospital evaluating the GE HealthCare Panda iRes Warmer. The brief described an audience of neonatal clinicians, biomedical engineers, and procurement staff, with concerns around workflow during neonatal resuscitation, bedside equipment, space, and infrastructure compatibility.

The model and request stayed the same. What I changed was the product knowledge available to the system. In both cases, the model still received the request and instructions.

Actual prototype output · screenshots from the talk

Select a screenshot to view it at full size.

Without product-specific knowledge, the output described a generic “Monitoring Interface”. With the product context available, it referred to ResusView ECG and SpO₂, a capability documented for this product.

The comparison also changed from an “Accessory Mounting Interface” to Integrated Resuscitation. The generic description sounded plausible; the product-specific version named a capability supported by the documentation.

What stood out was the move from plausible descriptions to documented product capabilities. That was a useful improvement in these examples, without establishing that retrieval would make every output accurate.

More source material ≠ better context

It was tempting to conclude that more product information would keep improving the result. A second comparison complicated that assumption.

The first version had a short, 7-page product brochure. Then we added a much larger technical manual of roughly 300 pages.

Actual prototype output · screenshots from the talk

Select a screenshot to view it at full size.

The additional technical material contained valid product information. What changed was which details the output prioritised.

With the brochure alone, the presentation highlighted the X-Ray Cassette Tray and Automatic Warm-Up. With the technical specification added, it highlighted the Side Bedside Panel and an RS-232 Connector.

Those details were real, and connection information could matter to the biomedical engineers described in the brief. My concern was the emphasis: for a presentation about workflow and practical adoption, I found those topics less useful as leading presentation steps. A detailed infrastructure discussion might call for a different selection.

Checking whether each detail appeared in the documentation could establish factual support. It couldn’t establish whether that detail deserved this much attention in this presentation.

Correct ≠ relevant

These outputs don’t tell me why the retrieval pipeline behaved this way, or exactly what context reached the model. The observation is narrower: adding valid source material changed what the output prioritised. A fact can be correct and still be the wrong thing to prioritise for the task.

The same selection problem appears at different scales. It might mean deciding which designs, files, or instructions to give an AI agent, or finding relevant information across an enterprise knowledge base.

This simple model only scratches the surface. RAG and context engineering cover a much broader field of specialised work, from evaluating retrieval quality to approaches such as GraphRAG and managed services that handle RAG infrastructure, often called RAG-as-a-Service (RaaS). I’m still learning about that wider field. What helped me get started was a more practical question: what does the model actually need for this task?

So… do I actually need RAG?

Not necessarily. Which brings us back to the PDF question.

If the information is small, relevant, relatively stable, and you already know what the model needs, just giving it the context may be the simplest and best solution. Retrieval starts becoming much more useful when the available knowledge is large, changes frequently, or when what matters depends on the task or the user.

I use this as a starting point for an architecture decision: do we have a context-selection problem that retrieval actually helps us solve?

Retrieval adds another place to fail

Adding retrieval also adds a step that can affect the quality of the result.

The model might be perfectly capable of answering the question, but if retrieval gives it the wrong information, too much irrelevant information, or not enough information, the final answer can still be poor.

There is more to evaluate and maintain, with potential latency and cost. We need to examine both the information selected and the output produced from it. Retrieval quality becomes part of the quality of the AI experience.

When context selection becomes part of the product

That selection depends on what the product understands about the customer.

We’re all getting access to increasingly capable foundation models, and many companies can use the same ones. But a general-purpose model doesn’t automatically know what’s specific to your customer: their internal knowledge, what they’re currently doing, what they’re trying to achieve, or which information matters at this particular moment.

Access to that information is only the beginning. The product still has to decide what matters right now.

What interests me is how well a product understands the customer’s knowledge, domain, and workflow, and how it turns that understanding into useful context for the task.

In our world, for example, it’s not enough for an AI to know generally how to create a presentation. A useful product might need to understand which product we’re talking about, which information and assets are relevant, who the presentation is for, and whether this is sales, training, or support.

My current thinking is that as models become more widely available, context and workflow understanding may become an increasingly important source of product differentiation.

What I took away

Three ideas changed how I think about the problem.

First, having information and finding what matters are different problems.

Second, more context isn’t automatically better context. What’s useful depends on the task you’re actually trying to complete.

Third, RAG is one tool for a much broader context problem, not something every AI product automatically needs.

Whether I’m choosing a file for an AI agent or thinking about a product that searches an enterprise knowledge base, Have → Find → Give helps me separate access, selection, and the context the model receives.

What does the model actually need for this task?

A post-talk note: how do we know the output is better?

During Q&A, someone asked how we define and measure output quality. I didn’t have a precise answer about the backend evaluation process. What I could describe was my part in the early prototype: reviewing outputs manually, sometimes using AI to help analyse or compare them, and working back through the prompts to iterate.

A later conversation with the team helped me see the direction more clearly: more systematic evaluation, using repeatable examples and clearer criteria for what makes an output useful.

Another engineer mentioned reward models after the talk. That raised a separate question to explore later. The immediate need is to define what a useful output looks like for the task and evaluate whether changes actually improve it.

Once context selection becomes part of the product, evaluating that selection becomes part of the engineering problem too.

A few things I found useful

Anthropic: Effective context engineering for AI agents

Anthropic: Contextual Retrieval

a16z: From Demos to Deals: Insights for Building in Enterprise AI