What RAG Actually Is (and Why Your LLM Doesn't Know Your Company Data)

If you've spent any time around chatbots, you've probably built up a simple mental model of how they work. You type a prompt, the LLM does its thing, and you get a response back. Ask "what's the weather today" and you get something like "sunny, mid-70s, rain later tonight." Straightforward.

That works fine for anything in the public domain. Sports scores, general tech questions, historical facts, whatever the model was trained on. Worth being precise here: a base LLM isn't searching the internet live when it answers. It's drawing on patterns learned once, during training, at some point in the past. If a chatbot looks like it's searching the web in real time, that's a separate tool bolted on top, not the LLM itself doing it. Either way, it breaks the moment you ask about something the model was never exposed to, like your company's internal data.

It helps to place this against the two things people usually compare it to. A search engine gives you a list of links and leaves you to read through them and pull out the answer yourself. A standard LLM skips that step, it reads everything it was trained on ahead of time and hands you a synthesized, plain-language answer directly. RAG does the same thing as the LLM, a direct synthesized answer, except it's pulling from your own private data instead of what the model happened to be trained on.

The problem

Say you want the LLM to help with some analytics on your company's numbers, or answer a specific question about a product only your team knows about. That information was never public, so it was never in the training data. No matter how good the model is, it simply doesn't know it.

The fix is conceptually simple: instead of sending just the user's question to the LLM, you send the question plus the relevant company data, combined into one prompt. The LLM then has everything it needs to answer properly.

That private data source can take a lot of forms. Sometimes it's an Excel file, sometimes it's a website, sometimes it's a folder of PDFs, sometimes it's a ticketing system or an internal database. The idea stays the same regardless of the source.

How does RAG actually work?

This is the part people gloss over, so let's slow down.

There are two phases: indexing and querying.

Indexing happens once, ahead of time. Your source data, say a company website, gets broken into chunks. Maybe one chunk covers finance, another covers a product spec, another covers HR policy. That splitting is usually just rule-based, by paragraph, by section, by token count, not something an LLM needs to do. Each chunk then gets run through an embedding model, which converts it into a vector, essentially a list of numbers that captures the meaning of that chunk. This embedding model is a separate, more specialized model from the general-purpose LLM that eventually answers the question. All these vectors, along with their original text, get stored in a vector database.

Querying happens every time a user asks something. Say someone types "do you have parking?" That question gets converted into a vector using that same embedding model, in real time. The system then runs a mathematical similarity search against every chunk-vector in the database. Because semantically similar chunks end up sitting numerically close to each other in that vector space, the search can pull back whichever handful of chunks are the closest match, usually a fixed number like the top five. Crucially, what gets pulled back is the original text of those matching chunks, not the vectors themselves.

That retrieved text then gets folded into what's effectively a prompt before the prompt. This piece usually carries system-level instructions, things like the role the model should play or the tone it should use, and it gets combined with the retrieved chunks and the user's original question into a single package. Instructions plus retrieved chunks plus the question, all in plain language, is what actually gets sent to the LLM.

The LLM never sees the vectors directly. Vectors only exist to make the search step fast and semantic, so the system can find "relevant" text even if the wording doesn't match exactly. Once the right chunks are found, it's back to plain text.

R, A, G

The name breaks down cleanly once you've seen the mechanism. Retrieval is the step where relevant chunks get pulled from your private data source based on vector similarity. Augmented refers to inserting those chunks into the prompt so the model has precise background to work with. Generation is the LLM actually producing a human-sounding answer built from what it just retrieved.

Why this matters

This is the whole trick behind what people call RAG, or Retrieval-Augmented Generation. It lets you bolt private, proprietary, or just very recent information onto a model that was never trained on it, without retraining the model itself. You're not changing what the LLM knows, you're changing what it sees at the moment you ask it something.

That's the entire mechanism. Chunk your data, embed it, store it, retrieve the relevant pieces at query time, and stitch it into the prompt before it ever reaches the model.