If you've spent any time around chatbots, you've probably built up a simple mental model of how they work. You type a prompt, the LLM does its thing, and you get a response back. Ask "what's the weather today" and you get something like "sunny, mid-70s, rain later tonight." Straightforward.
That works fine for anything in the public domain. Sports scores, general tech questions, historical facts, whatever the model was trained on. Worth being precise here: a base LLM isn't searching the internet live when it answers. It's drawing on patterns learned once, during training, at some point in the past. If a chatbot looks like it's searching the web in real time, that's a separate tool bolted on top, not the LLM itself doing it. Either way, it breaks the moment you ask about something the model was never exposed to, like your company's internal data.
It helps to place this against the two things people usually compare it to. A search engine gives you a list of links and leaves you to read through them and pull out the answer yourself. A standard LLM skips that step, it reads everything it was trained on ahead of time and hands you a synthesized, plain-language answer directly. RAG does the same thing as the LLM, a direct synthesized answer, except it's pulling from your own private data instead of what the model happened to be trained on.
The problem
Say you want the LLM to help with some analytics on your company's numbers, or answer a specific question about a product only your team knows about. That information was never public, so it was never in the training data. No matter how good the model is, it simply doesn't know it.
The fix is conceptually simple: instead of sending just the user's question to the LLM, you send the question plus the relevant company data, combined into one prompt. The LLM then has everything it needs to answer properly.
That private data source can take a lot of forms. Sometimes it's an Excel file, sometimes it's a website, sometimes it's a folder of PDFs, sometimes it's a ticketing system or an internal database. The idea stays the same regardless of the source.
How does RAG actually work?
This is the part people gloss over, so let's slow down.
There are two phases: indexing and querying.
Indexing happens once, ahead of time. Your source data, say a company website, gets broken into chunks. Maybe one chunk covers finance, another covers a product spec, another covers HR policy. That splitting is usually just rule-based, by paragraph, by section, by token count, not something an LLM needs to do. Each chunk then gets run through an embedding model, which converts it into a vector, essentially a list of numbers that captures the meaning of that chunk. This embedding model is a separate, more specialized model from the general-purpose LLM that eventually answers the question. All these vectors, along with their original text, get stored in a vector database.
Querying happens every time a user asks something. Say someone types "do you have parking?" That question gets converted into a vector using that same embedding model, in real time. The system then runs a mathematical similarity search against every chunk-vector in the database. Because semantically similar chunks end up sitting numerically close to each other in that vector space, the search can pull back whichever handful of chunks are the closest match, usually a fixed number like the top five. Crucially, what gets pulled back is the original text of those matching chunks, not the vectors themselves.
That retrieved text then gets folded into what's effectively a prompt before the prompt. This piece usually carries system-level instructions, things like the role the model should play or the tone it should use, and it gets combined with the retrieved chunks and the user's original question into a single package. Instructions plus retrieved chunks plus the question, all in plain language, is what actually gets sent to the LLM.
The LLM never sees the vectors directly. Vectors only exist to make the search step fast and semantic, so the system can find "relevant" text even if the wording doesn't match exactly. Once the right chunks are found, it's back to plain text.
R, A, G
The name breaks down cleanly once you've seen the mechanism. Retrieval is the step where relevant chunks get pulled from your private data source based on vector similarity. Augmented refers to inserting those chunks into the prompt so the model has precise background to work with. Generation is the LLM actually producing a human-sounding answer built from what it just retrieved.
Why this matters
This is the whole trick behind what people call RAG, or Retrieval-Augmented Generation. It lets you bolt private, proprietary, or just very recent information onto a model that was never trained on it, without retraining the model itself. You're not changing what the LLM knows, you're changing what it sees at the moment you ask it something.
That's the entire mechanism. Chunk your data, embed it, store it, retrieve the relevant pieces at query time, and stitch it into the prompt before it ever reaches the model.
Vector, hybrid, or graph: it depends on the use case
Everything up to this point describes plain vector search, and it's a fine default. But it's not the only retrieval strategy, and treating it as one-size-fits-all is where RAG systems start giving weird answers. The right approach depends less on the source, an enterprise doc versus a public FAQ, and more on the shape of the question being asked.
Being an enterprise doc doesn't automatically change my approach. It really just depends on what the use case is.
Say you've got simple info based prompts, like an FAQ. Someone asks "what's the exchange window on this product." Vector search should work fine here. These questions are conversational. They don't rely on matching an exact code or number.
Now think about IT support. A user asks "how do I fix VPN error 0x8007045D." Pure vector search can mess this up. The embedding model just sees the code as a random string of characters. It doesn't understand it as a specific error that needs an exact match. So it might give you a generic fix, or even the fix for a different error. Hybrid search fixes this problem. It uses something called BM25, which matches the exact text instead of guessing at meaning.
Graph approaches only come in when you need to connect a lot of different pieces together. Say you want to know "which vendors are tied to the same clause as Vendor X." Or you're building a report that pulls facts from many different linked documents. That's when this approach actually earns its place.