Which RAG architecture should you choose?

01: Why now

Why this decision matters now

In late 2025 I sat in a room with a mid-size insurer's engineering leads while they tried to explain, to their own compliance team, why a claims-lookup bot had cited a policy exclusion that didn't exist. Nobody could reconstruct which chunk it pulled from. Nobody had logged it. The engineer next to me just kept saying "we can rebuild the index and check," which was not the question anyone had asked.

RAG used to be something you bolted onto a chatbot to stop it from making things up. It still does that. But it's also quietly become the governed source of enterprise evidence your answers are grounded in, and that means the architecture you pick now shows up later: in audits, in incident reviews, in a cost line nobody flagged during the pilot because pilots don't run at production volume, and by the time they do, the person who chose the architecture has usually moved teams.

One thing it is not, and I would like this to stop appearing in vendor decks: an authorization boundary. The corpus governs the evidence the model is grounded in. It does not govern what the model knows. Everything pretraining put in there is still in there, along with your system prompt, the conversation so far, whatever a tool handed back, and the model's own willingness to bridge a gap you left open. Retrieval decides what you can point at afterwards. It does not decide what the model is capable of saying.

Treat it as a build decision, not a config flag.

FIG.01 Same technology, different weight-bearing role.

A short aside, because I get asked this constantly: no, you don't need a vector database with hybrid search and a reranker to answer "what are your support hours." Half the RAG builds I've reviewed were solving a problem three lines of a system prompt would have handled. Check that first.

Retrieval is where the failures usually start

Every postmortem I've sat through starts in the same place, and it's rarely the place the room expects. Hand a frontier model the wrong three paragraphs and it will write a confident, fluent, wrong answer. Hand a small, unremarkable, slightly embarrassing open-weight model the right three paragraphs and it usually gets there fine. That asymmetry is real, it is large, and it is why retrieval is the first thing I open.

First is not only, and I've watched a team burn a quarter on that distinction. A generator handed perfect context can still skip the middle of it, synthesize a claim nobody supplied, quietly prefer what it remembers from pretraining over the document sitting in front of it, follow four of your five instructions, or answer the question next to the one you asked. None of that is a chunking problem and none of it gets better when you re-tune your embeddings. So score the two halves separately: retrieval against what should have been found, the answer against what was actually retrieved. One blended quality number tells you the system is bad and nothing about which half to fix, which is how teams end up rebuilding an index that was working fine.

What people do instead is reach for the model swap first, every time, because it's the easy purchase: a line item, a vendor call, done in an afternoon, a Slack message to the team that says "we upgraded the model" and gets three thumbs-up emoji. Fixing retrieval, though, means someone has to actually sit with your chunking strategy and your embedding choice and your reranker and your failure logs, unglamorously, for weeks, and nobody claps for that in a roadmap review even though it's where the accuracy gains actually live. Ask your team, next time they want budget: does this improve what we retrieve? Or does it just make the changelog read better?

FIG.02 Retrieval is the first constraint — the generator still gets its own scorecard.

Unrelated, but I will die on this hill: the single best debugging tool for a bad RAG pipeline is a spreadsheet with the query, the retrieved chunks, and a human "yes/no, was this right" column. Not a dashboard. Not an eval framework with a logo. A spreadsheet. I built one for a client in about forty minutes once, after their $80K observability platform had been "still onboarding" for six weeks, and it found the bug in an afternoon. I don't fully know what to do with that fact. Make of it what you will.

03: The field

The architecture landscape

Six patterns show up repeatedly in production. What each buys you, what it costs. I'll skip the internals, since the developer-track article in this series covers those.

Hybrid + Reranker. Pairs keyword and semantic search, re-scores the merged results. That's it. That's the pattern. It's the unglamorous default and it's correct for most teams, and if you're building something fancier first, you'd better have a reason better than "it looked cool in the demo."

GraphRAG is having a moment right now, and I want to be careful here because I know people who do it well, but most of the implementations I've reviewed are expensive ways to store the same document chunks in a worse format. If your "graph" is a list of entities you pulled out with a regex and some cosine similarity, you don't have a graph. You have a spreadsheet with delusions of grandeur, and you're paying graph-database prices for spreadsheet-quality relationships. Real GraphRAG is genuinely good, and it's worth being precise about what it's good at, because "multi-hop" undersells it. Microsoft's original work was aimed at the questions ordinary chunk retrieval cannot answer at all: what are the recurring themes across this entire corpus, what changed between these two years of filings, who keeps showing up next to whom. Global sensemaking, not just a longer hop. Local entity lookups, path reasoning, and corpus-wide synthesis are three different jobs, and the graph approaches on the market solve different subsets of them. Pick for the job. It's just not what most people are actually building when they say the word.

Adaptive RAG routes easy questions to a cheap path and hard ones to an expensive one: worth it once you can prove your query mix actually varies, not before. Corrective RAG and Self-RAG both add checking, and the internet bundles them constantly, including in my table below. They are not the same technique and you should know which one is on your invoice. CRAG grades the retrieval: an evaluator scores what came back and, when it's weak, the pipeline does something about it, re-queries, widens, falls back to another source, throws the batch out. Self-RAG puts the judgment inside the generator: the model decides when to retrieve at all, criticizes what it got, and critiques its own draft while writing. A correction workflow around retrieval versus a model trained to reflect. Different failure modes, different bills, different debugging sessions at 2am. Where you see "Corrective / Self-RAG" below, read it as the self-checking family, and then choose a specific member on purpose. "Adaptive RAG" is the other umbrella people use for this, which is unhelpful, because it also covers plain query routing. Agentic RAG lets a model call retrieval repeatedly inside a reasoning loop: built for multi-step work, not lookups, though everyone wants to use it for lookups because loops feel more like "AI." Multimodal RAG extends retrieval to images, tables, and scans; skip it unless your knowledge actually lives outside plain text, which, check first, it might not.

Before the table, the assumptions, because ratings like this get quoted without them and then turn into architecture folklore. Everything below assumes a corpus in the low millions of chunks, a candidate set of roughly 50 to 100 documents going into any reranking stage, a mid-size cross-encoder on GPU, hourly-ish freshness, and single-digit queries per second with a two-second target. Change any one of those and the columns move. A cross-encoder reranker is one of the more expensive things in a retrieval path, and at a large enough candidate count it will dominate your latency budget outright. These are relative positions under one set of conditions, not properties of the architectures.

Architecture	Best for	Rel. cost	Latency	Op. complexity	Self-checking
Hybrid + Reranker	General-purpose default	Low–Med	Med	Low–Med	None
Adaptive RAG	Mixed difficulty under cost pressure	Low–Med	Variable	Medium	Routing only
GraphRAG	Multi-hop and whole-corpus sensemaking	High	Med–High	High	None
Corrective / Self-RAG	Accuracy-critical with unreliable retrieval	Med–High	High	Med–High	By design
Agentic RAG	Multi-step workflows & agents	High	High	High	Loop-dependent
Multimodal RAG	Images, tables, diagrams, media	Med–High	Medium	High	None

You'll notice there is no auditability column. That was deliberate, and it cost me an argument with an early reader. Auditability is not a property of any row in this table; it is a property of what you log, and every one of these six can be fully auditable or completely opaque depending on decisions you make elsewhere. More on that in a moment.

FIG.03 Six patterns, plotted by what they cost and what they buy.

How to actually decide, more or less

If you're in a regulated industry, your architecture is not what makes you auditable, and I have now had this argument with four separate compliance teams and won it four times. Auditability is a logging property. It is immutable records of the query, the retrieved chunks, the assembled prompt, the output, and the model version; provenance down to the individual chunk; a versioned corpus and index so you can reconstruct what the system saw last March; enforced access control; documented evaluation and change management; and a human review path for when it goes wrong. That's the list. NIST's Generative AI Profile is architecture-neutral and talks about lifecycle risk management. The EU AI Act's high-risk provisions talk about automatic event logging. Neither of them says "use Self-RAG," because neither of them cares which retrieval pattern you picked.

A model grading its own answer is a quality control. It is not an audit trail, and a self-critique score sitting in a log nobody can reconstruct is worse than no score at all, because it looks like evidence. Buy the logging first. It's cheaper than a grading loop and it's the thing the auditor actually asks for. Then add self-checking where the retrieval is genuinely unreliable, and be specific about which kind you're adding: CRAG when the documents coming back are the weak link, Self-RAG when the generator is the one over-reaching. Add it where a wrong answer is expensive enough to justify verifying it before it ships, which in my experience is real in pharma and parts of finance and much rarer everywhere else. The insurer I opened with did not need a grading pass. They needed to be able to answer "which chunk?" six weeks later, and no architecture on this list would have given them that.

GraphRAG: see above, and confirm with real example questions before you commit, not after. Agentic RAG is for genuine multi-step workflows, actions that chain into other actions; if what you actually have is a lookup wearing an agent costume, it'll show up in the bill before it shows up in the accuracy metrics. Multimodal RAG if your knowledge is genuinely in images, tables, or scans. Text-only retrieval will just quietly skip that content and nobody will notice until someone asks about a diagram. Adaptive RAG once you can point to an actual difficulty spread in your logs. And if none of that describes you, which is most of you, statistically, that's Hybrid + Reranker. Start there. Most teams end up there eventually anyway, just after a more expensive detour.

FIG.04 Roughly how the reasoning goes: treat it as a starting point, not a verdict.

What it actually costs

This is a composite of a few real cases: there's a well-documented pattern of agentic loops generating surprise five-figure bills (you can find postmortems on HN and in a handful of startup engineering blogs), and I've seen the same failure mode firsthand at roughly that scale. The industry and product details here are blended and anonymized, but the $40K-ish runaway bill from uncapped retries is very real: a loop that could retry up to twelve times on an ambiguous query before giving up, on traffic that turned out to be mostly people asking about part numbers in ways the classifier read as "ambiguous." Twelve retries. For a returns lookup. In at least one version of this, "intelligent retry logic" had been demoed to the board as a feature two weeks earlier.

The fix, in every version of this story, took about an afternoon: cap the retries, fall back to plain hybrid search after two attempts. Finding out it needed a fix is the part that took a very uncomfortable finance meeting.

That's the shape of the risk, generally: the self-checking family and Agentic patterns both add repeated model calls, GraphRAG and Multimodal both add infrastructure you now have to keep alive, and none of it shows up until you're at volume.

Anyway. On build versus buy: ask whether retrieval quality is something your product actually competes on, or whether it's table stakes you need to clear and move past. I've watched a team spend two quarters on a bespoke graph pipeline for a feature that, it turned out, nobody was choosing the product for. Buy the table stakes. Build the thing you're actually betting the business on.

FIG.05 The sticker price is the tip.

Which reminds me: has anyone else noticed that most of the Adaptive RAG papers benchmark against the same three or four public datasets? I went looking last winter for anything evaluating adaptive routing against a genuinely messy internal knowledge base: support tickets, half-finished wiki pages, that kind of thing, and mostly came up empty. Might be out there. Might not be a real problem. I never actually finished chasing this down, so take it as a question rather than a point.

06: Rollout

Don't commit on day one

Ship Hybrid + Reranker first. Add complexity only when you can point to the specific query that broke it: not a hypothetical one, an actual one, with a timestamp. That's most of the section, honestly.

FIG.06 Earn each layer of complexity before you add it.

The numbers to actually ask for

You don't need to run the evaluations yourself. You should know three numbers: faithfulness: is the answer grounded in what was retrieved, or is the model filling gaps on its own; answer relevance: does it address what was actually asked; and retrieval quality: did the system fetch the right material in the first place. Split them the way the system is split: retrieval quality is retrieval's report card, faithfulness and answer relevance are the generator's. Retrieval is where I'd look first when all three sag together, but a generator can absolutely tank faithfulness on context that was perfectly good, and if you average the three into one health score you will never see it happen.

I should admit something here. For a long stretch I told every team that started fresh to go straight to pure semantic search: embeddings felt like the modern, correct answer, and keyword matching felt like something you'd apologize for using in 2023. I was wrong, and it cost a client of mine about two weeks of debugging a pipeline that kept missing exact product codes and part numbers because dense retrieval does not reliably prioritize the difference between article number "SIL 10-9 120" and article number "SIL 10-9 12k". The embedding can represent both strings perfectly well. It just won't rank that one-character distinction above a paragraph of near-identical surrounding prose, especially for a rare identifier that barely appeared in training. BM25 is dumber and usually gets it right, because it matches the token rather than the vibe — assuming your analyzer doesn't shred the identifier first, which for anything with hyphens or odd spacing often means an untokenized keyword field, exact-match boosting, or character n-grams sitting next to it. Dates, part numbers, error codes, version strings: same story. Hybrid isn't just the safer recommendation: for me it's a little bit of a standing apology for being enamored with the elegant answer over the boring one that worked.

Two of those three you can measure this week, and the third is the one that needs the labelling project. Faithfulness and answer relevance are reference-free: you're checking the answer against the context you actually retrieved and against the question you were actually asked, and neither requires knowing the correct answer in advance. Retrieval quality is different. Recall against what should have been found needs relevance judgments, which means a labelled set, which means work. So build the golden dataset, real questions, known-good evidence, versioned like any other production asset, because it's the thing I've seen skipped most and it's always built under deadline pressure by someone who didn't choose to be there. But do not use "we don't have a golden dataset yet" as a reason to ship unmeasured for another quarter. Two of the three numbers are available to you today.

FIG.07 The three numbers to ask your team for.

I don't have a golden dataset for my own side project, for what it's worth. I've been meaning to build one for about eight months. So: do as I say.

Where this leaves you, probably

Start with Hybrid + Reranker. Add complexity only when you can name the specific query that broke it, and only after you know which half broke: retrieval and generation get separate numbers, always. If it was retrieval, decide deliberately whether you want CRAG grading the documents or Self-RAG grading the model's own draft, because those are two different builds. And whatever you end up running, the thing that makes it defensible in front of a regulator is the logging underneath it, not the shape of the diagram.

I post weekly: mostly what breaks in production, not what looked good in the demo. Next up is the hybrid + reranker implementation piece, the one that actually shows the tuning. Sign up to my newsletter and you read it early.

And if you do end up building something fancier: a graph, an agent loop, the whole stack, tell me. Comment, DM, whatever. I'm not always sure it was worth it, and I'd rather hear real numbers than guess.