RAG for Financial Data: Where Retrieval Breaks and What Works Instead

RAG, short for retrieval-augmented generation, answers a question by searching an index of document passages for the handful that look most relevant and having a language model write the answer from them. On financial data it works well as search and breaks as analysis, which is why Kepler, our enterprise intelligence platform for investment firms, uses retrieval only for finding passages and reading language: every number comes from facts extracted when each document arrives, resolved into your firm’s ontology and computed in code. Plain RAG falls short in predictable places: chunks separate tables from their headers, the top results miss the footnote that matters, the model retypes every digit, and it can’t return a complete list or say what was true on a date.

How RAG works

The idea comes from a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which paired a text generator with a search component over Wikipedia so the model could ground its output in passages it hadn’t memorized. Most document AI products still follow the same four steps:

1. Chunk. Split each document into passages, typically a few hundred tokens each.
2. Embed and index. Turn each chunk into a vector that captures its meaning, often with a keyword index (BM25) beside it.
3. Retrieve the top k. Turn the question into a vector, find the k nearest chunks, and often rerank them with a second model.
4. Generate. Put those chunks in the prompt and have the model write an answer, usually with citations to the chunks.

RAG is quick to stand up, needs no schema, and takes any question phrased in plain language. For a question whose answer sits in one passage, such as “what did management say about pricing on the last call?”, it does the job.

Where retrieval breaks on financial data

Financial questions rarely have their answer in one passage. They need the right number for the right entity and period, often from several documents, and sometimes for every company in a set. Each step of the pipeline drops something on the way.

Chunking separates numbers from their meaning

A chunk boundary can fall between a table’s header (“in millions, except per share data; three months ended September 30”) and its rows. Anthropic’s write-up on Contextual Retrieval (September 2024) gives the textbook case: a chunk that reads “The company’s revenue grew by 3% over the previous quarter,” naming neither the company nor the quarter. Its fix, adding document context to each chunk before indexing and pairing vector search with keyword search, cut failed retrievals by 49% on its tests, and by 67% with reranking. That’s a real gain in finding the passage. The model still has to read the number out of it correctly and type it into the answer.

The top k misses the footnote

Ask for a company’s off-balance-sheet commitments and the answer is spread across a commitments note, a lease table, an unconsolidated joint venture described in another note, and an exhibit. Retrieval returns the k chunks most similar to the question, and the passages that matter most often use different words from the question. FinanceBench, Patronus AI’s 2023 benchmark of questions on public filings, showed the pattern: configurations that handed the model long stretches of the relevant filing did better than retrieval from a vector store, which points at retrieval as the step where answers were lost. The authors also judged long context impractical for enterprise use, because of latency and because larger filings don’t fit.

Retrieving more chunks has its own cost

The obvious response is to retrieve more. Lost in the Middle (Liu and co-authors, 2023) found that models use information best when it sits at the beginning or end of a long context and significantly worse when it sits in the middle, including models built for long contexts. Retrieving twenty chunks to be safe puts most of them in the middle, where the model reads least reliably.

The model retypes every digit

Even with the right chunk in context, the number in the answer is generated token by token, beside a citation. That’s how a real figure lands under the wrong period or scale. Our guide to AI hallucinations in finance covers the five ways it goes wrong, and our research note The quieter hallucination explains why better models make the problem harder to see.

No complete lists

“Every company in our coverage that disclosed a customer above 10% of revenue in its latest 10-K” has one right answer, and it might be 40 companies. Retrieval returns k chunks by design, so the answer is whichever companies happened to rank highest, presented as if they were all of them. Microsoft Research’s post introducing GraphRAG (February 2024) describes the same limit from another angle: baseline RAG struggles with questions that aggregate across a whole dataset and with connecting facts spread across documents.

No entities and no time

A vector index knows text. It can’t tell that “Alphabet,” “Google” and “GOOGL” are one company, or that one manager’s Fund III has nothing to do with another manager’s Fund III. It also has no notion of when a fact was true. A 10-K prints the prior year, sometimes restated, beside the current one, so a search for last year’s revenue can return the restated figure from this year’s filing or the original from last year’s, and the answer won’t say which. Backtests and point-in-time models need the figure as it was known on the date, which a text index can’t supply. Our guide on checking AI-generated numbers includes a test for exactly this.

RAG vs knowledge graphs vs an ontology

Approach	What it stores	Good at	Where it falls short on financial data
Vector RAG	Text chunks and embeddings	Finding relevant passages fast	Tables, complete lists and time; the model types every number
Hybrid RAG	Chunks plus a keyword index, contextual chunks and reranking	Fewer missed passages	Still returns the top k; numbers still typed by the model
GraphRAG	Entities and relationships a model extracts, plus cluster summaries	Themes across a corpus, questions that hop across documents	The graph is only as good as the extraction; numbers still written by the model
Ontology with facts and code (Kepler)	Resolved entities, typed facts with period, unit and source, your firm’s definitions	Complete lists, numbers computed in code, any figure as of any date	Needs your firm’s ontology built first; our engineers build it in your first week

GraphRAG, as Microsoft describes it, has a model read the whole dataset to pull out entities and relationships, groups them into clusters, and summarizes each cluster ahead of time, so questions about themes and connections draw on the graph as well as the text. That’s a step toward structure. An ontology, which is what Kepler builds for each firm, goes further: every alias resolves to one record, every fact carries its period, unit and the sentence it came from, and the firm’s definitions live in the model as formulas. Our guide to what an ontology is covers the difference in depth.

What works instead

The architecture that holds up on financial data, and the one Kepler runs on, moves the hard work from question time to ingest time, and from the model to code.

1. Extract at ingest. Read every page of every document once, when it arrives, including scans, tables and exhibits. Store each fact with its value, scale, unit, sign, period, entity and source passage, so a follow-up question becomes a lookup.
2. Resolve entities into an ontology. Map every alias to one record, attach the firm’s definitions, and give every fact the dates it was true and the date it was recorded. That’s what makes “as filed” and “as restated” two separate answers.
3. Query for structured questions. “Every company with a customer above 10% of revenue” runs as a query over extracted facts and returns all of them, with a count you can check.
4. Compute in code. Margins, multiples and exposures run as code over cited facts, with the formula shown.
5. Check citations against sources. Confirm each cited passage contains the figure before the answer is shown, and strip any that doesn’t.
6. Keep retrieval for language. Finding where management discussed pricing, reading the wording of a new risk factor, or gathering the passages that bear on a qualitative question is what retrieval does well. It stays, as one input among several.

Our CEO, Vinoo Ganesh, made the wider case in Context Is the Easy Part: adjusting chunk sizes and retrieval parameters is prompt engineering with more inputs, and systems that have to hold up in production need provenance, versioning, validation and determinism around the context they hand a model.

When plain RAG is enough

RAG is the right tool when nobody needs an exact number or a complete list: questions over an internal policy library, a summary of a single document, finding which call discussed a topic, a first draft from a few known sources. Once the output feeds a model, a memo, an investor letter or a compliance check, it needs the pieces above. To connect an assistant to financial data through tool calls instead of passages, see MCP for financial data. If your team is weighing building a pipeline in house, build vs buy sets out what the full stack takes.

How Kepler does this

Retrieval is one layer of Kepler. It finds passages and helps the model read language, and it never supplies a number.

Every document read once, in full. Extraction models we train read every page at ingest: 26M+ SEC filings in full text and XBRL, 40M+ investor relations documents and earnings call transcripts, and your firm’s SharePoint, email, data rooms and models, connected read-only and kept in sync. Each fact is stored with its value, scale, unit, period and passage. Our own models read a document at roughly 700 times less cost than a frontier model, which is what makes reading everything practical (Stop Paying to Forget).

Your firm’s ontology. Our engineers build a living model of your firm from those documents. Every company, fund, manager and deal resolves to one record, your definitions carry one formula, and every claim points to the sentence it came from and keeps the dates it held. Lists come back complete because they’re queries over that model.

Numbers from the engine. Kepler’s deterministic engine pulls every figure from extracted facts and computes every derived value in code. Financial statements assemble from filers’ own XBRL with no model in the path, as filed or as restated, on any date. The model names facts by reference and never types a digit.

Retrieval where it helps. When a question turns on language (how management described demand, what a side letter says about fees, which risk factors are new), Kepler retrieves the passages and the model reads them, with each claim cited to its sentence. Every citation is checked against its source before you see the answer, and conflicting sources appear side by side. Agents that watch new filings overnight query the same extracted facts under the same access check on every read, so an alert quotes its sentence and every run is logged.

The right model for each step, everywhere you work. Small models we train do the reading, and frontier models handle the question and the write-up; we swap in better ones as they ship. The same answers reach Excel, Claude and ChatGPT through our MCP server, and your engineers’ apps through the API, with every number still cited. Our engineers built this layer before at Palantir.

Questions that need more than retrieval:

• “Every 10-K in our coverage disclosing a customer above 10% of revenue, with the sentence for each.”
• “Off-balance-sheet commitments in each of the last five 10-Ks, as filed.”
• “Which risk factors are new this year, and which figures in MD&A changed from the original filing?”

Try Kepler at app.kepler.ai, or book a demo and bring the question your current tool answers with the top ten hits.

Frequently asked questions

Is RAG good for financial data?

RAG is good at finding relevant passages and weak at numbers, complete lists and time. Kepler uses retrieval as one layer, for passages and language, while every number comes from facts extracted when each document arrives, resolved into the firm's ontology and computed in code, with each figure cited to its page, line or cell.

What is RAG in finance?

Retrieval-augmented generation splits filings, transcripts and fund documents into chunks, indexes them, retrieves the few chunks most similar to a question, and has a language model write the answer from them with citations. It's the default architecture behind most document chatbots and many AI research tools.

Why does RAG get financial numbers wrong?

Chunking can separate a table's rows from the header that gives their scale and period, retrieval can miss the footnote that holds the answer, and the model still retypes every digit it reads. Each step can lose context, and a citation to the right chunk doesn't guarantee the number came from it.

What is the difference between RAG and a knowledge graph?

RAG stores passages of text and finds them by similarity at question time. A knowledge graph stores entities and the relationships between them, so it can answer questions that connect facts across documents. An ontology adds the schema and definitions that give the graph meaning, and in finance it also needs time: what was true as filed, as restated and on any date.

When is RAG enough?

When nobody needs an exact number or a complete list: answering questions over a policy library, summarizing one document, finding which call discussed a topic, or drafting from a few known sources. Once the output feeds a model, a memo, an investor letter or a compliance check, it needs extracted facts, calculations in code and checked citations.

Can you make RAG more accurate on financial documents?

Partly. Adding document context to each chunk, combining keyword and vector search, reranking, and keeping tables whole all cut retrieval misses. None of them stops the model from retyping digits or returns a complete list, which is why platforms such as Kepler extract facts at ingest and compute numbers in code.