How to Check AI-Generated Numbers in Financial Research

To check an AI-generated number, open its source and confirm four things: the figure is there, the period is right, the units and scale are right, and the basis is the one you meant (GAAP or adjusted, as filed or as restated); then recompute anything derived, ask the question a second time, and test whether any list the tool gave you is complete. Kepler, which we build, is designed so every one of those checks passes: code pulls and computes every number, cites it to the exact page, line or cell, and checks each citation against its source before you see it. The ten checks below work on any tool, from a general chatbot to a finance platform, and take about ten minutes on a real piece of work; the last section shows how Kepler handles each one.

Why AI numbers go wrong

A language model doesn’t copy a number from a filing. It writes one, token by token, and usually gets it right. That’s the root of most failures, and it’s why better models make errors rarer without making them visible.

A financial fact is more than its digits. It’s a value plus a currency, a scale, a sign, a period, a concept and an entity. Change any one and you have a different fact with the same digits. Our research note The quieter hallucination shows this with LinkedIn’s 10-Q for the quarter ended March 31, 2016: the string 1,322,500 (in thousands) is tagged three times, as the principal of its convertible notes at March 31, 2016, the same principal at December 31, 2015, and the face amount on the day the notes were issued in 2014. Search for the digits and all three match.

That gives three ways for an AI number to be wrong:

• Wrong digits. The value changes on the way into the answer.
• Wrong fact. The digits are right and the context is wrong: thousands read as units, December’s column reported as March’s.
• Wrong choice. Every fact is right and the question needed a different one. Price Microsoft’s $196-a-share offer for LinkedIn using the balance-sheet carrying amount of the notes instead of the principal a buyer must repay, and enterprise value is off by $184 million, with a perfect citation.

Retrieval adds its own failures. Most AI tools read only the passages they retrieve for your question, so anything the search step misses is invisible to the answer.

Failure patterns, with documented examples

Each pattern below has happened in public. Our guide to AI hallucinations in finance covers why they happen; this section is about recognizing them.

Figures that aren’t in the source

In Microsoft’s February 2023 launch demo, Bing’s chatbot summarized Gap’s quarterly earnings release and gave an operating margin of 5.9%. The release showed 4.6%, and 5.9% appears nowhere in it, as CNBC reported after an independent researcher went through the demo line by line.

Retrieval that misses the answer

FinanceBench, published by Patronus AI in November 2023, is a set of 10,231 questions about public companies’ filings. On a sample of 150, GPT-4-Turbo paired with a retrieval system answered incorrectly or refused 81% of the time. Models have improved since, and the mechanism hasn’t changed: when the retrieval step brings back the wrong pages, the answer is built on them.

Citations that don’t hold up

In a study published in March 2025, Columbia’s Tow Center tested eight AI search tools on 1,600 queries, asking each to identify the source of a quoted article excerpt. Collectively they got it wrong more than 60% of the time, and they rarely signaled doubt. In October 2025, Deloitte agreed to partially refund the Australian government for a report that cited nonexistent sources and a fabricated quote from a court judgment; the revised version disclosed that a generative AI system was used in writing it.

Two numbers for one question

Thinking Machines Lab sampled 1,000 completions of the same prompt at temperature 0 from an open model running on a standard inference server in September 2025 and got 80 different completions. Settings that look deterministic often aren’t, so a figure that changes between runs is a sign the model is generating it rather than looking it up.

Errors that still add up

Wall Street Prep tested four AI tools by asking each to build Apple’s three-statement model in Excel from its latest 10-K and fourth-quarter press release (published February 2026, updated June 2026). Two of the four tools hallucinated significant portions of the historical data, in errors the testers called “subtle enough to be dangerous”: “slightly incorrect line items, all adding up to correct subtotals.” Their bottom line was that “even the best tool still underperforms a Junior Analyst.” For your own checks, a subtotal that ties doesn’t clear the lines under it, so test individual line items as well.

A ceiling on analyst tasks

On Vals AI’s Finance Agent benchmark (v2.1), built around the work of entry-level analysts on public-company filings, the top model, Claude Opus 5.5, scored 67.85% as of its October 29, 2026 update, which leaves about a third of the credit on the table.

Failure	What it looks like	The check that catches it	On Kepler
Invented figure	A plausible number with no match in the source	Open the citation and find the exact figure	Pulled from source; the model never writes a number
Wrong period or column	Right digits from the prior year or quarter	Read the column header and date	Period stored with every fact
Wrong scale or unit	Off by 1,000x, or per-share read as total	Check the table header and currency	Scale, unit and currency stored with every fact
Wrong basis	Adjusted vs GAAP, restated vs as filed	Confirm the basis the source states	As filed or as restated, on any date
Bad derived figure	Margin, growth or multiple that doesn’t recompute	Recompute from the cited inputs	Computed in code, formula and inputs shown
Incomplete list	Top hits presented as all of them	Compare against a list built by hand	Every document read at ingest; complete lists
Silent conflict	One of two disagreeing sources chosen	Plant a known conflict and see if it’s flagged	Both values shown with sources, conflict flagged

The checklist: ten ways to verify any AI tool’s numbers

1. Click ten citations. Choose numbers that drive the conclusion. For each, confirm the exact figure appears in the cited passage, page or cell. A citation that points to the right document but the wrong paragraph counts as a fail.
2. Check the period. Fiscal or calendar year, quarter or year-to-date, trailing twelve months or annual. Filings print several periods side by side; make sure the number came from the right column.
3. Check units, scale and currency. Look for “in thousands” or “in millions” in the table header, the reporting currency, and whether a figure is per share or total.
4. Check the basis. GAAP or adjusted, basic or diluted, consolidated or segment, as filed or as restated. Restated comparatives in a later filing can differ from the original; our guide to tracking restatements and reclassifications shows where to look.
5. Recompute derived figures. Ask for the formula and every input for each margin, growth rate, multiple and enterprise value, and rebuild one in a spreadsheet.
6. Ask twice. Run the same question in a new session. Any number that changes is a draft.
7. Test your definitions. Ask for a metric the way your firm defines it, such as adjusted EBITDA with your add-backs, or a portfolio company by its internal name. See whether the tool applies your definition or a generic one.
8. Test completeness. Build a small list by hand, such as every 8-K a company filed last year or every side letter in a data room, and ask the tool for the same list. Compare counts.
9. Plant a conflict. Pick a case where two sources disagree, like a press release and the 10-Q, or an original and a restated figure. A good tool shows both and says which it used.
10. Follow one number out of the tool. Export to Excel or a slide and check that the citation survives. A number that loses its source on the way into a deck has to be checked again by whoever reads the deck.

Running the checks in practice

Sample by risk. Every number that reaches an IC memo, client report or LP letter deserves a check. So do derived figures, outliers and anything read from a scanned page.

Keep a log. Record each number checked, its source, pass or fail, and the failure type. A month of entries tells you which tools to trust for which jobs. It also gives your compliance team a record of how AI output was reviewed, which examiners increasingly ask about; see AI governance for investment firms.

Raise the bar after a failure. If a sample of ten turns up one wrong figure, check every figure in that output, and tell the vendor.

Build the checks into vendor selection. Run all ten on each tool you’re considering, on the same work. Our 25 questions for evaluating AI vendors turn them into a two-week bake-off.

General assistants are useful for drafting and summarizing, as Can ChatGPT do financial analysis? explains, and they’re where these checks matter most. For filings work specifically, compare tools in the best AI for SEC filing analysis.

How Kepler removes the re-check

We built Kepler so the ten checks above pass by design, and analysts spend their time on judgment. Here is how it handles each failure.

On Kepler, the model never writes a number. Our deterministic engine pulls every figure from its source and computes every derived value as code over cited inputs. The model names a fact by reference, and the platform renders the filed value, so the digits in the answer can’t drift from the digits in the source. Ask the same question twice and you get the same number with the same sources. We match each step to a model, our own extraction models for reading and frontier models for planning and drafting the answer, and since none of them writes a figure, upgrading to a better model as one ships leaves every number where it was.

Every citation points to the line, and it’s checked first. Click any figure to land on the exact page, line or cell. A calculated figure opens as a small sheet showing the formula, each input and where each input came from. We check every citation against its source while the answer is written, and strip any that doesn’t match before you see it.

Context travels with the fact. Extraction models we train read every document in full at ingest and keep each fact with its value, scale, unit, period and passage. In our published tests, 99.4% of tagged facts were read at the right scale across 40 held-out filings (Stop Paying to Forget). Statements assemble from filers’ own XBRL with no model in the path, as filed or as restated on any date. Our paper Reclassified to Conform found 194 of 200 companies reshape a statement in ways XBRL tags can’t follow, and recovers the history.

Your definitions and your documents. Your firm’s ontology in Kepler holds your definitions, so adjusted EBITDA means yours every time. Your playbooks, such as the earnings update in your PM’s format, are saved once and run the same way every time, with every figure still cited. Because every document was read before you asked, lists come back complete. When two sources disagree, Kepler shows both and flags the conflict.

The citations go where the work goes. Kepler answers keep their sources in Excel, PowerPoint and Word, inside Claude and ChatGPT, and through our API. Agents that update models overnight record every run with its lineage. Our engineers, who built this layer before at Palantir, connect your firm’s own documents in the first week, so the same checks hold on your data as well as on public filings.

Run the ten checks above on Kepler at app.kepler.ai, see how enterprise intelligence fits the pieces together, or book a demo on your own coverage.

Frequently asked questions

Why does AI get financial numbers wrong?

A language model writes numbers rather than copying them, so digits can drift on the way into the answer. Kepler is built around that problem: code pulls and computes every figure, so the model never writes one. Even with the right digits, a frontier model on its own can attach the wrong period, scale or basis, read only the passages it retrieved, or pick one of two conflicting sources without saying so, and a citation attached after the answer is written doesn't prove the number came from the source.

Is a citation enough to trust an AI-generated number?

No. A citation shows that a source exists and is related to the answer. Open it and confirm the exact figure appears there, for the right period, in the right units and on the right basis. If the number was calculated, ask for the formula and every input, and recompute it yourself.

How many AI-generated numbers should I check?

When you first test a tool, check ten figures from a real piece of work, weighted toward the numbers that drive the decision and anything derived. If one fails, check every figure in that output. Once a tool computes numbers in code and cites them to the line, as Kepler does, a smaller spot check on high-stakes figures is enough.

Does asking the same question twice help?

Yes. Ask the same question in a fresh session. Language models can return different numbers to identical prompts, even at temperature zero. If the figures change, treat both answers as drafts and go to the source. A platform that computes numbers in code, such as Kepler, returns the same figure with the same source every time.

What is the fastest way to check an AI financial analysis?

Open the citation for the three to five numbers the conclusion depends on, confirm period, units and basis for each, and recompute any ratio or growth rate from its inputs. That takes about ten minutes and catches most errors that matter. For lists, compare the count against one you built by hand.

Is there an AI tool that doesn't generate the numbers itself?

Yes. Kepler, an enterprise intelligence platform for investment firms, pulls every figure from its source and computes derived values in code, so the model never writes a number. Each figure links to its exact page, line or cell, every citation is checked against its source before you see it, and the same question returns the same number.