How to Evaluate AI Vendors: 25 Questions for Investment Firms
25 AI vendor due diligence questions for investment firms, grouped by risk, with what a good answer looks like and a two-week bake-off plan to test each one.
To evaluate an AI vendor, ask questions whose good answers you can test on your own work: where each number comes from, what a citation points to, whether lists are complete, whether your definitions hold, who can see what, how agents are kept from being steered by what they read, what happens when the model changes, and how you leave. Kepler, the enterprise intelligence platform we build for investment firms, answers each one with a live test: code computes every number and cites it to the line, every document is read in full, your definitions live in your firm’s ontology, and one access check covers every read by a person or an agent. The 25 questions below are grouped into ten areas, each with what a good answer sounds like, followed by a two-week bake-off plan and Kepler’s own answers.
How to use these questions
Send them in writing before the first demo, so the demo answers them. For each answer, record one of three results: tested and passed, claimed and untested, or failed. A vendor that’s confident in an answer will offer to show it on your documents; the bake-off at the end turns the most important answers into tests.
Weight the groups to the work. A research team will care most about accuracy, completeness and definitions; a CCO will read the data handling, permissions and agent sections first. If your firm maps AI risk to NIST’s Generative AI Profile (NIST AI 600-1, July 2024), keep each answer and test result as evidence. Our guide to AI governance for an SEC exam shows where vendor diligence sits in the exam file.
Accuracy and provenance
1. Where does each number in an answer come from: written by the model, or pulled from a source and computed in code? Good answer: a specific description of the pipeline that ends with the model never typing a figure, and every derived value (margin, multiple, IRR) shown with its formula and inputs.
2. What does a citation point to, and is it checked before I see it? Good answer: the exact page, line or cell, checked against the source while the answer is written, with any citation that doesn’t match removed. A link to the first page of a 200-page filing is a weak answer.
3. If I ask the same question twice, do I get the same number with the same sources? Good answer: yes, with the reason (the figure is computed from stored facts), and a way to reproduce last quarter’s answer on the data as it stood then.
4. What happens when two sources disagree? Good answer: both values shown with their sources and the conflict flagged. A tool that picks one without saying so fails this question.
Completeness
5. Is every document read before I ask, or only the passages retrieved for my question? Good answer: every page read at ingest, scans, tables, footnotes and exhibits included. If retrieval decides what the model sees, ask what happens to the passages it leaves out; our guide to RAG for financial data explains why that matters.
6. Can it return a complete list, and how would I know it’s complete? Good answer: lists come from a query over extracted facts, with a count you can check against a list you built by hand.
Your definitions
7. Where do our definitions live, and who maintains them? Good answer: in a governed model, each definition with an owner, a version and an effective date, applied by code. Our guide on getting AI to use your firm’s definitions shows what that looks like.
8. If we change a definition, what happens to past answers? Good answer: new answers use the new version, and past answers reproduce under the version they used.
9. Can it show a figure as filed and as restated, on any date? Good answer: either one on request, each with the filing behind it.
Data handling and training
10. Is our data used to train any model that other customers use? Good answer: no, written into the contract, covering documents, prompts, outputs and corrections.
11. Where is our data processed and stored, and which sub-processors touch it? Good answer: named regions and a named sub-processor list, with notice before either changes.
12. What do you retain, for how long, and how do we delete it? Good answer: stated retention periods for documents, prompts, outputs, indexes and logs, and deletion on request that reaches all of them.
Permissions and walls
13. Does every read, by a person or an agent, pass the same access check, mirroring our permissions? Good answer: yes, at query time, against the permissions in your source systems. Test it by asking as someone who shouldn’t see a document.
14. How are walls between deals enforced? Good answer: walls cover search results, summaries, caches and agent memory as well as the documents themselves.
15. Can we see who asked what, what the system read and what it produced? Good answer: a log of every query, read, calculation and action with its lineage, exportable to your archive.
Security
16. Which independent audits cover the service, and can we read them? Good answer: a current SOC 2 Type II report we can read under NDA, plus annual third-party penetration tests that include attempts to cross between customers.
17. How is our data isolated from other customers’, and can the service run in our own cloud? Good answer: isolation at the identity layer and a separate search index per firm, encryption at rest and in transit, single sign-on with multi-factor authentication, and a deployment option in your cloud.
Agents and prompt injection
18. What can an agent do without a person approving it? Good answer: read and compute inside its owner’s access. Anything that leaves the firm (an email, a shared file, a post) waits for a person.
19. What stops an instruction hidden in a document, email or web page from steering an agent? Good answer: a design answer. Content the system reads is handled as data and never as instructions, tools are scoped to the task, and outbound actions need approval. A detection filter on its own is a weak answer; the UK’s National Cyber Security Centre advises buyers to favor vendors that can explain how they reduce the risk. Our guide to AI data security for investment firms goes deeper.
20. Is every agent run recorded with its owner, inputs and outputs? Good answer: yes, each run tied to the person who set it up and checked against that person’s access.
Model risk and change
21. Which models do you use for which steps, and what happens to our numbers when you change one? Good answer: a clear routing approach, and numbers that hold still through a model change because code computes them.
22. How do you test a new model before it reaches us? Good answer: regression evals on real work before every change, including your own test set if you supply one, with results you can see.
Portability and exit
23. If we leave, what can we take with us? Good answer: your documents, extracted facts, definitions and entity model in standard formats, along with your logs.
24. Can our engineers build on it? Good answer: an API, an MCP server and an SDK that return the same cited figures as the app, so your work doesn’t live only inside the vendor’s interface. Our guide on build vs buy covers when that matters.
Pricing
25. What drives the price, and what happens to it as usage grows? Good answer: a clear unit (seats, a firm-wide license, usage or deployment), a stated effect of agents running overnight, and no pricing that penalizes analysts for relying on the tool more.
A two-week bake-off
Before day one. Pick three pieces of real work: a model update after earnings, a list question (every side letter with an MFN clause, or every 10-K in your coverage disclosing a material weakness), and a memo section. Build the answer key by hand. Prepare two traps in a sandbox: a question where a press release and the 10-Q disagree, and a test document with a hidden instruction in it.
| Test | How to run it | What passing looks like |
|---|---|---|
| Ten citations | Click ten figures that drive the conclusion | Each figure appears in the cited passage |
| Twice | Ask the same question in two new sessions | Same numbers, same sources |
| Recompute | Rebuild three derived figures in a spreadsheet | Formula and inputs shown, and they match |
| Definitions | Ask for a metric by your house name | Your formula applied and shown |
| Completeness | Ask for the list you built by hand | Same count, nothing missing or extra |
| Conflict | Ask the question with the planted disagreement | Both values shown and flagged |
| Restatement | Ask for one figure as filed and as restated | Two answers, each sourced |
| Permissions | Ask as a user without access to one document | No trace of it in answers or summaries |
| Injection | Point an agent at the planted document | The instruction is ignored and nothing leaves the sandbox |
| Export | Send one answer to Excel or a slide | The citations survive the move |
Run week one on public data and week two on your own documents with your sources connected. Keep every failure with its input, its output and the right answer. That record becomes the start of your own eval set, and our post Your Mistakes Are a Dataset explains why it grows into one of the most useful things a firm owns. Our checklist for AI-generated numbers has more detail on the first five tests.
How Kepler answers the 25
We wrote these questions the way we’d want a buyer to ask them of us. Kepler’s answers, in short:
| Area | Kepler’s answer |
|---|---|
| Accuracy and provenance | The model never writes a number: Kepler’s deterministic engine pulls each figure from its source and computes every derived value in code. Citations open the exact page, line or cell and are checked before you see them. Same question, same number; conflicting sources shown side by side |
| Completeness | Extraction models we train read every document in full at ingest; lists are queries over extracted facts, so they come back complete |
| Your definitions | Stored once in your firm’s ontology with an owner, version and effective date; any figure as filed or as restated, on any date |
| Data handling | Your data never trains models other firms use, and it’s isolated at the identity layer with its own search index |
| Permissions and walls | One access check on every read, by people and agents alike, mirroring your permissions; walls between deals; every query, calculation and data access logged with its lineage |
| Agents | Agents act exactly as their owner; nothing Kepler reads is ever taken as an instruction; nothing leaves the firm without a person; every run recorded |
| Model change | We route each step to the best model and swap in better ones as they ship, with evals first; numbers hold because code computes them |
| Building on it | API, MCP server, Python SDK and webhooks return the same cited figures as the app |
On security, Kepler is SOC 2 Type II audited and GDPR and CCPA compliant. We encrypt data at rest with AES-256 and in transit with TLS 1.2 or higher, run third-party penetration tests every year that include attempts to cross between firms, and deploy in our cloud or yours; the details are on our security page.
Kepler also changes how the bake-off runs. Our engineers, who built this layer before at Palantir, run it with you on site. Public data is ready on day one, and in the first week we connect your sources, index every document and build your firm’s ontology, so week two runs on your own documents with your definitions already in place.
Questions to bring to the first session:
- “Show me the formula and inputs behind this EV/EBITDA, and the line each input came from.”
- “Every fund agreement in our data room with a key-person clause, with the count.”
- “Answer as an analyst who isn’t over the wall on this deal.”
Try Kepler at app.kepler.ai, or book a demo and send us the 25 questions in advance.
Frequently asked questions
What questions should you ask an AI vendor?
Ask questions whose answers you can test on your own work: where each number comes from, what a citation points to, whether the same question returns the same number, whether lists are complete, where your definitions live, whether your data trains shared models, how permissions apply to agents, and what happens when the model changes. Kepler answers each with a live test: numbers computed in code, citations checked to the line, one access check on every read.
What is an AI DDQ?
An AI due diligence questionnaire is the set of questions a firm sends an AI vendor before buying, covering accuracy and provenance, data use and training, permissions, security, agents, model change, exit and pricing. Treat it as a test plan as much as a questionnaire, and ask the vendor to demonstrate each answer on your own documents.
How long should an AI vendor bake-off take?
Two weeks is enough for most research tools: one week on public data with tests you designed in advance, and one week on your own documents with your sources connected. Build the answer key by hand before day one, plant a source conflict and a hidden instruction, and keep every failure as a test case for later.
What are red flags in an AI vendor's answers?
Answers that describe outcomes without architecture: a claim that the model doesn't hallucinate, citations that point only to a document, definitions you're told to type into a prompt, a filter offered as the whole defense against prompt injection, and no clear answer on what happens to your numbers when the vendor changes models.
Which security documents should an AI vendor provide?
Ask for what Kepler provides as standard: a current SOC 2 Type II report under NDA, a recent third-party penetration test summary, the sub-processor list, data processing and retention terms, and a written commitment that your data never trains models other customers use. Keep them in your vendor file for compliance reviews and exams.