How to evaluate AI vendors: a checklist

To evaluate an AI vendor, test it on your own data, not the demo. Ask who reviews each result, what happens to your data, how the tool connects to the systems you already run, and how you get out. Then compare price and references against the vendor's peers in the same market.

Key takeaways

• Test on your own samples. Buyers in Airframe's published markets ask for accuracy on their own invoices and their own code, not on a benchmark the vendor picked.
• Ask who checks the output. A good answer names the review step, the person who does it and the record it leaves.
• Read the loss reasons. Cost, setup effort and speed show up again and again in the loss reasons on Airframe's AI Audit Agent, FP&A and AP Automation market pages.
• Use a framework for the risk questions. NIST's AI Risk Management Framework is free and voluntary. It groups the work under four functions: Govern, Map, Measure and Manage.

Before you talk to vendors

1. Write down one workflow. Name the task, the inputs, the output and who uses it.
2. Collect a test set. Pull real past examples where you already know the right answer. Include the messy ones, because that's where tools break.
3. Name the owner and the reviewer. One person owns the decision. Someone else checks the output.
4. Decide what "good enough" means. Set the accuracy, time saved or error rate you need before you see a demo.
5. Decide build or buy first. If the workflow is unique to you, building may fit. See the Build-vs-Buy Map.

The checklist

1. Accuracy on your data

• Run the vendor on your test set, not theirs.
• Ask for accuracy by field and by document type, not one blended number.
• Ask what manual checks catch the errors.

Buyer questions from Airframe market pages:

• "What extraction accuracy do you hit on our invoice samples by field and type?" (AP Automation)
• "What is your measured false positive rate on our own repositories, and how do you prove that number in a pilot?" (AI Code Review)

2. How a person reviews the output

• Where does a person approve, reject or correct a result?
• Does the system change its behavior after a rejection? How?
• Is every action logged with a time stamp, and can you export the log?
• "How do you measure and cut comment noise, and how do future reviews change after our reviewers mark a comment as wrong?" (AI Code Review)

3. Your data

• How long does the vendor keep your data and prompts?
• Does any model provider train on your data?
• Can it run in your cloud, self-hosted or air-gapped? What stops working if it does?
• "How long do you keep our code and prompts, and does any model provider train on our source code?" (AI Code Review)
• "Can we run the platform self-hosted or air-gapped, and which review features stop working without internet access?" (AI Code Review)

4. Integrations

• Which systems does it connect to natively, and how deep does each connector go?
• Which APIs move the data, and on what schedule?
• "Which ERPs do you integrate with natively and how deep do field mappings go?" (AP Automation)
• "What native connectors move data from our ERP, HCM, warehouse, and BI stack?" (FP&A)

5. Risk and governance

Map the vendor's answers to the four functions of the NIST AI Risk Management Framework↗ (opens in a new tab): Govern, Map, Measure and Manage. For generative AI, NIST also publishes a Generative AI Profile (NIST AI 600-1). Both are voluntary, and neither certifies a product.

• Govern: Who at the vendor owns AI risk? What policy covers model changes?
• Map: What is the tool meant to do, and where shouldn't it be used?
• Measure: How does the vendor test accuracy, bias and drift, and how often?
• Manage: What happens when the output is wrong? Who gets told, and how fast?

6. Price and terms

• Is the price published, or quote only?
• What drives the bill: seats, volume or outcomes?
• What does it cost to leave? Can you export your data and configuration?

Some Airframe market pages list prices as published on each vendor's own pricing page. Where a vendor doesn't publish a higher tier, the page marks it "by quote."

7. Proof from peers

• Ask for references at companies your size, in your industry, on your stack.
• Read published case studies. Note what was measured and what was only claimed.
• Check who is winning switches in the category, and from whom.

Why AI vendors lose deals

Airframe's market pages list the reasons buyers walk away. Treat them as things to test early.

• AI Audit Agent: "Cost is high." "Speed is slow." "Time to value is late."
• FP&A: "Price runs high for small teams and setup stays hard." "Speed drops with very large models and complex data."
• AP Automation: "Set up is hard and ERP sync fails." "Cost is high and fees lack clarity."
• AI Code Review: "The buyer requires certifications that the vendor lacks." "The price per seat does not scale for large teams."

Run a short pilot

1. Use the test set from step 2. Agree the pass mark with the vendor in writing.
2. Run it on past work your team already checked, so you know the right answers.
3. Try to break it: missing fields, duplicates, the wrong period, a user without access.
4. Measure review time as well as accuracy. A tool whose every line needs checking saves little.
5. Decide: expand, narrow the scope, or stop.

Questions about evaluating AI vendors

What should an AI vendor evaluation checklist include?

Accuracy on your own data, how a person reviews output, data retention and training, integrations, risk governance, price and exit terms, and references from peers.

How do you test an AI tool's accuracy?

Run it on your own past cases where you know the answer. Report results by field or case type. Agree the pass mark before the pilot starts.

What is the NIST AI RMF?

A voluntary framework from the US National Institute of Standards and Technology for managing AI risk. Version 1.0 came out on January 26, 2023. Its core functions are Govern, Map, Measure and Manage.

What questions do buyers ask AI vendors?

Airframe publishes the top buyer questions for each market it tracks at airframe.ai/markets. AI code review buyers, for example, ask about false positive rates on their own repositories and whether model providers train on their code.

Should we build or buy AI?

It depends on how unique the workflow is and who will maintain it. Airframe's Build-vs-Buy Map shows where companies build instead of buy, category by category.