AI Vendor Evaluation Scorecard: 30 Questions, 10 Red Flags

In brief

Score an AI vendor on 30 governance questions across six categories, and check ten red flags that pause the deal whatever the score. Three cases sit behind it. Italy's regulator fined the maker of Replika EUR 5 million in 2025. A US court ruled in 2024 that a discrimination claim against Workday, as the agent of the employers using its screening tools, could go ahead. A Canadian tribunal held Air Canada responsible for what its chatbot told a customer. In each, a problem no demo would have shown reached a regulator or a court. The scorecard gives procurement, security and legal one shared instrument: six category scores from 1 to 5, a total from 6 to 30, four risk bands, and a rule that three red flags mean reject. Run it on every vendor that touches decisions about people, regulated data or customers, starting with a 12-week rollout.

Key Takeaways

→The scorecard asks 30 questions in six categories, scores each category from 1 to 5, and sorts a vendor's total of 6 to 30 into four risk bands.
→Ten red flags override the score: one pauses procurement and three mean reject, so a 28 with a red flag ranks below a clean 20.
→In Mobley v. Workday, a US court let a discrimination claim proceed in July 2024 on the theory that an AI screening vendor acts as its employer customers' agent. The case is still running.
→Under EU AI Act Article 25, a deployer that rebrands or substantially modifies a high-risk system becomes its provider, with fines of up to EUR 15 million or 3% of worldwide turnover.
→A 12-week rollout covers the vendors that matter: the inventory in weeks 1 to 2, Tier 1 scoring in weeks 3 to 6, and contract renegotiation in weeks 7 to 10.

Every AI vendor says it has enterprise-grade governance. A demo won't tell you whether that is true, and neither will an RFP response or a SOC 2 report. The vendor wrote the first two to be passed and set the scope of the third. You find out with questions the vendor didn't write.

What Is the Six-Category Governance Scorecard?

The Six-Category Governance Scorecard is a 30-question assessment that rates an AI vendor on six categories, each scored 1 to 5: model transparency, bias and fairness, data privacy and security, accountability and support, regulatory compliance, and lock-in and portability. Ten red flags sit beside the score, and any one of them pauses procurement whatever the total.

Score each question from 1 (no capability) to 5 (best in class).

Category	The five questions	Red flags in this category
Model transparency	1. Does the vendor provide model documentation (model cards, data sheets)? 2. Can you inspect training data composition and sources? 3. Are model updates communicated with change logs? 4. Can you access performance metrics and drift reports? 5. Is the decision logic explainable to non-technical stakeholders?	Refuses to disclose training data sources. Ships model changes without notification.
Bias and fairness	1. Has the vendor conducted fairness audits on the model? 2. Can you run independent bias testing on their system? 3. Are fairness metrics (demographic parity, equalized odds) available? 4. Is there a documented process for addressing bias when found? 5. Does the vendor publish bias audit results?	No bias audit has ever been conducted.
Data privacy and security	1. Where is your data stored (jurisdiction, encryption, isolation)? 2. Is your data used to train or improve the vendor's models? 3. Can you delete your data completely on request? 4. Does the vendor comply with GDPR, CCPA and relevant regulations? 5. What happens to your data if the contract ends?	Trains on your data without explicit consent.
Accountability and support	1. Is there a named individual accountable for AI governance at the vendor? 2. What is the incident response SLA for AI-related issues? 3. Does the vendor carry AI liability insurance? 4. Can you audit the vendor's AI governance practices? 5. Is there an escalation path for AI ethics concerns?	No incident response SLA for AI issues. No one accountable for AI governance. No independent audit access. No AI liability insurance.
Regulatory compliance	1. Is the vendor compliant with EU AI Act requirements? 2. Does the vendor align with NIST AI RMF? 3. Can the vendor provide conformity assessment documentation? 4. Does the vendor track regulatory changes proactively? 5. Is there evidence of third-party compliance certification?	Cannot provide EU AI Act conformity documentation for a high-risk use.
Lock-in and portability	1. Can you export your models, data and configurations? 2. Are APIs standardized (OpenAI-compatible, etc.)? 3. What is the switching cost estimate? 4. Does the vendor support multi-cloud deployment? 5. Are you dependent on proprietary formats or tools?	No data export or portability option.

The six category scores add up to a total between 6 and 30. Each category score is the average of its five answers, rounded to the nearest whole number. The total places the vendor in one of four bands: 25 to 30 is low risk and proceeds on a standard contract, 18 to 24 is moderate and needs added governance provisions, 12 to 17 is high and needs a remediation plan before any contract, and 6 to 11 is critical. The printable worksheet adds a red-flag checklist, a three-vendor comparison matrix and a sign-off block. Allow 30 to 45 minutes a vendor.

A vendor at 28 with one red flag is a higher risk than a vendor at 20 with none, because a red flag is a failure that strength elsewhere cannot offset. One red flag pauses procurement until it is resolved or an executive accepts the risk in writing. Three or more means reject.

Why Score AI Vendors at All?

When an AI product fails in front of people, the organisation that put it there answers for it, as three cases show.

In May 2025 Italy's data protection authority, the Garante, announced a EUR 5 million fine on Luka Inc., maker of the Replika chatbot, for lacking a legal basis, an inadequate privacy policy and no age verification. It had limited the processing of Italian users' data in 2023.

In February 2024 a British Columbia tribunal ruled against Air Canada after its website chatbot gave a customer wrong information about bereavement fares. The tribunal said the airline was in effect suggesting the chatbot was a separate legal entity, and called that "a remarkable submission". The award was CAD 812.02. The principle is worth far more. Mobley v. Workday, the third, has its own section.

A standard SaaS checklist covers encryption, uptime, data location, access and price. AI adds four risks it misses. A model update can move the outputs your controls were set against, with no release note. Training data you cannot describe means failure modes you cannot predict. Your own data may end up training the vendor's model. And a model can discriminate with no discriminatory rule written in it.

The Six Categories

1. Model Transparency

Can you explain the model's outputs to the people they affect? Watch for an old model card, silence about training data, or "it is a neural network" offered as an explanation. Missing documentation usually means missing governance, so it goes in the score. Full marks mean model cards updated with each major release, training data described with its provenance and limits, notice before behaviour changes, and drift metrics you can pull yourself.

2. Bias and Fairness

"We did not test for bias" and "our model is unbiased" are different statements, and vendors run them together. A vendor whose product makes decisions about people in hiring, lending, housing, healthcare, education or insurance and has never run a fairness audit does not know whether its model discriminates, and an indemnity won't change that. Top marks need published metrics, method and remediation history, plus independent testing.

3. Data Privacy and Security

The worst answers come in the politest language. "We use your data to improve service quality" can be innocent, or it can mean your data trains a model sold to your competitors. Ask the literal question. The floor is tenant isolation, no training on your data without explicit consent (an opt-out is not consent), deletion you can see working, compliance with evidence behind it, and verified destruction when the contract ends. Full marks need evidence for each floor.

4. Accountability and Support

If you can't name who owns AI governance at the vendor, the vendor may not know either. A 5 has a named lead, an AI incident SLA, evidenced cover, audit access and a working escalation path. On insurance, purpose-built AI liability policies exist (one launched with Lloyd's underwriters in May 2025), while some general liability forms now exclude AI. Ask what the policy covers when the claim is about an AI output.

5. Regulatory Compliance

Under the EU AI Act, the original prohibitions have applied since 2 February 2025 and the penalty regime since 2 August 2025. Regulation (EU) 2026/1744, the Digital Omnibus, moved the high-risk dates to 2 December 2027 for Annex III uses such as hiring and credit, and to 2 August 2028 for AI in Annex I products. Article 99 fines reach EUR 35 million or 7% of worldwide turnover for prohibited practices and EUR 15 million or 3% for operator obligations, whichever is higher (lower for SMEs). A vendor quoting the 7% tier is quoting one unlikely to apply to its product. The best name the articles that apply, hand over the documents and track changes before you ask. The Regulation Tracker keeps the dates current.

6. Lock-in and Portability

A vendor holding your data and workflows in formats only its tools can read gains pricing power every year you stay, so negotiate the exit before you sign. The top scorers export everything in standard formats through widely implemented APIs. Ask them for an export and they send consultants.

What Red Flags Should Stop an AI Vendor Evaluation?

A red flag is a failure of the vendor's governance as a system, so it isn't averaged into the score. The ten:

Transparency: refuses to disclose training data sources.
Bias and fairness: no bias audit ever conducted. The vendor doesn't know whether its model discriminates.
Privacy: your data trains their model without explicit consent.
Accountability: no incident response SLA for AI issues. "Best effort" has no deadline.
Accountability: no one at the vendor is accountable for AI governance.
Compliance: cannot provide EU AI Act conformity documentation for a high-risk use once those obligations apply. Under Article 25, some of that exposure can become yours.
Lock-in: no data export or portability option.
Transparency: model changes deployed without notification. Every silent update invalidates controls you calibrated to the old behaviour.
Accountability: no independent audit access.
Accountability: no AI liability insurance, or none that covers AI-related claims.

Four of the ten sit in one category, on purpose. A vendor weak on documentation can still be fixed. A vendor with no owner, no SLA, no audit access and no cover has nobody to fix anything.

After the Score: Choose, Then Contract

The worksheet's comparison matrix sets two or three scored vendors side by side with equal weights. Picking the highest total is the common mistake, so ask three more questions. Which vendor is strongest in the category this use depends on? For a hiring tool, that is bias and fairness. Which weaknesses can a contract repair? A missing incident SLA can be agreed, but a product built to train on customer data cannot contract its way out of its architecture. Which red flags can be cleared in your timeline? A bias audit can be done before signing, but portability can't be retrofitted in a quarter.

“Remediable weakness is a negotiation. Architectural weakness is a rejection.”

For moderate and high risk vendors, the worksheet lists ten contract provisions. Vendors resist four: change notification, data usage, audit rights and liability. Scope each to something the vendor can keep: thirty days' notice of material changes to decision logic, with "material" defined, or an annual third-party audit in place of audit on demand. For third-party risk, read The AI That Hurts You Is Not the AI You Own.

What Does Mobley v. Workday Change?

In Mobley v. Workday, filed in February 2023 in the Northern District of California, Derek Mobley alleged that he had applied to more than 100 jobs through employers using Workday's screening tools since 2017 and was rejected each time, and that the tools discriminated on race, age and disability.

On 12 July 2024 the court ruled, at the pleading stage, that Workday could plausibly be liable as an "agent" of its employer customers, because its tools were alleged to perform a traditional hiring function: rejecting candidates at screening. The same order dismissed the employment-agency theory and the intentional discrimination claims. No court has found that the tools discriminate.

On 16 May 2025 the court let the age claim proceed as a collective, the first of two certification steps, covering applicants aged 40 and over denied a recommendation through Workday's platform since 24 September 2020. Workday stated in its filings that 1.1 billion applications were rejected using its tools in that period. A motion to certify a class followed on 14 September 2026, with a hearing set for 9 March 2027.

An AI vendor can now face a claim as its customers' agent, and a vendor in litigation brings a legal question into your hiring process. My own reading, which no court has ruled on, is that the agent theory leaves the employer's liability where it was. An agent acts for a principal, and the principal here is you.

EU law reaches a similar place by statute. Under Article 25 of the EU AI Act, a deployer that puts its own name on a high-risk system, modifies it substantially so it stays high-risk, or repurposes a system so that it becomes high-risk is treated as its provider. Provider obligations then apply in full, and so does the operator fine tier in Article 99.

The 12-Week Rollout

Nobody can run 30 questions on every AI vendor, and nobody should. Run them on the vendors that cross a risk threshold:

Weeks 1 to 2, inventory. List every AI vendor in use or planned. Tier 1 decides things about people, handles regulated data or faces customers. Tier 2 supports internal decisions. Tier 3 is productivity tooling with no personal data.
Weeks 3 to 6, score Tier 1. Procurement leads, with security, legal and the AI governance lead.
Weeks 7 to 10, act. Negotiate with moderate and high risk vendors, and get remediation plans from the high risk ones. Reject critical vendors or take them to the executive committee with the risk written down.
Weeks 11 to 12, extend. A shorter assessment for Tier 2, a one-page intake form for Tier 3, and a yearly re-score for every Tier 1 vendor.

When Is the Scorecard the Wrong Tool?

It fits AI that touches consequential decisions, regulated data or customers. Three cases need adjusting.

Too much for low-risk internal tools. A transcription tool with no personal data needs a short intake form, or governance becomes the bottleneck people route around.
Too little for safety-critical AI. Medical devices, vehicles and critical infrastructure need domain regimes on top, such as the FDA's Software as a Medical Device resources or ISO 26262 for road vehicles.
A partial fit for foundation model APIs. Some items, such as audit access, won't be negotiable. Use the scorecard to record what you are accepting.

The scorecard doesn't replace legal review, technical due diligence or a pilot. For EU deployer duties, read The EU AI Act After the Omnibus, for Leaders Who Deploy. Use the worksheet on the next AI vendor that reaches your desk, then on the ones already in production that nobody scored.

Questions readers ask

What are the key AI responsibility red flags to avoid during vendor evaluation?

Ten, and any one of them pauses the purchase whatever the score. On disclosure and testing: the vendor will not disclose training data sources, or it has never run a bias audit. On your data: it trains on your data without explicit consent, or it offers no data export. On accountability: it sets no deadline for fixing AI incidents, nobody there owns AI governance, it refuses independent audit access, or it cannot show insurance cover for AI-related claims. On change and law: it ships model changes without notice, or it cannot produce EU AI Act conformity documentation for a high-risk use once those obligations apply. Three or more means reject.

What are the key AI ethics criteria I should include in our vendor evaluation scorecard?

Six categories of five questions each. Model transparency asks whether the vendor can document and explain the model. Bias and fairness asks whether it has tested for bias and whether you can test independently. Data privacy and security asks where your data sits and whether it trains the vendor's model. Accountability and support asks who owns AI governance at the vendor and what happens when something breaks. Regulatory compliance asks for evidence behind each claim, and lock-in and portability asks whether you can leave. Each category is scored on the same five-point scale, and the full list of questions is in the printable worksheet.

What is an AI vendor scorecard?

A structured assessment a buyer completes before signing with an AI vendor. This one asks 30 questions, scores each of six categories on a scale of 1 to 5, and adds the six scores into a total between 6 and 30. A total of 25 or more is low risk, 18 to 24 is moderate, 12 to 17 is high, and 6 to 11 is critical. Ten red flags sit beside the score and override it. Procurement leads, with security, legal and whoever owns AI governance. Allow 30 to 45 minutes a vendor.