Evaluating AI Vendors: The Six-Category Governance Scorecard

Three Vendors. Three Lessons. One Pattern.

In February 2023, the Italian data protection authority — the Garante — ordered Replika to stop processing the personal data of Italian users. The chatbot lacked age verification, processed minors' data without lawful basis, and exposed vulnerable users to harm. In May 2025, the Garante fined Replika's parent company Luka Inc. EUR 5 million for those violations. Every European enterprise that had quietly piloted Replika as a wellness tool inherited the headline.

In May 2025, a federal court in California granted preliminary collective certification in Mobley v. Workday — an age discrimination class action covering every job applicant aged 40 or older who was rejected by Workday's AI screening since September 2020. Workday represented to the court that 1.1 billion applications were processed by its software in the relevant period. The collective could include hundreds of millions of people. The court had earlier held that Workday — as the AI service provider — could be directly liable for employment discrimination under an agency theory. The customer is no longer the only defendant.

In February 2024, the British Columbia Civil Resolution Tribunal ruled against Air Canada after its chatbot invented a bereavement-fare policy that did not exist. The airline argued the chatbot was a separate legal entity. The tribunal disagreed: the bot was part of the airline's website, the airline owed a duty of care, and the customer was owed a refund. Air Canada built nothing. It bought a chatbot. The chatbot lied. Air Canada paid.

Three vendors. Three jurisdictions. Three different harms — privacy, discrimination, misrepresentation. One pattern. The vendor's failure becomes the deployer's failure, the deployer's failure becomes the regulator's case, and somewhere along the way a procurement team that ran a 90-minute demo and signed a master services agreement is on the front page of a trade publication trying to explain what an "AI alignment audit" is.

Every AI vendor will tell you they have enterprise-grade governance. Most of them are wrong, and a few of them know it. The difference between a vendor that earns a 26 out of 30 on a structured scorecard and one that earns 14 is the difference between a partner that compounds your Trust Premium and one that compounds your liability — and you cannot tell which is which from a demo, a SOC 2 report, or an RFP response. You have to ask better questions.

This article is the methodology behind the AI Vendor Assessment Scorecard. Six categories. Thirty questions. Ten red flags. A three-vendor comparison matrix. Ten contract governance provisions. And a 90-day rollout that slots into a Minimum Viable Governance sprint without slowing the business down.

Why the Old Procurement Template Fails for AI

Standard SaaS procurement asks five questions in five different forms. Is the data encrypted? What is the uptime SLA? Where is the data stored? Who has access? What is the price? These questions matter. They are also entirely insufficient for AI.

AI vendors introduce four risk surfaces that no traditional SaaS procurement template captures. First, model behavior changes silently. A SaaS app that updates its UI sends a release note. A foundation model that updates its weights ships behavior changes overnight, and your governance controls — calibrated to the old behavior — are invalidated without notice. Second, training data is the supply chain. The data the vendor trained on shapes every output their model produces. If you cannot inspect or characterize that data, you are buying a black box with unknown failure modes. Third, your data may train their model. In standard SaaS, your data stays your data. In AI vendor relationships, your data is potentially the next training set — for them and for every other customer. Fourth, the model can discriminate without anyone writing a discriminatory rule. The Earnest case proved this: race-neutral inputs (Cohort Default Rates, immigration status) produced a $2.5 million settlement for disparate impact. The model learned the bias from the data.

These four risk surfaces require four classes of question that standard procurement does not ask. They map directly to four of the six categories in the scorecard — Transparency, Bias and Fairness, Privacy and Security, Accountability. The remaining two categories — Regulatory Compliance and Vendor Lock-in — exist because the regulatory environment moved underneath everyone in 2024 and 2025, and because the architectural choices vendors are making today determine whether you can leave them in 2028.

The scorecard is not a checklist. It is a six-category structure for the conversation procurement should have been having for the last three years and was not.

The Six Categories

Category 1 — Model Transparency

Transparency is not a marketing term. It is the answer to a specific question: can you, the buyer, explain how this model makes decisions to the people the decisions affect? If the answer is no, every other governance control you build sits on sand.

The five questions in this category interrogate documentation (model cards and datasheets), training data composition, change logs for model updates, performance and drift reporting, and explainability for non-technical stakeholders. Watch for vendors who provide a model card that is three years old, who refuse to disclose anything about training data sources, or who treat "the model is a neural network" as an explanation. Disqualifier: a vendor that cannot or will not provide model documentation in 2026 is a vendor whose internal governance is too weak to produce it. The absence of documentation is not a documentation problem. It is a governance problem.

A vendor that scores well in this category publishes model cards updated with each major release, characterizes training data with provenance and known limitations, ships change logs for behavioral updates with at least 30 days of notice, exposes drift metrics through an API or dashboard, and can describe the model's decision logic in plain English to a non-technical reviewer. Most vendors in 2026 score 2 or 3. The leaders score 4 or 5. There are no excuses left for scoring 1.

Category 2 — Bias and Fairness

Bias and fairness is the category where vendors most often confuse "we did not test for bias" with "our model is unbiased." The two are not the same. The absence of testing is not the absence of bias. It is the absence of awareness. The Mobley v. Workday class certification — an alleged collective of hundreds of millions — is what untested bias becomes when it ships at scale.

The five questions ask whether the vendor has conducted fairness audits, whether you can run independent bias testing, whether fairness metrics are available, whether there is a documented remediation process when bias is found, and whether the vendor publishes bias audit results. The first question is the floor. The last question is the ceiling. Most vendors in 2026 will tell you they "monitor for bias internally." That is a 2 on this scorecard. A vendor scoring 4 publishes the metrics. A vendor scoring 5 publishes the metrics, the audit methodology, and the remediation history.

Disqualifier: a vendor whose product makes consequential decisions about people — hiring, lending, housing, healthcare, education, insurance — and has never conducted a fairness audit is a vendor that does not know whether their model discriminates. You cannot indemnify yourself out of that risk. The Workday precedent means the vendor is exposed too, but it does not mean the deployer escapes. Both parties end up in the caption.

Category 3 — Data Privacy and Security

Privacy is the category where vendors most reliably hide the worst answers in the most polite language. "We use your data to improve service quality" is sometimes innocent and sometimes a euphemism for "your proprietary data trains a model that we then sell to your competitors." You have to ask the literal question and read the literal answer.

The five questions cover data location and isolation, training-data usage, deletion on request, regulatory compliance (GDPR, CCPA, sector-specific regimes), and data fate at contract end. Each one has a non-negotiable floor. Data must be isolated by tenant — multi-tenant inference where prompts mingle is a failure mode. Your data must not train the vendor's models without explicit consent — opt-out is not consent. Deletion must be technically implementable, not just contractually promised. Compliance must be evidenced, not asserted. And contract end must result in data return and verified destruction, not a vendor email saying "trust us, it is gone."

Meta paid Texas $1.4 billion in 2024 for biometric processing without informed consent — the largest privacy settlement ever obtained by a single state. Every vendor selling biometric AI in 2026 has been put on notice. The question is whether yours has updated their practices, or whether they are still betting that no one will notice.

Category 4 — Accountability and Support

Accountability is the category that separates a governance partner from a governance theater performance. The five questions ask whether there is a named individual accountable for AI governance at the vendor, whether there is an incident-response SLA for AI-specific issues, whether the vendor carries AI liability insurance, whether you can audit the vendor's AI governance practices, and whether there is an escalation path for AI ethics concerns.

If you cannot name the person responsible for AI governance at the vendor, neither can the vendor. This is not a trick question. It is the question. A vendor that cannot produce an AI governance lead — by name, with title and tenure — has no AI governance, no matter what their compliance page claims. McKinsey's 2024 State of AI survey found that only 18% of organizations have an enterprise-wide AI governance council with decision-making authority. The percentage among AI vendors is not appreciably higher. Most vendors are governing AI ad hoc, the same way most enterprises are.

AI liability insurance is the leading indicator. A vendor that has obtained AI-specific liability coverage has been underwritten by an insurer who reviewed their practices and found them insurable. A vendor that has not either could not obtain coverage (a risk signal) or did not invest in it (a maturity signal). Either way, in a Workday-style precedent, liability flows downhill — to the deployer.

Category 5 — Regulatory Compliance

Regulatory compliance is the category that has changed most in the last 18 months. It is also the category most often answered with marketing copy and least often answered with evidence. The questions ask about EU AI Act compliance, NIST AI Risk Management Framework alignment, conformity assessment documentation, regulatory tracking, and third-party certification.

The EU AI Act citations matter here, and most vendors get them wrong. Article 5 prohibitions and Article 4 AI literacy obligations have been enforceable since February 2, 2025. General-purpose AI model obligations took effect August 2, 2025. High-risk system obligations under Article 6 and following are deferred to December 2, 2027 under the Digital Omnibus (provisional pending formal adoption in the EU Official Journal; originally August 2, 2026). Penalties under Article 99 are tiered: up to EUR 35 million or 7% of global annual turnover for prohibited practices, EUR 15 million or 3% for high-risk system non-compliance, and EUR 7.5 million or 1% for supplying incorrect information to authorities. A vendor that says "we are EU AI Act compliant, the penalties are 7% of turnover" is a vendor that has not read the regulation. Push back. Ask which articles. Ask for the conformity assessment documentation. Watch what happens.

The NIST AI Risk Management Framework is voluntary in the United States but increasingly the de facto standard for federal procurement and an emerging baseline for state-level regulation. ISO 42001 — the AI management system standard — is the audit-grade certification a serious vendor will pursue. Banking-sector vendors should additionally evidence SR 11-7 model risk management alignment. None of these are optional in 2026 for a vendor selling into regulated industries. Disqualifier: cannot provide EU AI Act conformity documentation for a high-risk use case once the high-risk obligations apply (December 2, 2027, deferred from August 2, 2026 under the Digital Omnibus; provisional). The deployer inherits the exposure.

Category 6 — Vendor Lock-in and Portability

Lock-in is the category most often skipped during the honeymoon. Five questions: can you export your models, data, and configurations? Are APIs standardized? What is the switching-cost estimate? Does the vendor support multi-cloud? Are you dependent on proprietary formats?

Portability is not a feature. It is a governance requirement. A vendor that traps your data, models, and workflows in proprietary formats is a vendor whose pricing power grows monotonically with your dependence. Three years in, when their renewal arrives at 3x the original price and you have nowhere to go, you will discover that the discount you negotiated at signing was paid back many times over in switching costs you cannot now afford. The leverage was always at signing. After signing, leverage flows the other direction.

The leaders in this category support OpenAI-compatible APIs (or other widely-implemented standards), allow full data and configuration export in standard formats, do not require proprietary tools to interact with the system, and offer multi-cloud or on-premise deployment options. The trailers ship proprietary inference APIs, store data in formats only their tools can read, require their consultants to extract anything meaningful, and offer one-cloud-only deployment that ties you to their infrastructure choices as well as their software choices.

The Ten Red Flags That Override the Score

A vendor scoring 28 out of 30 with one red flag is higher risk than a vendor scoring 20 out of 30 with no red flags. Red flags are not weaknesses to be averaged into a score. They are systemic governance failures that cannot be compensated for by strength elsewhere. Any single red flag should pause procurement until it is resolved or accepted with explicit, documented executive risk acknowledgment.

The ten red flags are listed in full in the scorecard worksheet. Each one maps to a category and to a specific failure pattern observed in real enforcement actions:

• Refuses to disclose training data sources — opacity signals legal exposure or governance absence. Mapped to Transparency.
• No bias audit ever conducted — the absence of testing is the absence of awareness. The Workday precedent is the cost. Mapped to Bias and Fairness.
• Your data trains their model without explicit consent — you are subsidizing their R&D with proprietary information. Mapped to Privacy.
• No incident response SLA for AI issues — "best effort" is not accountability. Mapped to Accountability.
• No one is accountable for AI governance at the vendor — governance without an owner is theater. Mapped to Accountability.
• Cannot provide EU AI Act conformity documentation for high-risk use cases once the high-risk obligations apply (December 2, 2027, deferred from August 2, 2026 under the Digital Omnibus; provisional) — under Article 25, deployer exposure follows. Mapped to Regulatory Compliance.
• No data export or portability option — captives are not customers. Mapped to Vendor Lock-in.
• Model changes deployed without notification — your governance controls are invalidated with every silent update. Mapped to Transparency.
• No independent audit access — "trust us" is not a governance position. Mapped to Accountability.
• No AI liability insurance — the underwriter's verdict on the vendor's risk posture. Mapped to Accountability.

Picking Among Three Vendors

Once you have scored two or three contenders, the comparison matrix in Section 5 of the worksheet does the next job. The mistake most procurement teams make is selecting the highest total score. The total score is the starting point, not the answer.

Three questions decide the matrix. First: which vendor's strengths align with your highest-risk use case? A vendor that scores a 5 on Bias and Fairness is the right pick for an HR application even if their total is lower than a competitor strong in Lock-in but weak in Bias. Categories are not fungible. Strength in your highest-risk dimension matters more than aggregate score.

Second: which vendor's weaknesses are contractually remediable? A vendor that scores a 2 on Accountability because they have no incident response SLA can sometimes negotiate one in. A vendor that scores a 2 on Privacy because they fundamentally use customer data to train their models cannot — that is an architectural choice, not a contractual term. Remediable weakness is a negotiation. Architectural weakness is a rejection.

Third: which vendor's red flags, if any, are addressable inside your timeline? A red flag for "no bias audit ever conducted" can be addressed by a vendor commitment to complete one before contract effective date. A red flag for "no data portability option" cannot be retrofitted in 90 days. The matrix surfaces the choice. The judgment is still yours.

The Ten Contract Provisions Vendors Will Resist

The scoring exercise tells you what the vendor's governance is. The contract tells you what the vendor's governance will be once they have your money. The ten contract governance provisions in Section 6 of the worksheet — audit rights, AI incident SLAs, model change notification, data usage restrictions, portability and exit, liability allocation, compliance maintenance, bias testing cadence, performance transparency, supply chain disclosure — are the difference between a master services agreement and an enforceable governance instrument.

Vendors will resist these. Plan for it. The most common resistance patterns: model change notification ("we cannot commit to 30 days notice for every weight update"), data usage restrictions ("we need flexibility to improve service quality"), audit rights ("we cannot give every customer audit access — that is not scalable"), and liability allocation ("AI outputs cannot be subject to traditional indemnity language").

The counter is not to back down. The counter is to scope the provision to what the vendor can commit to and document the carve-outs explicitly. "30 days notice for material model changes affecting decision logic, with a defined process for material" is sometimes acceptable when "30 days notice for all model updates" is not. "Annual audit by an agreed independent third party with results shared with the customer" is sometimes acceptable when "customer right to audit any time" is not. The principle is: the vendor commits to something, in writing, with consequences if they do not deliver. Vague commitments protect the vendor. Specific commitments protect you.

The Workday Precedent: Why the Math Has Changed

Until 2024, the prevailing assumption inside enterprise procurement was that AI vendor liability was contractually capped, that vendor indemnification protected the deployer, and that the worst-case scenario was a contract dispute settled out of court. Mobley v. Workday ended that assumption.

The case began in 2023. Derek Mobley, a Black male job applicant over 40, alleged that Workday's AI-powered applicant screening system rejected him from over 100 jobs over a seven-year period, on the basis of race, age, and disability. The procedural innovation came in 2024, when the court held that Workday — the AI vendor — could be held directly liable for employment discrimination under an agency theory. Workday was not just a tool provider. By making the screening decisions, it acted as an "agent" of the employers using it. Agents are liable for their discriminatory actions whether or not their principals directed the discrimination.

In May 2025, the court granted preliminary collective certification on the ADEA (age) claim, defining the collective as every applicant aged 40 or over who applied through Workday's platform from September 24, 2020 to the present and was denied employment recommendations. Workday represented that 1.1 billion applications were processed in that window. The collective could include hundreds of millions.

The Workday precedent does two things to the vendor selection math. First, it confirms that vendors face direct liability — not just contract disputes — for discriminatory AI outcomes. This pulls vendor incentives toward better governance, but it also means vendors with weak governance are now litigation targets in their own right. A vendor named in a class action is a vendor whose product becomes legally encumbered, whose insurance premiums spike, whose deployment in your environment becomes a reputational risk. You inherit that exposure even when you are not named in the suit.

Second, the precedent does not relieve the deployer. Both parties end up in the caption. Article 25 of the EU AI Act formalizes a parallel principle in European law: any deployer who substantially modifies a high-risk AI system, puts their name on it, or changes its intended purpose becomes a provider of that system — inheriting the full provider obligations under Article 16 and the full Article 99 penalty exposure. Vendor selection in 2026 is no longer "what does the contract say." It is "whose litigation exposure am I inheriting, and does our combined governance posture survive scrutiny."

This is the connective tissue between this scorecard and the Liability Ledger. Bad vendor selection accumulates accountability debt at a 1.5x compounding rate per six months. Good vendor selection — the kind this scorecard produces — converts vendor governance from a hidden liability into a known and managed one. The math has changed. The procurement template has not caught up. This article and the worksheet are the catch-up.

The 90-Day Rollout

The objection most often raised against rigorous vendor evaluation is speed. "We cannot run a 30-question scorecard on every AI vendor — the business will route around us." That objection is half right. You cannot run the scorecard on every vendor. You can — and should — run it on every vendor that crosses a risk threshold, and you can do it inside the 90-day Minimum Viable Governance sprint without slowing the business down materially.

Weeks 1 to 2: build the AI vendor inventory. You cannot evaluate what you cannot see. Pull procurement records, expense data, SaaS subscription audits, and shadow IT discovery to surface every AI vendor in use, planned, or under evaluation. Tier them by risk: Tier 1 (consequential decisions about people, regulated data, customer-facing), Tier 2 (internal decisions, employee-facing), Tier 3 (productivity tools, no PII).

Weeks 3 to 6: score Tier 1 vendors using the worksheet. 30 to 45 minutes per vendor. Procurement leads, with input from CISO, Legal, and the AI governance lead. Expect the first three vendors to take longer as your team calibrates. By vendor four, the time-per-vendor stabilizes. Document the scores, the red flags, and the gaps. This produces your Tier 1 risk map.

Weeks 7 to 10: renegotiate Moderate Risk Tier 1 contracts using the contract governance provisions checklist. Reject Critical Risk Tier 1 vendors or escalate to executive committee with documented risk acceptance. Pause expansions and renewals on High Risk vendors pending remediation. The negotiations will be harder than your team expects. Plan for two cycles per vendor. Most vendors come around. The ones that do not have told you something important.

Weeks 11 to 12: extend the scorecard to Tier 2, in lighter form. A 15-question abbreviated assessment for Tier 2 catches the top half of the risk surface without consuming the time budget that should be reserved for Tier 1. Tier 3 gets a one-page intake form. Build the calibrated triage, then operate it.

The output of the 90 days is not a one-time scorecard. It is an operating discipline: a vendor evaluation function inside procurement, with criteria, escalation paths, and a renewal cycle that re-scores every Tier 1 vendor annually. That discipline integrates directly into the Minimum Viable Governance framework — vendor governance becomes one of the five elements (inventory, risk tiers, owners, monitoring, escalation) operationalized. It is not additional work. It is the missing piece.

When This Framework Is Overkill, and When It Is Insufficient

A framework that hides its limitations is a framework you should not trust. The Six-Category Governance Scorecard fits a specific risk profile: enterprise procurement of AI systems that touch consequential decisions, regulated data, or external stakeholders. For two adjacent risk profiles, it is the wrong tool.

• Overkill for low-risk internal tools. A meeting transcription service used internally with no PII, no customer data, no consequential decisions, no external publication does not need a 30-question scorecard. A 5-question intake form is the right level of governance. Apply the full scorecard and your governance team becomes the bottleneck the business resents — and your Tier 1 vendors get less attention than they need.
• Insufficient for sovereign-grade or life-safety AI. Defense, intelligence, medical devices that diagnose, autonomous vehicles, critical infrastructure control systems — these require domain-specific evaluation regimes that go far beyond what this scorecard captures. The scorecard is necessary but not sufficient. Use it as a baseline; layer on the FDA's Software as a Medical Device guidance, ISO 26262 for automotive, NIST SP 800-218A for federal, or sector-specific equivalents.
• Conditional fit for foundation model APIs. Buying API access to a frontier foundation model (OpenAI, Anthropic, Google) is partly vendor evaluation and partly platform-risk acceptance. The vendor has more market power than the buyer; some scorecard items (independent audit access, contract bias-testing SLAs) are not realistically negotiable. Use the scorecard to characterize what you are accepting, then make the platform decision deliberately rather than letting it default.

The scorecard is also not a substitute for legal review, technical due diligence, or operational pilot testing. It structures the conversation between procurement, security, legal, and the AI governance function. The judgment still belongs to the people running the evaluation. A bad evaluator produces bad evaluations regardless of the scorecard. A good evaluator with the scorecard produces evaluations that survive scrutiny.

How This Connects to the Larger Picture

Vendor evaluation is one node in a connected governance system. The Minimum Viable Governance framework defines the organizational capabilities needed to hold vendors accountable — without an inventory, you cannot evaluate; without designated owners, you cannot escalate; without monitoring, you cannot detect drift in vendor performance. MVG is the prerequisite. The scorecard is the instrument.

The Liability Ledger quantifies the cost of getting vendor selection wrong. Accountability debt — the category most directly affected by vendor governance — compounds at 1.5x per six months and is the multiplier on every other category in the ledger. A poorly-evaluated vendor accumulates accountability debt silently, then converts it to litigation exposure when a precedent like Workday lands. The scorecard is the prevention mechanism. The ledger is the diagnosis when prevention has failed.

The Trust Premium Scoring Framework measures the upside of getting it right. Pillar 1 (Risk Avoided) and Pillar 2 (Performance Gained) both improve when vendor governance is rigorous. Well-evaluated vendors compound your Trust Premium. Poorly-evaluated ones erode it. The scorecard is one of the few governance disciplines that contributes simultaneously to risk reduction and performance improvement — most disciplines do one or the other.

The EU AI Act Strategic Guide — particularly the obligations under Article 26 for deployers of high-risk systems and Article 23 for importers placing third-country AI on the EU market — operates through vendor evaluation. You cannot meet Article 26 deployer obligations (instructions for use, human oversight, monitoring, recordkeeping) without first evaluating whether the vendor has built a system you can comply with. The scorecard is the prerequisite for Article 26 compliance.

Use the Scorecard

The AI Vendor Assessment Scorecard is a free, printable worksheet. 30 questions across the six categories. The 10-item red flag checklist. A three-vendor comparison matrix. Ten contract governance provisions. A sign-off section for procurement, CISO, Legal, and the Head of AI Governance.

Use it on the next AI vendor that crosses your desk. Then on the contract renewal that is due in the next quarter. Then on the three vendors already in production that you have never evaluated. Within 90 days, you will have a Tier 1 vendor risk map, a renegotiation queue, and an operating discipline that scales. That is the difference between accumulating vendor liability and managing it.

Related Frameworks

Pair this article with Minimum Viable Governance (the 90-day organizational capability sprint that this scorecard plugs into), The Liability Ledger (the compounding cost of getting vendor selection wrong), and Measuring Your Trust Premium (the upside of getting it right). For EU-specific obligations, see the EU AI Act Strategic Guide and the EU AI Act compliance template that runs alongside this scorecard.