Evaluate AI vendors on governance, transparency and risk before you sign. Thirty questions in six categories, ten red flags, a three-vendor comparison matrix and ten contract provisions.
By Ajay PundhirAskAjay.aiVersion 1.3 · Updated 2026-10-06
Who Should Complete This
Procurement, Head of AI, CISO, Legal
Time Required
30 to 45 minutes per vendor
Output
Governance-graded vendor comparison with risk flags
When to Use
Before any AI vendor contract, renewal, or expansion
Complete the Vendor Profile to capture the basics of the vendor and product under evaluation.
Score each of the 6 governance categories (5 questions each, scored 1 to 5). Use the scoring guide: 1 = No capability, 5 = Best-in-class. The category score is the average of its 5 questions, rounded to the nearest integer.
Run the Red Flag Checklist. Any single red flag pauses procurement, whatever the total score.
Complete the Scoring Summary to classify the vendor's overall risk level.
Use the Vendor Comparison Matrix to compare up to 3 vendors side-by-side.
Review with stakeholders. Share the completed scorecard with Legal, CISO, and the AI governance council before contract signature.
Why This Matters Every AI vendor claims "enterprise-grade governance." This scorecard gives you 30 specific questions to verify those claims. The difference between a vendor that scores 26/30 and one that scores 14/30 is the difference between a governance partner and a governance liability, and a demo or an RFP response will not show you which one you have.
Section 1: Vendor Profile
SaaS
On-Premise
Hybrid
Predictive
Generative
Agentic
Multi-modal
Assessment Details
Scoring Guide
For each question, score the vendor from 1 to 5. Be strict. The scorecard is only as useful as the scores are accurate.
1 No capability
2 Minimal / ad hoc
3 Developing
4 Established
5 Best-in-class
Brand commitments behind this scorecard
The vendor's failure becomes your failure. Mobley v. Workday, where a court ruled in July 2024 that an AI vendor could plausibly be liable as an agent of the employers using its tools, and the Air Canada chatbot ruling (Feb 2024) made this concrete. Vendor selection in 2026 is a decision about who carries liability. Score it like one.
Red flags override the score. A vendor at 28/30 with one red flag is higher risk than a vendor at 20/30 with none. Red flags are systemic governance failures, not weaknesses to be averaged into a number. Pause procurement until any red flag is resolved or accepted with documented executive risk acknowledgement.
Section 2: Governance Assessment
Category 1
Model Transparency
Evaluates whether the vendor provides meaningful visibility into how their AI models work, what data they use, and how they change over time.
1
Does the vendor provide model documentation (model cards, data sheets)?
2
Can you inspect training data composition and sources?
3
Are model updates communicated with change logs?
4
Can you access model performance metrics and drift reports?
5
Is the model's decision logic explainable to non-technical stakeholders?
Category 1 Score: Model Transparency
/ 5
Category 2
Bias & Fairness
Evaluates whether the vendor proactively tests for bias, makes fairness metrics available, and has processes for remediation when bias is found.
1
Has the vendor conducted fairness audits on the model?
2
Can you run independent bias testing on their system?
3
Are fairness metrics (demographic parity, equalized odds) available?
4
Is there a documented process for addressing bias when found?
5
Does the vendor publish bias audit results?
Category 2 Score: Bias & Fairness
/ 5
Category 3
Data Privacy & Security
Evaluates how the vendor handles your data: where it is stored, how it is protected, whether it is used to improve their models, and what happens when the contract ends.
1
Where is your data stored? (jurisdiction, encryption, isolation)
2
Is your data used to train or improve the vendor's models?
3
Can you delete your data completely on request?
4
Does the vendor comply with GDPR / CCPA / relevant regulations?
5
What happens to your data if the contract ends?
Category 3 Score: Data Privacy & Security
/ 5
Category 4
Accountability & Support
Evaluates whether the vendor has real accountability structures for AI governance: named individuals, incident response, liability coverage, and audit access.
1
Is there a named individual accountable for AI governance at the vendor?
2
What is the incident response SLA for AI-related issues?
3
Does the vendor carry AI liability insurance?
4
Can you audit the vendor's AI governance practices?
5
Is there an escalation path for AI ethics concerns?
Category 4 Score: Accountability & Support
/ 5
Category 5
Regulatory Compliance
Evaluates whether the vendor is aligned with major AI regulatory frameworks and can provide evidence of compliance, beyond its own claims.
1
Is the vendor compliant with EU AI Act requirements?
2
Does the vendor align with NIST AI RMF?
3
Can the vendor provide conformity assessment documentation?
4
Does the vendor track regulatory changes proactively?
5
Is there evidence of third-party compliance certification?
Category 5 Score: Regulatory Compliance
/ 5
Category 6
Vendor Lock-in & Portability
Evaluates how easily you can exit the vendor relationship: whether your data, models, and workflows are portable or trapped in proprietary systems.
1
Can you export your models, data, and configurations?
2
Are APIs standardized (OpenAI-compatible, etc.)?
3
What is the switching cost estimate?
4
Does the vendor support multi-cloud deployment?
5
Are you dependent on proprietary formats or tools?
Category 6 Score: Vendor Lock-in & Portability
/ 5
Section 3: Scoring Summary
Transfer each category score from above. The total determines the vendor's risk classification.
Category
Score (/5)
Risk Level
Notes
1. Model Transparency
2. Bias & Fairness
3. Data Privacy & Security
4. Accountability & Support
5. Regulatory Compliance
6. Vendor Lock-in & Portability
Risk Classification
Total Score_____ / 30
25 to 30: Low RiskProceed with standard contract
18 to 24: Moderate RiskNegotiate additional governance provisions
12 to 17: High RiskRequire remediation plan before contract
6 to 11: Critical RiskDo not proceed without fundamental changes
Scoring Interpretation
A Moderate Risk score does not mean "reject." It means the vendor has governance gaps that must be addressed in the contract, through additional SLAs, audit rights, liability provisions, or remediation timelines. A High Risk score means the vendor's governance is insufficient for production deployment today. A Critical Risk score means the vendor lacks the fundamental governance infrastructure required for responsible AI deployment.
Section 4: Red Flag Checklist
Immediate Disqualifiers
Any single red flag below should pause procurement, whatever the vendor's total score. A vendor scoring 28/30 with one red flag is higher risk than a vendor scoring 20/30 with no red flags. Red flags indicate systemic governance failures that cannot be compensated by strength in other areas.
1. Vendor refuses to disclose training data sources. Opacity about training data signals either legal exposure (copyright, privacy) or a governance structure that does not exist. If they cannot tell you what the model learned from, they cannot tell you what risks it carries.
2. No bias audit has ever been conducted. A vendor that has never tested for bias is a vendor that does not know whether their model discriminates. No test means no knowledge, either way.
3. Your data is used to train their models without explicit consent. If your data improves their product for all customers, you are subsidizing their R&D with your proprietary information. This is a data governance and competitive risk, not just a privacy concern.
4. No incident response SLA exists for AI issues. When their AI makes a wrong decision about your customers, how fast do they respond? "Best effort" is not an answer. No SLA means no accountability.
5. No one at the vendor is accountable for AI governance. If you cannot name the person responsible for AI governance at the vendor, neither can they. Governance without accountability is theater.
6. Cannot provide EU AI Act conformity documentation. If the vendor operates in or serves EU markets and cannot demonstrate conformity, you inherit their compliance risk. The EU AI Act's penalty exposure extends to deployers, not just providers.
7. No data export or portability option. If you cannot extract your data, models, and configurations, you are a captive. Treat portability as a governance requirement.
8. Model changes deployed without notification. If the vendor can change how their model works (its behavior, its outputs, its decision logic) without telling you, your governance controls are invalidated with every silent update.
9. No independent audit access. "Trust us" is not a governance position. If the vendor will not allow independent verification of their AI practices, the practices may not withstand scrutiny.
10. Vendor has no AI liability insurance. A vendor without AI liability insurance is a vendor that either cannot obtain coverage (a risk signal) or has not invested in it (a maturity signal). Either way, liability flows to you.
Red Flags Triggered: _____ / 10 Decision:
Proceed (0 red flags)
Pause & remediate (1+ red flags)
Reject (3+ red flags)
Section 5: Vendor Comparison Matrix
Use this matrix to compare up to 3 vendors side-by-side. Transfer scores from each vendor's individual scorecard assessment.
Category
Vendor A
Vendor B
Vendor C
Weight
1. Model Transparency
/5
/5
/5
Equal
2. Bias & Fairness
/5
/5
/5
Equal
3. Data Privacy & Security
/5
/5
/5
Equal
4. Accountability & Support
/5
/5
/5
Equal
5. Regulatory Compliance
/5
/5
/5
Equal
6. Vendor Lock-in & Portability
/5
/5
/5
Equal
Total Score
/30
/30
/30
Assessment
Vendor A
Vendor B
Vendor C
Risk Classification
Low / Moderate / High / Critical
Low / Moderate / High / Critical
Low / Moderate / High / Critical
Red Flags Triggered
_____ / 10
_____ / 10
_____ / 10
Contract Value (Annual)
Switching Cost Estimate
Recommendation
Proceed / Negotiate / Remediate / Reject
Proceed / Negotiate / Remediate / Reject
Proceed / Negotiate / Remediate / Reject
Comparison Decision Guide The highest-scoring vendor is not always the right choice. Consider: (1) Which vendor's strengths align with your highest-risk use case? (2) Which vendor's weaknesses are contractually remediable? (3) Which vendor's red flags, if any, are addressable within your timeline? The scorecard provides the data. The decision requires judgment.
Section 6: Contract Governance Provisions
For vendors scoring Moderate or High Risk, negotiate these ten provisions into the contract. Standard SaaS terms rarely cover them, because they were written for software that behaves the same way every day. Check each item once the contract language covers it. Version 1.3 folds in the ten clauses from The AI That Hurts You Is Not the AI You Own, merged with the provisions they duplicated.
Negotiate
Recommended Contract Provisions
1. Right to audit AI governance practices. Annual or on-demand audit rights covering AI governance, bias testing, data handling, security, and how the system reaches its decisions, with an evidence review at least once a year. Include the right to use an independent auditor of your choice.
2. AI incident response SLA. Set response and notification times for AI incidents specifically (model failures, drift, harmful or false outputs, a bias discovery, data exposure through the AI system), separate from the general security incident clause. Include an escalation path and named contacts.
3. Model change notification. At least 30 days' notice of material changes to the model, its training data or its decision logic, with "material" defined in the contract. Include the right to opt out of updates or roll back. Name the events that reopen your assessment and oblige the vendor to tell you when one occurs: a new model version, a change of sub-processor, an enforcement action against the vendor, or a significant drop in performance.
4. Data usage restrictions. No training, benchmarking or product improvement on your data, no commingling with other customers' data, and no retention beyond what the service needs, without your written consent. Include data isolation requirements. Your data feeds the vendor's service, not its model.
5. Data portability and exit. Full export in standard formats within 30 days of termination: your data, prompts and outputs, embeddings, fine-tuning data, and model weights and configurations you paid for. Include transition assistance for a fixed period, and say what happens to your data if the vendor is sold, closes the product or becomes insolvent.
6. Liability allocation for AI decisions. Allocate liability for AI outputs and decisions in writing. Include indemnification for AI-related discrimination claims, third-party intellectual property claims and regulatory penalties. For high-risk uses, size the cap to what you could owe, not to the fee you pay.
7. Compliance maintenance obligation. A warranty that the vendor complies with the laws that apply to its product (the EU AI Act, GDPR, sector rules) and keeps complying for the whole term, with the same obligations flowed down to its sub-processors and an annual written attestation. Include the right to terminate if compliance lapses.
8. Bias testing and remediation SLA. Bias testing at least quarterly, with documented results shared with you and remediation within 30 days of a finding. Include your right to run independent bias tests.
9. Performance transparency requirements. Model cards and evaluation reports, updated with each major release. Regular reporting on accuracy and quality against agreed baselines, drift and uptime, with access to monitoring dashboards. Remedies for degradation include the right to terminate, as well as service credits.
10. Subprocessor and supply chain transparency. A full list of third-party models (including any foundation model provider), data sources and infrastructure providers in the AI system. Require advance notice of additions, and the right to object or terminate if a new one is unacceptable to you.
Where vendors push back
Expect resistance on change notification, data usage, audit rights and liability. Scope each provision to something the vendor can keep, and write the carve-outs down: 30 days' notice for material changes to decision logic, or an annual audit by an agreed third party with the results shared, is often accepted where a broader version is refused.
Section 7: Assessment Notes
Key Findings
Governance Gaps Requiring Remediation
Recommended Next Steps
Sign-Off
Section 8: Related AskAjay Frameworks
This scorecard is part of a connected system of governance tools. Each addresses a different dimension of AI governance maturity and vendor management.
Definitions used throughout this scorecard. These align with the canonical methodology article.
Six-Category Scorecard. The 30-question, 6-category vendor governance assessment: Model Transparency, Bias and Fairness, Privacy and Security, Accountability, Regulatory Compliance, Vendor Lock-in. Five questions per category, scored 1 to 5. Total range: 6 (no governance anywhere) to 30 (best-in-class).
Red Flag. A systemic governance failure that overrides the score. Ten red flags listed in Section 4. Any single red flag should pause procurement until resolved or accepted with explicit, documented executive risk acknowledgement.
Risk Tier. Tier 1 (consequential decisions about people, regulated data, customer-facing) gets the full 30-question scorecard; Tier 2 (internal, employee-facing) gets a shorter assessment; Tier 3 (productivity tools, no PII) gets a one-page intake.
Model Card. A standardised document describing an AI model's purpose, training data characteristics, performance metrics, known limitations, and ownership. The minimum acceptable artifact for Category 1 in 2026.
Conformity Assessment. Under the EU AI Act, the documented evaluation evidencing that a high-risk AI system meets the requirements of Articles 8 to 15. Vendors must provide this; deployers should obtain and retain it.
Demographic Parity / Equalized Odds. Quantitative fairness metrics. Demographic parity = equal positive outcome rates across groups. Equalized odds = equal true-positive and false-positive rates across groups. Both have well-known trade-offs. The question is whether the vendor measures any of them.
Mobley v. Workday. N.D. Cal., filed February 2023, ongoing. On 12 July 2024 the court ruled, at the pleading stage, that an AI vendor could plausibly be liable for employment discrimination as an agent of its employer customers. On 16 May 2025 it let an age-discrimination collective proceed at the first of two certification steps; by Workday's own account the collective could run to hundreds of millions of applicants. No court has found that the tools discriminate.
Article 25 (EU AI Act). The provision under which a deployer becomes the provider of a high-risk system if it puts its own name or trademark on it, makes a substantial modification that leaves it high-risk, or changes the intended purpose of a system so that it becomes high-risk. Provider obligations under Article 16 then apply, with fines under Article 99 of up to EUR 15 million or 3% of worldwide annual turnover.
Evidence Base
This scorecard is the operational layer of a published, sourced framework. The methodology, vendor-failure precedents, and EU AI Act citations live in the canonical article below.
Canonical article:Your AI Vendor's Demo Was Built to Pass Your Questions: the scorecard, the red flags, Mobley v. Workday, EU AI Act dates and penalties, the 12-week rollout, and when the scorecard is the wrong tool.
Important Disclaimer
This scorecard is provided for informational and organizational purposes only. It does not constitute legal, procurement, or technical advice. AI vendor assessment requires domain expertise, legal review, and technical due diligence beyond what any scorecard can capture. Use this tool to structure your evaluation process, not to replace professional judgment. The questions and scoring framework reflect the author's understanding of AI governance best practices as of May 2026.