Evaluate AI vendors on governance, transparency, and risk — before you sign. 30 questions across 6 categories with immediate red flags and a 3-vendor comparison matrix.
By Ajay PundhirAskAjay.aiVersion 1.1 · Updated 2026-05-03
Who Should Complete This
Procurement, Head of AI, CISO, Legal
Time Required
30 – 45 minutes per vendor
Output
Governance-graded vendor comparison with risk flags
When to Use
Before any AI vendor contract, renewal, or expansion
Complete the Vendor Profile to capture the basics of the vendor and product under evaluation.
Score each of the 6 governance categories (5 questions each, scored 1–5). Use the scoring guide: 1 = No capability, 5 = Best-in-class. The category score is the average of its 5 questions, rounded to the nearest integer.
Run the Red Flag Checklist. Any single red flag triggers a procurement pause — regardless of the total score.
Complete the Scoring Summary to classify the vendor's overall risk level.
Use the Vendor Comparison Matrix to compare up to 3 vendors side-by-side.
Review with stakeholders — share the completed scorecard with Legal, CISO, and the AI governance council before contract signature.
Why This Matters Every AI vendor claims "enterprise-grade governance." This scorecard gives you 30 specific questions to verify those claims. The difference between a vendor that scores 26/30 and one that scores 14/30 is the difference between a governance partner and a governance liability — and you cannot tell the difference from a demo or an RFP response alone.
Section 1: Vendor Profile
SaaS
On-Premise
Hybrid
Predictive
Generative
Agentic
Multi-modal
Assessment Details
Scoring Guide
For each question, score the vendor from 1 to 5. Be honest — the value of this scorecard depends on accurate scoring, not generous scoring.
1 No capability
2 Minimal / ad hoc
3 Developing
4 Established
5 Best-in-class
Brand commitments behind this scorecard
The vendor's failure becomes your failure. The Mobley v. Workday class certification (May 2025) and the Air Canada chatbot ruling (Feb 2024) made this concrete. Vendor selection in 2026 is a liability allocation decision — score it like one.
Red flags override the score. A vendor at 28/30 with one red flag is higher risk than a vendor at 20/30 with none. Red flags are systemic governance failures, not weaknesses to be averaged into a number. Pause procurement until any red flag is resolved or accepted with documented executive risk acknowledgement.
Section 2: Governance Assessment
Category 1
Model Transparency
Evaluates whether the vendor provides meaningful visibility into how their AI models work, what data they use, and how they change over time.
1
Does the vendor provide model documentation (model cards, data sheets)?
2
Can you inspect training data composition and sources?
3
Are model updates communicated with change logs?
4
Can you access model performance metrics and drift reports?
5
Is the model's decision logic explainable to non-technical stakeholders?
Category 1 Score: Model Transparency
/ 5
Category 2
Bias & Fairness
Evaluates whether the vendor proactively tests for bias, makes fairness metrics available, and has processes for remediation when bias is found.
1
Has the vendor conducted fairness audits on the model?
2
Can you run independent bias testing on their system?
3
Are fairness metrics (demographic parity, equalized odds) available?
4
Is there a documented process for addressing bias when found?
5
Does the vendor publish bias audit results?
Category 2 Score: Bias & Fairness
/ 5
Category 3
Data Privacy & Security
Evaluates how the vendor handles your data — where it is stored, how it is protected, whether it is used to improve their models, and what happens when the contract ends.
1
Where is your data stored? (jurisdiction, encryption, isolation)
2
Is your data used to train or improve the vendor's models?
3
Can you delete your data completely on request?
4
Does the vendor comply with GDPR / CCPA / relevant regulations?
5
What happens to your data if the contract ends?
Category 3 Score: Data Privacy & Security
/ 5
Category 4
Accountability & Support
Evaluates whether the vendor has real accountability structures for AI governance — named individuals, incident response, liability coverage, and audit access.
1
Is there a named individual accountable for AI governance at the vendor?
2
What is the incident response SLA for AI-related issues?
3
Does the vendor carry AI liability insurance?
4
Can you audit the vendor's AI governance practices?
5
Is there an escalation path for AI ethics concerns?
Category 4 Score: Accountability & Support
/ 5
Category 5
Regulatory Compliance
Evaluates whether the vendor is aligned with major AI regulatory frameworks and can provide evidence of compliance — not just claims.
1
Is the vendor compliant with EU AI Act requirements?
2
Does the vendor align with NIST AI RMF?
3
Can the vendor provide conformity assessment documentation?
4
Does the vendor track regulatory changes proactively?
5
Is there evidence of third-party compliance certification?
Category 5 Score: Regulatory Compliance
/ 5
Category 6
Vendor Lock-in & Portability
Evaluates how easily you can exit the vendor relationship — whether your data, models, and workflows are portable or trapped in proprietary systems.
1
Can you export your models, data, and configurations?
2
Are APIs standardized (OpenAI-compatible, etc.)?
3
What is the switching cost estimate?
4
Does the vendor support multi-cloud deployment?
5
Are you dependent on proprietary formats or tools?
Category 6 Score: Vendor Lock-in & Portability
/ 5
Section 3: Scoring Summary
Transfer each category score from above. The total determines the vendor's risk classification.
12 – 17: High RiskRequire remediation plan before contract
6 – 11: Critical RiskDo not proceed without fundamental changes
Scoring Interpretation
A Moderate Risk score does not mean "reject." It means the vendor has governance gaps that must be addressed contractually — through additional SLAs, audit rights, liability provisions, or remediation timelines. A High Risk score means the vendor's governance is insufficient for production deployment today. A Critical Risk score means the vendor lacks the fundamental governance infrastructure required for responsible AI deployment.
Section 4: Red Flag Checklist
Immediate Disqualifiers
Any single red flag below should pause procurement — regardless of the vendor's total score. A vendor scoring 28/30 with one red flag is higher risk than a vendor scoring 20/30 with no red flags. Red flags indicate systemic governance failures that cannot be compensated by strength in other areas.
1. Vendor refuses to disclose training data sources. Opacity about training data signals either legal exposure (copyright, privacy) or a governance structure that does not exist. If they cannot tell you what the model learned from, they cannot tell you what risks it carries.
2. No bias audit has ever been conducted. A vendor that has never tested for bias is a vendor that does not know whether their model discriminates. The absence of testing is not the absence of bias — it is the absence of awareness.
3. Your data is used to train their models without explicit consent. If your data improves their product for all customers, you are subsidizing their R&D with your proprietary information. This is a data governance and competitive risk, not just a privacy concern.
4. No incident response SLA exists for AI issues. When their AI makes a wrong decision about your customers, how fast do they respond? "Best effort" is not an answer. No SLA means no accountability.
5. No one at the vendor is accountable for AI governance. If you cannot name the person responsible for AI governance at the vendor, neither can they. Governance without accountability is theater.
6. Cannot provide EU AI Act conformity documentation. If the vendor operates in or serves EU markets and cannot demonstrate conformity, you inherit their compliance risk. The EU AI Act's penalty exposure extends to deployers, not just providers.
7. No data export or portability option. If you cannot extract your data, models, and configurations, you are not a customer — you are a captive. Portability is not a feature. It is a governance requirement.
8. Model changes deployed without notification. If the vendor can change how their model works — its behavior, its outputs, its decision logic — without telling you, your governance controls are invalidated with every silent update.
9. No independent audit access. "Trust us" is not a governance position. If the vendor will not allow independent verification of their AI practices, the practices may not withstand scrutiny.
10. Vendor has no AI liability insurance. A vendor without AI liability insurance is a vendor that either cannot obtain coverage (a risk signal) or has not invested in it (a maturity signal). Either way, liability flows to you.
Red Flags Triggered: _____ / 10 Decision:
Proceed (0 red flags)
Pause & remediate (1+ red flags)
Reject (3+ red flags)
Section 5: Vendor Comparison Matrix
Use this matrix to compare up to 3 vendors side-by-side. Transfer scores from each vendor's individual scorecard assessment.
Category
Vendor A
Vendor B
Vendor C
Weight
1. Model Transparency
/5
/5
/5
Equal
2. Bias & Fairness
/5
/5
/5
Equal
3. Data Privacy & Security
/5
/5
/5
Equal
4. Accountability & Support
/5
/5
/5
Equal
5. Regulatory Compliance
/5
/5
/5
Equal
6. Vendor Lock-in & Portability
/5
/5
/5
Equal
Total Score
/30
/30
/30
Assessment
Vendor A
Vendor B
Vendor C
Risk Classification
Low / Moderate / High / Critical
Low / Moderate / High / Critical
Low / Moderate / High / Critical
Red Flags Triggered
_____ / 10
_____ / 10
_____ / 10
Contract Value (Annual)
Switching Cost Estimate
Recommendation
Proceed / Negotiate / Remediate / Reject
Proceed / Negotiate / Remediate / Reject
Proceed / Negotiate / Remediate / Reject
Comparison Decision Guide The highest-scoring vendor is not always the right choice. Consider: (1) Which vendor's strengths align with your highest-risk use case? (2) Which vendor's weaknesses are contractually remediable? (3) Which vendor's red flags, if any, are addressable within your timeline? The scorecard provides the data. The decision requires judgment.
Section 6: Contract Governance Provisions
For vendors scoring Moderate or High Risk, negotiate these governance provisions into the contract. Check each item as it is addressed in the contract language.
Negotiate
Recommended Contract Provisions
Right to audit AI governance practices — Annual or on-demand audit rights for AI governance, bias testing, data handling, and security practices. Include right to use independent third-party auditors.
AI incident response SLA — Defined response times for AI-specific incidents (model failures, bias discoveries, data breaches involving AI systems). Include escalation path and named contacts.
Model change notification — Minimum 30-day advance notice for any changes to model architecture, training data, or decision logic. Include right to opt out of updates and rollback capability.
Data usage restrictions — Explicit prohibition on using your data for model training, benchmarking, or improvement without written consent. Include data isolation requirements.
Data portability and exit clause — Full data export in standard formats within 30 days of contract termination. Include model weights and configurations if applicable.
Liability allocation for AI decisions — Clear allocation of liability for AI system outputs and decisions. Include indemnification for AI-related regulatory penalties and third-party claims.
Compliance maintenance obligation — Vendor obligation to maintain compliance with EU AI Act, NIST AI RMF, and applicable regulations throughout the contract term. Include right to terminate if compliance lapses.
Bias testing and remediation SLA — Regular bias testing cadence (quarterly minimum) with documented remediation within 30 days of discovery. Include your right to run independent bias tests.
Performance transparency requirements — Regular reporting on model performance metrics, drift reports, and accuracy benchmarks. Include access to real-time monitoring dashboards.
Subprocessor and supply chain transparency — Disclosure of all third-party models, data sources, and infrastructure providers used in the AI system. Include right to approve changes to the supply chain.
Section 7: Assessment Notes
Key Findings
Governance Gaps Requiring Remediation
Recommended Next Steps
Sign-Off
Section 8: Related AskAjay Frameworks
This scorecard is part of a connected system of governance tools. Each addresses a different dimension of AI governance maturity and vendor management.
Definitions used throughout this scorecard. These align with the canonical methodology article.
Six-Category Scorecard. The 30-question, 6-category vendor governance assessment: Model Transparency, Bias and Fairness, Privacy and Security, Accountability, Regulatory Compliance, Vendor Lock-in. Five questions per category, scored 1–5. Total range: 6 (no governance anywhere) to 30 (best-in-class).
Red Flag. A systemic governance failure that overrides the score. Ten red flags listed in Section 4. Any single red flag should pause procurement until resolved or accepted with explicit, documented executive risk acknowledgement.
Risk Tier. Tier 1 (consequential decisions about people, regulated data, customer-facing) gets the full 30-question scorecard; Tier 2 (internal, employee-facing) gets the abbreviated 15-question form; Tier 3 (productivity tools, no PII) gets a one-page intake.
Model Card. A standardised document describing an AI model's purpose, training data characteristics, performance metrics, known limitations, and ownership. The minimum acceptable artifact for Category 1 in 2026.
Conformity Assessment. Under the EU AI Act, the documented evaluation evidencing that a high-risk AI system meets the requirements of Articles 8–15. Vendors must provide this; deployers should obtain and retain it.
Demographic Parity / Equalized Odds. Quantitative fairness metrics. Demographic parity = equal positive outcome rates across groups. Equalized odds = equal true-positive and false-positive rates across groups. Both have well-known trade-offs — the question is whether the vendor measures any of them.
Workday Precedent. Mobley v. Workday (N.D. Cal.). Federal court held in 2024 that an AI vendor could be directly liable for employment discrimination under an agency theory. May 2025: court certified an ADEA collective potentially covering hundreds of millions of applicants. The case that made vendor selection a liability allocation decision.
Article 25 (EU AI Act). The provision under which a deployer who substantially modifies a high-risk vendor system, puts their name on it, or changes its intended purpose becomes a provider of that system — inheriting the full provider obligations under Article 16 and Article 99 penalty exposure.
Evidence Base
This scorecard is the operational layer of a published, sourced framework. The methodology, vendor-failure precedents, and EU AI Act citations live in the canonical article below.
Important Disclaimer
This scorecard is provided for informational and organizational purposes only. It does not constitute legal, procurement, or technical advice. AI vendor assessment requires domain expertise, legal review, and technical due diligence beyond what any scorecard can capture. Use this tool to structure your evaluation process, not to replace professional judgment. The questions and scoring framework reflect the author's understanding of AI governance best practices as of May 2026.