Instrument
Liability Ledger Audit
The Liability Ledger Audit scores an AI portfolio on 25 dimensions across five categories of ethical debt, from 25 (debt free) to 125 (critical). Each category compounds at its own rate every six months, and the audit puts the fastest-compounding debt at the top of the repair list.
Last verified
Scoring
| Dimension | 1 Well-Managed | 3 Partial | 5 Critical |
|---|---|---|---|
| D1 Bias Debt: compounds 2.0x every six months | |||
| D1.1 Fairness testing coverage | 1 Well-ManagedEvery user-facing model has had a fairness audit in the last six months, with automated tests in the deployment pipeline. | 3 Partial30 to 60% of user-facing models audited, with no plan to cover the rest. | 5 CriticalNo production model has ever been tested for fairness. |
| D1.2 Outcome monitoring | 1 Well-ManagedAutomated monitoring of outcome gaps between groups on every high-risk model, with alerts and drift tracked since launch. | 3 PartialOne or two flagship models are monitored. The rest are not. | 5 CriticalOutcomes are not tracked by group. You could not produce disparate-impact data if a regulator asked. |
| D1.3 Proxy variable audit | 1 Well-ManagedProxy risk analyzed for every high-risk model. Known proxies removed, or kept with a written reason and a control. | 3 PartialProxy analysis done only for the most visible model. | 5 CriticalNever done. "The model does not see race" is treated as proof. |
| D1.4 Remediation protocol | 1 Well-ManagedA written bias playbook, tested in at least one real or simulated event. | 3 PartialThe data science team investigates ad hoc, with no roles or communication plan. | 5 CriticalNo process and no precedent. |
| D1.5 Audit recency | 1 Well-ManagedA portfolio-wide audit within the past six months, with high-risk models audited more often. | 3 PartialLast audit 12 to 18 months ago. The schedule slips. | 5 CriticalNo fairness audit has ever been run. |
| D2 Transparency Debt: compounds 1.3x every six months | |||
| D2.1 Explainability coverage | 1 Well-ManagedEvery customer-facing and high-risk model gives explanations suited to its audience, tested with users. | 3 PartialFlagship or regulated models explain themselves. Most others are black boxes. | 5 CriticalNo model can explain a decision. |
| D2.2 Model documentation | 1 Well-ManagedA current model card for every production model, generated from the pipeline where possible. | 3 PartialThe central team documents its models. Business-unit and shadow AI go undocumented. | 5 CriticalNo documentation. Losing one data scientist would leave models unmaintainable. |
| D2.3 Stakeholder communication | 1 Well-ManagedPeople are told when AI shapes a decision about them, in wording tested for comprehension. | 3 PartialAI use appears in the privacy policy but not at the point of decision. | 5 CriticalAI use is hidden or never mentioned. |
| D2.4 Audit trail | 1 Well-ManagedTamper-evident logs of input, model version and output for every high-risk decision, retrievable within 48 hours. | 3 PartialLogs exist for some systems but miss inputs or versions. Retrieval is slow. | 5 CriticalDecisions are not logged. |
| D2.5 Regulatory readiness | 1 Well-ManagedMeets current transparency rules in every jurisdiction, with work under way on the rules still to come. | 3 PartialMeets the obvious requirements, with gaps elsewhere. Tracking is reactive. | 5 CriticalNo one has assessed which transparency rules apply. |
| D3 Governance Debt: compounds 1.5x every six months | |||
| D3.1 Governance structure | 1 Well-ManagedA cross-functional body with a charter, authority and a record of decisions, including stopping or retiring systems. | 3 PartialA part-time or advisory function that cannot block a deployment. | 5 CriticalNo structure. Whoever can deploy AI does. |
| D3.2 Policy coverage | 1 Well-ManagedPolicies cover development, deployment, procurement and use. Technical controls enforce them, and they are reviewed every year. | 3 PartialPartial, outdated or unenforced policies. Shadow AI is not covered. | 5 CriticalNo AI policy of any kind. |
| D3.3 Review cadence | 1 Well-ManagedEvery production system reviewed at least quarterly, high-risk systems monthly or continuously. | 3 PartialAnnual reviews for some systems, none for others. | 5 CriticalSystems run indefinitely without review. |
| D3.4 Incident response | 1 Well-ManagedAn AI incident playbook with severity levels, roles and message templates, rehearsed. | 3 PartialThe general IT incident process, with nothing specific to AI. | 5 CriticalAI failures are not recognized as incidents. |
| D3.5 Accountability assignment | 1 Well-ManagedEvery production system has a named owner with authority and resources, kept through handovers. | 3 PartialInformal owners, usually whoever built the system. No registry. | 5 CriticalNobody owns any system’s outcomes. |
| D4 Privacy Debt: compounds 1.8x every six months | |||
| D4.1 Consent management | 1 Well-ManagedConsent covers AI use by purpose, and a withdrawal reaches the training pipeline. | 3 PartialGeneral consent that does not name AI use. A withdrawal stops at the source data. | 5 CriticalData is used because it is available. No consent covers AI use. |
| D4.2 Data provenance | 1 Well-ManagedEvery training record traces to its source, consent status and permitted use. | 3 PartialSources are known for the main model. Consent status is mostly unknown. | 5 CriticalTraining data origins are unknown. |
| D4.3 Cross-border compliance | 1 Well-ManagedAI data flows are mapped and covered by transfer mechanisms in every jurisdiction. | 3 PartialRequirements are known, mapping is incomplete and compliance is assumed. | 5 CriticalData pooled across jurisdictions with no transfer assessment. |
| D4.4 Biometric data handling | 1 Well-ManagedMeets every applicable biometric law with specific written consent, or the organization has confirmed it processes none. | 3 PartialBiometric use is documented, but general consent is treated as enough. | 5 CriticalBiometric data is processed with no awareness of biometric law. |
| D4.5 Retention and minimization | 1 Well-ManagedEnforced retention periods, regular minimization, and deletion requests that reach training sets and trigger retraining where needed. | 3 PartialRetention rules are not applied to training data. Deletions stop at the source. | 5 CriticalTraining data is kept indefinitely, and nobody can say what is in it. |
| D5 Accountability Debt: compounds 1.5x every six months | |||
| D5.1 Human oversight | 1 Well-ManagedRisk-calibrated human review of every high-risk decision, by reviewers with the authority, time and information to override. | 3 PartialReview on headline systems only, mostly approving what the model recommends. | 5 CriticalConsequential decisions are made with no human review. |
| D5.2 Escalation pathways | 1 Well-ManagedAnyone affected can challenge an AI decision through a documented, tested path with timelines and a named decision-maker. | 3 PartialInformal paths that depend on knowing whom to call. | 5 CriticalAI decisions are final. |
| D5.3 Vendor liability allocation | 1 Well-ManagedVendor contracts carry AI-specific indemnity, audit rights, data-use limits and incident notice. | 3 PartialCloser review than for ordinary software, but generic indemnity. | 5 CriticalStandard liability caps with no AI terms. |
| D5.4 Decision attribution | 1 Well-ManagedLogs show whether the AI, a person or both made each decision, producible within 48 hours. | 3 PartialSome logs, but AI and human decisions are not told apart. | 5 CriticalNo record separates AI decisions from human ones. |
| D5.5 Redress mechanisms | 1 Well-ManagedA documented, tested process to put right harm done to a person, with timelines and defined remedies. | 3 PartialHandled case by case, depending on who picks up the complaint. | 5 CriticalNo internal route to a remedy. |
Score every dimension from 1 to 5 on evidence you can show. Scores 2 (Monitored) and 4 (Minimal) sit between the columns; the worksheet describes all five levels. For each dimension, keep the worst score across all your AI systems. Each category totals 5 to 25, and the five together give a score from 25 to 125. Lower is better.
| Total | Band | What it means |
|---|---|---|
| 25‑40 | BandDebt Free | What it meansDebt is measured and controlled, and monitoring is running. Keep the quarterly review. |
| 41‑65 | BandManageable Debt | What it meansKnown debts with a paydown plan under way. Check that no single category is close to its maximum. |
| 66‑90 | BandDangerous Debt | What it meansDebt is compounding unseen across several categories, and a lawsuit, inquiry or model failure would expose it. Start the 90-day plan now. |
| 91‑125 | BandCritical Debt | What it meansOne incident from crisis. Treat it as a board-level risk, with an executive sponsor and funding this quarter. |
Download: Liability Ledger Worksheet
The printable worksheet: every dimension, the score bands and room for your team's evidence.
Enter your email to get instant access. You'll also get The Brief, every other Tuesday.
Free. No spam. Unsubscribe anytime.
Why this instrument exists, and what the scores mean for a leadership team: AI Liability Compounds Like Credit Card Debt
How to run the audit
- List every AI system in production, including vendor AI, shadow AI and internal tools. If the list cannot be finished, that gap is itself Governance Debt: score D3 accordingly.
- Score all 25 dimensions for each system on evidence. A planned audit, a draft policy or a committee that never meets scores as if it did not exist.
- For each dimension, keep the worst score across the portfolio. One critical system sets the ledger.
- Add each category, then the five categories, and read the band.
- Rank every dimension scored 4 or 5 by its category rate: Bias first, then Privacy, then Governance and Accountability, then Transparency.
Read the categories before the total
A total can look comfortable while one category compounds at a high rate. An organization at 60 overall sits in Manageable Debt, yet with Privacy Debt at 20 of 25 it carries debt that grows 1.8 times every six months. Build the plan from the dimensions scored 4 or 5, and use the band to report progress.
The multipliers below are directional. They come from enforcement patterns, settlement trajectories and regulatory timelines, since no actuarial data on ethical debt exists yet. Your sector and jurisdictions may run faster or slower. The order holds: Bias Debt compounds fastest and Transparency Debt slowest.
| Category | Rate per six months | After 6 months | After 12 months | After 18 months |
|---|---|---|---|---|
| D1 Bias Debt | Rate per six months2.0x | After 6 months2.0x | After 12 months4.0x | After 18 months8.0x |
| D4 Privacy Debt | Rate per six months1.8x | After 6 months1.8x | After 12 months3.24x | After 18 months5.83x |
| D3 Governance Debt | Rate per six months1.5x | After 6 months1.5x | After 12 months2.25x | After 18 months3.38x |
| D5 Accountability Debt | Rate per six months1.5x | After 6 months1.5x | After 12 months2.25x | After 18 months3.38x |
| D2 Transparency Debt | Rate per six months1.3x | After 6 months1.3x | After 12 months1.69x | After 18 months2.20x |
Multiply what a fix costs today by the figure for its category and delay. A bias fix that costs $100,000 now costs about $800,000 after 18 months. The multipliers apply to cost only; the score stays on the 25 to 125 scale.
| Weeks | Phase | What the team does | Output |
|---|---|---|---|
| 1-2 | PhaseInventory | What the team doesComplete the AI inventory and score all 25 dimensions. Start no fixes until the whole ledger is visible, so a large hidden debt does not lose out to a small visible one. | OutputBaseline score, five category subtotals, and the cost of waiting on every 4 and 5 |
| 3-4 | PhasePrioritize | What the team doesRank every 4 and 5 by category rate. Estimate what each fix costs today and set it against the 6 and 12 month multipliers. Secure an executive sponsor and a budget. | OutputAn eight-week remediation plan with a named owner and a date for each item |
| 5-8 | PhaseRemediate | What the team doesWork down the list. Bias first: fairness audits on the three highest-risk models, then outcome monitoring. Privacy next: data classification, consent gaps, cross-border flows. Then governance: a governance body, policies, a review cadence and an incident plan. | OutputEvidence for each fix, ready to re-score |
| 9-12 | PhaseVerify | What the team doesRe-score all 25 dimensions on the new evidence. Report the change in score and the cost of waiting that was avoided. Set a review cadence for each category, quarterly at the least. | OutputA board report and the date of the next quarterly review |
Sources
- Temporal quality degradation in AI models (Vela et al., Scientific Reports 12, 11654). Nature Portfolio, 2022-07-08
- iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit. U.S. Equal Employment Opportunity Commission, 2023-09-11
- Attorney General Ken Paxton Secures $1.4 Billion Settlement with Meta Over Its Unauthorized Capture of Personal Biometric Data. Office of the Texas Attorney General, 2024-07-30
- AG Campbell Announces $2.5 Million Settlement With Student Loan Lender For Unlawful Practices Through AI Use. Massachusetts Office of the Attorney General, 2025-07-10
- Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). EUR-Lex, Official Journal of the European Union, 2024-06-13
- AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology, 2023-01-26
Ajay's views, from fifteen-plus years in the field. Not legal or compliance advice. See full disclaimers →
Published by AI Exponent LLC