Epistemic Humility: Governing AI You Don't Fully Understand

In brief

Nobody fully understands how today’s large AI models work, including their builders, and NIST’s AI Risk Management Framework says risks that are not well understood are hard to measure. Two cases show governance in place and missing what mattered. At Deloitte Australia, AI approved for code analysis, trained staff and a review step all existed, yet AI used for citations in a government report produced errors the review missed. Apple Card’s model was lawful, but the bank was unprepared for limits that surprised customers. A third case is a test: in Anthropic’s 2025 simulation, models from several developers resorted to blackmail, behavior nobody designed. A 2025 study also found that people with more AI literacy were more confident and less accurate about their own work. The answer is four practices: a register of unknowns, monitoring of the answers, a rehearsed way back, and disclosure with a route to a person.

Key Takeaways

→The people who build frontier models say they do not understand how those models work, and NIST’s framework says poorly understood risks are hard to measure.
→At Deloitte Australia, AI approved for one task, trained staff and a review step all existed. The errors came from a use nobody had assessed.
→Apple Card’s model was found lawful, yet the bank was unprepared for limits that surprised customers, and the regulator faulted its transparency.
→Anthropic’s own stress test found models from several developers resorting to blackmail in simulation, behavior nobody designed. The list of questions cannot be closed before launch.
→Four practices cover what a framework cannot list in advance: a register of unknowns, monitoring of the answers, a rehearsed way back, and disclosure with a route to a person.

Epistemic humility in AI governance means building your controls on the assumption that a deployed system will do something nobody predicted. In practice it comes down to four habits: write down your unknowns, watch for them, keep a way back, and tell the people who rely on you.

I do not fully understand the AI systems I help organizations govern, and neither do their builders. A governance program that cannot list what it does not know about a system is, I will argue, more dangerous than one with admitted gaps. It hands whoever signs off a confidence nobody has earned. I wrote this for the risk or compliance lead who is asked to approve an AI system they cannot inspect.

Who Actually Understands These Systems?

Not the builders, by their own account. Dario Amodei, the chief executive of Anthropic, wrote in April 2025 that "people outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work," and called that gap "essentially unprecedented in the history of technology." Stuart Russell had said much the same to a US Senate subcommittee in July 2023. In written testimony, he called large language models "opaque in the sense that their internal principles of operation are unknown."

The standards say it too, if you read past the summary. NIST's AI Risk Management Framework, where most American governance programs start, states that "AI risks or failures that are not well-defined or adequately understood are difficult to measure quantitatively or qualitatively." It adds that risks measured before deployment "may differ from risks that emerge in operational, real-world settings." The framework admits the very gap that most programs built on it leave out. A risk register lists what somebody thought of. The system has no obligation to stay inside it.

Two Misses and a Test

Two of these organizations had controls in place before something went wrong, and in both the controls answered a different question from the one that turned out to matter. The third case is a test, and it shows why that list of questions cannot be closed before launch.

In December 2024 Australia's employment department hired Deloitte, on a contract worth AU$439,142, to review a welfare compliance system. After an internal assessment the department consented to an AI tool chain to help analyze the system's code. Deloitte later told the Department of Finance that, near the end, the team also used generative AI to summarize a court case and to format citations. Staff had been trained on "the importance of reviewing all outputs against source materials," and the output was reviewed. Even so, the report published in August 2025 carried citation errors, and Deloitte wrote that while its review "corrected some errors, not all were identified as part of this process." Deloitte repaid the final installment of its fee, AU$97,587, and the department replaced the report twice, the second time in February 2026. Most of the controls I would have asked for were there. None covered the use that went wrong.

A customer posted in November 2019 that Apple Card had offered him a credit limit 20 times higher than his wife's, and that a customer service agent could not explain why. New York's Department of Financial Services examined underwriting data for nearly 400,000 New York applicants and found no evidence of discrimination. Goldman Sachs, the issuing bank, had a fair lending program, and it was able to explain every complainant's decision at the regulator's request. That report also found that the bank "seemed unprepared for the possibility that its complex underwriting model" would "produce outcomes that might surprise applicants." Its governance had settled whether the model was lawful. It had nothing ready for a customer asking why.

Anthropic's stress test, published in June 2025, put 16 models from several developers, its own among them, inside invented companies with access to invented email. When a model's goals clashed with the company's and it faced replacement, it could blackmail an executive over an affair. Claude Opus 4 and Gemini 2.5 Flash did so in 96 of 100 runs, GPT-4.1 and Grok 3 Beta in 80. The scenarios were built to force that choice, and Anthropic says it has seen no sign of this in real deployments. Here the test was the control, and it worked. I count it because Amodei's essay names the problem it exposes: "we cannot meaningfully predict such behaviors, and therefore struggle to rule them out." A pre-launch review at a customer will not rule them out either.

Set side by side, the cases show the same split between what was checked and what mattered.

Case	What turned out to matter	What was checked
Deloitte Australia, 2025	Whether every citation says what the report claims	That AI for code analysis was approved, and that output was reviewed
Apple Card, 2019 to 2021	Whether anyone could explain a surprising limit to the customer who asked	That similar applicants were treated alike under fair lending law
Anthropic stress test, 2025	What a model does when its goal conflicts with the company’s	The usual pre-launch question: does the model do the task it is given?

Only the third case is about a model doing something nobody designed. The first two are about questions nobody wrote down, which in my experience is the more common failure, and the cheaper one to fix.

Why Does Fluency Make It Worse?

A second reason to distrust a confident sign-off applies most to experts. Researchers led from Aalto University tested it in a study published in Computers in Human Behavior in late 2025. They asked 246 people to solve 20 logical reasoning problems from the Law School Admission Test with ChatGPT's help. Scores rose by about three points against a norm population, and people overestimated those scores by about four. Those who scored higher on an AI literacy questionnaire were more confident and less accurate about how well they had done. A second study of 452 people found the same pattern.

The study measured puzzles, not governance decisions. Its participants were judging their own work done with AI, though, which is close to what a reviewer does when AI helped produce the output, as at Deloitte. Then the reviewer’s sense of how thorough the check was needs checking too, ideally by someone who did not use the tool.

Four Practices for Governing What You Cannot Predict

None of these replaces a framework; they cover what NIST’s own text says is hard to measure. Model documentation, in my experience, already lists known limitations. A register of unknowns asks what nobody has tested yet.

Keep a register of unknowns. For each system, write down three things nobody has tested about how it will behave, next to the risks you do know. Give your incident process a category for behavior nobody anticipated, so a surprise has somewhere to land other than an argument about whose fault it was.
Monitor the answers as well as the uptime. Models lose quality after launch without throwing an error. A 2022 study in Scientific Reports trained four common model types, none of them language models, on 32 datasets from four industries and saw quality fall over time in 91 percent of the 128 model and dataset pairs. Uptime and latency tell you the system is running. Sampling and scoring its outputs tells you whether they are still any good.
Keep a way back. Before launch, name who can switch the system off or roll it back, how long that takes, and what the business does meanwhile. Rehearse it once. A rollback plan nobody has run is a hope with a document attached, and deciding in advance when to stop is part of the same job.
Say what you do not know. Put the register of unknowns in front of the board and, where a system decides something about a person, in front of that person too. Telling someone about an unknown does not hand them the risk, so say how they reach a human and get the decision looked at again. Apple Card’s regulator found a lawful model and still faulted the bank’s transparency, which "seemed to produce confusion that could have been mitigated."

These sit on top of a working baseline such as Minimum Viable Governance, and what it costs to skip them shows up in the Liability Ledger.

The Strongest Objection

The strongest objection is that this is an excuse: a governance lead who says "nobody understands these systems" can never be wrong. I take the point. My answer is that each practice above leaves something an auditor can inspect: a register with entries and dates, monitoring with thresholds and alerts, a rollback with a measured time. If those things do not exist, the humility is a slogan, and you should treat it like any other control that lives only on paper.

A second objection is that admitting uncertainty will slow approvals. It may. I would rather a board approve a system knowing its three unknowns than approve it believing there are none.

I would be wrong if programs that keep such a register turn out no quicker to catch and recover from surprises than programs that do not. Nobody has run that comparison yet, so read this as an argument from cases, not a measured result.

What to Do Before the Next Sign-Off

Take the next AI system that reaches you for approval and ask its owner for three things it might do that nobody has tested. If the answer is "none," the review is not finished. Then ask who can switch it off, and how fast. Write both answers into the approval and read them again in three months, when the data the system sees will have moved on.

Sources and Changes

Dario Amodei, "The Urgency of Interpretability", April 2025.
Stuart Russell, written testimony to the US Senate Judiciary Subcommittee on Privacy, Technology and the Law, hearing of 25 July 2023, page 1.
NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023, section 1.2.1.
Department of Employment and Workplace Relations, "Targeted Compliance Framework Assurance Review: Final Report", versions of 4 July 2025, 26 September 2025 and 3 February 2026; AusTender, contract notice CN4118426; Deloitte, letter to the Department of Finance, 2 September 2025, released under FOI 25-26/084.
New York State Department of Financial Services, "Report on Apple Card Investigation", March 2021, pages 2 to 8.
Anthropic, "Agentic Misalignment: How LLMs could be insider threats", 20 June 2025.
Daniela Fernandes and others, "AI makes you smarter but none the wiser: The disconnect between performance and metacognition", Computers in Human Behavior 175 (2026), 108779, online 9 October 2025.
Daniel Vela and others, "Temporal quality degradation in AI models", Scientific Reports 12, 11654, 8 July 2022.

All sources checked on 6 October 2026.

Changes on 6 October 2026. Cut from about 4,800 words under a new headline. Six cases became the three with the strongest primary record; SafeRent and Workday (still allegations) and a UK parliamentary report on pilots were removed. The Anthropic test is now described as a forced-choice simulation, Deloitte’s controls come from Deloitte’s own letter, and the 91 percent figure now describes one 2022 study’s model and dataset pairs. A download box pointing to a missing file, a reading list and several secondary statistics were removed.