The Governance Was Flawless. The Ethics Board Had No Windows

Start with a composite. It is anonymised and illustrative, built from a failure that recurs in lending, and no single firm stands behind its details. A lender builds model governance that would pass any audit: bias tests on every protected attribute it records, an ethics board that meets every two weeks, a model card for each model and a written reason for every decline. Everyone on the board comes from compliance, risk or engineering. Then a group of small-business owners, people who live with the lender's decisions, asks a question the tests never asked. Do inputs such as how long a business has banked with one institution, or how many other businesses share its address block, carry the history of its neighbourhood? No single variable discriminates. The combination might, and nobody inside had lived the experience that would make them suspect it.

An AI governance programme can pass every test it sets itself and still harm the people it decides about, because the tests come from the experience of the people who wrote them. The ethics board needs windows: people the system decides about, with a defined power to make the company look again. My claim is stricter than the usual advice on advisory panels, and founders tend to resist it. An advisory panel the company is free to ignore does not count. A window counts only when someone outside holds a power the company must answer to: a written reply, a forced review, a reversed decision or a public report.

What an outside group saw at Upstart

The real record is messier than the composite, and more useful. In February 2020 the Student Borrower Protection Center, an advocacy group, ran hypothetical applicants through the online lender Upstart's public rate tool, changing only the college attended. Its report, Educational Redlining, said a hypothetical Howard University graduate would pay about $3,500 more over a five-year loan than a similar New York University graduate. Upstart disputed the method, saying the applications differed in more than the school and that the analyst picked three results out of 26.

Upstart was not ungoverned. It had a no-action letter from the Consumer Financial Protection Bureau and reported fair lending tests under it. Writing in 2019, the Bureau said the results for minority, female and older applicants "show no disparities that require further fair lending analysis", adding that it had not replicated them. The same post said the model approved 23 to 29 percent more applicants than a traditional model across every tested race, ethnicity and sex segment.

In December 2020 Upstart, the SBPC and the NAACP Legal Defense and Educational Fund agreed to an independent monitor, the law firm Relman Colfax. After talks with the two groups, Upstart had already changed how it used education data, dropping a grouping of schools by average SAT and ACT scores. The monitor's second report, in November 2021, found "statistically and practically significant approval disparities for Black applicants" compared with non-Hispanic white applicants, and said this did not by itself show a violation. It also found that Upstart's inputs, standing alone, did not appear to be meaningful predictors of race. The third report identified what "would likely have been a viable less discriminatory alternative model". The final report, in March 2024, said Upstart had adopted "nearly all" of the monitor's recommendations and that the parties remained at an impasse over a key one.

The two sets of tests asked different questions. The results Upstart gave the Bureau compared its model with a traditional one. The monitor compared Black and white applicants inside Upstart's own model and asked whether a fairer model was available. Both answers can be true. The second question came from outside, and its answers are public.

Why rigour inside the building misses it

Proxies are the usual mechanism. In 2019 four researchers published an analysis in Science. It examined a commercial algorithm health systems used to pick patients for extra care, one "affecting millions of patients". It predicted health care costs as a stand-in for need. Because less money is spent caring for Black patients than for White patients, Black patients at a given risk score were considerably sicker. Remedying that would raise the share of Black patients receiving additional help from 17.7 percent to 46.5 percent. By some measures of predictive accuracy, cost looked like "an effective proxy for health". The authors worked with an academic hospital's data and had no contact with the manufacturer until their analysis was done. The manufacturer then replicated the result on its national dataset of almost 3.7 million patients.

Every test suite starts from what its authors expect to go wrong. Proxy harm lives in the gap between what a variable measures and what it means in someone's life. Engineers and compliance teams are good at the first. The second is usually noticed by people who have been declined or never referred, or by the advocates and researchers who work with them. In both cases here, outsiders found the problem. Internal rigour rarely closes that gap alone, because it sharpens the questions the team already has.

“A test suite is a list of the harms its authors can imagine. Those on the receiving end can imagine others.”

Four windows, and who can say no

Windows come in four kinds. They differ in cost, and more in who can say no.

Co-design is the cheapest. Show people the problem, the data and the trade-offs before there is a finished screen to react to, and pay them for the day. A standing panel is common; a panel with a power is rare. Give it one in writing: the right to require a review that the company must complete and answer before the next release.

Contest and redress is partly law already. In US lending, Regulation B requires a lender that declines an applicant to give specific principal reasons, and calls it "insufficient" to say the applicant failed to reach a qualifying score. In the EU, Article 22(3) of the GDPR gives people subject to some automated decisions "the right to obtain human intervention on the part of the controller, to express his or her point of view and to contest the decision". The law stops at the single case. The governance step is to count the appeals, read them for patterns and send them to whoever owns the model. One complaint is a support ticket. Forty from one neighbourhood are a finding.

An outside monitor is the strongest window and the most expensive. Upstart's ran for more than three years, past the two years first announced, and ended without agreement on a key recommendation. It also produced four public reports that a regulator, a customer or a competitor can read.

The test I put to founders is one question: who outside the company can make us stop or look again? If the answer is nobody, the board has no windows, however good its tests are.

The objections: faked, ignored, slow

Participation can be faked. New York's Attorney General reported in May 2021 that nearly 18 million of the more than 22 million comments the Federal Communications Commission received in its 2017 net neutrality proceeding were fake. A campaign funded by broadband industry players produced more than 8.5 million of them under real people's identities, though the report found no evidence the funders knew of the fraud. An open comment box is a target. Recruit through organisations that already have standing with the people affected, and know who is in the room.

Participation can also be ignored. OpenAI offered ten $100,000 grants in May 2023 for experiments in democratic processes for deciding the rules AI systems follow, and said the experiments were "not (at least for now) intended to be binding for decisions". That is an honest description of a consultation, and it marks the line between a panel and a window. Participation without power is performance.

And windows are slow. A panel that can force a review will delay some launches, and counsel will warn that its findings can become evidence. My view is that a window you design is easier to live with than one opened by somebody else's report. At Upstart the evidence surfaced anyway, in an advocacy group's report; the choice is whether you hear it first.

Trust as the moat

Trust in AI is scarce. YouGov found in December 2025 that 5 percent of Americans trust AI "a lot" to make recommendations or provide information (1,287 adults). In KPMG's January 2024 survey of 1,000 college-educated US consumers, 83 percent named the technology companies that develop generative AI as responsible for its ethical use.

People expect the builders to answer for AI and do not yet trust them to. I call the advantage that comes from closing that gap the Solidarity Moat. A competitor can copy a model within a release cycle and hire away the people who built it. It cannot copy years of outsiders who said no and were heeded. That is my judgment. I know of no study that measures it.

What to do this quarter

1. Name the three groups your system decides about most often.
2. Pick one window from the table and give it one power in writing, in a board resolution or contract the company cannot quietly withdraw, with a named executive who must reply by a set date.

The community engagement planner in the Responsible AI Playbook for Founders worksheet has a stakeholder map and a partnership tracker to start from. This is the last of four pieces for founders: Part 1 sets out the principles, Part 2 turns them into governance you can enforce, and Part 3, Responsible AI Is Decided at the Point of Use, finds the harms while the product is still being designed.

Rigour inside the building is necessary. The board still needs windows.

Sources

• Student Borrower Protection Center, Educational Redlining, 5 February 2020.
• Upstart, "Upstart's Commitment to Fair Lending", Paul Gu, 6 February 2020.
• CFPB, "An update on credit access and the Bureau's first No-Action Letter", 6 August 2019.
• SBPC, announcement of the agreement with NAACP LDF and Upstart, 1 December 2020; Relman Colfax, Upstart monitorship reports, 14 April 2021, 10 November 2021, 16 September 2022 and 27 March 2024.
• Obermeyer, Powers, Vogeli and Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations", Science 366(6464), 25 October 2019.
• 12 CFR 1002.9(b)(2) (Regulation B), eCFR, read 6 October 2026.
• Regulation (EU) 2016/679 (GDPR), Article 22(3), Official Journal L 119, 4 May 2016.
• New York State Office of the Attorney General, "Fake Comments: How U.S. Companies & Partisans Hack Democracy to Undermine Your Voice", May 2021.
• OpenAI, "Democratic inputs to AI", 25 May 2023.
• YouGov, "Most Americans use AI but still don't trust it", 9 December 2025 (1,287 US adults, 4 December 2025).
• KPMG, 2024 Generative AI Consumer Trust Survey release, 18 January 2024 (1,000 US consumers).

All sources checked on 6 October 2026.

Changes on 6 October 2026. Cut from about 3,300 words to an argument with a new title. The opening fintech story was told as one firm in Atlanta; it is now a labelled, anonymised composite, and the documented Upstart monitorship and the 2019 Science study carry the evidence. Removed: the orbit model, the quadrant chart and its outperformance claim, unsourced workshop and board costs, the Woebot example, an industry survey on business outcomes, claims about the number of founders advised, and the download box (the worksheet is linked in the text).