Responsible AI Is Decided at the Point of Use

In brief

Responsible AI is decided at the point of use, in design choices that look like product details. Google's diabetic retinopathy model, deployed in Thai primary care clinics from December 2018, was more sensitive than retina specialists on the images both could grade, in a study of 7,651 patients. Yet in its first six months it refused 21 percent of images, largely because of lit rooms and a faulty camera. A refusal, like a positive result, meant a referral to a hospital an hour away; uploads stalled on clinic internet; at one clinic half the patients declined to take part. Each choice was defensible alone. The harm came from the fit with the setting. Article 14 of the EU AI Act asks for oversight built into the interface, from 2 December 2027 for Annex III uses. Before launch, prototype the refusal, test at your worst site, design and log the override, and test with the people the output decides about.

Key Takeaways

→In Thai clinics, Google's diabetic retinopathy model refused to grade 393 of its first 1,838 images (21 percent), largely because of lit rooms and a camera that needed repair.
→The model was accurate: on images both could grade, 91.4 percent sensitivity for vision-threatening disease against 84.8 percent for retina specialists, in a study of 7,651 patients.
→What patients met was the design around it: a strict refusal threshold, a referral rule that meant an hour of travel, and uploads that stalled on clinic internet.
→Each choice would pass a responsible AI review on its own. The harm came from how they fitted the setting, which only fieldwork shows.
→Before launch, prototype the refusal, test at your worst site, design and log the override, and test with the people the output decides about.

Between December 2018 and May 2019, nurses at three clinics in Thailand's Pathum Thani province screened people with diabetes using a deep-learning system built by Google, which grades photographs of the retina for diabetic retinopathy. Instead of waiting weeks for an eye doctor to read the photos, a patient had an answer in the chair. Google's researchers sat in those clinics and published what they saw. In the first six months the system refused to grade about one image in five. Uploads stalled on clinic internet connections. A refused image, like a positive result, meant a referral to a hospital an hour's drive away, and at one clinic half the patients declined to take part rather than risk the trip. A nurse put it plainly: patients "are not concerned with accuracy, but how the experience will be."

Responsible AI is decided at the point of use, in choices that look like product details: what the system does when it cannot answer, where it runs, and who can overrule it. A policy can promise safety and fairness, but those choices decide who waits and who is sent away. That puts most of the ethics of an AI product with the team that designs it, and in my experience most governance programmes never look there.

What the clinics showed

The model was not the weak part. The full prospective study covered 7,651 patients at nine sites and was published in The Lancet Digital Health in 2022. On the images both could grade, it found vision-threatening retinopathy with 91.4 percent sensitivity, against 84.8 percent for the Thai retina specialists who read the same images. Its specificity matched theirs.

The trouble sat around the model. Emma Beede and her colleagues described it at the CHI conference in 2020, from observation and interviews at eleven clinics before launch and three after it. The system was designed, for patient safety, to grade only the highest-quality images. A little blur or a dark area and it rejected the photo, even where it might have made a strong prediction. Of 1,838 images put through it in its first six months, 393 failed that bar, 21 percent. Many of those refusals were fair: an expert panel in the larger study judged about six in ten of the refused images unreadable, and Beede traced the bad photos to rooms that could not be darkened, a camera that needed repair and no dilating drops. Only two of the eleven clinics had a screening room that could be darkened, and fluorescent light sometimes sat directly above the camera.

Nurses retook refused photos once, never a third time, because the flash hurt. "I'll do two tries. The patients can't take more than that," one said. Images went to the cloud while the patient waited, and on slow connections some took 60 to 90 seconds to upload. When the internet went down for two hours at one clinic, it screened 100 patients that day instead of 200. Some nurses tried to beat the refusals by photographing the top half of the retina in one shot and the bottom half in another, then reading the two together. The system took one image per eye and could not do the same.

The referral rule did the rest. Under the study protocol, a positive result or an image the system could not grade meant a referral to Pathum Thani Hospital. Not all patients had cars, and older patients needed a son or daughter to take a day off work to drive them. Some nurses started warning patients about the trip before they consented, and 50 percent of patients at one clinic opted out. Others worked around the rule. If a patient's history was clean and blood sugar was under control, a nurse would send them home without a referral.

Three of the design choices, and what the researchers proposed once they had seen the clinics:

Design choice	In the clinic	What the researchers proposed
Grade only the highest-quality images	393 of 1,838 images refused in six months. At one hospital with camera problems the system called 39 percent of images ungradable, where a specialist panel called 7.5 percent	Show the reason for each refusal, and set the threshold to fit local resources
Refer every ungradable image to hospital	An hour's drive for patients without cars. Half the patients at one clinic opted out, and nurses overrode the rule on their own judgement	Hold the referral until an eye doctor has read the image (adopted during the study)
Grade in the cloud while the patient waits	Uploads of 60 to 90 seconds, and a two-hour outage halved a day's screening	Better image compression, or a model that runs offline

Why no policy would have caught it

Each of those choices was defensible on its own. A strict quality threshold protects patients from a missed diagnosis. Referring uncertain cases is ordinary clinical caution. Grading in the cloud lets one model serve every clinic. A responsible AI review would have approved all three, and should have. The version used in the prospective study already carried a CE mark under EU product law.

The harm came from the fit between the choices and the setting: the light above the camera, a queue of 150 patients, a hospital an hour away and no car. None of that shows up in a model card or a policy. It shows up when someone stands in the clinic. The Lancet paper's summary says as much: "socioenvironmental factors and workflows must be taken into consideration" when such a system goes into a screening programme. Beede's team went further and called implementation "of equal importance to the accuracy of the algorithm itself."

Some readers will call this operations, not ethics. I disagree. Who bears the cost of a cautious threshold, in travel, lost wages and a missed day of screening, is a question about who gets care. In Thailand it was settled by a quality setting, a referral rule and an upload path. That makes those choices ethical ones, whoever made them.

It is fair to ask whether one study, partly written by the company that built the system, can carry an argument. I think the source makes it stronger. Google published the frictions of its own deployment, with numbers, and changed the protocol in response. Beede's paper notes that there are "no requirements for AI systems to be evaluated through observational clinical studies, nor is it common practice." Most deployments are never watched this closely, so their versions of the same problems go unrecorded.

The test I use is what I call the Empathy Radius: how far from the screen did you test? Level 1 is the people who use the interface, here the nurses. Level 2 is the people the output decides about, the patients. Level 3 is the communities who carry the burden, such as patients without cars and the families who drive them. Level 4 is the effect at scale. In my experience most product testing stops at Level 1. Google's team got partway to Level 2: after launch they watched about 50 patients go through screening, though what patients felt reached the paper through the nurses.

The law now asks for the same thing

The EU AI Act has written part of this into law. Article 14 says high-risk AI systems "shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons." The people assigned to oversee a system must be able to understand its limitations, interpret its output correctly, and "decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output" of the system.

Those are design requirements. The Thai nurses who sent a patient home despite a refused image were using the judgement Article 14 asks systems to make room for, with no button, no record and no way to check whether they were right.

“An override you do not design still happens. It just goes unlogged.”

After the Digital Omnibus, Regulation (EU) 2026/1744, these duties apply from 2 December 2027 for the uses listed in Annex III, such as hiring, credit scoring and education, and from 2 August 2028 for AI inside products covered by Annex I, which include medical devices.

What to prototype before launch

Four checks follow from the Thai clinics. Run them on a prototype, before anyone decides to launch.

Prototype the refusal. Design the screen for the moment the system cannot answer: what it tells the user, what happens to the person next, and who pays for that path. The Lancet authors suggested the system should say why it refused an image.
Test at your worst site, not your pilot site. Use the oldest device, the slowest connection and the longest queue you will actually serve.
Design the override and log it. Users will overrule the system whatever the protocol says. Give them a way to do it and keep the record, because that record is your best evidence of where the system does not fit.
Put the people the output decides about into the test. A system that has been tested only with its operators has a Level 1 radius.

The same fieldwork shows what asking early buys. Before deployment, the Thai nurses told the team the retinal photo had to sit next to the result: it let them check the right eye had been graded, and it gave them something to show a patient who felt fine but needed treatment.

The Responsible AI Playbook for Founders turns these into a twelve-item design ethics checklist, in Section 3, to run on each feature before it ships.

A responsible AI policy records what a company intends. What the product does to people is set by the refusal screen, the referral rule and the override. In Thailand the researchers learned that by sitting in the clinic, and the protocol changed during the study: a refused image now waited for an eye doctor's read before anyone was sent to hospital, though in the full study the median wait for that read was ten days. Most teams will not get a published field study of their product. They can still go and watch it being used before they decide it is ready.

This is Part 3 of a four-part series for founders. Part 1 sets the principles and Part 2 turns them into governance you can enforce. Part 4 asks who outside the company should have a say: The Governance Was Flawless. The Ethics Board Had No Windows.

Sources

Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk and Laura M. Vardoulakis, "A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy", CHI 2020, ACM, April 2020.
Paisan Ruamviboonsuk, Richa Tiwari, Rory Sayres and others, "Real-time diabetic retinopathy screening by deep learning in a multisite national screening programme: a prospective interventional cohort study", The Lancet Digital Health 4(4), April 2022. Funded by Google and Rajavithi Hospital.
Regulation (EU) 2024/1689 (AI Act), Article 14 and Annexes I and III, Official Journal, 12 July 2024; Regulation (EU) 2026/1744 (Digital Omnibus), amending Article 113.

All sources checked on 6 October 2026.

Changes on 6 October 2026. Rewritten as a shorter argument with a new title, opening on a published field study. Removed: the opening story of a diagnostic start-up in San Francisco and Madhya Pradesh, the fintech and mental health case studies, and other first-person advisory claims, none of which had a source on record; the Gemini market-value figure; a salary-bias study cited for a claim it does not make; the five-stage method, the methodology section and the 12-point audit, whose checklist now lives in the Responsible AI Playbook; and the download box.