Nobody Tells You When to Stop an AI Deployment

Your AI system is live, it earns money, and the evidence that it is doing harm is sitting in someone's inbox. Who decides to switch it off, and on what evidence? My bet is that your launch plan answers neither.

My argument is that most AI deployments that should be stopped are stopped late for a structural reason. The evidence usually arrives in time. What is missing is a stop condition written down before launch and a named person who is allowed to pull the plug.

A stop decision is a pre-committed rule that says which evidence pauses or ends a live AI system and who has the authority to act on it. If either half is missing, someone outside the organization will eventually make the decision instead.

What Do Zillow, Robodebt and MyCity Have in Common?

Each was stopped by an event: losses, a court case, a change of mayor. None was stopped by a rule written in advance.

Zillow. Zillow Offers bought homes using price forecasts and resold them. On 2 November 2021 Zillow reported a write-down of about $304 million on homes bought in the third quarter "at higher prices than the company's current estimates of future selling prices", and said it would wind the business down and cut about a quarter of its workforce. Its chief executive, Rich Barton, said "the unpredictability in forecasting home prices far exceeds what we anticipated". The full-year 2021 annual report put the Homes segment's loss before income taxes at $881 million, including $407.9 million of inventory write-downs. Stanford's Amit Seru has described Zillow bidding more for some homes than its own model predicted, to compete. The wind-down came within a quarter of the overpaying, and the reason its results release gave was forecasting error and the earnings and balance-sheet volatility it caused.

Robodebt. From July 2015 to November 2019, Australia's Commonwealth government raised welfare debts by averaging annual tax-office income across fortnights. In approving the class-action settlement in 2021, Justice Murphy of the Federal Court found that the Commonwealth unlawfully asserted debts of at least AU$1.763 billion against about 433,000 people, and that it "maintained the Robodebt system in the face of approximately 29 decisions" of the Administrative Appeals Tribunal, up to 30 May 2017, that rejected the method. It ended after the Commonwealth conceded a test case in November 2019. The Royal Commission later called it "a crude and cruel mechanism, neither fair nor legal".

MyCity. New York City launched a business-advice chatbot in October 2023. In March 2024 The Markup showed it telling employers they could take a cut of workers' tips and telling landlords they could refuse tenants with housing vouchers, both against the law. The city added disclaimers and kept it online. In January 2026 the new mayor, Zohran Mamdani, called it "functionally unusable", and the city took it down in early February.

What Did Other High-Stakes Fields Work Out First?

Three fields that stop dangerous operations routinely share one habit: they fix the stop conditions before the pressure to continue exists, and they give the call to someone that pressure cannot easily reach.

The FAA's Pilot's Handbook of Aeronautical Knowledge teaches the PAVE checklist (pilot, aircraft, environment, external pressures) as a risk check made before a flight and kept up during it. The FDA's guidance on data monitoring committees describes a group of experts, independent of the sponsor, that reviews accumulating data and can recommend that a trial be stopped, suspended or changed. In almost all cases the sponsor then decides. U.S. market-wide circuit breakers halt market-wide trading for 15 minutes when the S&P 500 falls 7 percent or 13 percent before 3:25 p.m., and for the rest of the day at 20 percent.

What Are the Six Red Lines?

A red line is a condition that pauses or ends a deployment whatever the business case for continuing. Six categories make a workable starting list for a live system. Write each one for the specific deployment, with a number where a number is possible, and sign it off before launch.

Three of the cases in the table were caught from outside. Waymo filed its recall after federal regulators opened an investigation in October 2025, prompted by media reports. Microsoft changed Recall after public criticism, promising to "improve privacy and security safeguards". The MyCity answers came from reporters. Amazon's recruiting model is the internal catch: Reuters reported that the company saw the bias by 2015 and disbanded the team by early 2017. Amazon said the tool "was never used by Amazon recruiters to evaluate candidates". Anthropic is the clearest case of a pre-committed capability line: its scaling policy names the safeguards a model must have before it ships, and the company applied them before it had concluded that the model needed them.

The point of writing these down early is that a threshold set under pressure tends to be set where it will not trip. Set it while the launch is still a plan.

Who Should Hold the Stop Key?

Give the stop to two keys. One belongs to the business owner of the deployment. The other belongs to a monitor who does not report to that owner, a pattern borrowed from the independent committees that watch clinical-trial data, with one change: here the monitor holds a real key.

• Either key can pause. If a red line is crossed, the owner or the monitor can stop the system without the other's agreement. A pause on a written red line needs no further justification.
• Restarting needs both keys. The system comes back only when both agree in writing that the cause is fixed or the line was misread.
• Automatic pauses come first where a metric exists. Wherever a red line can be measured live, wire it to pause the system on its own, the way a circuit breaker does, and let the two keyholders decide what happens next.
• The board hears about every red-line stop. Tell it within a set number of days: what tripped, who paused, and what happens next. Count the stops in its quarterly report as well; A Board AI Dashboard That Is Always Green Is Broken sets out the measures.

The second key matters because of an asymmetry in how organizations assign blame. Stopping a profitable system is a visible decision with one name attached. Letting it run is a non-decision spread across many people. The independent monitor exists so that the person who says "stop" is doing their job, not taking a career risk. Your board needs that switch too, and it needs to know who holds it.

What Do the Rulebooks Already Require?

Both main reference texts already expect you to be able to stop. The NIST AI Risk Management Framework asks for processes to decommission AI systems safely (GOVERN 1.7) and for mechanisms, with assigned responsibilities, to "supersede, disengage, or deactivate" systems whose performance or outcomes are inconsistent with intended use (MANAGE 2.4). The EU AI Act gives the stop to regulators: under Article 79, a market surveillance authority that finds a risky AI system out of compliance must require the operator to correct it, withdraw it from the market or recall it, within a deadline the authority sets. Neither text says what your trigger is or who in your organization pulls it. That part is yours to write.

Five Questions Before Your Next Launch Review

1. Are the stop criteria for this deployment written down? Specific, testable, and signed off before launch.
2. Who holds the second key? Name the person, and confirm they do not report to the deployment's owner.
3. Which red lines pause the system automatically? For each one that can be measured live, name the metric and the threshold.
4. Has your governance ever stopped or materially changed a deployment? If it has not in twelve months, ask whether it can.
5. Have you rehearsed a stop? Run one as a tabletop exercise, with the real escalation path, before you need it.

For the patterns that lead to these failures in the first place, see seven patterns behind AI project failures.

Sources and Changes

• Zillow Group, third-quarter 2021 results, Form 8-K, Exhibit 99.1, 2 November 2021; and Form 10-K for 2021, filed 10 February 2022.
• Stanford Graduate School of Business, "Flip Flop: Why Zillow's Algorithmic Home Buying Venture Imploded", 9 December 2021.
• Federal Court of Australia, Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, 11 June 2021.
• Australian Government, Final report of the Royal Commission into the Robodebt Scheme, 7 July 2023.
• The Markup, "NYC's AI Chatbot Tells Businesses to Break the Law", 29 March 2024; and "Mamdani to kill the NYC AI chatbot", with THE CITY, 30 January 2026.
• FAA, Pilot's Handbook of Aeronautical Knowledge (FAA-H-8083-25C), chapter 2, Aeronautical Decision-Making.
• FDA, Establishment and Operation of Clinical Trial Data Monitoring Committees, guidance, March 2006.
• U.S. SEC, Investor.gov, "Stock Market Circuit Breakers".
• Waymo, Part 573 safety recall report 25E-084, December 2025; NHTSA, preliminary evaluation PE25013, opened 17 October 2025.
• Reuters, "Amazon scraps secret AI recruiting tool that showed bias against women", 10 October 2018.
• Microsoft, "Update on the Recall preview feature for Copilot+ PCs", 7 June 2024, updated 13 June 2024.
• Anthropic, "Activating AI Safety Level 3 protections", 22 May 2025.
• NIST, AI Risk Management Framework 1.0 (NIST AI 100-1), January 2023: GOVERN 1.7 and MANAGE 2.4.
• Regulation (EU) 2024/1689 (the AI Act), Article 79 (Official Journal text, read 5 October 2026).

Changes on 5 October 2026. Cut from about 5,000 words to a reference with question headings; the six red lines and the two-key stop authority stay. Zillow's loss is now the $881 million segment figure from its 2021 annual report, and the claim that its algorithm was known to be failing for months is removed, because no source shows it. Robodebt's figures now come from the Federal Court judgment. Removed for length or for want of a primary source: survey failure rates, five smaller cases (Tay, Gemini, McDonald's, Bard and Clearview), the NASA comparison, the whistleblower and kill-switch sections, and the "trilogy" framing. Six custom diagrams went, and so did a download offer that linked to a missing file.