All writing

When to stop an AI initiative

When to stop an AI initiative

Nobody is promoted for stopping a project. That is precisely why the stop criteria have to be written down before the pilot starts, when the people writing them still have nothing invested.

Everything else in governance is expensive. This is free, takes forty minutes, and I have never seen a company do it unprompted.

The mechanism has been studied for fifty years

Barry Staw gave business students a company’s financial history and asked them to allocate research funding to one of two divisions. Then he told them how it had gone and gave them a second, larger allocation to make.

The students who had chosen a division that subsequently did badly put more money into it on the second round than students who were making the allocation fresh. Failure increased commitment rather than reducing it, and it did so most strongly among the people who felt personally responsible for the first decision (Staw, Organizational Behavior and Human Performance, 1976). The paper’s title, knee-deep in the big muddy, has outlived most of the literature around it.

Note what that result rules out. This is not a failure of information. The students knew the division had done badly. It is not a failure of intelligence or incentive design in any simple sense. It is what happens to a person who has to decide whether their own earlier judgment was wrong, which is a different and much harder question than whether the project is working.

Every AI pilot governance structure I have seen asks exactly the people who chose the pilot to assess whether it is working.

What this looks like in month nine

The project is not delivering. Everyone involved can feel it. What gets said instead is that it needs more data, or the use case needs refining, or the real benefit will come in phase two.

Phase two is the tell. Phase two is where projects go when the person running them cannot say stop and cannot say it is working.

The second tell is a change of metric. A project that was going to reduce handling time starts reporting adoption. Then it reports user satisfaction. Each substitution is individually defensible and the sequence is a project walking backwards away from the thing it promised.

The third is that the review meeting gets less frequent. Not by decision. It just does.

Writing the criteria

Before anything starts, in the same document as the business case, three things.

What specifically will be true by a specific date. Not a direction, a value. Handling time at or below nine minutes by 30 June, measured the way we measure it today.

What we will do if it is not. The options are extend once with a named change, or stop. Writing extend once prevents the infinite extension, because the second time the question comes up the document says stop and somebody has to actively override it rather than passively continue.

Who decides. One name, and it must not be the person who proposed it or the person running it. This is the part people resist and it is the part that does the work, because Staw’s finding is specifically about personal responsibility for the original choice.

Make stopping a good outcome, out loud

The criteria will not survive if stopping is career-damaging, because people are not stupid and will manage the metrics instead.

One client did this well enough that I now describe their approach to everyone. They ran eight explorations a year with the explicit expectation that six would stop. Stopping was reported in the same tone as shipping. The quarterly update listed what was learned from each closed exploration, by name, with the person who ran it credited.

Within a year people were stopping their own projects at month three without being asked, which is roughly the entire goal. The cost of a wrong idea fell to about a quarter of what it had been, so they could afford more ideas.

If your organisation cannot say out loud that six of eight will stop, you do not have an exploration budget. You have eight commitments you have not admitted to.

Where I stopped something that should have run

I am the enforcer of this rule and I have also applied it wrongly.

A pilot in a manufacturing client missed its stated threshold at the review date. Clear criteria, clear miss, and I argued for stopping on principle, partly because I wanted the rule to have teeth the first time it was tested.

The threshold had been set against a baseline measured in a different season. The process it was improving had genuine seasonal variation that nobody had thought about when writing the criteria, and against a like-for-like baseline it had actually cleared the bar.

We stopped a working project to protect a rule, which is a failure of exactly the kind the rule exists to prevent, just pointing the other way.

So the criteria need one more line, and mine now carry it: what would make this measurement invalid. If somebody can show the baseline was wrong, that is a reason to re-measure rather than to continue, and it has a deadline of its own so it cannot become a stalling tactic.

The forty minutes

Before your next pilot starts, write three sentences. The value and the date. Extend once or stop. The name of the person who decides, who is not you.

Then add the fourth: what would make this measurement invalid.

That is the cheapest governance available, and the only thing it costs is having the argument now instead of in month eighteen, when it will cost considerably more and somebody will be knee-deep.


Stop criteria are chapter six of The AI Advantage. This follows The business case nobody can audit. Next, why forty working tools do not add up to a capability: From forty tools to one layer.

More writing

5 min read

Price floors before the event, not during

Auto-buy means a promotion fires the instant you change a price. There is no pause button, no overnight reflection, no account manager ringing to check you…

5 min read

The number that keeps moving

AI-referred shoppers convert better than shoppers arriving from anywhere else. The figure is real, large, and not a plan. Here is the catch, and it matters…