AI Transformation Series · 04← Lakshmi Peri

The Pilot Graveyard: Why Experiments Don't Scale

Pilots die in the gap between "it works" and "it runs." Scaling was never the problem. The pilot and the production capability are different objects.

The pattern repeats everywhere. A pilot succeeds. The demo gets applause, maybe a town-hall mention. Six months later it's gone, and the autopsy never says the model was bad. It says the budget ended, or the champion changed roles, or people drifted back to the old way. Teams conclude they failed at scaling. They didn't. They built a demo and tried to promote it.

What a pilot proves

The model works

On selected cases, run by enthusiasts, beside the real workflow, with errors treated as findings and a project budget paying for it.

What production requires

The operation works

On every case, run by everyone, inside the real workflow, with errors treated as incidents and an operating budget paying for it. Nobody tested any of that.

The Six Gaps

Every dead pilot fell into at least one. Most fell into three or four at once.

GAP 1The Workflow Gap

In the pilot

The AI runs beside the work. People copy things into a separate tool, then copy results back.

In production

The AI runs inside the tools where work already happens. Nobody leaves their workflow to use it.

Where does this live when nobody is excited anymore?

GAP 2The Error Gap

In the pilot

Errors are interesting. Someone screenshots them for the retro.

In production

Errors are incidents. Someone has to catch them, fix them, and answer for them, at 4pm on a Friday.

When it's wrong, who notices, who fixes it, and how fast?

GAP 3The Ownership Gap

In the pilot

A champion drives it on borrowed time and personal credibility.

In production

An operator runs it as part of their actual job: monitoring, prompts, model updates, user questions.

Whose name is on this after the champion gets promoted?

GAP 4The Money Gap

In the pilot

A project budget with an end date pays for everything.

In production

An operating line pays for inference, maintenance, and the operator's time, every year, defended in every budget cycle.

Which budget line does this live in two years from now?

GAP 5The Measurement Gap

In the pilot

Success is a demo and some enthusiastic quotes.

In production

Success is a KPI moving against a baseline that was written down before the build started.

What number proves this is still worth running next year?

GAP 6The People Gap

In the pilot

Volunteers who wanted this to work used it carefully.

In production

Everyone else uses it, including the skeptics, the rushed, and the people who never read the guide.

Does it survive contact with someone who doesn't care?

The fix is sequencing. Design the production operation before the pilot starts: the owner, the error path, the budget line, the baseline. A pilot without a production design isn't an experiment. It's a demo with a deadline.

How to Use This

STEP 1

Audit the living

Run every active pilot against the six gaps. Score each gap open or closed.

STEP 2

Close before scale

An open gap is a workstream with an owner and a date. Scaling waits.

STEP 3

Change the entry rule

No new pilot starts without its production design on one page.

STEP 4

Retire honestly

A pilot that can't close its gaps gets a written end, and its lessons get kept.