A pilot that works in a demonstration and a system that runs every day are different objects. The distance between them is where a large share of corporate AI projects stop, and the reasons are more often organisational than technical.
A pilot is scored on its best output
Pilots are usually run by the people who chose the tool. They select the cases, write the prompts and read the results.
That selection is not dishonest, but it is favourable. The inputs are clean, the questions are ones the team already understands, and a poor answer gets discarded rather than counted.
Production has no such filter. Every input arrives, including the malformed, the ambiguous, and the ones written by someone in a hurry.
Measured quality therefore falls the moment the system meets real traffic, without anything about the model having changed.
Teams that anticipate this run the pilot against a random sample of last month's actual work rather than against chosen examples. The numbers they get are lower, but they survive contact with launch.
Integration costs more than the model
The model is bought. Everything around it is built.
A working deployment needs authentication, access controls that respect who may see which records, logging, rate limiting and a way to roll back to the previous behaviour.
It also needs connections into systems that were never designed to be read by anything other than a person: ticketing tools, document stores, ledgers carrying decades of accumulated local convention.
These are ordinary software engineering tasks and they consume ordinary software engineering time. A pilot skips all of them by having a person paste text into a window.
The budget approved for the pilot is usually a licence budget. The budget required for production is an engineering budget, and it is requested from a different person.
Someone has to own the wrong answers
In a pilot, a wrong answer is an interesting finding. In production it is a customer who was told the wrong thing.
That converts the project from a technology question into an accountability question, and accountability questions move slowly through organisations.
Legal wants to know what happens if the output is relied upon. Compliance wants to know what was retained. The department that will be blamed wants a veto.
None of these groups were in the room for the pilot, and each can return the project for another round of review.
Volume changes the economics
Per-request costs that are invisible across a hundred test queries become a line item across a million.
The comparison that matters is not the model's cost against zero, but its cost against the existing process, including the staff time the process already consumes.
That comparison often favours the model on simple cases and not on complex ones, because complex cases still require a person to check the work.
A deployment that automates the simple half and routes the rest to humans is usually viable. A deployment that promised to handle everything usually is not.
Projects fail here when the business case assumed full replacement and the technology only supports partial replacement.
Evaluation has to become continuous
A pilot is evaluated once. A production system drifts.
The vendor updates the underlying model, the company changes its policies, the incoming questions shift with the season, and behaviour that was checked in spring is not the behaviour running in autumn.
Continuous evaluation means keeping a fixed set of test cases with known good answers and re-running them on a schedule, so a regression is caught by the team rather than by a customer.
Few pilots build this, because a pilot does not need it. Retrofitting it after launch is harder than building it before.
The workflow rarely survives the change
Introducing a system that drafts a document changes what the next person in the chain receives.
A reviewer who previously read three paragraphs written by a colleague now reads two pages written by a machine, and the review takes longer rather than less time.
Savings that appear in one team therefore reappear as costs in another, and the net figure can be close to zero even when the tool works exactly as specified.
Redesigning the surrounding process is the expensive part, and it needs authority over more than one department.
Narrow deployments are the ones that survive
The projects that reach production tend to share a shape. They handle one task, for one team, with a clear definition of a correct output.
Classifying incoming messages, extracting fields from a standard form, drafting a first reply that a person then edits — each has a measurable before and after.
Narrow scope also makes failure cheap. If the classifier is wrong, a message is routed to the wrong queue and someone moves it.
Broad scope removes that safety. A general assistant with access to everything has no natural boundary on how wrong it can be.
Organisations that ship several narrow systems usually end up further ahead than those that spent the same period designing one comprehensive one.
Stalled pilots still leave something behind
A pilot that does not launch is not a total loss, though it is usually treated as one.
It produces a catalogue of where the company's data actually lives, which is information most organisations do not have written down anywhere.
It also produces a realistic sense of which tasks are genuinely rule-bound and which only appeared to be, and that distinction is what determines whether the second attempt succeeds.
The waste is not in running the pilot. It is in running the same pilot again eighteen months later because nobody recorded why the first one stopped.