Why most AI projects fail before anyone writes a line of code
The failure usually isn't technical. It's that nobody agreed what "working" would look like, so there was no way to tell when the system got there.
Every few weeks we get a call that starts the same way. A team spent three months on an AI feature, shipped it, and now nobody uses it. They want to know whether the model was the wrong choice.
It almost never was.
The missing first question
Before anything gets built, someone should be able to answer this: how well is this task done today, and how would we measure that it got better?
If the answer is a shrug, the project has already failed — it just doesn't know yet. Without a baseline there's no way to tell whether the system is helping, so the decision to keep or kill it gets made on vibes, six months and a budget later.
A baseline doesn't need to be sophisticated. For a document classification task it might be: of 200 documents that arrived last month, how many were filed correctly by a person, and how long did it take? That's an afternoon of work and it changes every conversation that follows.
"Better than nothing" is not a bar
The comparison that matters isn't model versus empty screen. It's model versus the process you already have.
A support triage system that routes 70% of tickets correctly sounds impressive until you learn the existing keyword rules got 85%. We've seen teams celebrate the first number for a quarter before anyone ran the second.
Where the cost of being wrong lands
The other question worth settling early: what happens when the system is wrong?
- If it misfiles a document, someone notices next week and moves it. Low cost — automate freely.
- If it tells a patient their results are normal when they aren't, the cost is unbounded. That path needs a person in it, permanently.
Most real systems have both kinds of decision in them. The work is separating them honestly, rather than applying one confidence level to everything because it's simpler to build.
What we actually do about it
On our own projects the evaluation set gets built before the feature does. Fifty to a hundred real examples with known-correct answers, drawn from actual data, not invented. Every prompt or model change gets scored against it.
It's unglamorous and it's the difference between "the new version feels better" and "the new version is four points better and two points worse on the edge cases we care about."
That's the whole trick. There isn't a more advanced one.