It’s Thursday evening. The AI-powered document summarizer has been running in production for six weeks and the team is mostly happy with it. Someone finds a cleaner prompt, tests it on eight examples, the outputs look noticeably better, and the change goes out just before close of day. Nobody sets a baseline. Nobody compares against the ten edge cases that caused complaints two months ago. The PR looks good, the tests pass, the deploy goes through.
On Monday morning you have three support tickets. The summarizer is producing outputs that are technically fluent but structurally wrong for a specific document type that accounts for about 20% of your users’ uploads. It was doing this before the change too, a little, but now it’s worse. You don’t know when it got worse. You don’t know how much worse. You don’t have anything to measure against.
This is not a story about a catastrophic failure. It is a story about the normal state of most teams shipping AI features in 2026.
The Two Wrong Answers
When engineering teams realize they have an eval problem, they tend to split into two camps, and both camps end up worse off than the middle.
The first camp ships and learns from production. The reasoning is reasonable: AI outputs are probabilistic, perfect testing is impossible, real user inputs will surface problems faster than any synthetic test set. Ship fast, watch the metrics, iterate when something breaks. The problem is that AI failures are often silent. A document gets summarized confidently and incorrectly. A classification runs, returns a plausible result, and is wrong 30% of the time. Nobody pages. Nobody raises a ticket. The users just quietly trust the wrong output, or quietly stop using the feature. By the time the failure is visible in your metrics, it has been running for weeks. You have no trace of when it started or why.
The second camp decides to build a proper evaluation framework before shipping anything else. A senior engineer reads a few papers, picks three open-source evaluation libraries, designs a synthetic dataset generator, maps out a custom scoring pipeline. Three months later, the eval framework covers a smaller surface than the SDK they could have installed in a day. The roadmap is behind. The team is now fighting over what the eval framework should actually measure. And the feature that prompted all of this is still shipping on vibes while the framework catches up.
Both of these teams are solving a real problem. Neither of them ships reliably.
Why AI Quality Is Different
With conventional software, a broken function usually tells you it is broken. An exception gets raised, a test fails, an error log lights up. The feedback is noisy in the good sense: it draws your attention to what needs fixing.
AI features break differently. The model does not raise an exception when it produces an incorrect output. It produces the output with the same confidence it would produce a correct one. The code that calls the model sees a 200 status code and a well-formed response. The downstream system processes it. The user never knows to complain because the output looks plausible.
This is why testing ten examples and declaring victory is such a reliable way to get yourself into trouble. Your ten examples happen to be the ones the model handles well. The ones it does not handle are sitting quietly in your user base.
A survey of over 500 enterprise AI practitioners by Galileo in early 2026 found that 84.9% of teams experience AI incidents within six months of deploying, and only 8.4% report zero incidents. What’s harder to read in those numbers is that many of the teams reporting zero incidents are not incident-free. They simply lack the visibility to detect failures that are already degrading the user experience.
The gap between “it worked on ten examples” and “it works reliably for the range of inputs your users actually send” is not a gap that intent closes. It is a gap that measurement closes.
What Evals Actually Are
The word “evaluation” carries a weight that makes it feel like a research project. It is not. At the startup level, an eval is a systematic answer to the question: does this feature behave the way we expect, across the inputs we care about, and does a new change make it better or worse?
That question can be answered with five things:
A golden set: a collection of real inputs, pulled from your production logs, that covers the range of what your users actually send, including the edge cases and the failures. Not 200 invented examples. Real inputs, with labels for what the correct output looks like.
A small number of rubrics: specific, checkable criteria that define what “good” looks like for your use case. For a document summarizer: does the output contain only information that was in the source document? Is the summary complete? For a classification agent: does it route to the right category? These are not vague quality scores. They are specific pass/fail checks tied to what your users expect.
A PR gate: an automated check that runs the golden set through your current model configuration and fails the deploy if quality regresses beyond a defined threshold. This does not need to be sophisticated. It needs to run before every change ships.
Production trace monitoring: a way to observe what your model is actually doing with real user inputs, not just the ones in your golden set. If something starts failing in production that your golden set did not cover, the trace tells you when it started and what it was doing.
Post-incident tests: every time a production failure surfaces something the golden set missed, a new test gets added before the incident is closed. This is the step that 48.3% of teams skip, which means the same failure mode can surface again months later.
None of this requires a dedicated eval team. It requires named ownership (one person who maintains the rubrics and the gate) and the discipline to make the golden set a living document rather than a launch artifact.
The Data Behind the Discipline
The numbers from the Galileo survey are worth sitting with. Teams that achieve 90-100% evaluation coverage report 70.3% excellent reliability. Teams below 50% coverage see only 32.4% reach that threshold. That is a 38-point reliability gap, and it is not explained by team size, funding, or model choice. It is explained by whether the team treats evaluation as infrastructure or as a to-do that never quite makes the sprint.
The specific number on “low-risk” behaviors is the one that surprises most teams: teams that skip evaluation for features they classify as low-risk experience 2.3 times more production incidents than teams that test comprehensively. The intuition that drives the classification, “this is simple, it is obvious, it does not need testing,” consistently fails in AI systems where edge cases proliferate beyond what any engineer imagines at design time. We covered how this plays out specifically for agents in When Your AI Agent Hits Production and the pattern holds whether you are evaluating a single model call or a multi-step workflow.
A date formatting function in a travel booking system looks low-risk at design time. In production, it interacts with timezone conversions, international date formats, and daylight saving transitions that nobody predicted. Every one of those edge cases became a ticket.
What the Practical Path Looks Like for a Startup
The teams that ship reliably are not doing something exotic. They are doing something consistent.
The first step is accepting that your current testing approach is a vibe check, not a quality gate. Five examples that look good to you is not evidence that the feature behaves well across your user base. It is evidence that the feature handles five examples you found easy to evaluate. That is useful information. It is not enough.
The second step is building the golden set before you think you need it. The best time to build it is during development, when you have the most clarity about what the feature is supposed to do and the most time to think carefully about edge cases. The second-best time is after you have three months of production logs to pull from. Either way: real inputs, real labels, not invented examples.
The third step is wiring a PR gate. The gate does not need to be strict on day one. It needs to exist. A gate that blocks deploys on regressions worse than two points from baseline, running against a 50-case golden set, will catch more problems than no gate at all, which is what most teams have.
The fourth step is making production visible. If your model is producing outputs in production and you cannot query those outputs to understand what it is doing, you are flying blind. Tracing the inputs, outputs, and context of every model call, and storing them in a queryable format, is the difference between debugging with evidence and debugging with speculation.
For teams in FinTech or HealthTech, this is not optional. It is a compliance requirement that shows up uninvited. An AI feature that produced incorrect outputs for six weeks and has no audit trail of what it said to whom is not a quality problem. It is a legal exposure. The AI-powered development work that gets to production reliably treats tracing as a first-class infrastructure concern, not a nice-to-have that arrives in the next sprint.
The Frame That Makes This Easier
Part of what makes teams avoid evals is the mental model that evaluation is something you do when you are ready, when the feature is stable, when you have time. That model produces the situation described at the top of this post.
The frame that works is simpler: every time you change a prompt, swap a model, or adjust the retrieval logic, you are making a change to the system. Changes need gates. Prompts are code.
That reframe is not a technical shift. It is a cultural one. If your engineering team has a discipline around not shipping code without tests, that same discipline applies to AI feature changes. The only difference is that the tests look different and the tooling is newer.
If you need guidance on where to start, a structured AI consulting engagement that begins with an audit of your current AI features and their observability gaps tends to surface the highest-priority issues quickly. Not because the gaps are hard to find. Because it is easier to act on someone else’s diagnosis than to make the case internally when everyone is already behind on the roadmap.
The Part You Do Not Want to Learn from Production
Here is the uncomfortable truth about the 8.4% of teams reporting zero AI incidents.
Most of them are not incident-free. They have simply defined “incident” as “something a user complained about,” which is the highest possible bar. The failures that produce plausible-but-wrong outputs, the ones that erode trust quietly and accumulate for months, do not appear in incident counts. They appear in churn rates and NPS drops, lagging indicators that are nearly impossible to trace back to a specific model change three sprints ago.
Building evaluation infrastructure before your first AI incident is not a comfort measure. It is the thing that keeps you from spending a week debugging a problem you cannot attribute to anything because the trace does not exist and the golden set was never built.
The teams building reliably in 2026 are not smarter or better funded. They treated the prompt as seriously as the code. The features that got quality gates stayed manageable. The ones that shipped on vibes accumulated debt that eventually stopped the team.
Your AI features are worth more than five examples. So is your user’s trust.