A founder I know shipped a document Q&A agent in March. Clean demo: the AI cited the right pages, answered with specifics, never fabricated a number. Three weeks after going live, a customer called to dispute a charge. The support agent, powered by the same model, cited a refund policy that had been updated twice in the preceding year. It wasn’t making things up. It was accurately describing a document that no longer applied.
That’s not a model failure. It’s a context failure. And the difference between those two diagnoses determines whether your next three months get spent yelling at the AI or fixing the architecture around it.
The Term You’re Going to Hear a Lot
In June 2025, Andrej Karpathy posted a clarification the AI industry had been avoiding for two years: the real skill in building production AI systems isn’t prompt engineering. It’s context engineering. “The delicate art and science of filling the context window with just the right information for the next step,” he called it.
The reaction was recognition, not surprise. Practitioners already knew this. They just hadn’t had a clean name for it.
Prompt engineering, as most teams practice it, means writing a good system prompt and tweaking the phrasing until the demo looks sharp. Context engineering means asking a harder question: what does this model actually need to know to answer correctly in production, at 3pm on a Tuesday, for a customer whose account has a flag on it, using a document that was updated last week?
Those are not the same problem. The second one takes longer to get right. It also doesn’t break the moment you leave the demo environment.
Why Agents Fail in Production
Ninety-five percent of AI agent pilots never make it to production, according to MIT researchers. Gartner added a sharper data point: more than 40% of agentic AI projects will be canceled by 2027, not because the models are too weak, but because the surrounding engineering is too shallow.
The failure modes are consistent once you’ve seen a few of them.
Stale context is the most common. An agent retrieves a document and answers based on it. The document was accurate when the agent was built. It isn’t accurate now. The model has no way to know this. It answers confidently because it’s doing exactly what it was designed to do: trust its context. This shows up constantly in NL-to-SQL pipelines, where the SQL generation is correct for a database schema that no longer exists. The model isn’t hallucinating. The schema drifted and nobody updated the retrieval layer.
Demo-to-production mismatch is the second. In a demo, the engineer controls what the agent retrieves. The data is clean, current, and representative. In production, retrieval hits every edge case the demo never did: ambiguous queries, partially updated records, missing metadata. The agent was tuned on the best version of the data, then deployed against the real version.
Tool failures with no recovery is the third. When an API call fails or returns malformed output, an under-engineered agent doesn’t stop and flag the issue. It improvises. It fills the gap with something plausible and continues the workflow. By the time a human notices the output is wrong, the agent has taken three more steps on top of corrupted context.
All three of these look like AI misbehavior from the outside. None of them are fixed by changing the model. Context architecture is one layer below the issues most teams diagnose.
Why the Million-Token Window Didn’t Solve This
When context windows reached one million tokens, a prediction circulated widely: RAG is dead. Just dump everything into the context and let the model find what it needs. This turned out to be wrong, and the data behind why it’s wrong is worth sitting with.
Researchers ran long-context windows against RAG pipelines on identical document sets. The long-context approach cost roughly 24 times more at scale. More importantly, accuracy started degrading well before the window was full. In a 128,000-token context, performance fell measurably beyond the 40% fill mark. The model doesn’t read its context the way you read a page. It attends to different parts of it with varying reliability, and information buried in the middle of a large context window gets lost more often than the same information retrieved cleanly through a well-structured RAG pipeline.
The engineering implication is uncomfortable: a bigger context window means you have more room to put the wrong things in. It doesn’t tell you what the right things are, or how to structure them, or what to do when they contradict each other. That judgment still lives with the humans who designed the system.
This is why 82% of IT and data leaders now say prompt engineering alone is no longer sufficient for production AI. The bottleneck has shifted. It’s not the model’s capability. It’s the quality, currency, and structure of what you feed into it.
What Context Engineering Actually Requires
The teams that get this right in production aren’t doing anything exotic. They’ve applied engineering rigor to a layer most teams treat as an afterthought.
The architecture that works combines three things. A retrieval layer that pulls relevant, current information for each specific query, usually RAG with hybrid search and a reranker rather than raw vector similarity. A routing layer that decides, for each request, whether retrieval is sufficient or whether additional context is needed before the model responds. And an observability layer that makes it possible to inspect, after the fact, exactly what entered each model request: what was retrieved, what was available but not retrieved, what got compressed, and what changed between the run that worked and the run that didn’t.
That third one is where most teams are under-built. Debugging an agent without context observability means working backwards from a wrong answer with no record of what the model actually saw. You can stare at the system prompt for hours and find nothing wrong, because the problem was in the retrieval output two steps upstream.
Building AI features that hold up in production requires treating context design as a first-class engineering problem, not a configuration step you revisit when something breaks. That means knowing which documents are authoritative and which are outdated, keeping retrieval indexes current as underlying data changes, writing eval suites that test what the model does when retrieval returns something ambiguous, and building rollback paths for when a context source changes in a way that breaks downstream behavior.
This is not a one-afternoon job. It’s an engineering discipline. Which is exactly what Karpathy meant by art and science: not a compliment, a description of the skill level required.
The Accountability Gap at Startups
Here’s the part that usually gets skipped in the context engineering conversation. The discipline is well-described now. The tools exist. The architecture patterns have been published. The failure modes are documented. And most startup engineering teams building AI features are still not doing this.
The reason isn’t ignorance. It’s accountability structure. When an AI agent produces a wrong answer, the instinct in almost every team is to adjust the prompt. It’s immediate, it’s concrete, and it feels like solving the problem. Sometimes it even fixes the symptom for a day or two. Context architecture, by contrast, requires pulling apart how retrieval works, auditing which documents are in the index, mapping the failure back to a specific information gap, and then fixing the information gap. That takes longer. It has no visible UI. And the engineer who does it doesn’t always get credit for the wrong answers that stop happening.
This is a management problem as much as a technical one. If your team’s definition of “the AI is broken” defaults to adjusting the system prompt and hoping for the best, you are optimizing for the thing that’s easy to measure instead of the thing that’s causing the failures.
The hardest part of AI consulting for teams already mid-build is this: the prompt isn’t your product. The information architecture around the prompt is your product. Most teams only realize this after they’ve spent two months tuning phrasing that has nothing to do with the actual problem.
What Good Looks Like
The teams that get context engineering right share a few things.
They treat their retrieval indexes the same way they treat their databases: with schema management, versioning, and migration plans. When a policy changes, the process for updating the index is as defined as the process for updating the policy document itself. Nobody ships a schema migration without a rollback plan. The same discipline applies here.
They have eval suites that run against their context layer, not just their model outputs. If a document gets updated and retrieval starts returning the old version, a test catches it before a customer does.
They’ve built their workflow automation around the agent’s information needs rather than assuming the agent will adapt to whatever information happens to exist. The agent is the consumer. The context architecture is the supply chain. A supply chain you never designed isn’t a supply chain. It’s a guess.
And when something breaks, the first question isn’t “what did the AI do wrong?” It’s “what was the AI working with when it went wrong?” That diagnostic shift changes what you find and what you fix.
The model your team is running is probably not the problem. GPT, Claude, Gemini: these are strong models. What they cannot do is compensate for a context layer that gives them outdated information, missing information, or too much information with no signal about what’s relevant. That’s the work. Not the prompts, not the model selection. The information architecture around the thing.
The good news is that this is tractable engineering, not machine learning research. If your team has built reliable data pipelines before, the same instincts apply. If you haven’t, the gap between a working demo and a reliable production agent will keep appearing in the same place: the context the model is drawing from, not the model itself. Fix the source. The agent follows.