The Hidden Reason: Why Your AI Gets Answers Wrong

“Context engineering: what it actually is, why it broke our system, and how we fixed it.”

We built an AI-driven observability and error-resolution system for a production platform.

The idea was clean. Capture every log. When an exception occurs, an AI layer reads it and takes action. If it recognises the problem, it pulls a resolution from a knowledge base. If it cannot solve it, it raises a properly formed support ticket. Deployment, pipeline, or security issues route to DevSecOps. Core faults route to engineering.

“On paper, elegant. In production, it failed. Repeatedly.”

It routed errors to the wrong teams. Security issues went to engineering. Pipeline failures went to support. And when it did find a resolution, the resolution often did not work.

We spent weeks assuming the model was not good enough. We were wrong. The model was fine. What we fed it and what it retrieved was broken. That distinction is the whole of context engineering.

Context Engineering Explained: What It Is and What It Isn’t

Here is the most common misunderstanding. Most people think context engineering means writing a better system prompt. Others think it means adding RAG. Both are pieces of it. Neither is the whole.

Prompt engineering is about instructing the model, crafting the wording of the request. Context engineering is about equipping the model, designing the system that assembles the right information, at the right time, in the right form, so the model can get the answer right in the first place.

The difference matters. A perfect prompt handed to a model with no context still fails. A modest prompt handed to a model with excellent context often succeeds. The leverage moved from the instruction to the information around it.

The industry felt this shift as soon as people moved from single questions to real agents, systems that use tools, remember conversations, pull in documents, and make decisions across many steps. At that point, clever wording stopped being the bottleneck. What the model could see became everything.

The trade-off: context engineering costs far more design effort than writing a prompt, in exchange for a system that is reliable rather than occasionally lucky.

What Actually Goes into the Context Window?

At any given moment, a model only knows what is inside its context window. That one snapshot is everything it has to work with. So context engineering is, concretely, the discipline of deciding what goes into that window, and what stays out.

The window is assembled from a handful of distinct ingredients:

• Instructions, the System Prompt: The standing rules that tell the model its role and constraints. This is the part most people think is the whole job. It is one ingredient.
• The User Request:  The actual task in front of the model right now.
• Memory: Both the short-term state of the current task and the longer-term memory of what happened before. Without it, the model starts from zero every time.
• Retrieved Information: What the system pulls from the knowledge base to ground the answer, the RAG layer.
• Tools: What the model can actually call to act, query a system, raise a ticket, take a step.

Here is the part most people miss. The window is finite. You cannot pour everything in. So context engineering is really an allocation problem: with limited space, what earns a place in the window for this specific task, and what does not? Too little and the model guesses. Too much and it drowns in noise and loses the thread. The skill is assembling exactly what the task needs, and nothing more.

In our system, the two failures mapped straight onto this. The misrouting was thin instructions and a missing signal on the input side. The failed solutions were poor retrieved information on the knowledge side. Same window. Wrong ingredients.

The trade-off: assembling the window per task is more engineering than dumping everything in, in exchange for a model that sees the signal instead of the noise.

How Context Fails?

We hit three failure modes. Naming them is half the battle, because each one looks like a model problem and is not.

1. Missing Context

The model does not have what it needs, so it fills the gap with a confident guess. This was our wrong-routing problem. We handed the model a bare error message, and it classified from a fragment, sending security issues to engineering. The information it needed existed. We just were not giving it to the model.

2. Stale Context

Information that was accurate once quietly goes out of date while still sounding authoritative. This sat underneath our retrieval problem. Knowledge changes, and a retrieval system that is not refreshed keeps serving yesterday’s answer with today’s confidence. Stale context does not throw an error. It just gets slowly, quietly wrong.

3. Wrong Retrieval

The system fetches something that looks relevant but is not, because of how the knowledge was split and matched. This was our chunking problem. The retrieval looked successful, the resolution looked plausible, and it did not work because the chunks pulled were the wrong ones.

Every one of these looks like the AI is not good enough. Every one of them is a context failure.

Read Also: Five Moves Every CTO Should Make Before 2027

The Model is Rarely the Bottleneck

Here is the uncomfortable truth for anyone still chasing the next model release.

The frontier models available today are extraordinary. [VERIFY AND INSERT: Claude’s latest, alongside models like MODEL, MODEL, and MODEL] can write production-grade code, reason across complex problems, and handle genuinely hard tasks. The raw capability sitting inside these models is almost certainly far beyond what most applications are actually using.

So when an AI product underperforms, the instinct is to reach for a newer, bigger model. It rarely helps. Because the limit was never the model. It was the context around it.

This is exactly what we saw in our own system. The model classifying our errors and proposing resolutions was highly capable. It was not failing because it lacked intelligence. It was failing because we handed it a bare error message with no surrounding signal, and it retrieved resolutions from a knowledge base that was poorly chunked and going stale. Same model throughout. We changed what it could see, and routing accuracy went from 70 percent to 90 percent.

Fix the context, and the same model that was producing mediocre output starts producing reliable output. Not because the model changed. Because what you fed it changed.

That is the line that separates teams shipping AI products that work in production from teams endlessly upgrading models and hoping.

The trade-off: getting the context right is slow, unglamorous engineering, in exchange for finally using the model capability you are already paying for.

What Actually Works?

• Enrich and Tag the Input

We stopped handing the model a bare error message. We improved log capture so it saw the fuller picture, and we tagged logs to map each one to its right context. Routing accuracy went from 70 percent to 90 percent. Same model. Better input context.

• Engineer the Retrieval, Do not just Enable it

Chunking was the core problem, so we reworked it. We added a graph layer so the system understood the relationships between pieces of knowledge, not just surface similarity. And we set a periodic refresh so the knowledge stayed current instead of rotting.

• Treat Context as Infrastructure, Not Setup

This is the deepest lesson, and it shows up in two habits worth adopting. First, treat your system prompt as code: versioned, tested, and reviewed, not a text file someone wrote once that everyone is now afraid to touch. Second, build the ability to measure context quality before you start tuning it, because context engineering done by gut feel is how you ship regressions that nobody notices until production does.

The trade-off: all of this is ongoing engineering effort, in exchange for an AI system you can actually trust in production.

The Takeaways

• When an AI system gets things wrong, check the context before you blame the model. The model is rarely the weak link.
• Context engineering is not prompt engineering and not just RAG. It is the discipline of assembling the right information for the model to succeed.
• A model only knows what is in its context window. That window is finite, so context engineering is fundamentally an allocation problem: what earns a place, and what does not.
• The window is built from instructions, the user request, memory, retrieved information, and tools. Most people think only about the first. All of them matter.
• Three failures recur: missing, stale, and wrong retrieval. Each looks like a weak model and is not.
• Enriching and tagging our input context took routing accuracy from 70 to 90 percent, on the same model.
• Treat your system prompt as code, and measure context quality before you tune it. Context is infrastructure, not setup.
• The question worth sitting with: if your AI gave a wrong answer tomorrow, would you know whether it was the model or the context? And if your instinct is to reach for a bigger model first, what is that assumption costing you?