Prompt Engineering Is Dead: 5 Layers of Context Engineering That Actually Work in Production

For a considerable time, prompt engineering was seen as one of the most important abilities in generative AI. Developers and consumers were trained to give precise prompts, assign a role, provide an example, state a format, and refine wording to come up with better responses from LLMs. The main idea was that one could enhance the prompt if the response was bad.

This method works; however, production AI is showing its limits. As AI systems are evolving from simple chatbots to new agents that are capable of research, coding, using instruments and making decisions, the responses are determined not only by how well the instruction is worded. The model still needs the correct documents, business data, history of conversations, memory, instruments, permissions, and real-time data.

And here comes context engineering. Context engineering is described by Anthropic as a methodology of collecting and keeping an optimal set of information available for the model during inference. This context can include system instructions, tools, external data, message history, examples and memory.

Thus, the expression “prompt engineering is dead” is intended to provoke. Prompt engineering is not dying; it merely becomes one of the layers of a broader discipline. The essential question is already changing from “What is the ideal prompt?” to “What does the AI require to make a correct decision right now?”

This shift is particularly essential in the environment of production. The chatbot can answer one question in a second; however, an AI agent can go through hundreds of steps, utilise external tools, find the appropriate documentation, analyse the outcome, and finalise its actions. Google has pointed out that context management is a significant problem for long-serving agents, noting that providing an agent with a wider context window does not solve the problem.

This new paradigm can be described in terms of five layers: instruction context, knowledge context, memory and state, tool and environment context, and governance and evaluation context.

1. Instruction Context: Tell the AI What It Is Supposed to Do

The first level is what most people think about when they hear the word “prompt engineering”: instruction. Any AI system must know its purposes, role, limitations, and expected outcome. A customer support agent must understand that it is there to solve problems for customers. A programming agent must know how to troubleshoot problems and safely change the software. A research agent must understand that its task is to search for the answer and deliver some evidence.

The mistake is in thinking that the system prompt should include all possible rules and exceptions. In reality, large prompts are difficult to work with. If the company changes its policy for refunds, it is unacceptable to change hundreds of lines of instructions.

The optimal way is to create a constant instruction but utilise changing details. A customer service agent can be instructed to answer queries according to the present company policy, verify the data using the tools provided by the company, and escalate the cases whenever there is some uncertainty.

Anthropic suggests keeping the instructions precise and straightforward, while avoiding overly complicated prompts and unnecessary edge-case instructions.

This shows that context engineering is akin to software engineering. Developers isolate application logic from mutable information and avoid hard-coding all information. AI systems can operate on the same principle: the instructions provide the goal while the context furnishes the relevant information.

2. Knowledge Context: Providing the Right Information to the Model

The next layer in the structure is that of knowledge context. It is during this stage that retrieval-augmented generation, enterprise search and document systems become significant.

Certainly, a foundation model possesses a lot of knowledge related to the fields of finance, technology and business. However, the model would not possess knowledge of the latest internal policies and procedures of the organisation. If an employee enquires, “Am I allowed to claim the expense?” a generic model will inform him of the general expense policies only but will not be able to provide information regarding the internal rules of the organisation without having the data.

To answer the employee’s question accurately, the AI will require various information, including the employee’s department, the current policy regarding expenses, the approval threshold, the type of expense, and the location. In this case, the answer is meaningful because the system finds the relevant data before generating the final answer with the help of the model.

This is what is crucial in the application of retrieval-augmented generation technology. It is not about just “linking an LLM to documents” but rather about retrieving appropriate information

Let’s say a company possesses 100,000 internal documents. A support agent providing a response to a question about the warranty for a product won’t need all the documents. Rather, he will need just the warranty policy, product specification, and the client’s purchase information.

Providing everything to the system can bring even worse results. Anthropic states that good context means having the maximum useful information and minimum irrelevant content.

This leads us to one of the most important principles of context engineering:

Having more context does not guarantee that you have better context.

3. Memory and State: Give AI Continuity

The third aspect is about memory and state. It becomes more relevant as AIs go from one-off dialogues to longer-term interactions.

Think about an agent trying to get a sale for a customer over a period of 6 months. Sometimes, the agent has to remember the needs, wants and past talks of its customer. Meanwhile, a programmer’s assistant has to remember previous efforts as well as all architectural decisions made during programming.

Still, memory does not mean storing everything.

A good architecture distinguishes between working context, the state of the session, and long-term memory. The first term stands for what is useful at the given moment. The state of the session means what has been learned during the work. Finally, long-term memory refers to what is expected to be useful in the future.

For example, when a customer says that they are considering an enterprise plan for 500 employees, the agent knows that this information will help its work right now. On the other hand, if during the conversation, a customer mentions their SSO requirement multiple times, the assistant knows that this data should be stored longer.

Consider a situation in which an employee communicates something to an AI like, “The CFO said to give a 30% discount.” Whether it was informal or wrong, it gets recorded, and later, whenever someone retrieves it six months later, it could be construed as an approval without knowing whether it is valid or not.

Thus, the importance of enterprise memory comes with provenance, timestamps, trust levels, expiration rules and updating procedures. Anthropic’s advancements in developing long-running agents help demonstrate efficient techniques such as data compaction and structured note-taking.

The question has changed from asking “What should AI remember?” to asking “What should AI remember, how trustworthy this memory is and when this memory should be verified again?”

4. Context of Tools and Environment: Making Connections Between AI and Reality

The fourth layer is that of the tools and the environment. This is where we can see an AI system transform from its chatbot role into that of an agent.

A chatbot can give you an answer. An agent can search the database, invoke an API, check a repository, conduct tests, get the document or modify the business system employed by an organisation.

Anthropic research indicates the importance of the tool and environment elements in enabling the model to perform complicated tasks.

Let us take an example of an AI coding agent trying to resolve the authentication error. It cannot be done successfully if some additional information, besides the error message, is not provided to the agent. In other words, the agent might require access to the repository, necessary source code files, dependencies, logs, recent commits, and test results.

The next step is to acquire tools that will allow searching through code, modifying files, testing and looking into failures.

The same model without the needed tools could only offer general suggestions.

The model has not become smarter, but its environment and context have become more beneficial.

That is why modern AI development resembles systems engineering. Now developers need to make decisions concerning what tools will be provided for the agent and what data the agent should use.

5. Context of Governance and Assessment: Indicating what AI should believe in

The fifth layer refers to governance. This component might be the least perceptible aspect of context engineering, but it is important when AI systems are engaged in real business processes.

Let us say that the AI system is aware of 3 pieces of information on a refund: old policy documents, a message from an employee, and the current policy database of the organisation. Which of them should this AI system rely on?

In this case, it does not matter only which information has been given to the AI system; it is also necessary to provide information regarding its authority and the way it was obtained.

Instead of just saying “Refund limit: ₹10,000″, the context can indicate the source of the information (the name of the document), the version of the policy, its start date, the relevant region, and the name of its owner.

This will enable the model to identify trustworthy information from informal and outdated information.

An example of OpenAI and its description of the workings of its internal data model represents a real example of the above-stated concept. The system combines different types of context, including table use.

Governance also regulates access to resources for any agent.

The finance agent cannot expect to automatically receive access to any employee’s medical records. The customer service agent does not need access to the payroll of the business. The coding agent does require access to a database, but not to the financial systems.

As the agents become increasingly independent, permissions become an inherent part of the architecture of the AI.

Example Case: The AI Customer Service Representative

Take, for example, a customer claiming:

“My order was delivered damaged, and I would like a refund.”

A conventional chatbot will likely apologise and respond with a standard return policy message.

However, a functional AI representative does much more.

The instruction level directs actions, explaining to the agent which method to follow regarding this complaint according to the rules that are in effect. The information level helps to reach the latest set of rules regarding lost orders and refunds. The memory level provides the agent with access to the previous experience with this customer. The environment level allows them to review the order in question and submit a refund request. Finally, the governance level determines whether the refund can be made or if a person should authorise it.

The outcome is not just a great answer. Instead, a controlled decision-making process is at work. Understanding this difference helps explain the rising importance of context engineering as businesses implement artificial intelligence in customer service, software programming, research, analytics, and business operations.

Data Agent by OpenAI Illustrates Relevance of Context

Another good indication of this change is OpenAI’s intra-company data agent work. The organisation has built a system that uses multiple data types, including database tables, annotations, and corporate wisdom, as well as memory and runtime environment data.

The importance of this is that the knowledge is not usually available in one system – it can spread across various sources such as databases, documentation, structures, human activity-related decisions and previous dealings.

Systems know the SQL language and may know how it works, but they do not grasp the meaning of the particular internal “active client” indicator.

Human beings usually have institutional knowledge that is not present in the data schemas. The context engineering tasks aim to make this knowledge available to AI.

We see a broader conclusion emerging: the model will not need to incorporate the organisation’s information if the surrounding infrastructure is capable of retrieving the information in time.

Why Larger Context Windows Are Not a Solution for All Challenges

Organisations working on AI systems have put significant amounts of money into increasing the size of context windows, and this technology is useful, but size does not equal quality.

If an AI agent processes a huge volume of useless documents, old outputs of tools, and repeated conversations, the model has plenty of information, but it is not necessarily better information.

Google’s recommendations provide for separating operations context, session state, memory, and other artefacts instead of treating them as one large information chunk.

Anthropic’s perspectives are similar when it comes to context compression and note structuring.

Thus, AI will most likely not evolve into technology that can work efficiently with an unlimited context window. Instead, in the future, many AI systems will face the same task – to choose what information to put in the context window, what to take out, and what to summarise.

The Paradigm of Context Engineering Will Become an Essential AI Skill

This shift will also transform the needed competencies of organisations.

Classic prompt engineering is associated with designing instructions, examples, formatting, and model behaviour. Context engineering includes more sophisticated knowledge of retrieval systems, databases, memory systems, APIs, tool calling, agent orchestration, security, and evaluation.

Thus, the new professional is likely to appear less as a “prompt creator” and more as a context engineer.

The crucial question is

“What data should an agent receive at the very moment to accomplish the task?”

It is important to note that answering this question requires thorough knowledge of the AI model and the surrounding business context.

Preparing for AI Agent Implementation

In creating AI agents, businesses should start with the mapping of the context architecture.

They need to establish what kind of information the agent will have access to in order to identify the sources of information, define which kind of information has to be dynamically retrieved, and determine which sources are authoritative.

They also need to be aware of how the information is retained and for how long. Some information should be retained only during the execution of a task, some information should be kept available for a session, while some information can be kept in memory for months. Each information category needs security policies and expiration rules.

Access to tools should also be controlled. Agents should have only those functions necessary for their work.

Lastly, companies have to evaluate context as well. They need to ask themselves whether the correct documents have been retrieved, whether expired information was used as context, whether the agent had used the most suitable tool and whether the system accessed unnecessary or sensitive information.

Prompt Engineering Is Not Truly Finished

The title may appear to be worded for effect, but reality is much more complex.

Prompt engineering is not dying out. It is transitioning into a larger engineering field.

This transition can be described as follows:

Prompt engineering → Context engineering → Agent engineering → AI system engineering

Prompt engineering is still useful since AI requires exact objectives and order. However, operational agents need more than prompts.

They need information, memory, instruments, authorisation, order and evaluations.

Hence, the new generation of AI systems will compete not so much in who formulates the smartest prompt but in who builds the strongest context system.

Final thoughts: The New Competitive Advantage

The first iteration of generative AI demonstrated how to get a model to talk. With the help of prompt engineering, it became possible to understand that communication can maximise the results of AI.

The second generation of AI reveals to the business world that it is about something else: for AI, it is critical to start doing work in an appropriate environment.

It means that an intelligent model devoid of valuable information may provide a confident but wrong answer. An agent without any memory will be doomed to repeat the same mistakes. An agent without instruments will lack the capacity to interact with real objects. An agent with no governance will be able to access and perform actions that it is forbidden to do.

Context engineering puts all these puzzles together.

It defines what the AI needs to know, remember, and retrieve; the instruments it requires to refer to; the information that can be believed; and the activities that may be allowed.

That is the reason why prompt engineering becomes a single function within the framework of AI development.

With prompt engineering, we learned how to prepare the instructions for the AI systems in the right way. Context engineering involves creating the right context for the AI systems to work in. 

As AI agents are taking over different professions, this becomes even more relevant. The enterprises that know how to use context will develop AI systems that will be efficient, even though they will be using the same technology as their competitors. 

Thus, the future of AI will not only rely on having the smartest model but also on building the right environment around it. 

So the main competitive advantage is no longer the prompt but the context.

Frequently asked questions

What is context engineering in AI?

Context engineering is a methodology for ensuring AI systems have access to the relevant information, tools, and governance needed during decision-making processes, enhancing their ability to perform complex tasks effectively.

How does context engineering differ from prompt engineering?

While prompt engineering focuses on creating effective prompts for AI, context engineering encompasses a broader scope that includes managing multiple layers of contextual information necessary for AI to function optimally in real-world applications.

Why is the shift to context engineering important for AI systems?

The shift to context engineering is crucial as AI evolves from simple tasks to more sophisticated operations. It allows AI to perform complex actions, make informed decisions, and improve its effectiveness based on relevant context rather than just prompt phrasing.

What are the five layers of context in AI?

The five layers of context engineering are: 1) Instruction Context, 2) Knowledge Context, 3) Memory and State, 4) Tools and Environment Context, and 5) Governance and Assessment, each playing a vital role in enhancing AI capabilities.

How can organizations implement context engineering effectively?

Organizations should map their context architecture, identifying knowledge sources, defining dynamic information retrieval processes, and implementing security policies to ensure efficient AI agent performance and accurate decision-making.