A 32k-token context charges you for every input token, then adds inference time as attention scales with context length. An agent that sends its entire history on every call pays quadratic attention costs for content that mattered two hours ago but no longer does. Efficient teams treat context as a managed resource, not an unlimited scratch pad.
This post covers practical methods for tightening agent context while preserving the information behind good decisions. They work across model providers and agent frameworks. I will also show where LaserData's data layer helps specifically.
What fills an agent's context
Start with an audit of a running agent's context. In most systems, the contents fall into several distinct categories:
System instructions: Persona, behavioral limits, and task framing. Usually static per deployment, from a few hundred tokens in simple agents to several thousand in complex ones. This is generally required and rarely suitable for pruning.
Conversation history: Earlier turns in the active task. Multi-step agents add to it at every step, while naive implementations retain everything. By step 20, old steps may outweigh information relevant to the current decision.
Retrieved context: Documents, memories, or reference data fetched for the current step. Effective retrieval returns relevant chunks. Without it, teams load everything and ask the model to filter, an approach that wastes resources at scale.
Tool results: Function outputs, API responses, and database reads. They can be oversized. Returning a complete JSON object when one field was needed is a common, avoidable example.
These categories offer different optimization potential. System instructions are usually fixed. Compression and pruning matter most for conversation history and retrieved context.
Sliding windows or semantic compression
The easiest history policy is a sliding window: retain the last N turns and discard older ones. It is simple and often sufficient for agents with short task horizons. Its weakness is recency-based loss. If a critical determination occurred at step 2 and the agent is at step 18, a window retaining the last 10 turns removes it.
Semantic compression takes another route. Periodically summarize older turns, replace the transcript with that condensed representation, and retain key decisions, constraints, and conclusions without the full path that produced them.
Teams we have worked with run summarization either inside the agent, as part of a step, or externally when history passes a threshold. Both can work. The external version is usually more dependable because it sits outside the primary task loop and can be audited separately.
Neither policy fits every agent. Use a sliding window when history beyond N steps cannot affect decision quality. Use summarization when earlier determinations still matter. A mismatch with the task horizon commonly degrades long-running agent behavior.
Selective retrieval versus loading everything
A common production pattern is to load all available context at the start of every step. It appears safer because nothing is forgotten. But irrelevant information also enters the prompt, increasing token cost and inference latency without improving the decision.
Selective retrieval loads only context relevant to the current decision. That requires a clear definition of relevance and an efficient retrieval path. Retrieval itself costs time. If it adds 15ms per step, token savings must exceed that latency cost for the tradeoff to work.
For most production agents we have seen, selective retrieval pays off beyond a certain step count. During the first three steps, full context may be reasonable because history is short. By step 50 of a complex task, undifferentiated history creates substantial unnecessary token cost.
In practice, selective retrieval often means several queries at each step: always include recent history, the last 2-3 steps; fetch reference data keyed to the current sub-task type; and retrieve standing constraints or earlier commitments that apply now. Each category uses different recency and relevance rules.
How external memory changes the problem
In-context memory is ephemeral. When a session ends, its history disappears. External memory, a persistent store the agent reads and writes during execution, removes that limitation. Findings, decisions, and derived knowledge can be stored outside the prompt and retrieved when needed, instead of retaining everything to prevent loss.
The question changes from "what belongs in context" to "what belongs in external memory, and what should be retrieved at each step." That calls for different reasoning and a memory layer capable of fast, selective reads at agent decision speed.
LaserData's context storage API follows this model. Agents write structured entries tagged by type, including decisions, constraints, discoveries, and task outputs, then retrieve by type or session recency. Retrieval is a query, not a full dump. The agent requests the category needed for the current step and receives that subset, keeping context tokens tied to relevance rather than total accumulated history.
Tool result trimming
Tool results often create hidden context bloat. A function returning a large JSON response places the entire response in context. If one field was needed, everything else adds length without helping the decision.
A direct fix is to process results before insertion: extract fields used by the agent, reduce large text blocks to the relevant finding, and remove fields never referenced downstream. This is straightforward when the tool is under your control, and harder with third-party integrations.
Another option is a tool interface that returns minimal, structured data instead of a raw API response. A wrapper calls the API, extracts relevant fields, and passes only those to the agent. The cost is tighter coupling, since the wrapper must anticipate which fields the agent's reasoning will require.
What context management cannot do
Context optimization lowers cost and inference latency, but it does not close fundamental capability gaps. We are not saying pruning can make an agent correct without the needed information; we are saying it can reduce waste when that information is available. Remove required information incorrectly, and the agent can still make the wrong decision.
Over-pruning is difficult to diagnose. Outputs still arrive, but quality declines in ways that look like unexplained forgetting. Debugging means checking whether the information ever entered context, whether the window had moved beyond it, or whether external retrieval missed it because the query was wrong.
Validate context policies against real task distributions, not only latency and cost. Halving token count while reducing task completion quality is not progress. The target is equivalent decision quality at lower cost, not lower cost that sacrifices correctness.