Vector databases cannot provide the exact, low-latency reads that much of agent memory requires. Teams typically embed conversation history, store the vectors, and retrieve context with cosine similarity. That works in demos and early prototypes. The failures emerge when more than a handful of agents run concurrently or each agent accumulates more than a few weeks of history.
The issue is not vector search itself. It is excellent for one access pattern: approximate nearest-neighbor retrieval across high-dimensional embeddings. Agent memory also needs several patterns vector databases were not built to serve. Making one the primary memory layer sends those requests through approximation search even when approximation is unnecessary.
What agent memory actually needs
Agent memory is not one data type. It contains at least four structurally distinct access patterns that sit close together in the reasoning flow.
Working memory: current session state, the active goal stack, and intermediate results. It is written every few seconds, read at each reasoning step, and never older than the session. Reads mean a session ID point lookup followed by a recency sort. Vector search is wrong here: return the exact current state, not a semantically similar state from three sessions ago.
Episodic memory: records of prior actions and outcomes. Before choosing an approach, an agent may need similar attempts and their results. Vector search fits this pattern because the goal is semantic similarity. Even here, pure vector retrieval falls short. Results usually need filters for recency, agent ID, success or failure, or task type before similarity ranking. Vector databases support metadata filters, but they add overhead and materially reduce throughput versus a native structured index.
Structured state: task queues, dependency graphs, constraint tables, and scratch-pad key-value stores. Complex task decomposition can generate thousands of structured reads and writes per session. These are exact known-key lookups or simple range queries. Using approximate nearest-neighbor search to fetch a task by known ID wastes compute and adds tens of milliseconds of latency.
Shared knowledge: data that survives sessions and is available to multiple agent instances, including a product catalog, allowed actions, or accumulated domain facts. Its access pattern most closely resembles a traditional database or document store. Vector search helps retrieve some relevant facts, but it is not the complete solution.
Why approximate nearest-neighbor search composes poorly
ANN methods, including dominant HNSW and common IVF variants, exchange recall for speed. At typical recall settings of 0.95 to 0.99, they find the true nearest neighbors most of the time. For agent memory, that qualification matters far more than it does in document search.
If vector retrieval supplies working context and ANN returns a result that is close but wrong, the agent reasons from an incorrect prior state. In document search, a near miss lowers relevance. In state retrieval, it creates a logical error: the reasoning chain builds a plan on a false premise.
ANN used for exact retrieval therefore needs an exact-search fallback or deduplication layer, erasing much of its latency benefit. Teams often end up pairing a vector database with a relational store for exact lookups, recreating the stitching problem they meant to avoid.
The index maintenance cost at scale
ANN indexes are built offline or updated incrementally. HNSW allows online insertion, but insertion is O(log n), with a constant that rises with index size. When each agent session creates thousands of embeddings, maintaining the index becomes non-trivial as agent count and session depth increase.
As an index expands, ANN precision falls without periodic re-indexing. Teams operating vector databases for agent memory often schedule rebuilds during off-peak hours, creating periods of weaker retrieval and writes that are not yet indexed. Continuously running agents may have no off-peak hours.
Compute is another cost. Embedding generation and ANN queries add up when agents, rather than people, drive access. A human-facing search product might perform 10,000 embedding lookups per day. An agent system might perform 10,000 per minute for each agent session. That is an order-of-magnitude difference in the cost model, and infrastructure must be sized for it.
What a purpose-built agent memory layer looks like
Separate storage from retrieval semantics. Put working memory and structured state in a store optimized for consistent point and range reads. Put episodic memory requiring semantic retrieval in a hybrid layer indexing structured fields and embeddings. Route each query to its matching index instead of sending everything through ANN.
At LaserData, documents can include embedded vector fields, while the query planner handles vector and structured queries independently. For a session ID filter plus embedding ranking, it uses a B-tree to filter first, creates a candidate set, then applies cosine ranking to that set. This costs far less than scanning the full embedding space and filtering afterward, the execution order used by most vector databases.
We are not saying vector search is wrong for agent memory. We are saying that making a vector database the sole storage layer sends every access pattern through ANN machinery that only some require. The resulting overhead accumulates where it most harms agent decision latency.
Signs your vector database is becoming a bottleneck
These recurring patterns suggest a vector database is no longer the right memory layer for your agent:
Your p99 context retrieval latency is high versus p50. ANN latency varies with graph traversal depth. A p99 at 3 to 5 times p50 often means the index is traversed farther than needed for workloads that do not require approximate retrieval.
You run exact-match lookups through the vector database. If application code sends "search by known ID" requests to the vector store simply because the data is there, you are paying ANN overhead for a query requiring no approximation.
Index rebuilds are visible. If a scheduled rebuild produces noticeable retrieval-quality differences before and after its window, the index is not keeping pace with writes.
You maintain Redis or Postgres tables alongside it for agent state. Adding a secondary store for data that does not fit the vector pattern means you are operating two systems where one purpose-built system could handle both workloads.
A more honest framing
Vector databases perform well within their intended scope. The problem is that AI tooling positioned them as universal memory for LLM applications, partly because semantic search was the first memory capability those applications needed, and partly because the vector database market had incentives to broaden that positioning. Before agent access patterns were understood, this framing became foundational to many early architectures.
The teams we have worked with that replaced vector-database-as-primary-memory say the architecture became simpler, not more complicated. Removing ANN from workloads that do not need it lowers latency and cost, and removes index maintenance requiring periodic off-hours windows. Vector retrieval remains available for episodic memory, where it fits, but is no longer on the critical path for workloads it was never designed to serve.