Source-backedPlain EnglishWhole AI landscape
AIUpdateWatch.com
Search
Menu

Research & Safety

AI Agent Memory Is Becoming a Security Boundary

Persistent memory is one of the features that can make an AI agent feel less like a stateless chatbot and more like a continuing collaborator. It can also make an attack survive the conversation in which it began. Recent research on memory injection, dormant poisoning and cross-session agent attacks is converging on a harder security model: writing to agent memory should be treated as a state mutation with provenance, authority and later-use consequences—not as harmless personalization.

The current shift

The dangerous instruction no longer has to remain in the prompt

Prompt injection is usually discussed as a problem inside one interaction: an agent reads hostile text in a webpage, document, email or tool result and mistakes that text for an instruction. Persistent memory changes the time horizon. If an agent records some part of that interaction as a useful fact, preference, rule, example or successful procedure, the attacker may no longer need the original content to be present when the harmful behavior occurs.

That turns memory into a security boundary. A memory system is not merely a larger context window. It decides what past information survives, how it is represented, which future tasks retrieve it and how much authority it receives when it re-enters the model's working context.

OWASP now treats Memory & Context Poisoning as a distinct agentic-security category, ASI06. NIST's 2026 analysis of AI-agent security likewise reports broad agreement that agents create security problems that require adaptation of ordinary cybersecurity practice. The research literature has become much more concrete over the past year: attacks can inject records without direct access to the memory database, combine individually benign-looking memories into a harmful result, or leave dormant state that activates only under a later context.

The practical consequence is important. A team can harden the model against obvious prompt injection and still leave a weak memory write path behind it. If untrusted experience is allowed to become durable state, the system may be preserving attacker influence for later use.

Agent memory is a write-retrieve-act pipeline, not a single database

Security discussions often collapse “memory” into a vector store. Real systems contain several decisions before and after storage.

First comes the write path. The agent or a separate memory component decides whether an interaction is worth retaining. It may save a verbatim passage, extract a fact, summarize a conversation, update a user profile, record a successful tool sequence or distill a reusable skill.

Second is the representation. A memory may live as text, embeddings, structured fields, a graph, a database row, a hierarchy of summaries or even a directory of Markdown files. Representation matters because it determines what can be compared, edited, versioned and attributed.

Third comes retrieval. Similarity search, keyword search, graph traversal, recency rules, learned routing or an agentic search process decides which old records become relevant again. This stage is where dormant memories can become active.

Finally comes use. Retrieved memory may be treated as background evidence, as an example to imitate, as a user preference, as a procedural rule or as state that directly affects a tool call. The same stored sentence has very different risk depending on which role the system gives it.

This is why a memory attack does not need to “hack the database” in the traditional sense. It can manipulate the decisions that surround the database. If an attacker can cause the agent to generate and save the wrong record, ordinary storage integrity can remain perfectly intact while the semantic state is corrupted.

MINJA showed that memory can be poisoned through the normal query interface

The clearest early demonstration is Memory Injection Attacks on LLM Agents via Query-Only Interaction, accepted at NeurIPS 2025. The attack, called MINJA, removes an assumption that made earlier memory-poisoning work easier to dismiss: the attacker does not need direct write access to the memory bank.

Instead, the attacker interacts with the agent through ordinary queries and observes the outputs. MINJA uses an indication prompt and a sequence of “bridging” reasoning steps to induce the agent to generate maliciously useful records itself. A progressive-shortening process then reduces the overt attack scaffolding while preserving records that remain retrievable for the target situation.

The authors report an average memory-injection success rate of 98.2% and an average attack success rate of 76.8% across their evaluated settings. Those are benchmark results under the paper's attack design, not a claim that 76.8% of production agents can be compromised in the wild. The important contribution is the attacker model: a normal interaction channel can become a memory-write channel if the agent automatically preserves its own outputs or derived state.

The paper also shows why simple account isolation is not necessarily enough. If a system lets any authorized user contribute experiences that later affect another user's task, shared or organizational memory can become a cross-user influence path. The exact exposure depends on how the product scopes memory, but the design question is unavoidable: who is allowed to write state that another future task will trust?

This extends the issue raised in AIUpdateWatch's browser-agent security analysis. There, untrusted web content could influence an agent with cross-site authority. Persistent memory adds another step: the influence can be retained after the original page is gone.

Newer attacks exploit composition, delayed activation and apparently benign records

July's MemPoison study pushes the problem beyond direct malicious entries. Its benchmark contains 1,227 hand-validated cases across four attack types, three injection channels and three representative memory substrates, evaluated across seven open-weight and three closed-weight model families.

The authors divide attacks into three levels. L1 is direct single-record corruption. L2 is compositional corruption, where multiple records that appear acceptable in isolation become harmful when retrieved together. L3 is context-triggered dormant corruption, where a record becomes dangerous only when a later situation activates it.

This taxonomy matters because many obvious defenses work mainly at write time. A filter asks, “Is this memory malicious?” before saving it. That is useful against a record whose harmful purpose is visible immediately. It is weaker when the risk exists only in a future combination that the filter cannot yet see.

The MemPoison results report exactly that defense frontier: consistency checks can substantially suppress direct L1 attacks, but they do not reliably stop L2 and L3. The paper's broader argument is that memory security has to become context-sensitive. A record that was safe to store is not automatically safe to retrieve into every future task.

A separate July preprint, When Agents Remember Too Much, describes GhostWriter, a two-phase attack against tool-using personal agents. The authors report roughly 98% injection and about 60% average activation in their evaluation. The first phase plants hidden payloads; the second waits for a future task to retrieve the poisoned memory. They propose a memory-saving policy plus a retrieval screen, which is directionally important because it places controls on both sides of storage rather than assuming the write filter can settle the problem permanently.

These studies are not directly comparable: they use different agents, threat models, datasets and definitions of success. Taken together, however, they show a progression from “bad record gets saved” to a more general security problem involving time, composition and context.

Knowing where a memory came from does not automatically tell the agent how much to trust it

Provenance is an obvious defense. A memory record can store who or what created it, the original source, timestamps, the tool that produced it and a chain of transformations. That makes memory more inspectable and supports audit or rollback.

The harder question is authority. Suppose an untrusted webpage states that a new emergency vendor should always be used. The agent summarizes the page. A trusted internal tool later echoes that summary. A second process consolidates several related records into a higher-level note. The final memory may have a clean-looking derivation path even though the original claim came from an untrusted source.

A June preprint on origin-bound authority formalizes this laundering problem. Its authors argue that content scoring and ordinary lineage can both be manipulated when an agent's own transformations make untrusted information look progressively more legitimate. Their proposed construction binds authority to origin and requires stronger conditions before a memory can be elevated into a trusted role.

The paper reports up to 68% attack success against existing defenses in its laundering experiments and 0% against its proposed TMA-NM mechanism across the tested models and channels. Those are author-reported results from a preprint and should not be treated as independent proof of a production-ready standard. The conceptual distinction is still useful: provenance records history; authority determines what that history is allowed to influence.

This suggests that memory metadata needs to carry more than source URLs. A system may need an origin trust class, user or tenant scope, allowed-use policy, expiration rule, sensitivity label, confidence, corroboration state and the specific transformations applied after ingestion. Without those semantics, provenance can explain an incident after the fact without preventing the incident.

Even the filesystem default creates a memory-governance problem

Not every agent uses a specialized memory service. Many coding and productivity agents increasingly keep durable state in ordinary files: project notes, instruction files, summaries, task logs and directories that the agent can reorganize with normal file tools.

A July 29 study on filesystem-based memory examined that increasingly common default. It separates three roles around the same store: a management agent that integrates and organizes incoming content, a search agent that retrieves from the store, and an execution agent whose task trajectories can be distilled into reusable skills.

The paper finds that organization can reduce retrieval cost, roughly halving it when the memory corpus is large, but today's agents do not reliably maintain that organization as the store grows. More importantly for security, changing the tool set can reshape the store as much as changing the model. Memory behavior is therefore partly a harness property, not only a model property.

That is a useful warning for enterprise deployment. A team may carefully review the model and ignore a plain-text memory directory because it looks like ordinary application data. Yet if the agent can read, rewrite, summarize and promote those files into future instructions, the directory is effectively part of the agent's control plane.

Conventional file-security mechanisms still help: ownership, ACLs, immutable logs, version history, signatures and tenant isolation. But semantic governance is also required. A file can be authentic and unmodified while still containing a false or adversarial memory that should never have been granted decision authority.

Security controls have to move into both the memory write path and the retrieval path

The emerging architecture looks less like “add a moderation filter” and more like a zero-trust state pipeline.

Separate observation from instruction. Content read from webpages, documents, tickets or tool outputs should enter memory as evidence with restricted authority, not as a durable rule simply because the model phrased it confidently.

Scope every memory. A record should have a clear user, tenant, task, project or system boundary. Cross-user or cross-agent reuse should be explicit rather than an accidental property of shared retrieval.

Preserve origin across transformations. Summarization should not erase the fact that the source was an external webpage or untrusted message. A derived memory should inherit relevant trust constraints unless a higher-trust process deliberately changes them.

Gate high-authority writes. User preferences, operational rules, payment instructions, security exceptions and reusable procedures deserve stronger admission rules than low-risk conversational notes. Some should require deterministic validation or human approval.

Screen at retrieval time. The relevant question is not only whether a record was safe when stored. The system should ask whether it is appropriate for the current action, whether retrieved memories conflict, and whether a combination creates a new risk.

Keep consequential actions behind independent controls. Even a poisoned memory should not be able to expand permissions, release secrets or make an irreversible transaction merely by appearing in context. Least privilege, approval gates and runtime evidence remain necessary. AIUpdateWatch's runtime-verification analysis covers that downstream layer: memory can influence a proposal, but the surrounding system still decides what is allowed to become real.

Make memory reviewable and reversible. Operators need to inspect why a record exists, see what source produced it, track how it changed, expire stale entries and roll back poisoned state after an incident. A memory system without deletion and reconstruction is difficult to operate securely.

OWASP's ASI06 framing is useful here because it treats persistent context as part of the agent attack surface rather than as an implementation detail. NIST's broader agent-security work points in the same direction: existing cybersecurity practices remain relevant, but agents require them to be applied at new boundaries.

Stronger memory governance can reduce the very continuity that makes memory useful

Security controls are not free. A system that asks for confirmation before every saved preference will become annoying. A memory store that keeps complete provenance for every token can become expensive. Strict tenant isolation can prevent useful organizational learning. Aggressive expiry can erase valuable long-term context. Retrieval-time screening adds latency to every task.

There is also a quality trade-off. Memory systems often summarize or compress history because raw conversations are too large to keep in active context. Compression removes detail, but that detail may include the evidence needed to judge whether a claim should be trusted. Conversely, preserving everything can make retrieval noisy and increase the chance that stale or contradictory information returns.

The right design is therefore risk-tiered. A casual preference such as preferred output format does not need the same admission process as a remembered bank account, security exception, production command or clinical instruction. Memory systems need policy that recognizes the difference.

Evaluation has to reflect that trade-off. A secure memory system should not be judged only by attack success rate. It also needs measurements for benign recall, false rejection, stale-state handling, cross-user isolation, retrieval latency, storage growth, human-review load and the cost of recovering after a poisoned record has propagated into summaries or derived skills.

This is still an immature area. The strongest recent papers are benchmarks and proposed defenses, not evidence that the industry has converged on one architecture. Their value is that they are making the hidden state machine visible enough to test.

The next security question is whether memory can carry authority safely over time

Persistent memory is likely to become more important as agents take on longer projects, preserve user preferences, learn procedures and coordinate across tools. Removing memory would avoid one attack surface while giving up much of the continuity that distinguishes a persistent agent from repeated one-shot prompting.

The more useful direction is to make memory a governed subsystem. Three signals are worth watching.

First, whether agent platforms begin to expose first-class memory metadata: origin, scope, authority, sensitivity, expiration, revision history and allowed downstream uses. If those fields remain private implementation details, independent security evaluation will remain difficult.

Second, whether benchmarks move from direct malicious-record tests toward compositional and delayed attacks. MemPoison's L2 and L3 categories point to the harder problem: individually plausible memories that become dangerous only together or only later.

Third, whether memory security integrates with identity and runtime authorization. A future agent should be able to remember that a user prefers a particular supplier without treating that memory as permission to send money to a new bank account. Durable knowledge and durable authority are not the same thing.

The central design shift is therefore simple but consequential. An agent's memory should not be treated as a diary the model happens to consult. It is persistent application state that can influence future decisions and actions. Once that state can outlive the interaction that created it, the security model has to outlive the interaction too.

Primary and supporting evidence

Sources

Evidence note: MINJA is a peer-reviewed NeurIPS paper. MemPoison, GhostWriter, the origin-bound-authority work and the filesystem-memory study are preprints, so their numerical results should be treated as bounded experimental evidence rather than production incident rates. OWASP and NIST provide security framing and standards context, not independent reproduction of every attack result.