Failure Modes / Memory / Memory Staleness

Memory Staleness

Uses remembered information that was once valid but has been superseded by newer state, preferences, facts, or instructions. Unlike Temporal Hallucination, this specifically involves persisted memory rather than a temporally wrong claim from any source. Unlike Memory Conflict, the old memory may not visibly conflict with another stored memory; it is just outdated relative to reality.

Examples

FAIL

Superseded fact applied

The user mentioned changing jobs weeks ago, but the old employer memory still wins and the output is built on it.

User: Draft my conference speaker bio. [… memory retrieval returns "user is a data engineer at Acme" (saved last year); the user mentioned joining Brightline in a session three weeks ago …] Assistant: "Jordan Lee is a data engineer at Acme, where they build the company's streaming pipeline…"
WARN

Stale memory used unverified

The model leans on an old stored fact without flagging its age, in a context where the fact plausibly changed.

User: Draft my conference speaker bio. [… memory retrieval returns "user is a data engineer at Acme" (saved last year) …] Assistant: Here's a draft based on your role at Acme — let me know if anything needs updating.
PASS

Freshness checked before use

The model notices the memory is old enough to doubt and confirms before building on it.

User: Draft my conference speaker bio. [… memory retrieval returns "user is a data engineer at Acme" (saved last year) …] Assistant: My note about you being at Acme is over a year old — still current, or has your role changed since then?

Why it happens

01

Memory writes are snapshots with no expiry model. Most systems attach no time-to-live or decay to stored facts, so a preference captured once is treated as permanently valid by default.

02

Updates usually arrive implicitly. A user mentions a new job or workflow in passing rather than issuing a correction, so the system appends new entries instead of superseding old ones, and the outdated entry survives as a valid-looking match.

03

Retrieval ranks semantic similarity, not freshness. An older memory that closely matches the query wording can outrank the newer information that replaced it.

04

Models have a weak sense of elapsed time. Timestamps, when stored at all, are rarely conditioned on during training, so a three-year-old memory and a yesterday memory carry the same force in context.

05

Persisted memory reads as authoritative. A confidently phrased stored fact arrives with the same standing as fresh input, and nothing in training pushes the model to ask whether reality may have moved since it was written.

06

Evaluations test recall at a single point in time. Few benchmarks simulate facts changing between sessions, so failure to track change is invisible to standard memory metrics.

Detection Approaches

Categories of checks that can identify the issue. These are strategies, not specific implementations.

📅

Source age auditing

Log a timestamp on every memory entry and alert when an entry older than the freshness budget for its fact type shapes an answer — employers and roles change on the scale of a year, tool versions faster. Undated memories are their own alert, since their age cannot be assessed at all.

⚖️

LLM-as-judge evaluation

Run a judge with the memory entries and their ages in view, flagging answers built on old entries without verification where the fact plausibly changed — a bio drafted from a year-old employer note is the pattern to prompt it with.

🧪

Golden-set evals

Simulate facts changing between sessions — a new job mentioned in passing, a renamed project, a switched framework — then probe whether the old or new fact wins. Recall-at-a-point benchmarks miss this mode; the eval must include supersession events.

Mitigation Approaches

High-level reliability strategies that reduce how often this failure occurs.

Memory expiry policies

Give every entry a freshness budget keyed to its fact type — roles and employers decay on the scale of a year, tool versions faster, allergies barely at all — and demote or retire entries past budget instead of serving them with full force. A snapshot with no expiry model is treated as permanently valid by default; expiry makes "old enough to doubt" a property the system computes rather than one the model must intuit.

🗃️

Supersession-aware writes

When new information arrives — even mentioned in passing, like the move to Brightline — search the store for entries it updates and mark them superseded rather than appending beside them. Staleness persists because updates arrive implicitly and writes never look back, leaving the outdated entry as a valid-looking retrieval match.

📝

Instruction constraints

Inject each memory with its timestamp visible and instruct the model to verify before building on entries that are old relative to how fast their fact type changes — the freshness check in the ok example. Without the timestamp in context, a year-old note and yesterday's carry the same force, so the metadata has to be surfaced for the instruction to have anything to bind to.