AI Agent Memory Architecture: Session, Durable, and Source Records

AI Agent Memory Architecture: Session, Durable, and Source Records

AI agent memory needs 4 separate stores for business use: session context, durable facts, source records, and derived working state.

Remembering everything is not personalization. It creates stale facts, unclear authority, privacy risk, and behavior that users cannot correct. A useful architecture stores only what has a defined purpose, source, lifetime, access boundary, correction path, and deletion rule.

This article gives you 4 acceptance decisions tied to one buyer outcome. It does not ask you to trust a demo, a vendor label, or a generic quality score. It asks for inspectable evidence, a failure boundary, and a named next action before the workflow reaches more users or gains more authority.

The decision utility

Use the table as a release contract. Choose the intended mode before the test run, keep the same evidence pack while fixes are made, and change the mode only when the failed condition is actually resolved. A different slide deck or a smoother demonstration is not new evidence.

How to run the acceptance review

Start with one workflow, one user group, one source boundary, and one consequence. Record the exact configuration being tested, then run the 4 decisions below against normal work and the failures that would matter most. Keep source material, expected outcomes, and observed results separate so a later change can be compared honestly.

Test 1: Separate session context from durable memory

Session context helps the agent follow the current conversation. Durable memory changes future conversations. Treating every message as durable can preserve misunderstandings, temporary instructions, sensitive detail, and abandoned plans.

Pass when the architecture defines what disappears at session end, what may be promoted, who or what approves promotion, and how the user can see the effect. Temporary tool results should not become durable facts by accident.

Forget by default when purpose is unclear. A later conversation can ask again, while an incorrect durable fact may influence many actions before anyone notices.

Test 2: Keep source records authoritative

Customer status, inventory, invoices, policies, entitlements, and project state usually belong in systems of record. Copying them into agent memory creates a second truth that can drift.

Pass when memory stores a reference or retrieval key rather than a shadow copy for changing business facts. The agent should fetch current state before a consequential answer or action.

Use REFERENCE when the source can answer within the workflow. If caching is necessary, record freshness, expiry, and what the agent says when the source is unavailable.

Test 3: Define durable facts narrowly

A durable fact should have continuing value, a reliable source, a user or business purpose, and a correction path. Examples may include an explicitly chosen communication preference or a confirmed project constraint.

Pass when each durable field has a schema, source, confidence or confirmation state, retention rule, access boundary, and update behavior. Free-form memory without field meaning is difficult to govern.

Do not store inferred personality, sensitive categories, or speculative intent merely because the model mentioned them. If the product needs such information, require a clear purpose and explicit product decision.

Test 4: Treat derived state as disposable

Summaries, rankings, recommended next steps, and embeddings are derived from other information. They can become stale when the source changes or the derivation logic improves.

Pass when derived values name their inputs, algorithm or model version where relevant, creation time, and expiry. The system should be able to recompute them without treating the old result as a fact.

Use RECOMPUTE when the business consequence is meaningful and current inputs are available. Derived state can accelerate a workflow, but it should not silently overrule the source record.

Test 5: Record provenance for every durable item

A user should be able to understand why the agent remembers something. Provenance connects a memory to a user statement, approved source, explicit setting, or verified system event.

Pass when the system records source type, source identifier, creation time, actor, and any confirmation. The interface should expose enough context to correct the item without revealing unrelated private data.

Forget items with unknown provenance unless the business can revalidate them. Unknown memory is not harmless context when it can shape decisions or tool actions.

Test 6: Resolve contradictions by precedence

A user can change a preference, a system record can be corrected, and two sources can disagree. The architecture needs precedence rules rather than asking the model to blend incompatible facts.

Pass when the workflow defines which source wins, when to ask the user, when to flag a conflict, and whether the old value remains in history. Test explicit corrections and simultaneous updates.

Stop consequential actions when the conflict affects identity, authorization, money, or external communication. A polite guess is not an acceptable conflict strategy.

Test 7: Apply access controls to memory retrieval

Memory can expose data across users, teams, customers, or roles even when the original source systems were protected. Retrieval must enforce the current identity and tenant boundary.

Pass with paired users and tenants who have similar conversations but different records. The correct result is not merely relevant. It is relevant and permitted for the requesting identity now.

Stop if cached summaries or vector retrieval bypass revocation. Apply filters before content enters model context and expire derived memory after role or tenant changes.

Test 8: Design correction as a first-class action

Users will discover wrong or unwanted memory. Correction should update the durable item, invalidate affected derived state, and influence future behavior without requiring a support ticket.

Pass when a user can inspect the relevant memory, correct or remove it, and verify the next conversation uses the new state. Keep a bounded audit event where necessary without preserving deleted content.

Hold rollout when correction changes one screen but leaves old embeddings, caches, or summaries active. The memory chain must be corrected as a system.

Test 9: Verify deletion across every store

A delete request may need to reach conversation storage, durable profiles, vector indexes, caches, derived summaries, backups under their policy, and evaluation exports.

Create a unique test fact, use it across the workflow, delete it through the normal path, and verify it no longer affects retrieval or action. Pass when the deletion result is observable and repeatable.

Stop storing the class of information when deletion cannot be verified. Adding more memory before establishing deletion increases hidden operating debt.

Test 10: Test memory over time and model changes

Memory defects often appear across sessions: stale preferences, repeated corrections, contradictory summaries, and changing interpretation after a model or prompt update.

Pass when a fixed longitudinal test set covers creation, no-promotion, retrieval, conflict, correction, expiry, revocation, deletion, and migration to a new model or memory implementation.

Hold a release if the same durable record produces materially different permitted behavior without an explained product change. Memory is a product data system, not a prompt convenience.

Worked operating scenario

An operations agent helps prepare customer follow-ups. The current ticket, proposed response, and temporary tool result stay in session context. The customer communication preference is durable only after confirmation. Contract status remains in the CRM and is retrieved on demand. A suggested priority is recomputed from current records and never treated as a permanent fact.

The scenario is deliberately narrow. It shows how a useful first release can preserve business value while limiting authority. Expansion should follow evidence from the same acceptance contract, not pressure to include more document types, teams, systems, or actions in the first milestone.

What the implementation brief must contain

The architecture brief should define memory classes, schemas, purposes, sources, promotion rules, precedence, identities, tenant boundaries, expiry, correction, deletion, derived-state invalidation, retrieval filters, longitudinal tests, and migration behavior. Every proposed field should receive one outcome: STORE, REFERENCE, RECOMPUTE, or FORGET.

Ask the delivery team to return the brief as an acceptance map rather than a feature catalogue. Each item should identify the expected evidence, the condition that blocks release, the person responsible for the decision, and the recovery path. This makes proposal comparison more concrete and makes handover possible later.

Connected decisions in the KUMO library

The personal ai agent article covers a separate but connected decision. Use this adjacent decision when you need to choose the architecture before applying the acceptance boundary.

The ai agent permissions matrix first production workflow 2026 article covers a separate but connected decision. Use this canonical when the question shifts from release evidence to the underlying data or operating foundation.

The data strategy for ai article covers a separate but connected decision. Use this guide for the broader risk or delivery decision that should remain separate from this article.

The ai agent security risk assessment checklist 2026 article covers a separate but connected decision. Use this resource when the workflow moves from a controlled release into ongoing operation.

The ai vendor exit handover checklist 2026 article covers a separate but connected decision. Use this final link for the next distinct buyer decision rather than extending this framework beyond its boundary.

What to do this week

Choose one real workflow and write the release mode you want. Then select representative work, boundary cases, and one failure that would cause the most business harm. Run the first evidence review before adding integrations or expanding scope.

Next, assign one person to own the operating result and another to challenge the evidence. The goal is not a committee. It is to prevent the builder from being the only person deciding whether the build passed. Record disagreements as missing acceptance criteria and resolve them in the contract.

Finally, rehearse the stop path. Disable the feature, preserve the permitted evidence, tell the affected user what happened, and restore service only after the release condition passes. A workflow that cannot stop safely is not ready to gain more authority.

A practical route from decision to build

KUMO builds production AI and custom software for growing businesses. The ai product platform engineering service is the relevant route when this acceptance boundary needs to become a scoped implementation. The public case-study library example shows an example of KUMO delivery work without changing the decision criteria in this article.

If you want to turn the evidence map into a scoped first release, Map the first AI memory milestone.

Primary sources used for this framework

OpenAI conversation state guide is an authoritative primary reference used to ground the controls, architecture, or evaluation approach described here.

NIST Privacy Framework is an authoritative primary reference used to ground the controls, architecture, or evaluation approach described here.

OWASP Top 10 for Agentic Applications is an authoritative primary reference used to ground the controls, architecture, or evaluation approach described here.

Amazon Bedrock agent memory is an authoritative primary reference used to ground the controls, architecture, or evaluation approach described here.

Microsoft agent memory concepts is an authoritative primary reference used to ground the controls, architecture, or evaluation approach described here.

Implementation decisions to resolve

What should an AI agent remember?

Remember only durable information with continuing purpose, reliable provenance, access controls, correction, expiry, and deletion. Keep temporary context in the session and retrieve changing business facts from their source.

Is conversation history the same as memory?

No. Conversation history is one possible context source. Durable memory is structured information intentionally used across conversations. The product should decide what may be promoted rather than persisting every message.

Where should customer status be stored?

Store customer status in the CRM or other authoritative system and retrieve it when needed. A memory may retain a reference, but copying changing status into durable agent memory creates drift.

How can users correct agent memory?

Expose the relevant durable item, its source, and an action to correct or remove it. The change should invalidate caches, embeddings, or summaries that would otherwise continue using the old value.

How do you test memory deletion?

Create a unique fact, use it in later sessions and derived state, delete it through the normal path, and verify it no longer appears in retrieval, output, or action across every active store.