How do you design memory systems for long-running AI agents?

Beginners
1

If an AI agent runs for a long time (like a chatbot or assistant), how do you design a system so it can remember things properly?

2

For long running AI agents, the simplest rule is:

Do not make the prompt the memory system.

Store memory outside the model, then only give the model the parts it needs for the current step.

A practical setup is:

  1. Store the full history, files, tool results, user preferences, decisions, and task state in a database.

  2. Before each model call, retrieve only the relevant facts and current working context.

  3. Let the model act on that small active context.

  4. After the model responds or uses a tool, write the important changes back into memory.

  5. Keep logs of what happened so the agent can recover, audit, or continue later.

Not every output should become memory. Some things should be verified first, some should be marked uncertain, and some should be ignored.

The model should be treated as the reasoning engine. The runtime should be the memory and state owner.

3

The concept sounds interesting. I wonder how the implementation would look like.

  1. Which part of the agent decides what to keep and what to through away aka forget/ignore? Programmed agent application logic or an LLM - the brain?
  2. How is the persistence part implemented? A database as container is still pretty abstract. What database type, specific product recommendations? How is the serialization done?
  3. Based on what criteria would the agent decide what persistent data records are important for the current session and context?

These are real implementation aspects I would be interested in. Can we discuss them here? Do you have references to open source examples or relevant articles?

Cheers,

Michael

4

Hi Michael, happy to discuss it here.

The main design rule I use is that the model is not the memory system. The runtime is.

For question 1, I would split the decision into layers.

The application should always make the final decision about what is allowed to become durable memory. The LLM can help classify or propose memory candidates, but I would not let it directly write permanent state without rules around it.

A practical flow is:

  1. Capture events from the run.

  2. Extract candidate memories or state changes.

  3. Classify them by type.

  4. Apply policy rules.

  5. Store only what is useful, verified, or needed later.

  6. Keep uncertain items marked as uncertain rather than treating them as facts.

For question 2, persistence does not need to start fancy.

A relational database is enough for many systems. SQLite is fine for local prototypes. Postgres is a good default once the system becomes serious. You can add vector search later for retrieval, but I would not make vector storage the whole memory system.

I usually think of storage as several categories:

  1. Event log.

  2. Current task state.

  3. Durable project state.

  4. User or operator preferences.

  5. Artifacts and files.

  6. Searchable summaries.

  7. Embeddings for retrieval when useful.

Serialization can be simple JSON at first, but the important thing is to use typed records. Each record should say what it is, where it came from, when it was written, what confidence it has, and whether it is still active.

For question 3, the agent should not retrieve everything. It should retrieve based on the current objective.

Useful criteria include:

  1. Is this needed for the current task?

  2. Was it created by this project, user, or run?

  3. Is it recent enough to matter?

  4. Is it still marked active?

  5. Is it verified or only a guess?

  6. Does it conflict with newer information?

  7. Is it instruction, preference, state, history, or evidence?

The pattern that worked best for me is:

Persist broadly, retrieve narrowly.

Store enough that the system can recover, audit, and continue later. But before each model call, build a small active context from only the pieces needed for the next step.

That is where long running agents become much more manageable. You stop treating the prompt as the memory container, and start treating the prompt as a temporary working view over external state.

5

Hi Pimpcat

Thank you very much indeed for this detailed answer and breakdown. I understand all of your points and recommendations from an engineering and architectural perspective and agree with them.

Usually I do understand these type of rules better if I can see them applied in a (sample) application or tutorial.

Would you have recommendations there?

Thank you very much indeed in advance.

Michael

6

@Pimpcat-AU Great advice - thanks.

7

Hi Michael,

I’ve been working on an open source project called Context Compiler that takes a similar approach to what Pimpcat described — the runtime owns authoritative state, and the model only receives a controlled working view each turn.

It maintains a small explicit state (premise + policies) outside the model. The host injects this state into the prompt before each call, and updates are handled deterministically from user directives rather than inferred by the model.

The motivation was exactly the issues you mentioned: constraint drift, corrections not sticking, and conversations accumulating contradictions instead of resolving them.

It’s not a full memory system (no retrieval layer), but a deterministic state layer that can sit alongside external storage or retrieval.

There are runnable demos comparing baseline vs compiler-mediated behavior across several models and providers. Across 7 models tested, baseline passes 26/42 scored scenarios, while the compiler-mediated path passes 42/42.

The demos are probably the most concrete way to see the pattern in practice.