--- title: "AI agent memory harvest · extraction, dedup, supersession, write pipeline · Agent Fieldbook" url: https://agentfieldbook.org/memory/harvest/ description: "How AI agents write and ingest memory. Multi-pass extraction, verification gates, classification routing, content-addressed dedup, supersession version chains, summarisation and lifecycle hooks. A source-graded field book to the write pipeline that turns raw conversation into durable, well-shaped memory." section: "Memory · Harvest" source: Agent Fieldbook — generated from the published page --- # How memory is written & ingested **Write · Both** The write pipeline. Raw conversation in, durable memory out: extract candidates (multi-pass), verify them, classify into a schema, dedup by content-address and supersede stale values: before anything is persisted. Each pattern is sourced to a framework doc, a first-party engineering post, or an arXiv paper. ## Trade-off - [↗](https://blog.cloudflare.com/introducing-agent-memory/) - [↗](https://arxiv.org/abs/2504.19413) - [↗](https://arxiv.org/abs/2310.08560) - [↗](https://arxiv.org/abs/2304.03442) - [↗](https://langchain-ai.github.io/langmem/concepts/conceptual_guide/) ## Field notes Extract, verify, classify, dedup, supersede: the write pipeline Harvest is the write path: what gets remembered, when, and in what form. It is the least-specified part of most agent stacks and the one that quietly determines everything downstream. Write too much and retrieval drowns in noise; write too little and the agent has no continuity. The approaches that hold up extract structured facts at the end of a run rather than storing raw transcripts, because a log is cheap to keep and expensive to search. - [Brandon Chaplin](https://www.linkedin.com/in/brandon-chaplin-digital-marketing-strategist) ## Common questions ### How does an agent decide what to remember? An extraction step reads the conversation and pulls out the parts worth keeping as separate facts, instead of storing the whole transcript. Each candidate is checked, given a type, and only then written. Everything else is dropped or left in the raw log. ### Should I store the whole conversation or just extracted facts? Both, for different jobs. Raw transcripts are cheap to keep and useful when debugging, but noisy and expensive to search. Extracted facts are what the agent actually retrieves. Most stacks keep the log and index the facts. ### When should memories be written, during the conversation or after? After the turn, or at the end of the run. Extracting mid-turn adds latency to the reply the user is waiting for. Running it after the response, or as a background job, keeps the conversation fast and lets the extractor see the whole exchange. ### How do you stop duplicate memories piling up? Derive each record's id from a hash of its content, so writing the same thing twice does nothing. That catches exact repeats only. Near-duplicates worded differently need a separate pass that compares meaning and then merges or replaces. ### What happens when a new memory contradicts an old one? The write step should settle it rather than store both. The usual approach is to look up existing memories on the same subject first, then explicitly add, update or delete. If contradictions are allowed to accumulate, retrieval returns both and the model picks one at random. ### When should a conversation be summarised? Once the history grows longer than is useful to keep word for word. Older messages fold into a short summary that stays in context while the raw turns move to storage. It keeps the gist inside the context window at the cost of detail you cannot recover. ## Next in the learning path - [Retrieval patterns The read pipeline that reads what you write](https://agentfieldbook.org/memory/retrieval/) - [Memory schemas The shapes harvest classifies candidates into](https://agentfieldbook.org/memory/schemas/) - [Memory providers Where the harvested records get persisted](https://agentfieldbook.org/memory/providers/) - [Agent coaching How reflection writes back into memory](https://agentfieldbook.org/memory/coaching/) ## Entries _9 entries listed on this page._ | name | description | subjectOf | | --- | --- | --- | | Content-Addressed ID + Dedup | Hash (e.g. SHA-256 of session+role+content) becomes the row id; INSERT OR IGNORE silently skips duplicates: re-ingesting a conversation is idempotent. | https://blog.cloudflare.com/introducing-agent-memory/ | | Multi-Pass Extraction | A full pass chunks the conversation (concurrent) for broad coverage; a detail pass runs overlapping windows to catch concrete values (names, prices, versions); results merge. | https://blog.cloudflare.com/introducing-agent-memory/ | | Verification Gate | Before commit, run entity / temporal / relational / support checks on each candidate memory → pass, correct, or drop. | https://blog.cloudflare.com/introducing-agent-memory/ | | Classification Routing | Classify each memory into its schema (fact / event / instruction / task) and route storage + indexing accordingly: e.g. tasks stay out of the vector index. | https://blog.cloudflare.com/introducing-agent-memory/ | | Write-Time Fact Extraction (ADD/UPDATE/DELETE) | An LLM extracts atomic facts on write and reconciles vs. existing memory as ADD / UPDATE / DELETE / NOOP. | https://arxiv.org/abs/2504.19413 | | Supersession (version chains) | A new memory with the same key supersedes the old via a forward-pointer version chain rather than deleting it; superseded vectors are pruned in parallel with new upserts. | https://blog.cloudflare.com/introducing-agent-memory/ | | Summarization / Compaction | Roll older conversation history into a running summary to stay within the context window. | https://arxiv.org/abs/2310.08560 | | Reflection / Self-Edit | The agent reflects on past experience and rewrites its own memory blocks or system prompt. | https://arxiv.org/abs/2304.03442 | | Lifecycle Hooks (pre / post-message) | Read/write memory at fixed points in the agent loop: before prompt assembly, after the response. | https://langchain-ai.github.io/langmem/concepts/conceptual_guide/ |