Agent memory benchmark on LoCoMo: retrieved memories vs full context
We stored every message of LoCoMo's 10 long conversations in KiviDB agent memory and answered 1,540 questions from retrieved memories: 62.3% correct with 12 times fewer tokens than the full...
LoCoMo is a benchmark of very long conversations: weeks of chat between two people, and questions about what was said. We stored every message in a KiviDB agent-memory store and answered each question twice, once from the memories a search returned and once from the whole history, with the same model.
1,540 questions, one model
- 62.3%: correct from 50 retrieved memories Against 65.8% when the model was given the entire conversation, about 13,400 words on average.
- 12×: fewer tokens per question 2,708 prompt tokens on average, against 32,024 for the full history. The model's bill falls in proportion.
- 55.1%: on questions about when Above the full history's 45.5%: each memory carries its session's date, next to the words that need it.
Answers by Amazon Nova Lite, graded against LoCoMo's answers by Amazon Nova Pro. Free-tier KiviDB database in AWS eu-central-1.
How we ran it
- The data LoCoMo's 10 conversations: 5,882 messages over 19 to 32 sessions each, and 1,540 questions in four kinds (single-hop, multi-hop, temporal, open-domain). Its fifth kind, unanswerable questions, is left out, as published evaluations usually do.
- Storing One memory per message, written as '[session date] Speaker: text', with a BAAI/bge-small-en-v1.5 vector computed on a laptop. One actor per conversation, so a search only sees its own. Loading all 5,882 took 167 seconds.
- Retrieving Hybrid search (by meaning and by words, fused) for each question, filtered to the conversation, returning the top 20 or the top 50 memories.
- Answering The same instructions both ways: answer briefly, work out relative dates from the message's date. Amazon Nova Lite on Bedrock, temperature 0.
- Grading Amazon Nova Pro compared each answer with LoCoMo's, counting a match in meaning or the same date in another format as correct, ten answers per call.
Every message, with its date
The locomo store holds one memory per message, each tagged with its conversation as the actor and its LoCoMo message id as a topic.
Agent memory · locomo · Memories
Where memory wins and loses
Share of questions answered correctly, by LoCoMo's question kinds.
| Memory, top 20 | Memory, top 50 | Full history | |
|---|---|---|---|
| Single-hop (841) | 62.5% | 72.9% | 81.6% |
| Multi-hop (282) | 32.6% | 46.5% | 51.1% |
| Temporal (321) | 46.4% | 55.1% | 45.5% |
| Open-domain (96) | 33.3% | 39.6% | 39.6% |
| All (1,540) | 51.9% | 62.3% | 65.8% |
What each answer reads
Mean prompt tokens per question, drawn to one scale.
| Prompt tokens | |
|---|---|
| Memory, top 20 | 1,184 |
| Memory, top 50 | 2,708 |
| Full history | 32,024 |
At Nova Lite's list price, 1,000 questions cost $0.07 (top 20), $0.17 (top 50) and $1.93 (full history) in model input and output. A larger model scales all three alike.
Three findings worth keeping
- How much you retrieve matters Going from 20 memories to 50 raised accuracy from 51.9% to 62.3%, and the share of LoCoMo's evidence messages that were retrieved from 60% to 71%. It still cost 12 times fewer tokens than the whole history.
- Dates belong in the memory Each memory starts with its session's date. On questions about when, retrieval beat the full history, where the model has to find a date many lines above the sentence that needs it.
- Multi-hop is the gap Questions that join facts from several sessions lost the most. Storing raw messages is the simplest form of memory; extracting facts and summaries with a model, which KiviDB agent memory can do, is what we did not measure here.
Every figure in one table
| Context given | Correct | Prompt tokens | Evidence retrieved | Search, median |
|---|---|---|---|---|
| Memory, top 20 | 51.9% | 1,184 | 60.3% | 30 ms |
| Memory, top 50 | 62.3% | 2,708 | 71.1% | 33 ms |
| Full history | 65.8% | 32,024 | all | - |
Search time is measured inside the KiviDB Cloud API in eu-central-1: a hybrid search over 5,882 memories filtered to one conversation; its 95th percentile was 97 ms (top 20) and 100 ms (top 50). Evidence retrieved: the share of the messages LoCoMo marks as each answer's evidence that were among the memories returned.
Questions
Why Amazon Nova and not a larger model?
Nova Lite is inexpensive enough to answer 1,540 questions with the whole history in every prompt, which is the comparison that matters here; the full study cost under $4 in model calls. A stronger model would raise every row.
How does this compare with published LoCoMo results?
Published results use other answer models, other judges and other prompts, so their numbers are not comparable with these. The comparison that holds is within this page: the same model, prompt and judge with and without memory.
Did KiviDB extract facts from the conversations?
No. The store's extraction model was off, and every message was stored as it was written. That is the baseline; a store with an extraction model condenses conversations into facts and summaries, which we have not benchmarked yet.
Whose data is this?
LoCoMo (Maharana et al., 2024), from Snap Research. We report scores only and do not republish the conversations.
Other case studies
- RAG retrieval on SciFact: Keyword, vector and hybrid search scored on 300 scientific claims.
- Semantic cache on Banking77: 13,083 support questions: model calls saved against wrong answers.
- PostgreSQL to KiviDB, live: Change data capture timed from commit to readable, and a fix it led to.
Give your agent a memory
Create a free database, add an agent-memory store, and recall what matters before each model call.