I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.
This aligns with the "zero tokens" approach described in this paper. :)
I tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.
I was doing something similar where I saved user input/model output in a multi-depth node style storage system (each depth having more precise details) with the focus on the model having accurate user fed information. I was mostly focused on retrieval of accurate / useful information based on user query (injecting the database node as additional, high confidence information)
Once this(Zero-mem) passes it's peer review, I may have to see if my system can handle something similar instead/in addition.
I'm quite excited to see growth in these different ways of eliminating token's.
Long winded aside, @langs, have you published your work on this?
Using attention for retrieval was inspired by a comment I saw in here long time ago: Prediction and retrieval are two sides of the same coin; to predict better, you must retrieve more accurately.
I'm still working on the improvement of algorithms, my tests shows the performance and accuracy will be improved a lot in the next release.
I am not an expert on LLMs, but what's preventing you from treating the context as 'virtual memory', and using the attention matrix to 'blank out' tokens which have very low weights and will not contribute much to the input? I imagine most tokens are like this, and you can skip computation on 95% of an 1M (or practically infinite) context.
Hah I built a similar thing, stored a few million tokens chunked and precomputed winth Qwen a3e (best ratio of kv-size to tokens after chunking).
Some custom kernels and I was able to find all the relevant paragraphs with full force of qwen reasoning within 0.3s, and with a summary round within 0.7s.
Downside - required 200GB ram/vram ;) A few GBs for model and most of it for caching kvs.
This is actually quite easy to implement at the harness level and the NER can be way more naive because of the typical nature of LLM dialogue (programming, long running tasks etc).
Can confirm. I did a clean-room implementation of this in Rust just by reading the paper. Will update once they release the reference implementation. https://github.com/ptaranat/zeromem
The useful contribution is not zero token cost; it is removing generative rewriting from memory. Preserving original traces avoids a subtle auditability failure: once an LLM compresses an interaction, retrieval is grounded in the summary's omissions rather than the evidence.
I would still want a harder benchmark around mutation and contradiction. If an entity changes attributes across sessions, can the graph and temporal hierarchy preserve both states, surface the conflict, and show which trace justified the answer? The 57.6% time reduction is compelling, but for production agents I would measure unsupported-answer rate and evidence recall under stale, conflicting, and adversarial traces. Encoder compute and index-maintenance cost should also sit beside token cost; otherwise "zero-token" risks being read as "free."
I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.
This aligns with the "zero tokens" approach described in this paper. :)
I tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.
I was doing something similar where I saved user input/model output in a multi-depth node style storage system (each depth having more precise details) with the focus on the model having accurate user fed information. I was mostly focused on retrieval of accurate / useful information based on user query (injecting the database node as additional, high confidence information)
Once this(Zero-mem) passes it's peer review, I may have to see if my system can handle something similar instead/in addition.
I'm quite excited to see growth in these different ways of eliminating token's.
Long winded aside, @langs, have you published your work on this?
https://github.com/AttemorySystem/attemory/ stars and issues are welcome :)
Using attention for retrieval was inspired by a comment I saw in here long time ago: Prediction and retrieval are two sides of the same coin; to predict better, you must retrieve more accurately.
I'm still working on the improvement of algorithms, my tests shows the performance and accuracy will be improved a lot in the next release.
What the difference of doing this vs semantic search over indexed fact chunks? This is RAG right?
Yes, it uses modern technology to tackle an old problem: retrieval.
"A local Qwen3.5 retrieval model attends over the indexed memory"
Hrm.
I am not an expert on LLMs, but what's preventing you from treating the context as 'virtual memory', and using the attention matrix to 'blank out' tokens which have very low weights and will not contribute much to the input? I imagine most tokens are like this, and you can skip computation on 95% of an 1M (or practically infinite) context.
Hah I built a similar thing, stored a few million tokens chunked and precomputed winth Qwen a3e (best ratio of kv-size to tokens after chunking).
Some custom kernels and I was able to find all the relevant paragraphs with full force of qwen reasoning within 0.3s, and with a summary round within 0.7s.
Downside - required 200GB ram/vram ;) A few GBs for model and most of it for caching kvs.
This is actually quite easy to implement at the harness level and the NER can be way more naive because of the typical nature of LLM dialogue (programming, long running tasks etc).
Can confirm. I did a clean-room implementation of this in Rust just by reading the paper. Will update once they release the reference implementation. https://github.com/ptaranat/zeromem
Care to elaborate?
Handling state and memory continuity without heavy token overhead is a massive bottleneck for modern agents. Very interesting approach.
[flagged]
The useful contribution is not zero token cost; it is removing generative rewriting from memory. Preserving original traces avoids a subtle auditability failure: once an LLM compresses an interaction, retrieval is grounded in the summary's omissions rather than the evidence.
I would still want a harder benchmark around mutation and contradiction. If an entity changes attributes across sessions, can the graph and temporal hierarchy preserve both states, surface the conflict, and show which trace justified the answer? The 57.6% time reduction is compelling, but for production agents I would measure unsupported-answer rate and evidence recall under stale, conflicting, and adversarial traces. Encoder compute and index-maintenance cost should also sit beside token cost; otherwise "zero-token" risks being read as "free."
[flagged]