I don't get it. Why benchmark the latency instead of recall/precision? Optimizing for millisecond-level latency is meaningless in the context of LLM calls. Accuracy is the tool's greatest value, yet there is no testing for it?
Has anyone else benchmarked all these tools for precision/recall? I too want to know if agent memory is something I should add. I only do session memory for now and that is quite useful.
Benched cold start recently: 19k LongMemEval sessions laid down in the real ~/.claude and ~/.codex layouts, 100 questions whose answer sits in exactly one session, scored by whether that session comes back (bias: I built deja, one of the six)
deja: 29s to index, 24ms query, 18/100 hit@1, 67 found@50. Plain BM25, no vectors
agentmemory: 95s import, 14 hit@1, 65 found@50, plus a worker and engine on four ports
MemPalace: ~3h mining, 2.6s query, 14 hit@1
CASS: 56m index, every NL query fails with "query fuel exhausted" on the release build (fixed on their main)
claude-mem: no-op out of the box, only records forward from install
funes: the documented 1 min first pass indexed 189 of 19k sessions (0/100); full index still embedding, ~3 sessions/s
Numbers look low because 19k sessions is brutal; on the standard 500-session LongMemEval-S the same BM25 gets ~85% hit@1
The funny thing is BM25 basically ties embeddings here at 1/100th the cost. The real cliff is reranking (found@50 67 vs hit@5 35) and staleness. Vector search has zero concept of "superseded info" only fix I found was letting explicit user corrections outrank the transcript.
Yes. See LongMemEval, LoCoMo. Tons of research here.
But precision/recall is relatively "solved". What nobody has gotten close to solving is maintenance and provenance - what goes into memory, what qualifies as truth, how stale memory gets invalidated/superseded.
We're now in the phase of re-discovering 30+ years of pain of knowledgebases.
I'm building an AI agent that plays Slay the Spire. It currently reaches A20 Act 3 consistently and occasionally defeats the A20 Heart. It use two search: spare graph search for deck building, and MCTS for combat.
Parallelly, I'm working on an attention-based memory retrieval system that achieved SOTA on LongMemEval, LoCoMo, and code retrieval benchmarks. https://github.com/AttemorySystem/attemory/
Maybe later I'll build a live-streaming AI agent that plays Slay the Spire while conversing with viewers and remembering everything in chat!
Simply feeding the Slay the Spire game state directly to an LLM is currently insufficient to achieve stable deck-building or consistent combat performance.
I categorize the game's actions into three types: combat, deck-building, and other interactions.
For combat, MCTS is used. Yes, it's similar to the approach used in Go AI.
For deck-building, a sparse graph search is used: the goal is to rapidly identify winning deck templates within the graph structure.
Only the remaining aspects are delegated to the LLM to make reasoned decisions.
I am working on the same thing right now. However, unlike storing conversations in an external retrieval system, I use a local LLM to store the conversation's KV cache and perform retrieval directly on that cache. The method involves running a prefill pass and, after obtaining the attention scores, filtering for the corpus segments that received attention.
This aligns with the "zero tokens" approach described in this paper. :)
I tested it on the LoCoMo used in this paper, and also LongMemEval, both achieved SOTA results.
I was doing something similar where I saved user input/model output in a multi-depth node style storage system (each depth having more precise details) with the focus on the model having accurate user fed information. I was mostly focused on retrieval of accurate / useful information based on user query (injecting the database node as additional, high confidence information)
Once this(Zero-mem) passes it's peer review, I may have to see if my system can handle something similar instead/in addition.
I'm quite excited to see growth in these different ways of eliminating token's.
Long winded aside, @langs, have you published your work on this?
Using attention for retrieval was inspired by a comment I saw in here long time ago: Prediction and retrieval are two sides of the same coin; to predict better, you must retrieve more accurately.
I'm still working on the improvement of algorithms, my tests shows the performance and accuracy will be improved a lot in the next release.
I am not an expert on LLMs, but what's preventing you from treating the context as 'virtual memory', and using the attention matrix to 'blank out' tokens which have very low weights and will not contribute much to the input? I imagine most tokens are like this, and you can skip computation on 95% of an 1M (or practically infinite) context.
Hah I built a similar thing, stored a few million tokens chunked and precomputed winth Qwen a3e (best ratio of kv-size to tokens after chunking).
Some custom kernels and I was able to find all the relevant paragraphs with full force of qwen reasoning within 0.3s, and with a summary round within 0.7s.
Downside - required 200GB ram/vram ;) A few GBs for model and most of it for caching kvs.
Pi’s minimalism reveals a simple truth: the LLM is the core of any agent harness. Consequently, much of current harness tuning will become redundant(or even a hindrance) with next-gen models.
I always thought that the frontier coding models were specifically trained to perform well in the harness from their providers. I imagine this will always be the case to some extent.
From what I understand pre-training is totally irrelevant to this and as far as post training goes there will be multiple steps, for claude and codex and the like that ship with a harness, the harness is definitely included in evaluation. However, they will definitely include evaluation from a variety or even none, and settle on something that works the "best" for a release.
Writing code is indeed the bottleneck for same resource constrained companies.
Rapid code development creates more opportunities for trial and error, providing companies with more information for decision making, that previously might have been addressed by meetings.
Of course, this might bring other problems, but it might not right to generally speaking that writing code is not a bottleneck.
I’m very suspicious of this objection, because when Claude first landed the same people now saying “code is not the bottleneck” were saying “the generated code doesn’t work.” Smacks of moving goalposts.
The only solid objection to “AI is going replace developers” is “AI is an accelerant.” It helps developers move faster. I haven’t seen anywhere it has fully replaced developers.
Whether this leads to a large number of job losses depends on whether you think we can increase software output by the same factor as the acceleration and still be profitable. I think we can, latent software demand is extremely high. I also think we’re nearing the limit of capability with current models.
Situation could change if more advanced models emerge, but some of the more foreseeable advances probably have compute requirements beyond today’s hardware.
"mandatory safety testing" is an impractical ideal, it's not workable in real world.
Like any technology, LLMs are dual-use tools capable of both beneficial and malicious applications—a fundamental reality that human intent cannot change.
That’s a good analogy, but I think the reason we currently need the "nurse" role is the need to interact with the physical world. Most software products don't require this step, so the demand for “nurses” or "doctors" will likely decrease.
If robotics technology continues to advance, the number of nurses in real world will also decrease in the foreseeable future.
Nurses will likely be one of the last professions to be significantly impacted by robotics. Drivers, factory, warehouse, agriculture workers, cashiers etc will be automated away way before them.
It's just hard to design robots that can handle patients that may be simultaneously fragile, mentally handicapped and aggressive in a way that doesn't hurt them and respects their rights. It can take multiple human nurses to do a seemingly simple thing like changing a diaper.
The demand for nurses and doctors is only going to go up. Not just int their current roles, but everything adjacent to them as well. Everything involved with running trials of health care products. Humans aren't going away, and they need to be taken care of.
reply