A benchmark published by Tony Dzi finds his AI memory system refused to answer more questions than Mem0 when information was missing, but also refused more often when answers were available. The results suggest that making an AI agent cautious is not a simple win: the same rule can prevent made-up answers and block useful ones.
Watch Desk analysis
What happened
Dzi tested sqlite-graph-memory against other systems using a synthetic vault of 202 notes and 91 questions. In his benchmark, his system correctly returned nothing for 10 of 22 questions with no answer in the vault, compared with three for Mem0. It also refused 18 of 69 answerable questions, against nine for Mem0.
Dzi says a single threshold in his system drives the trade-off. Switching it off, while keeping the same retrieved results, reduced correct refusals from 10 to three and false refusals from 18 to six. These are results from the configurations tested on this benchmark, not a general ranking of memory products.
Key findings
- The refusal rule helps, but has a price
The threshold caught seven additional unanswerable questions compared with its disabled setting, while also rejecting 12 additional answerable ones. - The test is deliberately small and synthetic
It uses 202 notes about invented people, projects and decisions, with 91 questions. Results may differ on real notes and other question mixes. - A simple entity check did well on this test
An outside contributor’s pipeline refused all 10 questions about entities absent from the vault, but had lower Recall@5 and struggled with paraphrased questions. - The author reports a separate limitation
Dzi says his system trails Mem0 on the LoCoMo evaluations he ran, which measure other memory tasks and exclude questions designed to test whether an answer is absent.
Why it matters
Memory systems for AI agents need to retrieve useful information and avoid inventing details when the record is silent. This benchmark shows why “refuse more” is not an adequate measure of reliability on its own: a refusal can be prudent, or simply a failure to find or express an answer.
The useful takeaway is to examine both sides of the ledger. A system that avoids confident errors but frequently withholds available information may still frustrate users, while a system that answers more often may be bluffing. The benchmark’s small, invented vault makes it a case study, not a verdict on how these tools behave in everyday use.
Our read
This is a worthwhile, candid benchmark because it reports the misses as well as the win, and even identifies a simple outside approach that performed better on one narrow task. But the comparison bundles retrieval, refusal rules and answer extraction, so it cannot tell us which product is best in general.
Treat the figures as a useful question to ask of any AI memory tool: how often does it stay quiet when the answer is absent, and how often does it do so when the answer is there? The source article provides the benchmark files and scripts for readers who want to inspect the test itself.
What to watch
- Whether the proposed entity-existence check improves results when tested on a fresh benchmark.
- Whether future tests use real-world notes and different proportions of answerable questions.
- Whether the project separates retrieval failures, refusal decisions and answer-extraction failures in later evaluations.
Discussion spark: When an AI memory system lacks an answer, should it err on the side of refusing, even if that means withholding more answers it could have found?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.