The gist: Memory systems for agents fail on 86 percent of queries where the correct fact lacks direct linguistic match, despite being able to retrieve the fact when it is directly visible.
Researchers have released a benchmark comprising 125 tasks that exposes a systematic weakness in agent memory systems: they cannot retrieve stored facts when the search query contains no literal overlaps. A classic example is a nut allergy that should be relevant in a macaron query but is not automatically connected.
The InMind Benchmark specifically tests the implicit association blind spot problem. Rather than merely measuring whether memory systems can retrieve facts, it examines whether they can retrieve facts when the query does not contain the same keywords as the stored information. The benchmark encompasses 125 tasks across ten life domains, with 113 tasks supported by citable public sources.
The results reveal a dramatic gap: when the relevant memory is directly available in context, backbone models achieve 84.0 percent correct answers. However, when these same memories must be located via a retrieval mechanism, six tested memory systems (vector-based, graph-based and agent-based approaches) drop to a maximum of 14.4 percent — despite being able to retrieve the same facts with 100 percent accuracy on direct query. This shows that the problem lies not in storage or factual knowledge, but in the interface between query and retrieval.
Further tests point to the core issue: even an eightfold increase in embedding dimensionality does not close the gap. A minimal diagnostic procedure that keeps memory visible before the query arrives could, however, resolve most of the performance gap. This suggests that routing — the decision about which facts remain continuously available — is the open development challenge.
For engineering domains such as chatbot and agent development, the benchmark is relevant because it shows that higher-dimensional embeddings or larger memory capacity alone are insufficient. Instead, systems must actively decide which information should be pre-activated before a query arrives — for instance through context prefetching or explicit relevance modeling.
Source: arxiv.org · Published 26 July 2026
Lumi AI News — AI-assisted curation according to Article 50 EU AI Act. Paraphrase and classification by Lumi News Pipeline v1.7.3.