Benchmarks
Numbers, with the methodology.
hippo's retrieval, measured on LongMemEval (opens in new tab) (ICLR 2025), the public 500-question memory benchmark, and on LoCoMo (opens in new tab), 10 long multi-session conversations. The harness, the data hash, and the per-question results for both are in the repo so you can rerun them.
Standard task: per-question haystack
Each question ships its own haystack of about 48 conversation sessions, and the job is to retrieve the session that holds the answer. This is the standard LongMemEval-S setup, the same one published systems report. Recall at 5, on the _s split:
| Embedder | R@1 | R@5 | R@10 |
|---|---|---|---|
| MiniLM-L6 (local, optional install) | 88.4 | 98.0 | 99.4 |
| voyage-3-large (opt-in, June 2026) | 94.6 | 99.8 | 99.8 |
For reference, gbrain reports 97.6% R@5 on this split with a paid frontier embedder. hippo reaches 98.0% with a free local embedder, the best of five settings in its benchmark scripts; at 500 questions that is a tie. The June 2026 build scored 98.6%, and the re-measure (opens in new tab) shows both runs. On the standard task, retrieval recall is effectively saturated, so the embedder is a swappable part, not the differentiator.
Large store: one unified memory
Point retrieval at a single store of all 19,195 sessions, with no pre-scoped haystack, closer to how an agent's memory actually accumulates. Recall stops being free (measured June 2026):
A stronger embedder helps here (47 to 56) but neither is usable on its own: the answer drowns among thousands of distractors. This is where we expect the memory lifecycle to matter, and measuring it is the next benchmark on the roadmap. It is not shown yet: on our tests so far, decay made no measurable difference and sleep cost recall.
Reranking: what a paid model bought, and what it did not
hippo recall "<query>" --reranker jev sends the top 40 candidates to the hosted TypeSafe Jev (opens in new tab) API and reorders them. It is off by default and the default recall path makes no network call. Ranking on a private 300-query developer store, so unlike the two benchmarks above this one is not redistributable and you cannot rerun it. All figures aggregated over 2,000-draw paired bootstrap runs, with 98.75% intervals:
| Metric | Base | Cross-encoder | Jev | Jev minus cross-encoder |
|---|---|---|---|---|
| R@1 | 0.260 | 0.413 | 0.617 | +0.203 [0.133, 0.273] |
| R@5 | 0.460 | 0.613 | 0.740 | +0.127 [0.080, 0.177] |
| MRR | 0.358 | 0.509 | 0.672 | +0.163 [0.113, 0.217] |
| recall at the token budget | 0.693 | 0.740 | 0.747 | +0.007 [0.000, 0.020], tied |
The R@1 margin held in 20 of 20 bootstrap seeds, and a permutation null with arm labels shuffled inside each query reached it in 0 of 200 runs. The R@1 margin replicated on LongMemEval at n=500, +0.0700 [0.0200, 0.1200]; R@5 tied there. Cost is about 0.0004 USD a recall, p50 295 ms over 300 calls.
What it did not do
Better ranking did not produce better answers. Three graded-answer tests on LongMemEval all tied against the free local cross-encoder, and those three share one 150-question set, so they are not three independent tries. What Jev buys today is a shorter context: 2 Jev-ranked memories answer as well as 5 ranked by the cross-encoder, roughly 900 fewer tokens a query. Turning it on also sends your query and candidate memory text to a third party.
Method, every interval, the cost table and the full limits list: the Jev reranker eval (opens in new tab).
Tested, and local by default
Reproduce it
The data is longmemeval_s_cleaned.json (SHA-256 d6f21ea9...), 500 questions over 19,195 sessions. Retrieval is turn-level dense plus BM25, fused with reciprocal rank fusion and max-pooled to session, by the scripts in benchmarks/longmemeval rather than hippo recall. Embeddings are L2-normalized; the local embedder is MiniLM (an optional install), with an opt-in pluggable provider for frontier embedders.
Every measurement hippo has published, pre-registrations kept next to their results, is indexed in docs/evals (opens in new tab). That index includes the runs that failed and the one claim we retracted.
LoCoMo: conversational evidence recall
LoCoMo (opens in new tab) is 10 long multi-session conversations, 5,882 turns and 1,986 questions. hippo scores it deterministically: did the gold evidence turn land in the top 5 recalled memories? No LLM judge. Measured once on v1.25.0 (2026-07-05) with the zero-dependency default embedder, MiniLM-L6:
| Category | n | Evidence R@5 |
|---|---|---|
| single-hop | 282 | 0.239 |
| multi-hop | 321 | 0.491 |
| temporal-reasoning | 92 | 0.169 |
| open-domain | 841 | 0.450 |
| adversarial | 446 | 0.226 |
| overall | 1,982 | 0.363 |
A point estimate, not an exact value: n=1, and the run predates the v1.26.0 determinism fix and a harness fix that lost tags on 0.9% of stored rows. Not comparable to the LLM-as-judge numbers Mem0 and Letta publish. Informational only, gates no feature. Protocol, caveats and regeneration commands: LOCOMO_INVESTIGATION.md (opens in new tab).