Benchmark Finds Hybrid RAG Outperforms Dedicated AI Memory Engines Under Noise

Summary

AAI Labs benchmarked Hindsight, Mem0, Honcho, OpenViking and Graphiti against hybrid RAG, full-context and no-memory baselines on 1,000 questions drawn from four conversation and email corpora. Unrelated history increased from 0% to 50% to 90%, while the model, embedder, answer prompt and judge remained constant; most runs used an 8K-token context budget. Hybrid RAG, which combines keyword and vector search over the original transcripts, achieved 96.8% accuracy at a total cost of $1.70 and lost only 1.7 percentage points as noise rose from 0% to 90%. Hindsight was the strongest dedicated engine at 89.9% and $20.39, and tied RAG statistically at a 2K context budget, but its median retrieval latency rose from 2.8 to 15.6 seconds as the store grew. OpenViking scored 82.7% at $20.73 and kept returned context below 1,000 tokens on average, but its compact quotas limited recall. Mem0 reached 81.5% at $7.03; its top-20 limit crowded out facts and extraction sometimes lost speaker names. Honcho scored 69.7% after its build completed, improving to 77.5% with tuned retrieval, while Graphiti scored 53.6% at 50% noise because ten graph facts often lacked enough evidence for multi-part answers. A full-context baseline fell from 98.0% to 52.4% when older evidence no longer fit. Extraction systems were 6–18 points worse on facts mentioned once, and memory construction consumed 90–98% of total cost for four extraction engines at 90% noise. The authors recommend preserving original history, establishing hybrid RAG first, and testing single mentions, speaker attribution and older conversations before adopting a dedicated memory layer. The conclusions apply to conversational question answering, not every product feature, and small score differences should not be treated as a definitive ranking.