benchmarkAI Evaluation3d ago
Analysis of LoCoMo benchmark sensitivity to prompt and judge selection in coding agents
Facts
Researchers analyzed how variations in answer prompts and LLM judges significantly impact LoCoMo benchmark scores for long-term conversational memory. The study tested the Becos team memory system, demonstrating that context-optimized memory performs competitively or better than full-conversation baselines depending on the evaluation configuration.
- ▸Changing the evaluation judge can move LoCoMo benchmark scores by up to 7.2 points.
- ▸Changing the answer prompt can shift scores by up to 9.3 points.
- ▸The Becos team memory system uses 90% less context than full-conversation baselines.
- ▸Becos outperformed the full-conversation baseline with gpt-4o-mini but showed parity with gpt-4.1-mini.
- ▸The researchers released the full dataset, including answers, verdicts, and scoring scripts on arXiv.
- Occurred
- 2026-11-14 06:41 UTC
- First reported
- 2026-11-14 06:41 UTC
- Last updated
- 2026-11-17 03:11 UTC
- Event type
- benchmark
- Region
- Global
Entities
Categories
Source timeline · 1
- arXivFirst-party2026-11-14 06:41
Community vote
Sign in to vote on importance and sentiment.
Platform Signals
Impact score
52
AI sentiment
NeutralEvent significance30%0
Entity importance20%0
Source credibility20%1
Novelty15%1
Corroboration15%1
Confidence1
Novelty1
Corroboration1
Community
ImportanceNo votes
Strategic sentiment
Positive 0%Neutral 0%Negative 0%
Platform signals are algorithmic and community-derived indicators. They are not investment recommendations and do not predict valuation or performance.