Feed
benchmarkAI Evaluation3d ago

Analysis of LoCoMo benchmark sensitivity to prompt and judge selection in coding agents

Facts

Researchers analyzed how variations in answer prompts and LLM judges significantly impact LoCoMo benchmark scores for long-term conversational memory. The study tested the Becos team memory system, demonstrating that context-optimized memory performs competitively or better than full-conversation baselines depending on the evaluation configuration.

  • ▸Changing the evaluation judge can move LoCoMo benchmark scores by up to 7.2 points.
  • ▸Changing the answer prompt can shift scores by up to 9.3 points.
  • ▸The Becos team memory system uses 90% less context than full-conversation baselines.
  • ▸Becos outperformed the full-conversation baseline with gpt-4o-mini but showed parity with gpt-4.1-mini.
  • ▸The researchers released the full dataset, including answers, verdicts, and scoring scripts on arXiv.
Occurred
2026-11-14 06:41 UTC
First reported
2026-11-14 06:41 UTC
Last updated
2026-11-17 03:11 UTC
Event type
benchmark
Region
Global

Entities

Categories

Source timeline · 1

  • arXivFirst-party2026-11-14 06:41
Community vote

Sign in to vote on importance and sentiment.

Platform Signals

Impact score
52
AI sentiment
Neutral
Event significance30%0
Entity importance20%0
Source credibility20%1
Novelty15%1
Corroboration15%1
Confidence1
Novelty1
Corroboration1
Community
ImportanceNo votes
Strategic sentiment
Positive 0%Neutral 0%Negative 0%

Platform signals are algorithmic and community-derived indicators. They are not investment recommendations and do not predict valuation or performance.