benchmarkAI Agents7d ago
Introduction of ExpertCrowd-SWE Benchmark for Evaluating Autonomous Coding Agents
Facts
ExpertCrowd-SWE is a new benchmark featuring 100 expert-authored software engineering tasks designed to test autonomous coding agents in large, unfamiliar codebases. The benchmark evaluates performance across multiple frontier models, revealing significant performance gaps and persistent challenges in solving complex repository-wide engineering tasks.
- ▸ExpertCrowd-SWE consists of 100 multi-file software engineering tasks spanning 87 open-source projects and 11 programming languages.
- ▸Claude Opus 5.5 and GPT-6 Astra were identified as top performers among seven models tested.
- ▸Performance scores ranged from 32% to 67.3%, indicating that even leading frontier models struggle with repository-level engineering requirements.
- ▸The benchmark includes 24 open-source tasks with the remaining 76 held out to prevent model contamination.
- ▸Evaluation criteria include weighted, partially executable requirements to simulate real-world code maintainer standards.
- Occurred
- 2026-11-09 21:20 UTC
- First reported
- 2026-11-09 21:20 UTC
- Last updated
- 2026-11-11 03:05 UTC
- Event type
- benchmark
- Region
- Global
Entities
DeepSeek-V4.1-Flash modelGrok 4.6 modelQwen 3.8 Max modelClaude Fable 5.1 modelClaude Opus 5.5 modelGPT-6 Astra model
Categories
Source timeline · 1
- arXivFirst-party2026-11-09 21:20
Community vote
Sign in to vote on importance and sentiment.
Platform Signals
Impact score
74
AI sentiment
NeutralEvent significance30%1
Entity importance20%1
Source credibility20%1
Novelty15%1
Corroboration15%1
Confidence1
Novelty1
Corroboration1
Community
ImportanceNo votes
Strategic sentiment
Positive 0%Neutral 0%Negative 0%
Platform signals are algorithmic and community-derived indicators. They are not investment recommendations and do not predict valuation or performance.
Related events
- 18hIntroduction of SciResearch-Eval Benchmark for AI Agent Scientific Reasoning
- 21hRelease of CAE-Bench to Evaluate AI Agents in Computer-Aided Engineering Solver Development
- 4dResearchers Introduce LuthorSQL: A System 1 Text-to-SQL Model to Reduce LLM Agent Costs
- 7dSciMARS Framework Improves Multi-Agent Scientific Discovery
- 8dResearchers develop zero-search reinforcement learning agent for Slay the Spire 2
- 9dGradReason Research Demonstrates LLM Agents Optimizing Neural Network Training Schedules