Feed
benchmarkAI Agents7d ago

Introduction of ExpertCrowd-SWE Benchmark for Evaluating Autonomous Coding Agents

Facts

ExpertCrowd-SWE is a new benchmark featuring 100 expert-authored software engineering tasks designed to test autonomous coding agents in large, unfamiliar codebases. The benchmark evaluates performance across multiple frontier models, revealing significant performance gaps and persistent challenges in solving complex repository-wide engineering tasks.

  • ▸ExpertCrowd-SWE consists of 100 multi-file software engineering tasks spanning 87 open-source projects and 11 programming languages.
  • ▸Claude Opus 5.5 and GPT-6 Astra were identified as top performers among seven models tested.
  • ▸Performance scores ranged from 32% to 67.3%, indicating that even leading frontier models struggle with repository-level engineering requirements.
  • ▸The benchmark includes 24 open-source tasks with the remaining 76 held out to prevent model contamination.
  • ▸Evaluation criteria include weighted, partially executable requirements to simulate real-world code maintainer standards.
Occurred
2026-11-09 21:20 UTC
First reported
2026-11-09 21:20 UTC
Last updated
2026-11-11 03:05 UTC
Event type
benchmark
Region
Global

Entities

Categories

Source timeline · 1

  • arXivFirst-party2026-11-09 21:20
Community vote

Sign in to vote on importance and sentiment.

Platform Signals

Impact score
74
AI sentiment
Neutral
Event significance30%1
Entity importance20%1
Source credibility20%1
Novelty15%1
Corroboration15%1
Confidence1
Novelty1
Corroboration1
Community
ImportanceNo votes
Strategic sentiment
Positive 0%Neutral 0%Negative 0%

Platform signals are algorithmic and community-derived indicators. They are not investment recommendations and do not predict valuation or performance.