SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation
Abstract
High-quality labeled datasets are crucial for training and evaluating foundation models in software engineering, but creating them is often prohibitively expensive and labor-intensive. We introduce SPICE, a scalable, automated pipeline for labeling SWE-bench-style datasets with annotations for issue clarity, test coverage, and effort estimation. SPICE combines context-aware code navigation, rationale-driven prompting, and multi-pass consensus to produce labels that closely approximate expert annotations. SPICE’s design was informed by our own experience and frustration in labeling more than 800 instances from SWE-Gym. SPICE achieves strong agreement with human-labeled SWE-bench Verified data while reducing the cost of labeling 1,000 instances from around $100,000 (manual annotation) to only $5.10. These results demonstrate SPICE’s potential to enable cost-effective, large-scale dataset creation for SE-focused FMs. To support the community, we release both SPICE tool and SPICE Bench, a new dataset of 6,802 SPICE-labeled instances curated from 291 open-source projects in SWE-Gym (over 13x larger than SWE-bench Verified).
BibTeX
@inproceedings{Oliva-al:ASE25,
author = {Gustavo Ansaldi Oliva and
Gopi Krishnan Rajbahadur and
Aaditya Bhatia and
Haoxiang Zhang and
Yihao Chen and
Zhilong Chen and
Arthur Leung and
Dayi Lin and
Boyuan Chen and
Ahmed E. Hassan},
title = {{SPICE:} An Automated {SWE-Bench} Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation},
booktitle = {ASE},
pages = {2325--2337},
publisher = {{IEEE}},
year = {2025},
}