Evaluating Large Language Models in Class-Level Code Generation
Abstract
Recently, many large language models (LLMs) have been proposed, showing advanced proficiency in code generation. Meanwhile, many efforts have been dedicated to evaluating LLMs on code generation benchmarks such as HumanEval. Although being very helpful for comparing different LLMs, existing evaluation focuses on a simple code generation scenario (i.e., function-level or statement-level code generation), which mainly asks LLMs to generate one single code unit (e.g., a function or a statement) for the given natural language description. Such evaluation focuses on generating independent and often small-scale code units, thus leaving it unclear how LLMs perform in real-world software development scenarios.
BibTeX
@inproceedings{Du-al:ICSE24,
author = {Xueying Du and
Mingwei Liu and
Kaixin Wang and
Hanlin Wang and
Junwei Liu and
Yixuan Chen and
Jiayi Feng and
Chaofeng Sha and
Xin Peng and
Yiling Lou},
title = {Evaluating Large Language Models in {Class-Level} Code Generation},
booktitle = {ICSE},
pages = {81:1--81:13},
publisher = {{ACM}},
year = {2024},
}