An Empirical Study on Numerical Bugs in Deep Learning Programs

Gan Wang, Zan Wang, Junjie Chen, Xiang Chen, Ming Yan

Abstract

The task of a deep learning (DL) program is to train a model with high precision and apply it to different scenarios. A DL program often involves massive numerical calculations. Therefore, the robustness and stability of the numerical calculations are dominant in the quality of DL programs. Indeed, numerical bugs are common in DL programs, producing NaN (Not-a-Number) and INF (Infinite). A numerical bug may render the DL models inaccurate, causing the DL applications unusable. In this work, we conduct the first empirical study on numerical bugs in DL programs by analyzing the programs implemented on the top of two popular DL libraries (i.e., TensorFlow and PyTorch). Specifically, We collect a dataset of 400 numerical bugs in DL programs. Then, we classify these numerical bugs into nine categories based on their root causes and summarize two findings. Finally, we provide the implications of our study on detecting numerical bugs in DL programs.

BibTeX

@inproceedings{Wang-al:ASE22,
  author    = {Gan Wang and
               Zan Wang and
               Junjie Chen and
               Xiang Chen and
               Ming Yan},
  title     = {An Empirical Study on Numerical Bugs in Deep Learning Programs},
  booktitle = {ASE},
  pages     = {173:1--173:5},
  publisher = {{ACM}},
  year      = {2022},
}

An Empirical Study on Numerical Bugs in Deep Learning Programs

Abstract

Related papers