Prioritizing Test Inputs for DNNs Using Training Dynamics

Abstract

Deep Neural Network (DNN) testing is one of the most widely-used techniques to guarantee the quality of DNNs. However, DNN testing typically requires the ground truth of test inputs, which is time-consuming and labor-intensive to obtain. To relieve the labeling-cost problem of DNN testing, we propose TDPR, a test input prioritization technique for DNNs based on training dynamics. The key insight of TDPR is that bug-revealing samples exhibit different learning trajectories compared to normal ones. Based on this, TDPR constructs a learning trajectory for each test input, which characterizes the evolving learning behavior of DNNs. Then, TDPR extracts features from these learning trajectories and applies learning-to-rank techniques to build a ranking model, which can intelligently utilize the generated features to prioritize test inputs. To evaluate TDPR, we conduct extensive experiments on 8 diverse subjects, considering various domains of test inputs, different DNN architectures, and diverse types of test inputs. The evaluation results demonstrate that TDPR outperforms 7 baseline approaches in both prioritizing test inputs and guiding the retraining of DNNs.