Lightweight programming experiments without programmers and programs: an example study on the effect of similarity and number of object identifiers on the readability of source code using natural texts

Abstract

It is often said that controlled experiments should check the effect of programming languages or styles on programming. But it is also often said that running controlled experiments is a very time consuming and error prone task – that's why a lot of researchers do not run such experiments. Both arguments are plausible, but there is potentially an alternative: lightweight experiments where the effort to run such experiments is low and which (still) fulfill the requirements of controlled experiments with two exceptions: First, these experiments do not use programmers as subjects, and second, the experiments do not contain programming tasks. Instead, such experiments try to find analogies from other domains where the topic to be studied is (still) close enough to the original target domain but where it is easier to find participants and experimental setups. This paper illustrates such a lightweight experiment by introducing a study on the effect of number of identifiers and similarity of identifiers on (code) readability – without using source code and without programmers as subjects. The result of the experiment is comparable to other experimental results which gives a first indicator that it is possible to run such lightweight experiments that approximate the results of full-blown experiments. This paper argues that such lightweight experiments could be useful in the process of experimentation – they cannot and should not supersede full-blown experiments, but they can help in early stages of experimentation.