The main contribution of this paper is in quantifying the benefits of autotools such as GNU Autoconf: showing how much they help the portability of large software packages and illustrating their limitations.
We also showed several useful metrics for measuring the complexity of code when it comes to its portability: the average distribution of CPP conditionals in the code, the number of new Autoconf tests that had to be written, and the portion of those tests that were static and therefore beyond the normal capabilities of Autoconf. Using these metrics we showed how packages with more lines of code may not be as difficult to port as packages that use a more diverse set of system features.
In analyzing the evolution of the Am-utils package, we showed how the package benefited from autotooling: its code size was reduced dramatically and the code base was made cleaner, thus allowing rapid progress on new feature implementation.
The GNU autotools we used also have limitations. First, maintainers must become intimately familiar with the autotools. Second, building autotooled packages takes longer. Third, even autotools cannot solve all portability problems; large-package maintainers usually have to write their own custom tests. Fourth, certain tests cannot be executed in a cross-compiled environment. Nevertheless, in the long run, once an initial autotooling effort has taken place, an autotooled package is easier to maintain on existing systems and port to new or diverging systems.
Although a count of code lines had been used for years as a metric of code complexity, we believe that metric oversimplifies the process of portable software development. In the future we would like to automate the process of quantifying the complexity of a package. We plan on building tools that will parse autotool files and related C code, separately evaluating conditionally-included code from unconditionally-included code. This will allow us to evaluate how much of a given autotooled package is highly portable code and how much code is densely populated with system-specific code. In addition, we would like to account for package-specific portability complexities (i.e., how new architectures affect Gcc and Binutils, new file systems affect Amd, and new windowing systems affect Emacs and Tk).