Instruction visibility in SPEC CPU2017

Abstract When analyzing the impact of compiler optimizations upon the execution of benchmarks, it is important to know which portion of the executed code can be influenced by the compiler, and which portion cannot. From the compiler perspective, the program text is visible. Pre-compiled library code is invisible, because the compiler can only improve the program text. Thus, an important question, from the point of view of compiler developers, is what proportion of the instructions executed by these benchmarks are visible when they are executed with reference inputs. This paper answers this question for the 43 programs of SPEC CPU2017 benchmark suite, when compiled with gcc and executed on an x86 processor. This information helps both computer architects and compiler designers to assess the potential gain of application-level code transformations. On average, 92.2% of the instructions executed by SPEC CPU2017 are visible. In contrast, 10.8% of the instructions executed in a typical run of GNU CoreUtils are visible. Different optimization levels tend to preserve these ratios. This stark contrast indicates that developers should exercise caution when projecting performance results from compiler-based code transformations measured in SPEC CPU2017 onto other applications. Amdahl’s law is a reminder that the performance improvement to a system is limited by the portion of execution time that cannot be affected by the change.