Screening the most highly cited papers in longitudinal bibliometric studies and systematic literature reviews of a research field or journal: Widespread used metrics vs a percentile citation-based approach

Abstract There is a literature gap regarding the period representativeness bias associated with sample selection in longitudinal bibliometric studies. The purpose of this paper is to analyse and compare, in terms of period representativeness, the common methods used for selecting a sample of the highly impactful papers in a field/ journal. Using 92 593 papers (Information Science & Library Science area, 1977–2016), we compared, in terms of the number of papers/year, samples of the 100 most impactful papers, obtained with different selection options. We repeated the analysis also for Top500, Top2000, and Top20000. This study shows that the frequently used metrics to compare the impact of papers and to select a sample of "most impactful papers" published in each year and each field may privilege specific periods while neglecting others. The main result of our study is that the percentile citation-based method reduces this "year of publication" representativeness bias. This paper draws attention to the importance of the sample selection, in bibliometric studies, and to the period representativeness bias associated with different choices to select the "most impactful papers".