What Evidence We Would Miss If We Do Not Use Grey Literature

Context: Over the last years, Grey Literature (GL) is gaining increasing attention in Secondary Studies in Software Engineering (SE). Notably, Multivocal Literature Review (MLR) studies, that search for evidence in both Traditional Literature (TL) and GL, is particularly benefiting from this raise of GL content. Despite the growing interest in MLR studies, the literature assessing how GL has contributed to MLR studies is still scarce. Objective: This research aims to assess how the use of GL contributed to MLR studies. By contributing, we mean, understanding to what extent GL is providing evidence that is indeed used by an MLR to answer its research question. Method: We conducted a tertiary study to identify MLR studies published between 2017 and 2019, selecting nine MLRs studies. Using qualitative and quantitative analysis, we identified the GL used and assessed to what extent these MLRs are contributing to MLR studies. Results: Our analysis identified that 1) GL provided evidence not found in TL, 2) Most of the GL sources were used to provide recommendations to solve or help in some problem, explain a topic, classify the findings, and provide solution proposals, and 3) 19 GL types, mainly produced by SE practitioners, (including blog posts, slides presentations, or project descriptions) were used among the studies. These findings show the importance of GL to MLR studies and increase state of the art by pilling additional evidence on this topic. Conclusions: We evidence how GL was used to contribute to MLR studies, showing that if these studies have not used the GL, several findings derived from the practice of SE would have been omitted. We described the challenges involved when conducting this investigation, along with potential ways to deal with them, which may help future SE researchers.