A landscape of the adoption of empirical evaluations in the brazilian symposium on human factors in computing systems

Context: Empirical evaluations have been widely applied as a formalism to validate and ensure the credibility of research works. In the context of Human-Computer Interaction (HCI) field, the usage of empirical methods plays a vital role due to its focus on evaluating the end-user and usability of software solutions. Even though the adoption of experimental evaluation techniques has gained popularity in recent years, its application is still questioned both qualitatively and quantitatively. Goal: To analyze how empirical research has evolved in the Human-Computer Interaction Brazilian community. Method: We performed a quasi-experiment, using published papers over the 17 editions in Brazilian Symposium on Human Factors in Computing Systems. Our experiment was divided into two phases, in order to evaluate the quality assessment and classify the type of realized empirical study, respectively. Results: From the sample of 231 studies, 113 reached satisfactory quality in the first phase. It was found that empirical studies were becoming more present in lastly years of the conference. Moreover, in the second phase, it was found with 95% confidence that the quality of the empirical studies increased, by comparing two different divisions of the 20-year conference period. Findings: The assessment questionnaire used in our research and the learned lessons about each kind of empirical study can support the conduction of new empirical evaluations in the HCI community, improving their quality.