The Other Human in The Loop – A Pilot Study to Find Selection Strategies for Active Learning

Gathering data becomes increasingly simple whereas the labeling of collected instances remains difficult. Active learning provides methods to reduce the labeling effort by intelligent selection of instances. In contrast to building mathematical models or developing heuristics to solve this task, we pursue another approach: We let humans select the instances which should be labeled. Participants are asked to learn to predict the sex of 18 abstract illustrations of bugs as either male or female. This article describes the design, goal and the execution of this study with 14 groups (71 participants). In this exploratory study we analyze humans’ balance between exploration and exploitation, the participants’ learning behavior, the collaboration within the group as well as the question when to stop querying. The comparison of human performance with baseline active learning algorithms provides promising results which indicate that machine active learning might benefit from incorporating human strategies. Additionally, we provide the complete data and extracted spreadsheets for download.