Forecasting Crowd Work Quality via Multi-dimensional Features of Workers

Abstract : Modeling changes in individual crowd workerperformance over time offers new ways to improvethe quality of crowd labels, such as bydynamically routing label annotation tasks toworkers more likely to produce reliable labels.Whereas prior crowd annotator models have typicallyadopted a single generative approach, weformulate a discriminative, flexible feature-basedmodel. This allows us to combine multiple generativemodels and integrate additional behavioralevidence, enabling better adaptation to temporalvariance in worker accuracy. Experimentswith a public crowdsourcing data show that ourmodel improves prediction accuracy by 26-36%across workers, enabling 29-47% improved qualityof crowd labels to be collected at 17-45%lower cost. Furthermore, we confirm that ourproposed model shows significantly accurate predictionthan baselines under limited supervision.