CQM: coverage-constrained quality maximization in crowdsourcing test

Mobile app testing is challenging since each test needs to be executed in a variety of operating contexts including heterogeneous devices, various wireless networks, and different locations. Crowdsourcing enables a mobile app testing to be distributed as a crowdsourced task to leverage the crowd in a community. However, a high test quality and expected test context coverage are difficult to achieve in crowdsourcing test. Upon distributing a test task, mobile app providers can neither know who will participate and submit a high-qualified test report nor predict whether all expected test contexts can be covered during the test. To address this problem, we put forward a novel research problem called Coverage-constrained Quality Maximization (CQM) for crowdsourcing test. Given a mobile app test task, our objective is to discover and recommend a set of potential workers from available crowd workers such that they can accomplish the task achieving expected test context coverage and the possible highest test quality. We prove that the CQM problem is NP-Complete and then introduce two efficient greedy algorithms. Based on a real dataset of the largest Chinese crowdsourcing test platform, our evaluation shows that the proposed algorithms are effective and efficient, and can be potentially used as online services in practice.