Robust Cumulative Crowdsourcing Framework Using New Incentive Payment Function and Joint Aggregation Model

Abstract

In recent years, crowdsourcing has gained tremendous attention in the machine learning community due to the increasing demand for labeled data. However, the labels collected via crowdsourcing are usually unreliable and noisy. This issue is mainly caused by: 1) non-flexible data collection mechanisms; 2) non-incentive payment functions; and 3) inexpert crowd workers. We propose a new robust crowdsourcing framework as a comprehensive solution for all these challenging problems. Our unified framework consists of three novel components. First, we introduce a new flexible data collection mechanism based on the cumulative voting system, allowing crowd workers to express their confidence for each option in multi-choice questions. Second, we design a novel payment function regarding the settings of our data collection mechanism. The payment function is theoretically proved to be incentive-compatible, encouraging crowd workers to truthfully disclose their beliefs to get the maximum payment. Third, we propose efficient aggregation models, which are compatible with both single-option and multi-option crowd labels. We define a new aggregation model, called as Simplex Constrained Majority Voting ( extit{SCMV}), and enhance it via using probabilistic generative model. Furthermore, fast optimization algorithms are derived for the proposed aggregation models. Experimental results indicate higher quality for the crowd labels collected by our proposed mechanism without increasing the cost. Our aggregation models also outperform the state-of-the-art models on multiple crowdsourcing datasets in terms of accuracy and convergence speed.

Publication
IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
Date
Links