Crowdsourcing Gold-HIT Creation at Scale: Challenges and Adaptive Exploration Approaches (Position Paper) - Abraham, Ittai; Alonso, Omar; Kandylas, Vasilis; Patel, Rajesh; Shelford, Steven; Slivkins, Aleksandrs; Wu, Hai (Microsoft Research, 2013)
Gold HITs — Human Intelligence Tasks with known answers — are commonly used to measure worker performance and data quality in industrial applications of crowdsourcing. We suggest adaptive exploration as a promising approach for automated, scalable Gold HIT creation. We substantiate this with initial experiments in a stylized model.