Exploring Crowdsourced Worker Evaluation Methods in Open-Ended Tasks
Paper Title: Exploring Crowdsourced Worker Evaluation Methods in Open-Ended Tasks Authors: Ryuya Itano, Honoka Tanitsu, Motoki Bamba, Ryota Noseyama, Akihito Kohiga and TakahiroKoita, Doshisha University, Japan Abstract: Crowdsourcing assumes a transient relationship between task requesters and workers, which makes it hard for workers to improve their skills. In addition, crowdsourcingtasksare shifting from simple to more complex and open-ended, highlighting the importance of training workers to handle such tasks. Although various methods have been proposed to train workers, a method to evaluate their skill levelsin open-ended tasks has not yet been established. Direct evaluation by requesters is desirable, but scaling up tasks is difficult due to the requester’s heavy workload.This study aims to explore methods for evaluating workers without increasing requesters’ workload, comparing and verifying apeer-basedmethod and anLLM-based automated method.The experiment inves...
