Exploring Crowdsourced Worker Evaluation Methods in Open-Ended Tasks

 Paper Title: 

Exploring Crowdsourced Worker Evaluation Methods in Open-Ended Tasks


Authors: 

Ryuya Itano, Honoka Tanitsu, Motoki Bamba, Ryota Noseyama, Akihito Kohiga and TakahiroKoita, Doshisha University, Japan


Abstract:

Crowdsourcing assumes a transient relationship between task requesters and workers, which makes it hard for workers to improve their skills. In addition, crowdsourcingtasksare shifting from simple to more complex and open-ended, highlighting the importance of training workers to handle such tasks. Although various methods have been proposed to train workers, a method to evaluate their skill levelsin open-ended tasks has not yet been established. Direct evaluation by requesters is desirable, but scaling up tasks is difficult due to the requester’s heavy workload.This study aims to explore methods for evaluating workers without increasing requesters’ workload, comparing and verifying apeer-basedmethod and anLLM-based automated method.The experiment investigated the alignment between evaluations fromthe twomethods andthose from requesters, thereby clarifying the characteristics of each method.The experimental resultsdemonstrated the applicability of LLMs to evaluating workers in open-ended tasks, revealing both their strengths in consistency and limitations in capturing subtle human judgments.


Keywords

Crowdsourcing, Worker Training,Worker Evaluation, Amazon Mechanical Turk, LLM-as-a-Judge


Volume URL: https://airccse.org/journal/ijite/vol15.html


Abstract URL: https://aircconline.com/abstract/ijite/v15n1/15126ijite03.html


Pdf URL: https://aircconline.com/ijite/V15N1/15126ijite03.pdf


#Crowdsourcing #WorkerTraining #WorkerEvaluation #AmazonMechanicalTurk #LLMasaJudge #callforpapers #researchpapers #cfp #researchers #phdstudent #education #learning #online #researchScholar #journalpaper #submission #journalsubmission



Comments

Post a Comment

Popular posts from this blog

Using Socrative to Enhance In-Class Student Engagement and Collaboration

Image
Sam M Dakka Department of Engineering & Math, Sheffield Hallam University, Howard Street, Sheffield S1 1WB,United Kingdom  ABSTRACT Learning and teaching experiment was designed to incorporate SRS-Student Response System to measure and assess student engagement in higher education for level 5 engineering students. The SRS system was based on getting an immediate student feedback to short quizzes lasting 10 to 15 minutes using Socrative software. The structure of the questions was a blend of true/false, multiple choice and short answer questions. The experiment was conducted through semester 2 of yearlong engineering module. The outcome of the experiment was analyzed quantitatively based on student performance and qualitatively through student questionnaire. The results indicate that using student paced assessments method using Socrative enhanced student’s performance. The results showed that 53% of the students improved their performance while 23% neither improved...
Read more

Using Socrative and Smartphones for the support of collaborative learning

Image
International Journal on Integrating Technology in Education  (IJITE) ISSN : 2320 - 1886(Online; 2320 - 3935(Print)  http://airccse.org/journal/ijite/home.html Abstract The integration of new technologies in the classrooms opens new possibilities for the teaching and learning process. Technologies such as student response system (e.g. Clicker) are getting popularity among teachers due to its effects on student learning performance. In this study, our primary objective is to investigate the effect of Socrative with combination of smartphones on student learning performance. We also observed the benefits of interactivity between the teacher and the students and among classmates, which positively influences collaborative learning and engagement of students in the class. We test these relationships experimentally in a community college class environment using data from a survey answered by students in information technology associate degree. The results of our st...
Read more

Technology-Driven Education: Insights from Collegiate Esports

Image
 Paper Title:  Technology-Driven Education: Insights from Collegiate Esports Authors:  Rochell McWhorter, Julie Delello and Rob E. Carpenter, University of Texas at Tyler, USA Abstract: This case study investigated how collegiate eSports players navigate the physical, mental, social, and academic challenges associated with competitive gaming, with a focus on its implications for adult education. Through in-depth interviews, the study explored players' diverse gaming routines, perceptions of excessive gaming, and strategies for balancing competing demands. While eSports participation fosters cognitive skills, social bonding, and career opportunities, it also presents risks such as physical inactivity, poor dietary habits, stress, and exposure to toxic gaming behaviors. A key finding is that the eleven students independently developed strategies to address these challenges, emphasizing the importance of integrating comprehensive support systems—such as physical activity ini...
Read more