Overview

Mr. Yamamoto from our lab gave a research presentation at The 23rd IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology!


Presenter:Ryotaro Yamamoto
Title: 「Task Assignment in Crowdsourcing Using Deep Knowledge Tracing
Abstract:Crowdsourcing is used to create language resources for low-resource languages. In such crowdsourcing endeavors, the variability in workers' abilities makes quality control a critical issue. Specifically, mismatches between workers and creation tasks can lead to the creation of incorrect language resources, resulting in frequent rework of creation and evaluation tasks and increased costs. Therefore, this study proposes a task assignment method that considers task difficulty, using a Deep Knowledge Tracing model trained on workers' work history to assign tasks that workers are likely to complete successfully. Specifically, Word2Vec is used to automatically assign knowledge tags to tasks based on word semantics, and a Deep Knowledge Tracing model is constructed. The constructed Deep Knowledge Tracing model calculates the success probabilities for tasks for each knowledge tag and worker. These probabilities are used as costs in an extended many-to many Hungarian method for optimal assignment. To verify the effectiveness of the proposed method, an experiment was conducted in which 500 bilingual evaluation tasks between Indonesian and Minangkabau were assigned to 20 workers. The experimental results showed that the proposed method improved the accuracy rates by 2.4% - 5.6% compared to random assignment and assignment based on past accuracy rates.