CrowdAdvisor: A Framework for Freelancer Assessment in Online Marketplace

TLDR

Hiring in online labor marketplaces is challenging due to the large number of freelancers and the reliance on often skewed ratings, unlike traditional hiring which involves full‑time or contract employees. The authors propose a multidimensional assessment framework to evaluate freelancers across multiple dimensions. The framework incorporates both current freelancer information and past job performance, and is evaluated using data from a popular online marketplace. Analysis of 7,254 jobs and 96,271 applicants demonstrates that the proposed framework outperforms the baseline algorithm, revealing that ratings alone are insufficient.

Abstract

Hiring is one of the important challenges in the context of online labor marketplace. Unlike traditional hiring, where workers are hired either as a full time employee or as a contractor, hiring from online marketplaces are done for individual jobs of short duration. As these marketplaces are open for anyone, hiring becomes challenging due to the large number of freelancers applying for a posted job. Quite often, clients use ratings of the freelancers while hiring. However, we have observed that ratings are skewed towards higher values and do not provide valuable insights about freelancers' abilities to do a quality work. Therefore, we propose a multidimensional assessment framework which evaluates freelancers on several dimensions. The proposed framework, not only uses the current information about the freelancer, but also utilizes the past jobs he has performed. The framework is evaluated on the data collected from a popular online marketplace. Our analysis, performed on 7254 jobs and 96,271 applicants, shows that the assessment made by the proposed framework outperforms the baseline algorithm.