2015 · 21 citations · 18 references
EducationEducational AssessmentHigher EducationAssessment For LearningOnline Higher EducationPerformance AssessmentAssignment ProblemsMassive Open Online CoursesOnline AssessmentPeer AssessmentRanking ProblemsStudent AssessmentAggregationMeasurement Of Relative PerformanceOnline RankingOrdinal AnalysisPartial RankingPeer GradingRanking Aggregation AlgorithmsRelative Performance Analysis
We investigate the potential of using ordinal peer grading for the evaluation of students in massive online open courses (MOOCs). According to such grading schemes, each student receives a few assignments (by other students) which she has to rank. Then, a global ranking (possibly translated into numerical scores) is produced by combining the individual ones. This is a novel application area for social choice concepts and methods where the important problem to be solved is as follows: how should the assignments be distributed so that the collected individual rankings can be easily merged into a global one that is as close as possible to the ranking that represents the relative performance of the students in the assignment? Our main theoretical result suggests that using very simple ways to distribute the assignments so that each student has to rank only k of them, a Borda-like aggregation method can recover a 1-O(1/k) fraction of the true ranking when each student correctly ranks the assignments she receives. Experimental results strengthen our analysis further and also demonstrate that the same method is extremely robust even when students have imperfect capabilities as graders. Our results provide strong evidence that ordinal peer grading can be a highly effective and scalable solution for evaluation in MOOCs.
18
The PageRank Citation Ranking : Bringing Order to the Web
Lawrence M. Page, Sergey Brin, Rajeev Motwani et al. · 1999 · 12.7K citations
Learning to rank for information retrieval
Tie‐Yan Liu · 2010 · 2K citations
Rank aggregation methods for the Web
Cynthia Dwork, Ravi Kumar, Moni Naor et al. · 2001 · 1.8K citations
Weighted sums of certain dependent random variables
Kazuoki Azuma · Tohoku Mathematical Journal · 1967 · 1.1K citations · Full text
C. L. Mallows · Biometrika · 1957 · 782 citations