Concepedia

Multi-evaluator Agreement

Multi-Evaluator Agreement is a metric in heuristic evaluation that quantifies the degree of consensus among multiple evaluators in identifying usability issues by measuring the overlap or consistency of their findings.

10

Publications

1.1K

Citations

38

Authors

19

Institutions

Publications per year

2017–2026

2

Parents

Publications

10

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Morten Hertzum, Niels Ebbe Jacobsen · International Journal of Human-Computer Interaction · 2001 · 423 citations

, , +14

The ventilatory threshold: Method, protocol, and evaluator agreement

Masayoshi Shimizu, Jonathan Myers, Nancy Buchanan et al. · American Heart Journal · 1991 · 135 citations

, , +12

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann et al. · 2024 · 131 citations · Full text

, , +15

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Morten Hertzum, Niels Ebbe Jacobsen · International Journal of Human-Computer Interaction · 2003 · 125 citations

, , +14

Do some evaluators report consistently higher or lower PCL-R scores than others? Findings from a statewide sample of sexually violent predator evaluations.

Marcus T. Boccaccini, Darrel B. Turner, Daniel C. Murrie · Psychology Public Policy and Law · 2008 · 116 citations

, , +11

1–5 of 10

Authors

38

  1. NE
    Niels Ebbe Jacobsen

    Carnegie Mellon University, United States

    3 publications · 589 citations

    3

    h-index

  2. MH
    Morten Hertzum

    Roskilde University, Denmark

    3 publications · 589 citations

    3

    h-index

  3. BB
    Babette Brumback

    University of Florida, United States

    1 publications · 24 citations

    1

    h-index

  4. NB
    Nancy Buchanan

    Long Beach Medical Center, United States

    1 publications · 135 citations

    1

    h-index

  5. MB
    Michel Bédard

    Lakehead University, Canada

    1 publications · 24 citations

    1

    h-index

1–5 of 38

Institutions

19

  1. Software Engineering Institute

    Pittsburgh, United States

    1 h-index

    1 publications

    25 citations

  2. International Computer Science Institute

    Berkeley, United States

    1 h-index

    1 publications

    131 citations

  3. Universidad Rey Juan Carlos

    Madrid, Spain

    1 h-index

    6 publications

    120 citations

  4. University of California, Berkeley

    Berkeley, United States

    1 h-index

    4 publications

    524 citations

  5. Japan External Trade Organization

    Tokyo, Japan

    1 h-index

    1 publications

    423 citations

  6. Fraunhofer Institute for Experimental Software Engineering

    Kaiserslautern, Germany

    1 h-index

    1 publications

    25 citations

  7. Université de Montréal

    Montreal, Canada

    1 h-index

    1 publications

    131 citations

  8. Nordic Laboratory for Luminescence Dating

    Roskilde, Denmark

    1 h-index

    1 publications

    41 citations

1–8 of 19

Venues

  1. 2

    h-index

  2. Journal of Quantitative Analysis in Sports

    1 publications · 107 citations

    1

    h-index

  3. BMC Medical Education

    1 publications · 20 citations

    1

    h-index

  4. Psychology Public Policy and Law

    1 publications · 116 citations

    1

    h-index

  5. American Journal of Occupational Therapy

    AJOT · 1 publications · 24 citations

    1

    h-index

1–5