A Weighted Kappa Coefficient for Three Observers as a Measure for Reliability of Expert Ratings on Characteristics in Handball Throwing Patterns

Jörg Schorer, Christel Weiβ

Measurement in Physical Education and Exercise Science · 2007 · 12 citations · 15 references

Concepts

Abstract

Many methods for the identification of key variables in movement patterns have the problem that direct transferability back to coaches or players is not easy; this is because the variables used are not presented in the language of most sport participants. Using a research strategy—from the coaches back to the coaches—proposed by Roth (1996) Roth, K. 1996. Techniktraining im Spitzensport: Alltagstheorien erfolgreicher Trainer, Köln: Sport und Buch Strauss. [Technical training at high-performance sport: Everyday life's theories of successful coaches]. [Google Scholar], this study tries to identify discriminative variables by an expert rating. The statistical problem was to analyze the reliability between three coaches. A new, weighted Kappa coefficient for multiple observers is introduced as an extension of what Fleiss proposed in 1971, which takes into account the different types of disagreement. It is concluded that the high variability among different variables´ reliability can be considered as a measurement of the difficulty for the expert ratings. The results also indicate that simple movement features could be identified and transferred as instructions for goalkeepers back to the coaches.

References

15