A Study of the Reliability of Adjudicator Ratings at the 2005 Virginia Band and Orchestra Directors Association State Marching Band Festivals

Abstract

Abstract The purpose of this study was to examine the reliability of adjudicator ratings at the 2005 Virginia Band and Orchestra Directors Association State Marching Band Festivals. Judge evaluation sheets for 124 bands were collected and tabulated from five festival sites throughout Virginia during October of 2005. The subsequent data were analyzed for the reliability of the final ratings, caption ratings, reliability at each site, differences in reliability for three versus five judges, and differences between evaluations for small and larger bands. Inter-correlations and Cronbach Alphas indicated that the overall judge reliability was very good, alpha = .94. There was also no significant variation in reliability among the five sites or between three or five judges. The inter-correlations of the caption ratings were very high, suggesting one global evaluation factor. A one-way analysis of variance indicated that smaller bands were rated significantly lower than larger bands. These findings were in general agreement with other research in this area. The high inter-correlations of the caption ratings suggested that the evaluation form as currently utilized may lack diagnostic validity. Study Participants in marching band festivals are frequently concerned about the reliability of the ratings received at these events. Band directors may question whether their groups were evaluated according to equitable standards and whether or not the judges agreed with one another. Research tends to support these concerns. Rickles (2006) found that larger band ratings in Arizona were significantly different than small band ratings, and Brakel (2004) reported a difference in ratings consistency between advanced and less experienced bands. Sullivan (2003) suggested that differences in ratings between large and small bands might be due to adjudicator bias or the particular classification system. Several studies of band and orchestra adjudication have indicated differences between the reliability of the final and caption ratings (Burnsed, Hinkle and King, 1985; Burnsed & King, 1987; Garman, Boyle and DeCarbo, 1991). The results of these studies suggest that adjudicators generally agree on the final rating an ensemble receives. There is often, however, disagreement between adjudicators on caption ratings. This would suggest that adjudicators are really offering the final rating as a global assessment and are making the caption ratings fit the overall rating (Burnsed and King, 1987). The focus of much of the judge reliability literature has been the evaluation of solo, concert festivals, or recorded examples of performances (Fiske, 1983; Burnsed and King 1987; Garman, Boyle and DeCarbo, 1991; Bergee, 2003; Norris and Borst, 2007; Smith and Barnes, 2007). Only a few of these studies have used actual data from festivals and very little research has been conducted on the adjudication of marching band festivals. The purpose of the present study was to examine the reliability of adjudicator ratings at the 2005 Virginia Band and Orchestra Directors Association State Marching Band Festivals. We wanted to see if some of the oftenmentioned concerns about judging and festivals were evident at the 2005 marching festivals sponsored by the Virginia Band and Orchestra Directors Association (VBODA). This study attempted to address the following research questions about the 2005 VBODA Marching Band Festivals: 1 . What is the reliability of the adjudicator final ratings? 2. What is the reliability of the adjudicator caption ratings? 3. Was the reliability of ratings different at each site? 4. Was the reliability of ratings different for three and five adjudicators? 5 . Did larger bands receive higher ratings than smaller bands? Method The Virginia Band and Orchestra Directors Association sponsored marching festivals at five sites in Virginia during October of 2005. …