Application of Generalizability Theory to Concept Map Assessment Research

Abstract

In the first part of this article, the use of Generalizability (G) theory in examining the dependability of concept map assessment scores and designing a concept map assessment for a particular practical application is discussed. In the second part, the application of G theory is demonstrated by comparing the technical qualities of two frequently used mapping techniques: construct-a-map with created linking phrases (C) and construct-a-map with selected linking phrases (S). Some measurement facets that influence concept-map scores are explored and how to optimize different concept mapping techniques by varying the conditions for different facets is shown. It is found that C and S are not technically equivalent. The G coefficients for S are larger than those for C under the same condition. Furthermore, a decision(D) study shows that fewer items (propositions) would be needed for S than C to reach desired level of G coefficients if only one occasion could be afforded. On the other hand, C seems to reveal students' understanding about different concepts than S better. For practical purposes, one might prefer S because it is easier to score and produces higher reliability. However, this efficiency comes at the cost of validity. We would trade off validity and reliability for efficiency by including more propositions in C map.