A Direct Method for Teaching and Measuring Engineering Professional Skills: A Validity Study

Abstract

Abstract A Direct Method for Teaching and Measuring Engineering Professional Skills: A Validity StudyProficiency in engineering professional skills (Table 1) is critical for success in themultidisciplinary, intercultural team interactions that characterize 21st century engineeringcareers. In 2006, the XXX University College of Engineering and Architecture partnered withthe Office of YYY to create an innovative, direct method to teach and measure the ABETprofessional skills simultaneously called the curricular debrief (CD) method. The CD method isan authentic performance task and the Engineering Professional Skills Rubric (EPS), an analyticrubric to measure the quality of the student performance. No direct method for teaching andmeasuring these skills simultaneously has existed in the literature prior to the CD method.Table 1. ABET Criterion 3 Professional Skills Student Learning Outcomes 3d Ability to Function on Multidisciplinary Teams 3f Understanding of Professional and Ethical Responsibility 3g Ability to Communicate Effectively 3h Understanding of the Impact of Engineering Solutions in Global, Economic, Environmental, and Cultural/Societal Contexts 3i Recognition of and Ability to Engage in Life-Long Learning 3j Knowledge of Contemporary IssuesThe four years of on-going college-wide, program-level research conducted at WSU to establishthe initial reliability and validity of this method has shown its potential for significant nationwideimpact. Yet, more research is needed to provide strong evidence that the CD method as aperformance task and the EPS Rubric as a measurement instrument are reliable and valid. Thisstudy’s primary research goal is to rigorously establish the reliability and validity of the CDmethod and the EPS Rubric. This project will directly contribute to fundamental research inengineering education on a problem of national importance and interest.Our research questions are: 1) To what extent does the CD method as a performance task equally elicit students’ consideration of engineering professional skills when implemented in different course types and at different points in a program’s curriculum? 2) Do EPS Rubric scores reliably provide information about students’ engineering professional skills proficiency levels? 3) What is the correlation coefficient between the EPS Rubric’s scores and scores from other established instruments that measure the same or similar skills?A descriptive collective multi-site case-study methodology will be used. This methodology willallow the investigators to understand and examine the contexts in which parallel performancetasks are implemented in three distinct sites and four distinct course-type settings. A completerandomized design will be used to sample students into control and experiment groups withineach course offering, where each course offering is a block of the analysis. There will be 70experimental teams and 66 control teams for a total of 136 teams (796 students).The studentteam will be the primary unit of analysis within a block.Pattern matching will be used to establish the performance assessment’s reliability and validity,complemented by these statistical analytic techniques: (a) a generalizability study; (b) ANOVA; ത(c) correlation coefficient; and (d) task difficulty using the formula p = ܺ/ܺ௠௔௫