The Impact of Structured Student Debates on Critical Thinking and Informatics Skills of Second-year Medical Students

Abstract

Yet it has become increasingly difficult to keep abreast of and to assimilate the investigative reports which accumulate day after day…. (O)ne suffocates… through exposure to the massive body of rapidly growing information. —BERNHARD VON LANGENBECK, Address at the First Congress of Surgery, April 10, 18721 Among its many facets, the field of medical informatics encompasses the use of technology to access and manage scientific information. The Association of American Medical Colleges (AAMC), through the Medical Informatics Objectives of the Medical School Objectives Project (MSOP) has identified five informatics-related roles of the physician and has established objectives for each of these roles. The lifelong learning role incorporates skills relating to information retrieval, evaluation, and reconciliation. Without activities specifically targeting these skills, it is an act of faith that students will graduate with adequate preparation in these areas. To explicitly address these curricular goals, second-year students in Endocrinology and Reproduction Course at the University of Texas Medical Branch in Galveston were required to participate in debates on controversial topics in these fields. This exercise provided a structured task for developing and improving skills in literature searching, critical thinking, including evaluation of the quality of studies, reconciling results of conflicting studies, teamwork, formal presentation and communication, and spontaneous scholarly discussion. A search of the Medline database produced only one article describing the use of student debates for acquiring content and developing critical thinking and communication skills in health science education.2 The paper describes a first-year pharmacy curriculum that incorporated debates on socioeconomic topics relevant to pharmacy practice. While these debates required critical analysis of issues, the primary focus was on content rather than cognitive or informatics-related skills.2 Published accounts of debates in a college chemistry course3 and business school4 provide qualitative descriptions of the beneficial effects of such exercises on critical thinking, updating knowledge, and communication skills. In a more quantitative approach, Allen et al.5 conducted a meta-analysis of the impact of formal instruction in communication skills (including debates) on critical thinking ability. Such training resulted in 44% increase in scores on tests of critical thinking. Compared with training in other communication skills, participation in “forensics” (i.e., competitive debates) produced the greatest improvement, although the differences did not achieve statistical significance.5 Finally, Johnson et al.6 performed a meta-analysis of the effects of a method they call “academic controversy” on a variety of cognitive outcomes. This method, which shares many features of the debates described in the current report, has produced “increased achievement and retention, higher-quality problem-solving and decision-making, more frequent creative insight, more thorough exchange of expertise, and greater task involvement” by students.6 The current report describes the implementation of structured debates and the evaluation by students and faculty of the degree to which the informatics objectives were accomplished. Method The 174 second-year students were divided into six sections of approximately 30 students each and were assigned to teams of three within each section. The debate topics represented areas of controversy in endocrinology and reproductive science. Students received assigned topics at the beginning of the course, and each student participated in one debate. When not presenting, students were expected to attend their section's debates. Each team researched background information, identified the main issues, found and analyzed relevant studies, developed arguments on both sides of the topic, developed a “rational compromise” after weighing the evidence, and prepared to present each side of the topic. Each team was assigned to present the pro (supporting) or con (opposing) perspective immediately before the debate. Each of the six students gave a five-minute presentation of one of the following segments: pro background and arguments; pro supporting data; con background and arguments; con supporting data; pro “rational compromise”; con “rational compromise.” All team members participated in a ten-minute rebuttal segment prior to the “rational compromises” and a ten-minute question-and-answer session after the final presentation. A faculty Moderator kept the session on schedule, participated in the question-and-answer portion, and evaluated the students' performances. The students were assessed individually on presentation skills, contributions to the rebuttal and question-and-answer segments, and professionalism. The teams were evaluated for the accuracy of information and appropriateness of conclusions, and on written summaries and references turned in at the debate. Individual and team scores were combined to generate a letter grade for each student. Debates presented in each of the six student sections in a given week generally revolved around a single theme in order to provide similar learning experiences for all students attending. For example, one set of debates addressed related facets of the role of insulin resistance in producing disease: (1) Hyperinsulinemia causes hypertension; (2) Insulin resistance increases the risk of thrombosis; (3) Insulin resistance increases the risk of coronary artery disease; (4) Insulin resistance causes the polycystic ovary syndrome; (5) Obesity is an independent risk factor for coronary artery disease; (6) Intensive treatment of type 2 diabetes mellitus lowers the risk of coronary artery disease compared with conventional treatment. Other themes included menopausal hormone replacement therapy, HIV infection and pregnancy, growth hormone therapy in non—growth-hormone-deficient children, and the diagnosis and management of thyroid and parathyroid neoplasia. The effectiveness of this exercise was evaluated by three modalities. First, all students (n = 174) were requested to complete a survey following their debates. Second, faculty Moderators (n = 17) were surveyed to obtain their impressions of the students' skills and the educational value of the debates. Finally, two focus groups of randomly selected students (n = 4 per group) met with facilitators midway through the course to discuss the debates and other course aspects. The facilitators were educators not directly involved in the course. Summaries and anonymous comments from the focus groups were reviewed and approved by the students. The student and faculty questionnaires were parallel instruments and were administered following the debates. Section A asked the students to use a three-item scale—“major resource,” “minor resource,” and “not used”—to describe the importances of ten resource types. Faculty had one additional category, “can't judge.” Section B addressed 13 specific objectives of the debates, 11 of which corresponded to skills identified in the MSOP (Table 1). Students used a scale from 0 to 6 (0 = not used/not applicable; 1 = complete novice; 2 = minimally competent; 4 = moderately competent; 6 = expert) to retrospectively rate their pre- and post-debate skills. The faculty scale replaced “not used” with “can't judge.” Section C used Likert-like scales to assess the importances of skills fostered by the debates, and the usefulness and timing of debates for promoting skill development. Finally, section D asked for comments and suggestions.TABLE 1: Student Self-assessments and Faculty Ratings of Skills Developed during Preparation and Presentation of Structured Student Debates, University of Texas Medical Branch at Galveston, 1999–2000*Results Of the 174 participants, 114 (65.5%) responded to the survey. They did not differ from the non-respondents with regard to individual debate scores (33.6 ± 0.2 versus 33.6 ± 0.3, p >.9), team debate scores (29.2 ± 0.3 versus 29.0 ± 0.4, p >.7), or scores on the final course exam (90.3 ± 0.8 versus 88.3 ± 1.4, p >.2). Six faculty, three clinicians and three basic scientists, who had moderated 19 of the 30 debates (63.3%), responded to the faculty survey. These six included all four who had moderated more than one debate. Among the students responding, 78 (67%) indicated that the skills acquired through the debates would be “important” or “very important” in their careers, while all six faculty rated the importance of these skills in the highest category. Seven students (6%) felt the skills would be “not important at all.” Seventy students (60%) agreed or strongly agreed that the debate had been a valuable learning exercise, while 23 (20%) disagreed or strongly disagreed. The students were evenly divided as to whether one (n = 33), two (n = 35), or three-to-four (n = 33) similar exercises would be required to “promote adequate development” of the skills. Four faculty (66.7%) felt three or four times would be appropriate, one felt four to eight times would be needed, and one felt two times would be adequate. One faculty member and 61 students (52%) felt the preclinical years were the most appropriate place in the curriculum for such exercises. Seventeen students (15%) felt they should be limited to the clinical years, and five faculty (83%) and 23 students (20%) indicated that the exercises should occur throughout the four-year curriculum. The results of the student and faculty surveys of skill development are presented in Table 1. The students' self-assessments increased significantly for all skills, with mean ratings of post-debate skills generally near a score of 4, or “moderately competent.” However, the increase in mean score was greater than one level for only one skill (weighing and reconciling conflicting information), while for two skills (impromptu reasoning; working effectively as a team) less than 40% of the respondents reported any increase. Although the sample sizes (114 students, 6 faculty) preclude statistical comparisons, faculty ratings of student skills appeared lower than student self-ratings for all but four skills. Table 2 summarizes the students' responses regarding resource utilization. Review articles (88.9%) and primary research articles (86.3%) were most frequently identified as major resources.TABLE 2: Numbers and Percentages of Second-year Medical Students Rating Information Resources as Major or Minor in Preparing for Structured Debates, University of Texas Medical Branch in Galveston, 1999–2000*Focus-group summaries corroborated the generally favorable survey findings. Specifically, the debates were perceived more as exercises in critical thinking than as exercises in content acquisition, had been effective in promoting literature-searching and research-analysis skills, and had been “interesting and enjoyable.” The most common criticism was the amount of preparation time required. Comments from the faculty survey, while overall extremely favorable, suggested several areas for improvement: students' overreliance on reviews and published expert opinion, a tendency for students to want “to win” the debate rather than come to a balanced judgment based on the evidence, and the need to couple specific instruction in these skills with the debates. Discussion This report describes the method and evaluation of structured student debates for promoting the development of several cognitive and informatics-related skills, many of which are embodied in the MSOP. The data reported confirm that this exercise accomplished most of its goals. The central goal of promoting the development of skills in analyzing research studies and weighing and reconciling contrasting results was realized. The specific objectives related to this goal (A-D in Table 1) showed the greatest mean increases in self-ratings as well as the greatest proportions of students reporting improvement. Furthermore, primary research articles were among the two most important resource categories, corroborating the value of this exercise in stimulating critical analysis of research reports. However, the comparable emphasis on review articles raises concern that the exercise could deteriorate into general summaries rather than critical evaluations of the literature. In order to focus students' attention on the primary literature, the debate format and evaluation emphasized the use of data to support arguments. Some reliance on review articles was to be expected, as most students had neither extensive backgrounds in the topics nor much experience in reconciling conflicting research. Faculty impressions were lower than the students' self-ratings in these areas, especially with regard to “critically reviewing published research,” with mean ratings below the “moderately competent” level. Thus, at the completion of this exercise, the faculty perceived lower abilities and, therefore, a greater need for further skill development than did the students. The students also indicated improvement in literature searching, weighing risks and benefits of treatments, making evidence-based decisions, and understanding cost—benefit issues. For the other self-rated skills there were lower proportions of students improving and smaller increases in mean scores, although all increases were statistically significant. Although faculty assessments of most skills were lower than students', faculty rated the students at comparable or higher levels in literature searching, presentation skills, impromptu reasoning, and teamwork. The retrospective nature of the student survey, in which the students rated both their pre- and post-debate skills after completing the debate, may be viewed as a weakness in the study design. Nonetheless, the increase in scores indicates that the students felt the exercise did, in fact, promote skill development. While the significant increases in mean scores indicate progress in students' skill development, the magnitudes of changes were generally small and the percentages of students reporting improvement varied by skill. These findings suggest that one such exercise is insufficient for adequate skill development. All faculty and most students acknowledged the importance of these skills and indicated that additional exercises were necessary. Consensus among faculty was for three or four exercises throughout the four-year curriculum, while the students' varied recommendations are best summarized as two debates during the preclinical years. In summary, we have found that structured student debates among second-year medical students promoted development of critical thinking and informatics skills identified in the MSOP Medical Informatics Objectives. A series of exercises distributed throughout the curriculum, targeting progressively more advanced skills and coupled to instruction in these skills, may achieve these objectives more fully.