Characterization of the Council of Emergency Medicine Residency Directors' Standardized Letter of Recommendation in 2011-2012

Abstract

The Council of Emergency Medicine Residency Directors (CORD) introduced the standardized letter of recommendation (SLOR) in 1997, and it has become a critical tool for assessing candidates for emergency medicine (EM) training. It has not itself been evaluated since the initial studies associated with its introduction. This study characterizes current SLOR use to evaluate whether it serves its intended purpose of being standardized, concise, and discriminating. This retrospective, multi-institutional study evaluated letters of recommendation from U.S. allopathic applicants to three EM training programs during the 2011–2012 Electronic Residency Application Service (ERAS) application cycle. Distributions of responses to each question on the SLOR were calculated, and the free-text responses were analyzed. Two pilots, performed on five applicants each, assisted in developing a strategy for limiting interrater reliability. Each of the three geographically diverse programs provided a complete list of U.S. allopathic applicants to their program. Upon randomization, each program received a list of coded applicants unique to their program randomly selected for data collection. The number of applicants was selected to reach a goal of approximately 200 SLORs per site (n = 602). Among this group, comprising 278 of 1,498 applicants (18.6%) from U.S. allopathic schools, a total of 1,037 letters of recommendation were written, with 724 (69.8%) written by emergency physicians. SLORs represented 57.9% (602/1037) of all LORs (by any kind of author) and 83.1% (602/724) of letters written by emergency physicians. Three hundred ninety-two of 602 SLORs had a single author (65.1%). For the question on “global assessment,” students were scored in the top 10% in 234 of 583 of applications (40.1%; question not answered by some), and 485 of 583 (83.2%) of the applicants were ranked above the level of their peers. Similarly, >95% of all applicants were ranked in the top third compared to peers, for all but one section under “qualifications for emergency medicine.” For 405 of 602 of all SLORs (67.2%), one or more questions were left unanswered, while 76 of all SLORs (12.6%) were “customized” or changed from the standard template. Finally, in 291 of 599 of SLORs (48.6%), the word count was greater than the recommended maximum of 200 words. Grade inflation is marked throughout the SLOR, limiting its ability to be discriminating. Furthermore, template customization and skipped questions work against the intention to standardize the SLOR. Finally, it is not uncommon for comments to be longer than guideline recommendations. As an assessment tool, the SLOR could be more discerning, concise, and standardized to serve its intended purpose. Tras su creación en 1997, la carta de recomendación estandarizada (CRE) del Council of Emergency Medicine Residency Directors (CORD) se ha convertido en una herramienta crítica para evaluar a los candidatos para la formación en la medicina de urgencias y emergencias (MUE). Ésta herramienta no se ha evaluado desde los estudios iniciales asociados con su introducción. Este estudio caracteriza el uso actual de la CRE para evaluar si sirve su propósito intencionado de ser estandarizada, concisa y discriminatoria. Estudio retrospectivo multinstitucional que evaluó las cartas de recomendación de los candidatos alopáticos americanos en tres programas de formación de MUE durante el ciclo de solicitudes Electronic Residency Application Service (ERAS) durante 2011–2012. Se calculó la distribución de las respuestas de cada pregunta en la CRE y se analizaron las respuestas de texto libre. Se realizaron dos estudios piloto en cinco candidatos cada uno, que fueron de ayuda en el desarrollo de una estrategia para limitar la validez interevaluador. Cada uno de tres programas geográficamente distintos proveyó una lista completa de los candidatos alopáticos americanos a su programa. Tras la aleatorización, cada programa recibió una única lista de candidatos codificada a su programa seleccionado aleatoriamente para la recogida de datos. El número de candidatos se seleccionó para alcanzar un objetivo de aproximadamente 200 cartas de recomendación por sitio (n = 602). Este grupo lo formaron 278 de los 1.498 solicitantes (18,6%) a los colegios alopáticos, para los cuales se escribieron un total de 1.037 cartas de recomendación de las que 724 (69,8%) fueron escritas por urgenciólogos. Las CRE fueron 602, lo cual representó el 57,9% de las 1.037 cartas de recomendación escritas por cualquier autor y el 83,1% de las 724 cartas de recomendación que habían sido escritas por urgenciólogos. Trescientas noventa y dos de las 602 cartas de recomendación tuvieron un único autor (65,1%). Para la pregunta “valoración global,” los estudiantes fueron puntuados en el decil más alto en 234 de 583 solicitudes (40,1%; la pregunta no fue contestada por algunos), y 485 de 583 (83,2%) de las solicitudes fueron clasificadas por encima del nivel de sus pares. De la misma forma, más del 95% de todas las solicitudes se clasificaron en el tercio más alto en comparación con los pares para todas las secciones excepto la de “calificaciones para MUE”. Para 405 de las 602 CRE (67,2%), una o más preguntas se dejaron sin contestar, mientras que 76 de todas las CRE (12,6%) se personalizaron mediante el cambio de formulario estándar. Finalmente, en las 291 de las 599 CRE (48,6%), el número de palabras fue mayor que el máximo recomendado de 200. El grado de exageración que contiene la CRE limita su capacidad discriminativa. Es más, la personalización del formulario y el saltarse preguntas va en contra de la intención de estandarizar la CRE. Finalmente, no es infrecuente que los comentarios sean más extensos que las recomendaciones de la guía. Como herramienta de valoración, la CRE podría ser más discriminatoria, concisa y estandarizada para servir al propósito para el que se diseñó. For many years emergency medicine (EM) program directors (PDs) struggled with the limitations inherent in traditional narrative letters of recommendation. In 1995, in response to growing concerns, the Council of Emergency Medicine Residency Directors (CORD) established a standardized letter of recommendation (SLOR) task force whose goal was to create “a method of standardization for letters of recommendation.”1 After a 2-year development process, the SLOR was introduced in 1997. The goal of the task force was to develop an assessment instrument with three basic tenets: 1) standardization—the same essential questions are asked and answered for every candidate; 2) time-efficient to review—a clear and concise synopsis for efficient evaluation; and 3) discriminating—a template to convey comparative performance data on candidates in specific areas important to clinical practice.1 Of paramount importance, the SLOR's format was designed to limit “clerkship grade and adjective inflation,” which was believed to be “rampant” at the time. In the first few years after the adoption of the SLOR there were a number of commentaries and studies related to its use.2-6 In general, it appeared that the SLOR accomplished many of its intended goals. Since that time, the SLOR has been universally adopted and become an expected component of any application for EM training. The SLOR template used today contains minor changes from the original format released in 1997. Approximately 15 years after its introduction, the SLOR task force was reestablished to evaluate whether or not the SLOR, as it is used currently, adheres to its original tenets. Secondary goals included characterization of SLOR use and identification of opportunities for improving this assessment tool. This article reports these results. This was a retrospective, multi-institutional study based on U.S. allopathic applications to EM from the 2011–2012 Electronic Residency Application Service (ERAS) application cycle. Institutional review board (IRB) approval was obtained from each of three participating institutions prior to the collection of data. Due to the anonymous nature of this study, each affiliated IRB approved the study as exempt from informed consent. Data were collected from applicants to three geographically diverse EM training programs: Medstar Georgetown University Hospital/Washington Hospital Center, Washington, DC; Oregon Health & Science University, Portland, Oregon; and The University of Cincinnati, Cincinnati, Ohio. Two of these programs have 3-year formats and the third is a 4-year program. To first determine interrater reliability, the SLORs of five applicants were randomly selected using a computerized random-number generator that was then checked for duplicate entries by the American Association of Medical Colleges (AAMC) identifier. Each of the three primary study authors independently reviewed the letters of recommendation, and the investigators' coding of the data were compared. There were 11 SLORs in this pilot, with 597 data points. All three authors agreed in 85.9% of outcomes. Disagreements were resolved by consensus, and the coding protocol was amended to avoid future discrepancies. A second pilot of five additional candidates applying to all three programs was randomly selected (using the same method described). Once again all three authors independently evaluated the letters of recommendation. In this cohort there were 13 SLORs with 705 data points for comparison. All three authors agreed in 94.2% of outcomes. A final meeting focused on remaining issues and resulted in a consensus plan for dealing with ongoing areas of disagreement. For the study itself, we targeted a minimum sample size of 600 SLORS. We projected that with 600 letters, we would have power of 0.8 to detect differences in proportions of 11% to 14%, depending on the numbers of subgroups compared (i.e., two to four projected subgroups). An initial, informal review of 20 random applicants yielded an average of two SLORs per applicant on two separate occasions. This estimate was used when randomly generating 100 unique applicants from each of the three participating programs with Microsoft Excel for Macintosh, v. 14.2.3 (Microsoft Corp., Redmond, WA) to obtain approximately 200 SLORs from each site. Descriptive statistics including means, proportions, and 95% confidence intervals (CIs) were calculated using IBM SPSS Statistics for Macintosh, v. 20 (IBM Corporation, Armonk, NY). To reach the goal of approximately 600 SLORs, the ERAS applications of 287 allopathic applicants were reviewed. This represents 18.6% of the 1,498 U.S. allopathic seniors applying to EM (287 applicants minus nine interns) in 2011–2012.7 There were 1,037 letters of recommendation. Emergency physicians authored 724 (70%) of these letters, 602 of which were SLORS. Of the 602 SLORS, 198 (32.9%) were from The University of Cincinnati, 203 (33.7%) were from Medstar Georgetown University Hospital/Washington Hospital Center, and 201 (33.4%) were from Oregon Health & Science University. The SLORs were based on EM rotations 98.3% (592 of 602) of the time, with only 1.7% (10 out of 602) based on related EM electives (e.g., research, ultrasound). On occasion, a SLOR is written for an intern rotating through the emergency department (ED). Fourteen of 588 SLORs (nine applicants) were written for interns or residents, of which only four were completely based on rotations in the ED as a resident. The other nine were based on rotations during medical school the year before, four of which included additional information since that rotation. The nature of contact was “extended direct clinical contact” in 73.2% (431 of 589) of all SLORs, in 76.3% (95% CI = 71.8% to 80.5%) of single-author SLORS, and in 67.0% (95% CI = 60.0% to 73.5%) of group SLORs. The majority of authors (57.9%, 319 of 551) indicated that they knew the applicants for longer than 1 month. Single-author SLORs made up 65.1% (392 of 602) of all SLORs. The most common authors of this type of SLOR were clerkship directors (CDs; 36.2%, 142 of 392), EM faculty (29.6%, 116 of 392), assistant/associated PDs or assistant clerkship directors (APDs/ACDs; 11.5%, 45 of 392), PDs (10.2%, 40 of 392), and emergency physicians not affiliated with the training program (5.9%, 23 of 392). Group SLORs made up 34.9% of all SLORS with the most common contributing authors being CDs (80%, 168 of 210), PDs (79%, 166 of 210), and APDs (47.6%, 100 of 210). The vast majority of SLORs reported some variation on the honors/pass/fail grading system (95.8%, 566 of 591), with 4.2% of the SLORs coming from rotations that grade on a pass/fail system. Table 1 provides a breakdown of EM clerkship grading as well as a reported “historic” grading breakdown from the previous academic year. Clerkship data reported that 29.9% (160 of 535) of the SLORS were from institutions that have 50 or fewer participants in the EM rotation per year, 39.1% (209 of 535) between 51 and 99, and 31.0% (166 of 535) have 100 or more students participate annually. Section B of the SLOR template pertains to “qualifications for EM.” The results for SLOR questions B1–B4b can be found in Table 2. Question B5a asks, “How much guidance do you predict this applicant will need during residency?” Authors of the SLOR answered “almost none” of of the time, in of of and the remaining 4.2% of indicated that guidance was The results of “global or section of the SLOR are in Table In SLORs were evaluated to determine whether the was related to the of a of was Table these results. A review of the written comments focused on the total by number of words. In of of SLORs, the word count was greater than the recommended This was in of of single-author SLORs and of of group SLORs. A list of faculty comments from the rotation was included in of of longer than 200 and in of of that were comments the for clinical or clerkship were included in of of longer than 200 and in of that were In of of the written comments were Table the of each question was not Question which the of year, was skipped or not the of the time. The SLOR template was “customized” or changed in of of the the final was not checked that the applicant not the to review the SLOR. The goal of the original SLOR was to create a letter that was standardized and concise while data. after the SLOR was and PDs and found that the most important in were the clinical clerkship EM rotation and letters of recommendation. This was by and that for EM the EM rotation grade and clinical were for the that time, the SLOR that one of the of the SLOR was that it two of the three a letter of recommendation and the grade on an EM EM rotations during the year of medical school as a these are not on the or Medical In a study of applicants to EM training from a single program that of all letters of recommendation were Since the SLOR has become to applications for training in the comprising of all letters written in of applications and 83.1% of the letters written by emergency physicians. The for all SLORs is a clinical rotation in the the SLOR has is that in most authors their on “extended direct contact” with the than the author has the applicant for longer than 1 the for additional and that could the SLOR. The ERAS application is with information an including The SLOR to an of as they to the ED through as the ED and to peers. The SLOR author is in an to on the and (e.g., as of and ability to work with a important to in can be to from the ERAS application the most important of the SLOR is its ability to the performance of applicants to one in specific data that grade inflation is an in the SLOR. For many of the questions throughout the SLOR, comparative information to be through to peers. are based on to For represents the the the and the in to other EM A review of “qualifications for EM that in the of to and the vast majority of applicants received or than received level of or of candidates were ranked as to peers. are to of the data from the task force of SLOR authors that do SLOR authors they In SLOR authors that in some they use their to the adjective of to the associated This all applicants are as or the SLOR that an is the level of their or It is that most applicants faculty to author their single-author SLORs based on the of faculty The results in Table that the a in they are when a “global of The data that as become more in SLORs, they become more in grading PDs are the most of the SLOR An of this based on author type is important for SLORs. It is that the most authors of candidates as in the top 10% of The has with grade inflation As a they have a number of changes including the the intention of being an and not a The SLOR has been intended to be an assessment tool. to it as a a for the author is whether or is assessing or the not the of grade of applying were of or the for their EM In SLOR authors reported that of students received of the previous year This be in by the that EM is a clerkship at some and not the performance of all students and not applying to EM As an instrument designed to performance to EM peers, the SLOR of the that clerkship are more applicants than they A second of the SLOR is to a concise synopsis of the this associated with the SLOR have that the narrative written comments be at a maximum of work by that the average to a SLOR was compared to for a narrative In the SLOR interrater than traditional of the level of of the are important in a when many programs applications annually. The current study found that the written comments of approximately of all SLORS were the there has been a years to additional information in this section that a list of faculty comments from the EM rotation and additional information the clinical or The data from this study that the of random faculty comments from the rotation is three more common in written comments greater than 200 words. comments of that an important a list of comments of the concise the written comments were intended to an of grading or the clinical at that is three to four more common in written comments that were longer than the this information the it the on the that SLORs to A written narrative the SLOR and an to this is a of as and that are important in future as a and this narrative any questions or issues in the SLOR. not in this study, it has been common from for faculty to that are in the written Finally, the SLOR was as a standardized instrument to limit the based on and found in narrative letters, as well as to comparative data performance for every issues were that to have an on of the SLOR template is one these changes are template the of SLORs be they be based on consensus that changes are adopted by all SLOR authors and not based on The majority of SLORs have one or more questions that are left unanswered, which There are questions have been 1) the author could have an 2) the question be as to be the with which was skipped of the and 3) the be is being For the “global assessment not the complete of the time. which questions for the authors to through changes made in specific questions or authors as to the intention and of each question The group SLOR is a format that approximately of all SLORs. there is some group SLORs are written by of the program the and to consensus an In to single-author SLORs, group SLORs are to their on “extended direct clinical group SLORs use from other faculty or are based on more was that group SLORs to be more in For group SLORs provided fewer top 10% than any single-author SLOR with the of from PDs In a of made up of group SLOR the or made up of all single-author SLORs. for this is that applicants the single-author SLOR by the authors Group SLORs, on the other the grading for whether a letter was or to be for the issues in the SLOR. Two related studies are to evaluate the of the SLOR authors and PDs evaluate SLORS. The intention is to a on issues related to this is the is it by the issues in the current and specific changes need to be are by the It is clear that the template could be as could and ongoing on to complete a SLOR. be as to whether there be a to letter of which is a more and for this in the and and additional all have the to with the three basic of the SLOR. grade inflation is the most is that a we as and the we will be to this The majority of applicants to EM have the to in In programs in a The SLOR would 1) that every applicant has unique and 2) the to and written and 3) that is in and from and not or It is that there were in data as the data were not after the every was made to interrater reliability, it is during the The study is in its ability to SLOR authors as they in out the letters, the studies of SLOR authors and PDs this All three of the training programs participating in this study were results have to The results of this study that the standard letter of recommendation in its current use is in its ability to be There is a question of whether changes in its use have its ability to be concise and be of these as they to an applicant