Construct: Every six months, residency programs report their trainees' Milestones Level achievement to the Accreditation Council for Graduate Medical Education (ACGME). Milestones should enable the learner and training program to know an individual's competency development trajectory. Background: Milestone Level ratings for residents grouped by specialty (e.g., Internal Medicine and Emergency Medicine) show that, in aggregate, senior residents receive higher ratings than junior residents. Anesthesiology Milestones, as assessed by both residents and faculty, also have a positive linear relationship with postgraduate year. However, these studies have been cross-sectional rather than longitudinal cohort studies, and studies of how individual residents progress during the course of training are needed. Longitudinal data analysis of performance assessment trajectories addresses a relevant validity question for the Next Accreditation System. We explored the application of learning analytics to longitudinal Milestones data to: 1) measure the frequency of "straight-lining"; 2) assess the proportion of residents that reach "Level 4" (ready for unsupervised practice) by graduation for each subcompetency; 3) identify variability among programs and individual residents in their baseline Milestone Level and rates of improvement; and 4) determine how hypothetically constructed growth curve models fit to the Milestones data reported to ACGME. Approach: De-identified Milestone Level ratings in each of the 25 subcompetencies submitted semiannually to the ACGME from July 1, 2014 to June 30, 2017 were retrospectively analyzed for graduating residents (n = 67) from a convenience sample of five anesthesia residency programs. The data reflected longitudinal resident Milestone progression from the beginning of the first year to the end of the third and final year of clinical anesthesiology training. The frequency of straight-lining, defined as the resident receiving the same exact Milestone Level rating for all 25 subcompetencies on a given 6-month report, was calculated for each program. Every resident was evaluated six times during training with the possibility of six straight-lined ratings. Findings: The number of residents in each program ranged from 5-21 (Median 13, range 16). Mean Milestone Level ratings for subcompetencies were significantly different at each six-month assessment (p < 0.001). Frequency of straight-lining varied significantly by program from 9% - 57% (Median 22%). Depending on the program, 53%-100% (median 86%) of residents reached the graduation target Level 4 or higher in all 25 anesthesiology subcompetencies. Nine to 18% of residents did not achieve a Level 4 rating for at least one subcompetency at any time during their residency. Across programs, significant variability was found in first-year clinical anesthesia training Milestone Levels, as well in the rate of improvement for five of the six core competencies. Conclusions: Anesthesia residents' Milestone Level growth trajectories as reported to the ACGME vary significantly across individual residents as well as by program. The present study offers a case example that raises concerns regarding the validity of the Next Accreditation System as it is currently used by some residency programs.

Jay Roby

Publications