Assessing competency in residents and fellows—how do you measure up?
Key Takeaways
In residency, ACGME Competencies and Milestones aim to bolster outcomes-based education.
In partnership with faculty guidance, residents and fellows can use Milestones to identify opportunities for growth.
Concerns have been raised about the use of Milestone data to remediate resident performance, with high variance of such measures pointing to program-related (and not trainee-specific) factors and idiosyncrasies.
Trainee competency is integral to robust and high-level care. In 1999, the Accreditation Council for Graduate Medical Education (ACGME) promoted a set of competencies to establish foundational skills every practicing physician should possess. These six ACGME Core Competencies focus on educational skills and professional attributes that recapitulate patient care and prepare trainees for the quotidian requirements of clinical practice.
Wherever you are in your medical training, it’s worthwhile to refresh yourself on the Core Competencies and associated Milestones.
The Core Competencies
The ACGME Core Competencies measure a physician’s ability to provide high-level care, offer advice and resources to prevent disease, and provide emotional support to patients and their family members, according to an article in NEJM Knowledge+.[]
According to the authors of an article in JAMA Network Open, “Competencies and milestones are intended to fulfill the aim of outcomes-based education, in which desired outcomes of training, rather than content or hours in particular rotations, drive curriculum planning.”[]
The six Core Competencies are as follows:
Practice-Based Learning and Improvement
Patient Care and Procedural Skills
Systems-Based Practice
Medical Knowledge
Interpersonal and Communication Skills
Professionalism
As the ACGME explains in its Milestones Guidebook for Residents and Fellows, milestones expatiate specialty-specific skills, knowledge, and behaviors that represent progression within each of the six ACGME Core Competencies.[] Milestones are a roadmap for trainee development as they progress in terms of clinical skills, knowledge, and values.
Every specialty must form its own Milestones. For instance, the patient care skills demonstrated by an internal medicine resident will differ from those of an interventional radiology fellow, and so forth.
In recent years, the ACGME has paired with the American Board of Medical Specialties (ABMS) member boards to create specific milestones for each specialty that bulwark the 6 ACGME Core Competencies.
Milestones as stepping stones
Milestones can assist residents and fellows with gauging their growth and help determine where they stand in their educational program. They also help trainees conceptualize what each stage of development looks like. The ACGME offers an apt analogy to setting off on a hike.
“A map and mile markers are essential to helping you get to the end of the trail. Similarly, for any profession, knowing where you are now in your abilities and what you need to focus on next helps you along a path to mastery. The Milestones can help residents and fellows intentionally focus on each step in becoming the best physicians they can be,” they write.
Milestones communicate a lingua franca of sorts to be utilized by faculty and program directors. When residents inquire about their development in a specific competency area (eg, patient care) or subcompetency (eg, clinical reasoning), this common language can help trainees and clinical coaches to forge an alliance and work together.
“Like a swimmer getting coached on how to push off the wall, or a pianist on intonation, having a shared understanding and language for getting better improves the learning experience,” states the ACGME.
Sensemaking
The authors of the invited commentary in JAMA Network Open take an honest look at some of the issues with the Milestones ratings. ACGME requirements invite programs to gather much data and make sense of it at their discretion. Faculty define and generate evidence, such as assessment data, that explains the trainees’ learning progress as demonstrated via Milestones. These ratings are then submitted to the ACGME.
The process of interpreting and synthesizing this data is referred to as sensemaking. When sensemaking, stakeholders form their own narratives to organize and interpret data.
The authors explain the potential flaws when faculty and program directors create their own narratives. “Although the role of Clinical Competency Committees (CCCs) is to standardize the process of implementing the Milestones framework for assessment and rendering judgments about trainee progress, CCCs vary in what data they have available and how they make sense of that data as a reflection of trainee Milestones progression and areas for program improvement. The multistep process of collecting and synthesizing data for the CCC and then determining residents’ competence level introduces many opportunities for program-specific variability and idiosyncrasy.”
Questions about accountability
Intriguingly, research shows that programs are mostly responsible for the magnitude of their trainees’ development. With regard to FM (family medicine) Milestone ratings, for instance, program-level factors contributed to 35% of the variance. This variance was greater than the influence of individual residents themselves, which accounted for 22% of the variance. Ideally, residents would control their own fates, with 100% of the variance attributable to individual performance. Program-level factors could include clinical schedules assigned to residents or program-specific strategies.
In the aggregate, raters and programs may interpret individual competencies and Milestones differently, which could explain why FM milestone ratings for medical education and patient care, for example, more accurately predicted growth than other competency Milestones.
“Trainees and their advisors may rightly question the interpretation, accuracy, or value of using individual Milestone ratings to inform individualized learning plans when ratings may seem to make greater sense as a marker of program strengths, opportunities, gaps, or challenges,” the JAMA Network Open authors wrote. With regard to medical knowledge, educators harbor robust experiences measuring this construct and its various correlates, such as scores on licensing and in-training examinations.
High variance in ratings for many Milestones secondary to program-specific signals, argue the authors, calls into question accountability for trainee performance. Program-level variability was highest for professionalism and interpersonal communication skills; this implies that there’s an ambiguous understanding across programs of what these Milestones mean and what observable behaviors delineate developmental progression with regard to professionalism and interpersonal communication skills.
In other words, remediation required for falling short of such Milestones may represent program flaws and failures, rather than the trainee’s own.
Instead of using Milestones to signify remediation, they could be used to identify inflection points for early intervention and the provision of novel learning opportunities, as well as stimulate practice and coaching, suggested the authors. This strategy may better celebrate the spirit of mastery and obviate the need for remediation later on.
Programs could thus flip the script and render a growth-oriented program that fosters alliances between trainees and faculty “to identify curricular gaps and design new or improved rotations, teaching sessions, and assessment approaches to support all residents.”
Nevertheless, the authors conceded that Milestone rates constitute rich information and propel data-driven actions.
What this means for you
Concerns have been raised over the interpretation and accountability of remediation for residents based on Milestone data. Experts point out that a lot of the variability is due to programs themselves and not trainees. Nevertheless, assessments of such data provide rich insights that can facilitate data-driven actions. For their part, trainees should regularly review Milestones and discuss areas of growth or further learning with faculty mentors and program directors.