August 20th, 2019
Bias in the Residency Ranking Process
Scott Hippe, MD
Scott Hippe, MD, is a Chief Resident at Family Residency of Idaho in Boise.
“Can we please try to be objective about this!” I said these words to myself over and over during this year’s interview season as we formulated our residency rank list. At my institution, the residents and faculty have equal sway in forming the rank list. The chief resident facilitates the resident half of the process. As the hours wore on during our last meeting, the discussion gradually deviated from assessing the applicants’ clinical potential. The focus shifted to peripheral elements of character: “[Applicant] is just really cool and I want them to be my friend,” and similar arguments. I was concerned.
One’s ability to get along with other residents is important. However, it isn’t the only factor to consider when ranking residency applicants. You want a clinically solid incoming class who know their stuff and can keep up with the rigors of residency. Trying to move away from haphazard character judgments, I focused on applicants’ board scores and clerkship grades. Consider a situation where applicants “A” and “B” were equally liked by resident interviewers and were similarly involved in extracurriculars. The fair thing to do would be to use their “stats” to sort out who deserved to be ranked higher. Right? And then I started thinking about those criteria.
I realize that the Insights blog is read by individuals of many different backgrounds and at varying stages of medical training. Depending on who you are, the ensuing comments might be controversial or maybe they are old news. The first, potentially obvious observation: Objective measures aren’t perfect at predicting real-life clinical ability.
“Objective” measures aren’t the be-all, end-all to ranking
- In my experience, USMLE and COMLEX scores (especially Step 1) correlate only loosely with clinical ability. Step 1 isn’t written to be a primary discriminating factor in the residency selection process. Beyond pass/fail, we should stop caring as much about the three-digit score. Seriously, how many practicing physicians need to remember the Krebs cycle? I am not the first one to suggest Step 1 be de-emphasized (Acad Med 2016; 91:12).
- Although of questionable clinical significance, board scores certainly do correlate with demographic characteristics. Namely, higher board scores are associated with having a non-minority background and speaking English as a first language. Incorrect use of board scores in the ranking process solidifies bias against nontraditional and diverse applicants.
- I argue that clerkship grades are not much better. Often they are heavily weighted by standardized testing and thus subject to similar biases as is the USMLE. Clerkship standards also vary from institution to institution. Clerkship performance has some validity internal to the specific medical school, but I would argue that it has less external validity when comparing across schools.
Toward eliminating bias
I cannot say whether there is one completely perfect way to conduct applicant ranking. The Match process provides a decent place to start. I especially appreciate the standardization it brings to the process. But the Match doesn’t eliminate all sources of bias, especially when it comes to test scores and clerkship grades. Despite many efforts to increase diversity in medical education, the proportions of under-represented minorities in medical schools haven’t changed much since 1980.
Specialties by offered positions in the 2015 U.S. Residency Match
Consider the imperfections inherent in the metrics we use to evaluate medical students. This should beget humility in the resident selection process. Humility that discovers and celebrates applicants’ life experiences. Humility that de-emphasizes the “objective” measures we use to evaluate our applicants. Humility that breeds a culture of curiosity rather than exclusivity.
The Match this year happened months ago, but soon enough, fall will return and with it, another interview season. If you are involved in a residency, I hope these thoughts challenge next year’s interview process. Lower your USMLE thresholds for interviews. See past the test scores. Look for character. Above all, I propose emphasizing a central evaluation criterion: An applicant’s potential to serve their community and the common good with their medical training.
7 Responses to “Bias in the Residency Ranking Process”
2021-2022 Chief Resident Panel
Abdullah Al-abcha, MD
Mikita Arora, MD
Madiha Khan, DO
Khalid A. Shalaby, MBBCh
Brandon Temte, DO
Resident chiefs in hospital, internal, and family medicine
Learn more about Insights on Residency Training.
NEJM Journal Watch – Recent General Medicine Articles- Triptan Use in Patients with Migraine and Cardiovascular Disease or Elevated CV Risk
- Paracentesis Timing and Outcomes in Hospitalized Patients with Cirrhosis and Ascites
- Correcting Severe Hyponatremia: Don't Go Too Slow
- Is Prostate Cancer Overtreatment on the Rise?
- Which Is the Preferred Catheter for Outpatient IV Antibiotics?
Thank you. That’s is why we need to look at the entire application and weigh all aspects evenly. This fascination about board scores is beyond me. The scores also change as the years go by.
We want people who have excelled in medical school and have shown devotion and good work ethics during clerkship along with great letters to support their character.
Character is very important, but without skill and knowledge, it is not enough. All are equally important. USMLE scores are a pretty objective measure.
Hi Scott! Pretty good and accurate analysis. I totally agree with you, I have passed trough 3 matches and Always you can notice the Bias, I understand that They are defending their Status Quo, but considering other more relevant factors can make the difference in the Match process, Thanks for showing this reality!
As a retired psychiatrist who devoted many years to C-L psychiatry my exposure to a variety of physicians across specialties was an exceptionally large sample. What I found was that a substantial number of physicians find it a chore to “play nice with others.” This includes recognizing the need for he extra effort required to maintain open, real time communication with other physicians. An especially disheartening observation is the failure on both ends if the relationship to recognize that PCP’s ought to be the hub for consolidating a patient’s information and orchestrating care* in conjunction with other physicians and medical professionals. With EHR it is even easier to ignore the problem assuming that’s what EHR does. As one who is now only a patient, I can tell you it doesn’t. A most compelling discussion of the factors that the most functional “societies” ( here used to describe any social group) have in common is to be had in “Blueprint: The Evolutionary Origins of a Good Society” by Nicholas Christakis M.D., PhD, MPH, Sterling Professor of Social and Natural Science in the Departments of Sociology, Medicine, Ecology and Evolutionary Biology, Statistics and Data Science and Biomedical Engineering at Yale. Yes, this is one person (!) eminently qualified to discuss characteristics that make for successful social structures and which social structures fail for lack of certain characteristics. Depending on politicians to construct a health care system is “like taking an accordion on a deer hunt”-Donal Rumsfeld speaking unkindly of a NATO ally. But it seems to be that leaving it to politicians is what is happening. I’m not sure what the metric might be to select candidates with personality traits that would increase the likelihood of medicine reforming its social structure from within but one is sorely needed. This entertaining and informative read by someone more than “just qualified” might spark some thinking about how the process might start to develop such a metric. It might also lead to research that allows for the evaluation to start at the undergraduate institutional level for those pursuing pre-med curriculums. The biggest problem I foresee is a general lack of humility in physicians as a group (generalization from many years of observation) and little emphasis on a systems approach to thinking about the role of physicians as simply one part, not the most crucial part, of a functional “medical society” (society used generically, not for AMA, etc)
*too often MD’s think about “diagnosis and treatment” without including “care” as the nursing profession does.
amen.
Nice article. Mostly everyone shows their best face at interviews. How do I evaluate character RELIABLY?
Having moved between GME and UME, the same issues arise in both arenas. Standardized testing is not a good objective measure. Some of us perform better on such testing than others of us of equal intelligence. Certainly, why someone is pursuing a medical career or a specialty is key to success (and possibly an opportunity to decrease an individual’s burnout potential). On the other side of the issue is how to evaluate and stratify a group of residency candidates in a pass-fail system without using the bias of standardized testing.