I would have to disagree with this writer. As a writer and reader of letters of recommendation (hundreds read every year), and a core faculty member at a residency program, the letters of recommendation are one of the MOST USEFUL pieces of information that we have to differentiate our applicants. With the move to pass/fail for exams, we often interview before an actual scored exam is available. These exams ARE PREDICTIVE of whether students will struggle with standardized testing, which will occur 5 times during residency (ITEs + Step 3 + Boards). This is an important consideration for us as a program. Additionally, applications often include very similar lists of charitable activities, volunteering, and clinical experience that make it extremely difficult to stratify them. Personal statements are sometimes very helpful, but often equally boilerplate.
Despite bias that certainly occurs by the letter writers, even bad letter writers can give useful information even when they don't know they are doing so. Short ambiguous letters may indicate an unengaged student, boilerplate letters are often correlated with average, and highly personalized letters appear when a preceptor was highly involved with the student, whether their impression was good or bad.
Sometimes these are helpful, sometimes they are not, but they're one of the last things we have in an ever progressive quest by schools to make everyone look exactly the same on paper, so I think they have their place for now.
With all due respect to your experience, in this particular discussion, you seem to have missed the whole point of the writer's essay.
Writer obviously underlines the fact that those LOR's once expected to reflect the students' general standing/involvement with the mentor-clinic (as you assume so by roughly classifying letters as ''short ambiguous-boilerplate-highly personalized'' indepent of their positive-negative content) fails to do so nowadays when those writing the letters hand out free personalized letters. ''they curdle and congeal into a platitudinous mush'' vividly expresses the point writer is making by describing an inflation of high regards in LOR's, effectively making those words less valuable, and in general, making LOR's less utilizable in evaluation of the student's standing.
Writer also clearly stated how at times residents accepted with perfect LOR's turn out to be ingrained with personality defects, bringing more harm than good to the team. This also correlates with another paragraph telling how medical students act deliberately obsequious to secure a good LOR, and as stated before, how mentors hand out LOR's in a less selective manner.
What you have written, on the other hand, seemed very unsettling to me. The tone implies personal statements often ''boilerplate'', which terrifyingly implies that a student's statement is not taken seriously enough, and applications include ''very similar lists of ...''. This way of thinking in evaluation just might be expected to devolve into looking for what is ''flashy'' in applications. When you disregard what you see as ''very similar'' and claim it makes harder to ''stratify'', you really serve exactly the thing you said that was harmful: ''making everyone look the same on paper'' while they obviously are not. Then you get medical students trying to APPEAR AS hotshots instead of humbly working and properly focusing on learning the practice of medicine itself in fear of being regarded as ''boilerplate'' or being lost in ''stratification''. Then you get students with dozens of publications under the belt, that were created in a time frame that makes you question your own capability, which mostly turn out to be a dishonest type of teamwork when you further question the student. Then you get 270+'s with volunteering + research but no sense of teamwork and a practically baseline EQ.
I propose reconsidering the writer's opinion in a different light.
Fully agree. Well stated. It's the only place you might pick up on the humanity of the applicant, something extremely important in a physician. Nothing is perfect but in the days of savage meritocracy this offers the possibility of humanity.
Thank you very much for your valuable insights. However, without intending to be a "wet blanket," I would like to raise a point that perhaps there is a systemic issue, one that extends far beyond the criteria for admission to medical residencies. I am unsure of your perspective on this, but there are those (myself perhaps among them) who argue that medicine, within the modern biomedical model, has prematurely specialized in the containment of increasingly sick and complex patients in hospitals. This has necessitated the creation of subspecialties, supra-specialties, ultra-specialties, and micro-specialties, which (pardon me if I offend anyone) have fragmented not only the medical episteme but also the human being in their wholeness, dividing the biological body from everything else, and creating as many specialties as there are organs, systems, and age groups—each disjointed and existing in their own worlds.
In its net effect on professional practice, this premature specialization has demonstrated an inability to address the leading causes and effects of morbidity, mortality, and disease burden, restricting research and coverage to the containment of acute events within the framework of chronic non-communicable conditions, trauma, and communicable diseases, which in turn depend on these chronic conditions. In our clinical experience in tertiary care, for instance, we have not encountered any approach that evaluates or addresses the risk factors for developing acute appendicitis, despite it being the most prevalent acute surgical event in pediatrics. This curious lack of inquiry into the implications of removing a lymphoid organ reveals a broader ignorance of its unknown effects on human well-being.
Premature specialization has not only fragmented medical knowledge; it has also eroded professionals' skills and abilities to address health problems holistically. Suppose a patient with metabolic syndrome (obesity, hypertension, hypertriglyceridemia, hyperglycemia, hyperuricemia, elevated CRP, etc.) bypasses their family doctor to see a cardiology specialist. Physicians who once focused on their patients' hearts have been displaced by cardiologists, who now concentrate solely on this organ, making it almost inaccessible for general practitioners. With the advancement of diagnostic and therapeutic knowledge and technology, cardiology itself has splintered into small parts: some specialists focus on coronary arteries, while others deal with areas like heart valves, cavity contractility, the heart's electrical conduction system, or related pathologies.
The intersection of cardiology and nutrition is not an arbitrary association but accounts for nearly 80% of today's global disease burden and mortality. Yet, what can a cardiology sub-specialist do for a patient's heart condition if its origin, prevention, mechanism, and effective treatment are tied to poor nutrition—an area outside their expertise and governed by other specialties that have established invisible yet impenetrable boundaries, based on deeply flawed epistemological foundations?
Evidence shows that such a fragmented vision and over-specialization is particularly inadequate in an increasingly complex world where, paradoxically, a few live very well while problems persist. To paraphrase the closing line from David Epstein’s book "The Range": "If you want a problem to grow and become more complex, just focus on one aspect of it."
I agree. Worthless. I would give some credence to a verbal evaluation given to me by someone I know personally and whose judgement I trust. Probably opinions from the residents who worked directly with the student would have more value than those of attendings that just made rounds with them for a few weeks.
Hmmm….not sure about this. Accepting students into med school…..doling out residency spots….and even hiring onto staff….are about predicting an individual’s future fitness in that field, and interpersonal compatibility with the incumbent group. And as someone once said, ‘humans are bad at prediction, especially about the future’. There is an element of “crap-shoot” that is unavoidable.
You could also argue how one’s ability to rote memorize the Kreb’s cycle in undergrad biochemistry adequately predicts for those things mentioned above. I’d argue it doesn’t. But it’s another small data point among many other data points, and at end of the day, it culminates in your “best guess”.
I’d argue that “references” (however flawed and biased in the ways mentioned in this OP) is one of the few channels to acquire some information on the interpersonal aspects of an applicant. Do you run the risk of hiring a good suck-up rather than a good doc? Sure. But I don’t know how eliminating this would meaningfully improve the process over the status quo.
I do agree though that the form-letter format for reference letters is a step in the wrong direction. I’d rather read about something unique that the referee observed (which doesn’t have to be in flowery prose) rather than a simple rehash of something I can get from their GPA or residency evaluations.
This took me back to my grad school letters. I had gotten 10 sealed, signed over the envelope letters from our governor who was a rising star at the time. I got sober right after undergrad - and got my first interview for grad school while at the tail end of opiate withdrawal; but I was honest about all of it. I recently did the “post retirement” clean out of my safe and found all 10 letters in a bundle. I had sadly convinced our governor what a great guy I was at the peak of my addiction; and the grad school admissions coordinator took a still shaky junkie because of true honesty. Still sober today, by the way. Letters of rec indeed SHOULD be meaningless. Because they are.
This reminds me of an appalling experience I had when I was a junior faculty member.
Several med students asked me to write letters of recommendation. Not a problem until this one guy asked me for a recommendation letter to several psychiatry programs. Now, I’m sensitive about who gets into psychiatry programs and why—probably numerous jokes about the effectiveness and purpose of psychiatry over the years had contributed to that sensitivity.
I wrote the recommendation letter and he asked to see a copy. It was a good letter as those letters go, so I didn’t see a problem.
He came back with an EDITED version of the letter!! I was furious, and told him that not only was I not going to use his edited version but that I wasn’t going to send out the original at all.
He had always been obsequious—one of the annoying things about him. With my refusal, I thought he was going to kneel before me begging me to understand WHY his version WAS BETTER!! There was no apology, and NO sense that he had perhaps overstepped.
Finally, I told him (at a high volume) to get out of my office RIGHT NOW or I would be calling security. He continued to beseech me calling me by my first name instead of calling me Dr. Wall.
I walked to the door, opened it and pointed out into the hall. “NOW,” I said, and he left, but not without glancing back with reproachful and sorrowful looks.
If you did that now as a junior faculty member, the student would likely report you for harassment and accuse you of being a bad teacher. The bad apples have always tried to blame others but now they know they will get a supportive response from the medical schools who are fearful of administrative complaints, EEO and lawsuits.
The PPV is low, but the NPV is high if two or more of the letters have the tells most folks recognize as mediocrity: "competent job," "was always present," "participated in rounds," etc. If culling the bottom 5% is an LOR function, they are effective. However, as you allude to above, is the enterprise juice worth the squeeze, and are there other means to ascertain the same info.
Okay but with med school grades increasingly pass fail, including USMLE part 1, evaluations scrutinized for gender or racial bias what does “evaluation and recommendation” mean anymore? I can’t complain about a system where an older physician advocates enthusiastically for a younger one. Bring on the hyperbolic superlatives. If I have more questions for the candidate it will be in an interview
Remember this one? Lake Wobegon, “where all the women are strong, all the men are good-looking, and all the children are above average.” I would say most such letters are difficult to interpret unless one knows the author and has read previous letters written by them for comparison. This essay also reminds me of an opinion article in the NEJM many years ago on the worthlessness of the Deans Letter.
I have found the best indication of worthiness can be gleaned by asking a supervisor this one question: Would you hire this person...it is a yes/no question and covers every aspect you need to know. If they give a 'qualified' response, the answer is no.
When I write a letter I try to include concrete details to support the effusive praise. I try to include areas that I thought were important, going down a list that includes fund of knowledge, relation w patients and families, whether they are liked by other colleagues, efficiency w tasks, ability to synthesize complex information, etc.
I also volunteer to write writers for students I find exceptional, and declined when asked by students who weren’t bad but weren’t great.
I think if everyone did this then the letter could be, at least, less useless.
Unfortunately all schools have evolved from trying to teach and develop people . Now its all about the money what’s worse my Father was correct again it always is about the money
I would have to disagree with this writer. As a writer and reader of letters of recommendation (hundreds read every year), and a core faculty member at a residency program, the letters of recommendation are one of the MOST USEFUL pieces of information that we have to differentiate our applicants. With the move to pass/fail for exams, we often interview before an actual scored exam is available. These exams ARE PREDICTIVE of whether students will struggle with standardized testing, which will occur 5 times during residency (ITEs + Step 3 + Boards). This is an important consideration for us as a program. Additionally, applications often include very similar lists of charitable activities, volunteering, and clinical experience that make it extremely difficult to stratify them. Personal statements are sometimes very helpful, but often equally boilerplate.
Despite bias that certainly occurs by the letter writers, even bad letter writers can give useful information even when they don't know they are doing so. Short ambiguous letters may indicate an unengaged student, boilerplate letters are often correlated with average, and highly personalized letters appear when a preceptor was highly involved with the student, whether their impression was good or bad.
Sometimes these are helpful, sometimes they are not, but they're one of the last things we have in an ever progressive quest by schools to make everyone look exactly the same on paper, so I think they have their place for now.
With all due respect to your experience, in this particular discussion, you seem to have missed the whole point of the writer's essay.
Writer obviously underlines the fact that those LOR's once expected to reflect the students' general standing/involvement with the mentor-clinic (as you assume so by roughly classifying letters as ''short ambiguous-boilerplate-highly personalized'' indepent of their positive-negative content) fails to do so nowadays when those writing the letters hand out free personalized letters. ''they curdle and congeal into a platitudinous mush'' vividly expresses the point writer is making by describing an inflation of high regards in LOR's, effectively making those words less valuable, and in general, making LOR's less utilizable in evaluation of the student's standing.
Writer also clearly stated how at times residents accepted with perfect LOR's turn out to be ingrained with personality defects, bringing more harm than good to the team. This also correlates with another paragraph telling how medical students act deliberately obsequious to secure a good LOR, and as stated before, how mentors hand out LOR's in a less selective manner.
What you have written, on the other hand, seemed very unsettling to me. The tone implies personal statements often ''boilerplate'', which terrifyingly implies that a student's statement is not taken seriously enough, and applications include ''very similar lists of ...''. This way of thinking in evaluation just might be expected to devolve into looking for what is ''flashy'' in applications. When you disregard what you see as ''very similar'' and claim it makes harder to ''stratify'', you really serve exactly the thing you said that was harmful: ''making everyone look the same on paper'' while they obviously are not. Then you get medical students trying to APPEAR AS hotshots instead of humbly working and properly focusing on learning the practice of medicine itself in fear of being regarded as ''boilerplate'' or being lost in ''stratification''. Then you get students with dozens of publications under the belt, that were created in a time frame that makes you question your own capability, which mostly turn out to be a dishonest type of teamwork when you further question the student. Then you get 270+'s with volunteering + research but no sense of teamwork and a practically baseline EQ.
I propose reconsidering the writer's opinion in a different light.
Fully agree. Well stated. It's the only place you might pick up on the humanity of the applicant, something extremely important in a physician. Nothing is perfect but in the days of savage meritocracy this offers the possibility of humanity.
The paragraph containing "My best student in 20 years", and "A True Star", reminded me of the "story":
Young Man [to clerk]: Do you have any cards that say 'To the only Woman I ever loved?"
Clerk: "Yes, we do."
Young Man: "Great, I'll take four of them!"
Thanks for your good posts.
Dear Dr. Cifu,
Thank you very much for your valuable insights. However, without intending to be a "wet blanket," I would like to raise a point that perhaps there is a systemic issue, one that extends far beyond the criteria for admission to medical residencies. I am unsure of your perspective on this, but there are those (myself perhaps among them) who argue that medicine, within the modern biomedical model, has prematurely specialized in the containment of increasingly sick and complex patients in hospitals. This has necessitated the creation of subspecialties, supra-specialties, ultra-specialties, and micro-specialties, which (pardon me if I offend anyone) have fragmented not only the medical episteme but also the human being in their wholeness, dividing the biological body from everything else, and creating as many specialties as there are organs, systems, and age groups—each disjointed and existing in their own worlds.
In its net effect on professional practice, this premature specialization has demonstrated an inability to address the leading causes and effects of morbidity, mortality, and disease burden, restricting research and coverage to the containment of acute events within the framework of chronic non-communicable conditions, trauma, and communicable diseases, which in turn depend on these chronic conditions. In our clinical experience in tertiary care, for instance, we have not encountered any approach that evaluates or addresses the risk factors for developing acute appendicitis, despite it being the most prevalent acute surgical event in pediatrics. This curious lack of inquiry into the implications of removing a lymphoid organ reveals a broader ignorance of its unknown effects on human well-being.
Premature specialization has not only fragmented medical knowledge; it has also eroded professionals' skills and abilities to address health problems holistically. Suppose a patient with metabolic syndrome (obesity, hypertension, hypertriglyceridemia, hyperglycemia, hyperuricemia, elevated CRP, etc.) bypasses their family doctor to see a cardiology specialist. Physicians who once focused on their patients' hearts have been displaced by cardiologists, who now concentrate solely on this organ, making it almost inaccessible for general practitioners. With the advancement of diagnostic and therapeutic knowledge and technology, cardiology itself has splintered into small parts: some specialists focus on coronary arteries, while others deal with areas like heart valves, cavity contractility, the heart's electrical conduction system, or related pathologies.
The intersection of cardiology and nutrition is not an arbitrary association but accounts for nearly 80% of today's global disease burden and mortality. Yet, what can a cardiology sub-specialist do for a patient's heart condition if its origin, prevention, mechanism, and effective treatment are tied to poor nutrition—an area outside their expertise and governed by other specialties that have established invisible yet impenetrable boundaries, based on deeply flawed epistemological foundations?
Evidence shows that such a fragmented vision and over-specialization is particularly inadequate in an increasingly complex world where, paradoxically, a few live very well while problems persist. To paraphrase the closing line from David Epstein’s book "The Range": "If you want a problem to grow and become more complex, just focus on one aspect of it."
Kind regards,
Jairo Echeverry-Raad
I agree. Worthless. I would give some credence to a verbal evaluation given to me by someone I know personally and whose judgement I trust. Probably opinions from the residents who worked directly with the student would have more value than those of attendings that just made rounds with them for a few weeks.
Hmmm….not sure about this. Accepting students into med school…..doling out residency spots….and even hiring onto staff….are about predicting an individual’s future fitness in that field, and interpersonal compatibility with the incumbent group. And as someone once said, ‘humans are bad at prediction, especially about the future’. There is an element of “crap-shoot” that is unavoidable.
You could also argue how one’s ability to rote memorize the Kreb’s cycle in undergrad biochemistry adequately predicts for those things mentioned above. I’d argue it doesn’t. But it’s another small data point among many other data points, and at end of the day, it culminates in your “best guess”.
I’d argue that “references” (however flawed and biased in the ways mentioned in this OP) is one of the few channels to acquire some information on the interpersonal aspects of an applicant. Do you run the risk of hiring a good suck-up rather than a good doc? Sure. But I don’t know how eliminating this would meaningfully improve the process over the status quo.
I do agree though that the form-letter format for reference letters is a step in the wrong direction. I’d rather read about something unique that the referee observed (which doesn’t have to be in flowery prose) rather than a simple rehash of something I can get from their GPA or residency evaluations.
This took me back to my grad school letters. I had gotten 10 sealed, signed over the envelope letters from our governor who was a rising star at the time. I got sober right after undergrad - and got my first interview for grad school while at the tail end of opiate withdrawal; but I was honest about all of it. I recently did the “post retirement” clean out of my safe and found all 10 letters in a bundle. I had sadly convinced our governor what a great guy I was at the peak of my addiction; and the grad school admissions coordinator took a still shaky junkie because of true honesty. Still sober today, by the way. Letters of rec indeed SHOULD be meaningless. Because they are.
This reminds me of an appalling experience I had when I was a junior faculty member.
Several med students asked me to write letters of recommendation. Not a problem until this one guy asked me for a recommendation letter to several psychiatry programs. Now, I’m sensitive about who gets into psychiatry programs and why—probably numerous jokes about the effectiveness and purpose of psychiatry over the years had contributed to that sensitivity.
I wrote the recommendation letter and he asked to see a copy. It was a good letter as those letters go, so I didn’t see a problem.
He came back with an EDITED version of the letter!! I was furious, and told him that not only was I not going to use his edited version but that I wasn’t going to send out the original at all.
He had always been obsequious—one of the annoying things about him. With my refusal, I thought he was going to kneel before me begging me to understand WHY his version WAS BETTER!! There was no apology, and NO sense that he had perhaps overstepped.
Finally, I told him (at a high volume) to get out of my office RIGHT NOW or I would be calling security. He continued to beseech me calling me by my first name instead of calling me Dr. Wall.
I walked to the door, opened it and pointed out into the hall. “NOW,” I said, and he left, but not without glancing back with reproachful and sorrowful looks.
Wow, just wow!!
I don’t think I wrote another letter after that.
If you did that now as a junior faculty member, the student would likely report you for harassment and accuse you of being a bad teacher. The bad apples have always tried to blame others but now they know they will get a supportive response from the medical schools who are fearful of administrative complaints, EEO and lawsuits.
Well, I finally did sue two egregious doctors for sexual harassment and it was settled out of court in my favor.
And that was not the first episode of sexual harassment that I endured in my 30 year medical career.
I’d counter-sue and I would do everything in my power to make such an upstart student regret the day he’d been born!!
Enough is enough!!
Sexual harassment is real and these days pretty clearly defined so your theoretical student wouldn’t win.
The PPV is low, but the NPV is high if two or more of the letters have the tells most folks recognize as mediocrity: "competent job," "was always present," "participated in rounds," etc. If culling the bottom 5% is an LOR function, they are effective. However, as you allude to above, is the enterprise juice worth the squeeze, and are there other means to ascertain the same info.
More anecdotal evidence of trading money for credentials to elite status.
Okay but with med school grades increasingly pass fail, including USMLE part 1, evaluations scrutinized for gender or racial bias what does “evaluation and recommendation” mean anymore? I can’t complain about a system where an older physician advocates enthusiastically for a younger one. Bring on the hyperbolic superlatives. If I have more questions for the candidate it will be in an interview
Remember this one? Lake Wobegon, “where all the women are strong, all the men are good-looking, and all the children are above average.” I would say most such letters are difficult to interpret unless one knows the author and has read previous letters written by them for comparison. This essay also reminds me of an opinion article in the NEJM many years ago on the worthlessness of the Deans Letter.
I have found the best indication of worthiness can be gleaned by asking a supervisor this one question: Would you hire this person...it is a yes/no question and covers every aspect you need to know. If they give a 'qualified' response, the answer is no.
When I write a letter I try to include concrete details to support the effusive praise. I try to include areas that I thought were important, going down a list that includes fund of knowledge, relation w patients and families, whether they are liked by other colleagues, efficiency w tasks, ability to synthesize complex information, etc.
I also volunteer to write writers for students I find exceptional, and declined when asked by students who weren’t bad but weren’t great.
I think if everyone did this then the letter could be, at least, less useless.
Unfortunately all schools have evolved from trying to teach and develop people . Now its all about the money what’s worse my Father was correct again it always is about the money
ALWAYS.