A prominent medical journal embarrrasses itself by publishing a flawed study from the American Board of Internal Medicine. Critical appraisal lessons are obvious.
I took my first IM boards in 1992 and my first subspecialty boards in 1994. 31 years and three recertifications later and I am due again. Now that I am in my 60's, I have no interest in taking another 8 hour exam to "prove" myself worthy of practicing. I currently practice medical center occupational health exclusively and serve as a senior medical director. The ID boards have little relevance to my current practice, so I will be sitting them out and retiring. Enough is enough!
I am a 66 yo Hospitalist who is thinking about getting out. My next accreditation deadline is 2026. I was working full time until recently and my current work performance currently would be called out rapidly by my colleagues, nurses, sub specialists and length of stay concerned administrators nearly immediately if not adequate. Yet ABIM holds the expensive final hammer. How screwed up is that? They could avoid at least pretentious easily invalidated attempts to prove their own worth.
Pediatricians have faced the same problems. Thankfully, most older pediatricians grandfathered in. I had to take pediatric pulmonology boards every seven years which varied from in-person to take home to adding clinical goals over time. Cost of renewing board certification has skyrocketed over the years. Many of my friends sat on the board and contributed questions but the whole enterprise wreaked of self-aggrandizement.
Absolutely correct. More physicians are now waking up to the gross deception Big Medicine and Big Pharma have on the practice of medicine. Bogus research, deceptive journalism, sponsoring politicians, paying for 70% of the FDA budget, and direct advertisement. Follow the money. This has got to stop!
ABIM may be the most egregious but they are not alone. Many of the governing bodies force physicians into paid practice questions and restricted CME requirements with the theory that they keep people current. I don't believe they do. I think that those who are motivated to stay current will do so regardless of whether CMEs are attached to their learning. Whereas those who don't care to stay current will always have a way to complete CMEs without actually learning. Time on task is not the same as learning.
Unlike the FDA, which really ought to have a fiduciary duty to not license/approve/permit devices/therapies in the absence of proper evidence of efficacy, I suppose no such fiduciary duty exists for journal editors in terms of requiring the papers they publish to consist of proper evidence and/or rigorous science…or perhaps even just the absence of overt bias and conflict of interest . But you would’ve thought that a modicum of self respect would compel at least such a standard among these editors, even if only on a voluntary basis. Sadly, it appears “you would be wrong”.
I wonder when docs will throw up their hands in disgust and refuse to take one. more. darn. test.
Multiple choice tests test only one kind of knowledge—the ability to take multiple choice tests.
If one practices in an area for a few years or less, one rather quickly finds out which docs can be trusted and which have accumulated a legacy of mistakes or other bad behavior. This is usually transmitted by word of mouth.
There are few experts in medical practice since what you call medical practice is extremely narrow minded and narrowly taught and pays no attention to root causes or making people healthy without drugs.
Excellent article. IN ADDITION to all that..... the board exams, in my experience, do not adequately assess and reflect what a practicing physician needs to know. I am double boarded in IM and CCM. Have re-certified twice (last time 2017 IM and 2020 CCM). Passed every time - but the tests and questions were ridiculous. PAP SMEARS, CHEMOTHERAPY AND ORGAN TRANSPLANT REGIMENS, etc.... These exams need to assess the expected standard of care practice of each specialty - not be some BS power trip for the academic elitist question writer's. Anyway, I am considering pursuing NBPAS certification for myself and hospital. I love what I do and I'm good at it.
I have taken 3 of the 10 year exams. Absolute waste of time and money, especially because one has to take the day off to test.
The Texas Medical Association spoke against allowing competing MOC organizations, hurting its own members. The conflicted leaders of that organization were allowed to influence and make all the policy decisions.
I am board certified but also am NBPAS certified—which turned out to be a far more reasonable process more reflective, I thought, of my actual experience and competence.
The entire US economy has a monopoly problem. I’ve been saying for years that we no longer have a capitalist economy we have turned into a monopolistic economy! We need universal healthcare and a FTC that keeps Lina khan long enough to get the job done of breaking up these monopolies!
One of the major problems with this study is having categorized continuous data. Although commonly done, this practice should be avoided. Categorizing continuous data results in substantial loss of statistical power. It also causes boundary problems. Are the individuals just on one side or another of the dividing line between one quartile or another really all that different?
A third problem with categorization and comparing a high or low category with other categories is the undue influence of outliers.
In this analysis the low quartile 7-day mortality was 53.0/1000. This was compared to Q2: 49.1/1000, Q3: 49.1/1000 and Q4 48.9/1000. The 3 higher quartiles had essentially the same mortality. If there was an effect of the intervention it should have been "dose-dependent" with progressively lower mortality with increasing quartiles. This was not seen. Given that only the lower quartile had an effect, it seems like it had outliers that explain the statistical outcome.
I bet that if a proper regression was done using continuous instead of categorical variables no statistically significant difference would be found.
Herer is a nice review summarizing why categorization of variables should not be done.
Why Quantitative Variables Should Not Be Recoded as Categorical
The real issue that ABIM has to answer with the LKA/10-year exam - can they prove that someone who "fails" their MCQ test is not competent to practice in their specialty? The purpose of their process as currently structured is not to correlate score with competency. It is to identify the bottom 2-10% of test takers and fail them. This is the critical fallacy of their MOC process.
Does it need to be dose dependent? Beyond a certain level of knowledge, real world performance equilibrates and we see diminishing returns at the higher quartiles. The outliers you mention are the lowest decile hospitalists that are causing a disproportionate number of bad outcomes. I'm not saying that's the correct interpretation since the whole thing could be a mirage as JMM is implying.
I'm using common sense reasoning not statistical reasoning here. Forgive me, gurus.
Nice appraisal, and good points. I would argue MOST published research is marketing disguised as science (not just pharmaceutical) and suggest that the overwhelming majority of observational studies are as reliable as a coin flip. Even when they haven’t been gamed and engineered the way this one was. Thank you for this.
You said that a drug study with 3 of 4 authors working for the drug company would get rejected. I don't share your confidence. When you have ACIP members voting for vaccines on which they hold a patent, no level of corruption is unattainable.
I took my first IM boards in 1992 and my first subspecialty boards in 1994. 31 years and three recertifications later and I am due again. Now that I am in my 60's, I have no interest in taking another 8 hour exam to "prove" myself worthy of practicing. I currently practice medical center occupational health exclusively and serve as a senior medical director. The ID boards have little relevance to my current practice, so I will be sitting them out and retiring. Enough is enough!
I wonder how many other studies fall into that category? Anyone notice other areas?
I am a 66 yo Hospitalist who is thinking about getting out. My next accreditation deadline is 2026. I was working full time until recently and my current work performance currently would be called out rapidly by my colleagues, nurses, sub specialists and length of stay concerned administrators nearly immediately if not adequate. Yet ABIM holds the expensive final hammer. How screwed up is that? They could avoid at least pretentious easily invalidated attempts to prove their own worth.
Pediatricians have faced the same problems. Thankfully, most older pediatricians grandfathered in. I had to take pediatric pulmonology boards every seven years which varied from in-person to take home to adding clinical goals over time. Cost of renewing board certification has skyrocketed over the years. Many of my friends sat on the board and contributed questions but the whole enterprise wreaked of self-aggrandizement.
This is a fantastic piece. I am enrolled in the LKA, and I do like it. It does teach a few valuable things, and has a lot of stupid things too.
And, we should conduct our own RCT: have docs enroll in the study and see if being in LKA truly makes a difference.
Absolutely correct. More physicians are now waking up to the gross deception Big Medicine and Big Pharma have on the practice of medicine. Bogus research, deceptive journalism, sponsoring politicians, paying for 70% of the FDA budget, and direct advertisement. Follow the money. This has got to stop!
Very scary stuff.
ABIM may be the most egregious but they are not alone. Many of the governing bodies force physicians into paid practice questions and restricted CME requirements with the theory that they keep people current. I don't believe they do. I think that those who are motivated to stay current will do so regardless of whether CMEs are attached to their learning. Whereas those who don't care to stay current will always have a way to complete CMEs without actually learning. Time on task is not the same as learning.
Unlike the FDA, which really ought to have a fiduciary duty to not license/approve/permit devices/therapies in the absence of proper evidence of efficacy, I suppose no such fiduciary duty exists for journal editors in terms of requiring the papers they publish to consist of proper evidence and/or rigorous science…or perhaps even just the absence of overt bias and conflict of interest . But you would’ve thought that a modicum of self respect would compel at least such a standard among these editors, even if only on a voluntary basis. Sadly, it appears “you would be wrong”.
I wonder when docs will throw up their hands in disgust and refuse to take one. more. darn. test.
Multiple choice tests test only one kind of knowledge—the ability to take multiple choice tests.
If one practices in an area for a few years or less, one rather quickly finds out which docs can be trusted and which have accumulated a legacy of mistakes or other bad behavior. This is usually transmitted by word of mouth.
It seems to be mostly reliable.
There are few experts in medical practice since what you call medical practice is extremely narrow minded and narrowly taught and pays no attention to root causes or making people healthy without drugs.
Excellent article. IN ADDITION to all that..... the board exams, in my experience, do not adequately assess and reflect what a practicing physician needs to know. I am double boarded in IM and CCM. Have re-certified twice (last time 2017 IM and 2020 CCM). Passed every time - but the tests and questions were ridiculous. PAP SMEARS, CHEMOTHERAPY AND ORGAN TRANSPLANT REGIMENS, etc.... These exams need to assess the expected standard of care practice of each specialty - not be some BS power trip for the academic elitist question writer's. Anyway, I am considering pursuing NBPAS certification for myself and hospital. I love what I do and I'm good at it.
I have taken 3 of the 10 year exams. Absolute waste of time and money, especially because one has to take the day off to test.
The Texas Medical Association spoke against allowing competing MOC organizations, hurting its own members. The conflicted leaders of that organization were allowed to influence and make all the policy decisions.
Shameful.
I am board certified but also am NBPAS certified—which turned out to be a far more reasonable process more reflective, I thought, of my actual experience and competence.
The entire US economy has a monopoly problem. I’ve been saying for years that we no longer have a capitalist economy we have turned into a monopolistic economy! We need universal healthcare and a FTC that keeps Lina khan long enough to get the job done of breaking up these monopolies!
One of the major problems with this study is having categorized continuous data. Although commonly done, this practice should be avoided. Categorizing continuous data results in substantial loss of statistical power. It also causes boundary problems. Are the individuals just on one side or another of the dividing line between one quartile or another really all that different?
A third problem with categorization and comparing a high or low category with other categories is the undue influence of outliers.
In this analysis the low quartile 7-day mortality was 53.0/1000. This was compared to Q2: 49.1/1000, Q3: 49.1/1000 and Q4 48.9/1000. The 3 higher quartiles had essentially the same mortality. If there was an effect of the intervention it should have been "dose-dependent" with progressively lower mortality with increasing quartiles. This was not seen. Given that only the lower quartile had an effect, it seems like it had outliers that explain the statistical outcome.
I bet that if a proper regression was done using continuous instead of categorical variables no statistically significant difference would be found.
Herer is a nice review summarizing why categorization of variables should not be done.
Why Quantitative Variables Should Not Be Recoded as Categorical
https://www.scirp.org/journal/paperinformation?paperid=93794
The real issue that ABIM has to answer with the LKA/10-year exam - can they prove that someone who "fails" their MCQ test is not competent to practice in their specialty? The purpose of their process as currently structured is not to correlate score with competency. It is to identify the bottom 2-10% of test takers and fail them. This is the critical fallacy of their MOC process.
Does it need to be dose dependent? Beyond a certain level of knowledge, real world performance equilibrates and we see diminishing returns at the higher quartiles. The outliers you mention are the lowest decile hospitalists that are causing a disproportionate number of bad outcomes. I'm not saying that's the correct interpretation since the whole thing could be a mirage as JMM is implying.
I'm using common sense reasoning not statistical reasoning here. Forgive me, gurus.
Nice appraisal, and good points. I would argue MOST published research is marketing disguised as science (not just pharmaceutical) and suggest that the overwhelming majority of observational studies are as reliable as a coin flip. Even when they haven’t been gamed and engineered the way this one was. Thank you for this.
You said that a drug study with 3 of 4 authors working for the drug company would get rejected. I don't share your confidence. When you have ACIP members voting for vaccines on which they hold a patent, no level of corruption is unattainable.