Excellent summary of a complicated topic regarding the RCT, medicine's major tool for determining the true benefit and risks of proposed interventions, devices, drugs, lifestyle changes, and surgery.
I feel critical analysis of the scientific literature should be a required course in high school and I nominate Dr. Cifu to organize its establishment.
This was so good. I’m glad you plan to put this out there for more folks can critically look at themselves and others’ practice skillsets as the literature is more and more catering to those that fund rather than results that matter. I learned a lot here!
Excellent introduction. Knowledge of all the factors that introduce bias reinforces the concept that the small differences that are frequently cited as being "statistically significant" can be ignored. Good point about how the language used by the authors of a study can be subtle indicators of bias as well.
Good review, and some of us (speaking for myself) learned a couple of new points. Here are a few additional points:
- Untoward effects are typically understated to make the intervention look better. E.g. The benefit of the intervention is often reported as a relative risk reduction, but the harms reported as an absolute risk increase.
- The authors may focus on a single clinical outcome such as fewer acute cardiac events, but de-emphasis a more important one, such as all-cause mortality. My favorite illustration of this is the Helsinki Heart Study demonstrating reduced MI, but increased all-cause mortality from gemfibrizil.
- "P hacking": The researchers report a benefit for a subgroup of subjects, among a vast number of possibilities, especially if not mentioned in the study protocol. They really really want to show a benefit of the intervention for someone - anyone!
- My favorite: The study authors' choice of language that subtly convey the desirability of an intervention. E.g. The phrase "horrors of covid" or any use of the word "crisis" has an emotional appeal to imply that the tested intervention is desirable. Or reporting an untoward effect by the phrase "Only 1% of subjects were hospitalized for GI bleeding.". The word "Only" is meant to bias the reader to think the negative effect is trivial.
Perhaps Dr Cifu is planning a column on the ways by which researchers frame a study to promote an intervention rather than report data and let the reader decide its benefits and harms. To me, these practices are pervasive in the literature and undermine the conclusions of many, if not most, studies.
I am happy to see someone else pointing out the blatant bias in the report on the Helsinki trial. I reviewed over 500 articles on cholesterol and cardiovascular disease before writing my book, The Cholesterol Delusion. I included an appendix on how to read a medical journal report that was only a few pages. Then I reprinted both the Lipid Research Clinics trial and the Helsinki trial followed by a point by point critique of both. I picked these two studies because they were the ones most often cited by the proponents of the cholesterol theory as having provided the definitive proof of the theory. The initial paragraphs in the part titled How to Read and Interpret the Helsinki Heart Study stated: "As mentioned before, the Helsinki Study was a virtual clone of the Lipid Research Clinics (LRC) trial. Reading the two sequentially, one is struck by the similarities. The structures of both are about the same and even the language is remarkably similar.......The results are summarized in an interesting fashion. We are given the results in terms of number per 1000 rather than simple percentages. Because all one needs to do is move the decimal point one space to the left to get the percentage, one may wonder why this peculiar method is employed. Remember the principle that when results are presented in a confusing or unnecessarily complex way, the purpose is to mislead the reader. In this case , it is likely that the figures are given this way to make a very small difference appear larger; thus 2.7% and 4.1% are listed at 27 and 41per thousand, respectively."
Then, after showing that the study reported differences in absolute risk of only 0.1% to 1.4%, I recommended to the readers: You be the judge of the veracity of their concluding paragraph:
"In conclusion, specifically modifying the lipoprotein profile with gemfibrozil resulted in a MARKED reduction in the incidence of coronary heart disease... and furnished additional and CONCLUSIVE evidence of the role of lipid modification in preventing coronary heart disease."
Excellent. Thanks so much. After this series we are planning one on "how to not get fooled by the medical literature" and all your points fit PERFECTLY there.
Excellent summary of a complicated topic regarding the RCT, medicine's major tool for determining the true benefit and risks of proposed interventions, devices, drugs, lifestyle changes, and surgery.
I feel critical analysis of the scientific literature should be a required course in high school and I nominate Dr. Cifu to organize its establishment.
This was so good. I’m glad you plan to put this out there for more folks can critically look at themselves and others’ practice skillsets as the literature is more and more catering to those that fund rather than results that matter. I learned a lot here!
I also immediately check who funded the study.
Excellent introduction. Knowledge of all the factors that introduce bias reinforces the concept that the small differences that are frequently cited as being "statistically significant" can be ignored. Good point about how the language used by the authors of a study can be subtle indicators of bias as well.
Good review, and some of us (speaking for myself) learned a couple of new points. Here are a few additional points:
- Untoward effects are typically understated to make the intervention look better. E.g. The benefit of the intervention is often reported as a relative risk reduction, but the harms reported as an absolute risk increase.
- The authors may focus on a single clinical outcome such as fewer acute cardiac events, but de-emphasis a more important one, such as all-cause mortality. My favorite illustration of this is the Helsinki Heart Study demonstrating reduced MI, but increased all-cause mortality from gemfibrizil.
- "P hacking": The researchers report a benefit for a subgroup of subjects, among a vast number of possibilities, especially if not mentioned in the study protocol. They really really want to show a benefit of the intervention for someone - anyone!
- My favorite: The study authors' choice of language that subtly convey the desirability of an intervention. E.g. The phrase "horrors of covid" or any use of the word "crisis" has an emotional appeal to imply that the tested intervention is desirable. Or reporting an untoward effect by the phrase "Only 1% of subjects were hospitalized for GI bleeding.". The word "Only" is meant to bias the reader to think the negative effect is trivial.
Perhaps Dr Cifu is planning a column on the ways by which researchers frame a study to promote an intervention rather than report data and let the reader decide its benefits and harms. To me, these practices are pervasive in the literature and undermine the conclusions of many, if not most, studies.
I am happy to see someone else pointing out the blatant bias in the report on the Helsinki trial. I reviewed over 500 articles on cholesterol and cardiovascular disease before writing my book, The Cholesterol Delusion. I included an appendix on how to read a medical journal report that was only a few pages. Then I reprinted both the Lipid Research Clinics trial and the Helsinki trial followed by a point by point critique of both. I picked these two studies because they were the ones most often cited by the proponents of the cholesterol theory as having provided the definitive proof of the theory. The initial paragraphs in the part titled How to Read and Interpret the Helsinki Heart Study stated: "As mentioned before, the Helsinki Study was a virtual clone of the Lipid Research Clinics (LRC) trial. Reading the two sequentially, one is struck by the similarities. The structures of both are about the same and even the language is remarkably similar.......The results are summarized in an interesting fashion. We are given the results in terms of number per 1000 rather than simple percentages. Because all one needs to do is move the decimal point one space to the left to get the percentage, one may wonder why this peculiar method is employed. Remember the principle that when results are presented in a confusing or unnecessarily complex way, the purpose is to mislead the reader. In this case , it is likely that the figures are given this way to make a very small difference appear larger; thus 2.7% and 4.1% are listed at 27 and 41per thousand, respectively."
Then, after showing that the study reported differences in absolute risk of only 0.1% to 1.4%, I recommended to the readers: You be the judge of the veracity of their concluding paragraph:
"In conclusion, specifically modifying the lipoprotein profile with gemfibrozil resulted in a MARKED reduction in the incidence of coronary heart disease... and furnished additional and CONCLUSIVE evidence of the role of lipid modification in preventing coronary heart disease."
(emphasis added).
Excellent. Thanks so much. After this series we are planning one on "how to not get fooled by the medical literature" and all your points fit PERFECTLY there.
Thank you. Looking forward to this posting.