Sensible medicine appreciates that treating patients is a " regimen" , a combination of treatments, drugs, interventions and not just one drug . For sure RCTs are far from this reality , so let Doctors be doctors , treat pateints , in the real environment ofreal medicine
>>When was the last time the FDA approved a novel drug because it demonstrated superiority in a “pivotal systematic review?” That would be nonsensical, because our regulatory agencies (correctly) require experimental evidence and (correctly) interprets the evidence themselves.
Really valid points by the author, but also somewhat overstated.
The pyramid is a kind of lens in and of itself, and it’s important to be mindful of that and not overestimate the value of any study based on an abstract ranking structure. That said, it seems that what is really being pointed out here is not a deficiency of the pyramid, but the vital importance of the work of the authors of any study or review. Bad work can always be misleading and distorting; good work can be informative, pointed, and bring clarity. The failings of bad work don’t necessarily invalidate the premises of a hierarchy of evidence. In the end, it’s really about the quality of the work, and the duty of the community to scrutinize and critique all work, no matter the journal, or authors, or funders, or position in the pyramid.
The last 3 years have taught me is not to trust RCTs and Big pharma. I look for doctors who practice medicine, are in the trenches treating successfully thousands of patients ( not in hospitals - bonuses for Remdesivir). Medicine has been corrupted by the pharmaceutical industry, in collaboration with doctors and medical systems that put profits over patient health and RCTs is their tool of control and corruption.
“... systematic reviews and meta analyses (SRMAs) are emphatically not the highest level of evidence. … but rather a lens through which we view actual evidence from RCTs and observational studies. They depend upon the quality of the review process (many are very low quality) and the evidence that they appraise ... many RCTs are poorly conducted, underpowered, and susceptible to bias from the pharmaceutical industry that (generally) runs them … And most of all, read and judge each paper by its individual merits not by its strata on a colorful triangle. “
. . .
Agreed. I would also add that it's not always wise to assume just because a RCT appears in a “high impact” journal (e.g. NEMJ ), that it must be true.
And, Mills's IVM Together Trial is an example of a “poorly conducted, underpowered, and susceptible to bias” RCT (ironically, it won “Trial of the Year” by the Society of Clinical Trials for 2022!).
See https://pierrekory.substack.com/p/the-publication-of-fraudulent-ivermectin : “I have written a ton about this bad boy, compiling not only my own analysis, but those of my deeply expert colleagues. My three part series on just this one trial is here, here, and here. … If you prefer a more “sober”, “objective” analysis, please read this review of the trial published by the Cato Institute. It is masterful, and almost looks like they re-wrote my own review but in a more “academic” fashion and tone.”
The problem with this world view is that the point of view of a single individual is more likely to be biased than a group of individuals. To choose a recent example: this heuristic would make ivermectin look like a fantastic treatment for COVID-19.
Excellent post, thank you. I train new medical writers and this has given me some ideas on teaching them to better evaluate published research.
Overall, I think regulatory authorities agree with you, except they generally wil not recognize or weight significantly the contributions from observational trials, even in settings where an RCT is not feasible or ethical. Despite real world evidence initiatives at regulators, the push for "un-runnable" RCTs stall potential new therapies. So, there remains the challenge of properly putting observational data and RWE in the proper context.
Well written and thought-provoking. But probably the best to obtaining more "real" evidence, at least for relatively common conditions, is to run many more large, real-world national or multinational clinical trials independent of industry. They also need to be independent of Key Opinion Leaders and to guarantee appropriate access to individual patient data for qualified groups to re-validate (or invalidate) the reported results. They should use the most appropriate active comparators and at times placebos to re-validate efficacy/effectiveness where it may have been "faked". If governments supported this, we would save tens of billions/year globally and have better outcomes. Obvious candidates now in adult medicine include direct comparisons of SGLT2-i, GLP-1 agonists, insulins, drugs for heart failure, anticoagulants, new antihypertensives, treatments for Covid19, and in future for tropical diseases or TB. For TB, such trials have already shown how to reduce length of treatment.
For conditions where the typical "outcomes" are highly subjective (psychiatry, pain) we need RCTs that measure objective function (e.g. ability to complete education, to work, social function like family life), as well as objective measures of health (medical visits, costs, hospitalization, all cause mortality). This will never come from industry or KOLs because the risks of "failure" (losing to a competitor, or losing totally) in a RCT are too high. It cannot come from "disease groups" either, because they are compromised by the framework they've often been co-opted to use in viewing the world. Think: dementia drugs, diabetes treatment, anything psychiatric.
Does any Sensible Medicine reader disagree? Why do so few governments support spending public money to learn how best to spend health care dollars?
Tom Perry MD, FRCPC (internal medicine/clinical pharmacology, University of BC)
I think animal studies are often best. It is difficult or impossible to really do science with actual people. I look at good animal studies are the gold standard. Most human studies are worthless because they are so easily manipulated. The meta studies racket and the peer review racket make sure that the drug companies get what they want.
Sadly, judging individual studies on their merits is not something most doctors have time to do. Many physicians get their information from standard of care, as determined by panels of experts, rather than reading “unfiltered” medical literature.
In their defense- there really is too much information out there. No one can figure it all out for themselves.
Maybe, devil’s advocate here- instead of more admonitions to think critically, we need better guidance when it comes to outsourcing critical thought? Where is the practical advice for selecting one’s experts?
[Insert mandatory early 2023 statement about how in the near future AI will probably help with evaluating evidence.]
A great reflection on the evidence pyramid with relevance well beyond medicine (I am a social scientist and this applies every bit as much to my discipline).
Good posting, Mike. The bottom line is that the responsible modern physician needs to be a well-armed skeptic, skilled in the art of criticism and committed to ongoing willingness to listen to other critics. Let us consider the recent Cochrane review on the physical means to stop the spread of respiratory viruses. By definition, all studies have to be performed over time in different populations. Is it likely that a randomized person in the masking arm will act differently around others during a pandemic than a person in the control arm? Of course. There is no definitive way to control for this. Even if the subjects behave the same, how would we control for the way family members and others in their home communities will interact with them? We cannot. That is why the Cochrane Library culls the literature for many RCTs in different populations. If a hypothesis like masks effectively prevented the spread of respiratory viruses, then it should be demonstrable in different populations in different locales. If not, we consumers of the literature have to consider the limitations of the studies. In this case, it would not be entirely unreasonable for an establishment person to conclude that a definitive, pure study has never been and will never be accomplished and his/her common sense recommendation is that we should nevertheless recommend masking as many individuals in our society, for cultural reasons East Asians, seem to prefer that choice. Right now the medical establishment has not been ideally transparent about the ongoing requirement for masking in many doctors' offices, clinics and hospitals. All this makes for ongoing erosion of trust and respect for the medical establishment.
Sensible medicine appreciates that treating patients is a " regimen" , a combination of treatments, drugs, interventions and not just one drug . For sure RCTs are far from this reality , so let Doctors be doctors , treat pateints , in the real environment ofreal medicine
>>When was the last time the FDA approved a novel drug because it demonstrated superiority in a “pivotal systematic review?” That would be nonsensical, because our regulatory agencies (correctly) require experimental evidence and (correctly) interprets the evidence themselves.
*dies suddenly, laughing*
Sure they do, doc. Sure they do.
Really valid points by the author, but also somewhat overstated.
The pyramid is a kind of lens in and of itself, and it’s important to be mindful of that and not overestimate the value of any study based on an abstract ranking structure. That said, it seems that what is really being pointed out here is not a deficiency of the pyramid, but the vital importance of the work of the authors of any study or review. Bad work can always be misleading and distorting; good work can be informative, pointed, and bring clarity. The failings of bad work don’t necessarily invalidate the premises of a hierarchy of evidence. In the end, it’s really about the quality of the work, and the duty of the community to scrutinize and critique all work, no matter the journal, or authors, or funders, or position in the pyramid.
Quite like this. I do think there are problems with the idea of an evidence pyramid.
The last 3 years have taught me is not to trust RCTs and Big pharma. I look for doctors who practice medicine, are in the trenches treating successfully thousands of patients ( not in hospitals - bonuses for Remdesivir). Medicine has been corrupted by the pharmaceutical industry, in collaboration with doctors and medical systems that put profits over patient health and RCTs is their tool of control and corruption.
“... systematic reviews and meta analyses (SRMAs) are emphatically not the highest level of evidence. … but rather a lens through which we view actual evidence from RCTs and observational studies. They depend upon the quality of the review process (many are very low quality) and the evidence that they appraise ... many RCTs are poorly conducted, underpowered, and susceptible to bias from the pharmaceutical industry that (generally) runs them … And most of all, read and judge each paper by its individual merits not by its strata on a colorful triangle. “
. . .
Agreed. I would also add that it's not always wise to assume just because a RCT appears in a “high impact” journal (e.g. NEMJ ), that it must be true.
Popp's 8/21 Cohrane review of IVM is a good example of a review showing a low “quality of the review process.” See Tess Lawries rebuttal: “The Uses and Abuses of Systematic Reviews – The Case of Ivermectin in Covid-1 https://osf.io/mp4f2/ and https://pierrekory.substack.com/p/the-publication-of-fraudulent-ivermectin-da1
And, Mills's IVM Together Trial is an example of a “poorly conducted, underpowered, and susceptible to bias” RCT (ironically, it won “Trial of the Year” by the Society of Clinical Trials for 2022!).
See https://pierrekory.substack.com/p/the-publication-of-fraudulent-ivermectin : “I have written a ton about this bad boy, compiling not only my own analysis, but those of my deeply expert colleagues. My three part series on just this one trial is here, here, and here. … If you prefer a more “sober”, “objective” analysis, please read this review of the trial published by the Cato Institute. It is masterful, and almost looks like they re-wrote my own review but in a more “academic” fashion and tone.”
The problem with this world view is that the point of view of a single individual is more likely to be biased than a group of individuals. To choose a recent example: this heuristic would make ivermectin look like a fantastic treatment for COVID-19.
this is a strong take and one with which i very much agree.
"lying with meta studies" has become something of an art form and the selective assembly of poor and mediocre data gets used to swamp the good data.
it's the equivalent of badly balancing cohorts in CT and them trying to back out bias at the end with some sort of cox model.
your study is not longer a study. it's not only as good as its modeling parameters.
what you want is the best overall study done sufficiently soundly that adjustment is not needed.
that and only that is data.
the rest is estimation and skullduggery.
been seeing A LOT of this in epidemiology. they seem to use statistical tool predominantly to occulde and not to reveal.
feels like a badly broken field.
Excellent post, thank you. I train new medical writers and this has given me some ideas on teaching them to better evaluate published research.
Overall, I think regulatory authorities agree with you, except they generally wil not recognize or weight significantly the contributions from observational trials, even in settings where an RCT is not feasible or ethical. Despite real world evidence initiatives at regulators, the push for "un-runnable" RCTs stall potential new therapies. So, there remains the challenge of properly putting observational data and RWE in the proper context.
Well written and thought-provoking. But probably the best to obtaining more "real" evidence, at least for relatively common conditions, is to run many more large, real-world national or multinational clinical trials independent of industry. They also need to be independent of Key Opinion Leaders and to guarantee appropriate access to individual patient data for qualified groups to re-validate (or invalidate) the reported results. They should use the most appropriate active comparators and at times placebos to re-validate efficacy/effectiveness where it may have been "faked". If governments supported this, we would save tens of billions/year globally and have better outcomes. Obvious candidates now in adult medicine include direct comparisons of SGLT2-i, GLP-1 agonists, insulins, drugs for heart failure, anticoagulants, new antihypertensives, treatments for Covid19, and in future for tropical diseases or TB. For TB, such trials have already shown how to reduce length of treatment.
For conditions where the typical "outcomes" are highly subjective (psychiatry, pain) we need RCTs that measure objective function (e.g. ability to complete education, to work, social function like family life), as well as objective measures of health (medical visits, costs, hospitalization, all cause mortality). This will never come from industry or KOLs because the risks of "failure" (losing to a competitor, or losing totally) in a RCT are too high. It cannot come from "disease groups" either, because they are compromised by the framework they've often been co-opted to use in viewing the world. Think: dementia drugs, diabetes treatment, anything psychiatric.
Does any Sensible Medicine reader disagree? Why do so few governments support spending public money to learn how best to spend health care dollars?
Tom Perry MD, FRCPC (internal medicine/clinical pharmacology, University of BC)
IVM
Thank you, really interesting read.
I think animal studies are often best. It is difficult or impossible to really do science with actual people. I look at good animal studies are the gold standard. Most human studies are worthless because they are so easily manipulated. The meta studies racket and the peer review racket make sure that the drug companies get what they want.
Sadly, judging individual studies on their merits is not something most doctors have time to do. Many physicians get their information from standard of care, as determined by panels of experts, rather than reading “unfiltered” medical literature.
In their defense- there really is too much information out there. No one can figure it all out for themselves.
Maybe, devil’s advocate here- instead of more admonitions to think critically, we need better guidance when it comes to outsourcing critical thought? Where is the practical advice for selecting one’s experts?
[Insert mandatory early 2023 statement about how in the near future AI will probably help with evaluating evidence.]
A great reflection on the evidence pyramid with relevance well beyond medicine (I am a social scientist and this applies every bit as much to my discipline).
Good posting, Mike. The bottom line is that the responsible modern physician needs to be a well-armed skeptic, skilled in the art of criticism and committed to ongoing willingness to listen to other critics. Let us consider the recent Cochrane review on the physical means to stop the spread of respiratory viruses. By definition, all studies have to be performed over time in different populations. Is it likely that a randomized person in the masking arm will act differently around others during a pandemic than a person in the control arm? Of course. There is no definitive way to control for this. Even if the subjects behave the same, how would we control for the way family members and others in their home communities will interact with them? We cannot. That is why the Cochrane Library culls the literature for many RCTs in different populations. If a hypothesis like masks effectively prevented the spread of respiratory viruses, then it should be demonstrable in different populations in different locales. If not, we consumers of the literature have to consider the limitations of the studies. In this case, it would not be entirely unreasonable for an establishment person to conclude that a definitive, pure study has never been and will never be accomplished and his/her common sense recommendation is that we should nevertheless recommend masking as many individuals in our society, for cultural reasons East Asians, seem to prefer that choice. Right now the medical establishment has not been ideally transparent about the ongoing requirement for masking in many doctors' offices, clinics and hospitals. All this makes for ongoing erosion of trust and respect for the medical establishment.