The Meta-research Study I Want to See
The Meta-research Study I Want to See
I hope you are listening, Drs. Ioannidis, Prasad, Jena...
Everyone’s goal for future of medical research is more good studies and fewer bad ones. Even our best study design, the randomized, placebo-controlled trial, frequently produces questionable results. Is there a way we can rid the literature of poorly done RCTs that produce misleading data? What follows is a proposal for a meta-research study that I am absolutely convinced will identify where unreliable research is published and decrease the demand for this research.[i] If this study were to be done, it would improve the quality of medical research, reduce the clutter of medical journals, and get more doctors back to the job of seeing patients.
The Funnel Plot
The funnel plot, common in meta-analysis, is my favorite figure. Its intention is to identify publication bias. The funnel plot graphs individual studies with their effect size on the X-axis and a measure/correlate of their sample size, usually the standard error, on the Y-axis. A classic funnel plot looks like this one from an article published in Annals of Internal Medicine on the efficacy of lumbar epidural steroid injections for sciatica.
Here is the actual funnel you are supposed to be reminded of:
The graph from the Annals article does not identify publication bias. When publication bias exists, the graph will lack one part of the funnel, because usually smaller, negative studies have not been published. Therefore, studies that fit this description will be missing – those in the oval below:
A New Use for the Funnel
What if we use the funnel plot in a different way – to identify journals that reliably publish inaccurate studies? In a meta-analysis we have identified the truth -- also known as the mean effect size.[ii] The results of large, well-done studies and well-done small studies will closely approximate truth. Poorly done studies, and a random selection of small studies, will produce results farther from the truth. So, let’s instead see the funnel plot not as a funnel but as a superimposed pipette and colander:
What we now have is the “pipette of truth” and the “colander of distraction.” I’d hypothesize that if we attached some measure of journal quality (probably the impact factor) to each point (study) on the original funnel plot we would find that the higher quality journals routinely publish studies that fill the pipette of truth while lower quality journals routinely publish articles whose results fill the colander of distraction – either because they are smaller or because they are large but of low quality.[iv]
OK, so what is the meta-research I want to see done? Take 100 meta-analyses. In each one, label each point in the funnel plot with the impact factor of the journal that published the study. (The points in the funnel plot from the Annals meta-analysis represent studies published in journals with impact factors ranging from 4.2-9.3). Next, define what counts as being within the pipette or within the colander. This is probably just a measure of “distance” away from the mean effect size. I would hypothesize that we find a strong correlation between lower impact factors and likelihood of being in the colander rather than the pipette. As you go from pipette to colander (or spout of the funnel to its bowl) you go from higher quality journals to lower quality ones.
The results of our study will allow us to identify an “unreliable impact factor” (UIF, pronounced whiff). The UIF correlates strongly with the tendency to publish RCTs that fill the colander of distraction. Journals with an impact factor below the UIF are not trustworthy. These are misleading journals.[v]
Using the Results
How would I hope that academic medicine would respond to these results? Misleading journals could either be forced to cease publishing altogether or they could be limited to publishing critiques, thought pieces, and reflections.[vi] The physician-researchers who primarily publish in these journals could spend more time improving medical care by seeing patients.
I will leave it to the researchers who take up this project to comment on the flaws in my reasoning here (I can’t imagine there are any).[vii] These researchers could also comment on the off-target consequences of limiting what lower quality journals could publish.[viii] I look forward to reading the final paper. I would appreciate an acknowledgement or perhaps even being included as author six of seven.
[i] This is, in fact, the only thing about which I am absolutely convinced.
[ii] I write this with my tongue placed firmly in my cheek.
[iv] Yes, I know I am filling an upside-down colander, which by definition has holes in it, but just go with me here.
[v] I’d refer to these as MJs, but as a Chicagoan I can’t do that.
[vi] You know, the things that fill a good portion of my CV.
[vii] See ii and iii.
[viii] I am helping you here with your discussion.
One more consideration came to my mind. I would not for example believe that this study works with antidepressants. All the antidepressant studies say there's a small and arguably clinically insignificant effect. The problem is it's all due to bias and antidepressants don't actually work. There's unblinding bias, publication bias, sponsorship bias, something called the telephone game bias, cold turkey bias, short trial duration bias which stops before the median time of natural remission of untreated depression. Then there's the fact that the non-responder subgroup actually have their depression scores do worse than placebo. It goes on. Basically it's an anti-science field and yet the studies have a consensus. I would actually believe the studies that are outliers and not the ones that align. So the presumption of this kind of study design is that we actually have a tendency towards quality to begin with which I am afraid may oftentimes be untrue
Scanned over the BGC contract withe DoD. Thanks to Sasha, we're all getting a much clearer picture as to where thus is all going to lead: back to the psychopaths at the DoD.