I think the problem is much of medical "journalism" is not about assessing or probing the facts. And there is frequently a lack of understanding about how studies and trials are set up, the parameters that are measured, and the questions that need to be asked - a common confusion is the difference between relative and absolute risk.
Medical journalism is often more about explaining how something is supposed to work or the theories behind an idea. It doesn't provide the same level of scrutiny or examination as many other areas of journalism such as politics or business do.
So if, for example, a sugar company published a study saying we should all eat more sugar that would be rightly viewed with scepticism and questions about conflicts of interest asked. But those same questions are not asked, if a pharmaceutical company publishes a press release about a new drug. That is not to say that there is not important and valuable work published by the pharmaceutical industry or indeed many other groups, but simply that this should be held to the same level of scrutiny and it's not.
Here's a good example that I reported on a couple of months ago:
In 2008, a meta-review was published by Kirsh et al, this looked at clinical trials submitted to the US Food and Drug Administration for the licensing of four new-generation antidepressants in adults. This paper received a great deal of publicity at the time, because the researchers reported they found no meaningful difference between antidepressants and placebos. Using the standardized mean difference (SMD) to measure the effect size between antidepressants and placebos, Kirsch found an effect of 0.32 which his study claimed was too small to be clinically significant. So, it was widely reported as evidence that antidepressants don’t work.
Fast forward to 2018 and another antidepressant meta-review hits the headlines, this time the study is by Ciprani et al, this looks at 522 clinical trials of antidepressants in adults. However, now it’s reported that this paper suggests antidepressants do work after all. But the puzzling fact that was completely overlooked in the media coverage was that Ciprani actually found a slightly smaller effect than Kirsch. He reported a 0.30 SMD between the placebo and antidepressant group.
This means that, somewhat counterinituitively, the group that claimed that antidepressants do work found a marginally lower difference between the placebo and the active drug, than the group that claimed that antidepresssants didn’t!
This point went completely unreported in the media.
I think you could have found much worse examples of churnalism. But as always, your analysis is enlightening for a layperson like me. I think the worst omission in the NPR interview was the failure to explain that all the participants in the study had at least 1 sister diagnosed with breast cancer. That fact should have been mentioned. However, I give Alexandra White credit for clarifying that correlation does not imply causation. It would be interesting to give both the host and Dr. White an opportunity to respond to your criticisms.
I give the researcher some credit for gently restraining NPR's jump-to-conclusions reporter. At least she pointed out that the absolute risk was 1.6%, so a "doubled" risk was still not overwhelming. But the fact that everyone in the study group had a sister with breast cancer? Failure to explain that makes this a first-class example of Churnalism. I like Ayesha Rascoe, but maybe someone should invite her to sign up for a Healthcare Evidence 101 class. Can Gary Schwitzer recommend one?
My first thought was that women who used these products should be aware-but-not-terrified -- but I worried about hairdressers who worked with the stuff every day! The fact that their risk was actually lower than that of the "user" group was interesting (and also not mentioned on NPR). What precautions do hairdressers typically use -- gloves, masks, Vaseline? And could these tell us anything about how to mitigate the risk?
Of the seven sins, Doctor, I would rank "incuriosity" as the most deadly.
Oh, and thanks for the morning chuckle; your relating the exhortation "go write an article and leave us in peace" reminds me of the old "why don't you go fix the broken turnsignal switch in the car and leave us in peace" sort of urging that I remember from days gone by. The corollary being "you promised to fix the washing machine this weekend and the day isn't getting any younger."
Regarding NPR; I first began to suspect that their "oh so calmly reasonable" stylistic presentation masked an appetite for absurdity, when they started to take Krugman seriously, back in the nineties. It's been difficult to think NPR a credible source of information ever since.
Good analysis, Dr. Cifu! I used to listen to NPR quite frequently, and like you, often (too often) found myself ranting at the seeming lack of critical thinking skills by the journalist. Or perhaps NPR journalists just refused to ask probing questions for fear of challenging a guest/interviewee. Either way, I stopped listening or contributing to public radio as a result. I highly recommend it—restore your peaceful Sunday mornings! They do wonders for the soul (and your health).
Oct 26, 2022·edited Oct 26, 2022Liked by Adam Cifu, MD
I understand your point and frustration. Kudos for running us through the "7 Deadly Sins" treatise. Bullseye. It was definitely worth reading. Thank you, Dr. Cifu. Perfect takedown of "churnalists" who continue to insult our intelligence and abuse those out there who lack critical thinking skills. It's sad to see what's become of NPR. It may be a false equivalency to say but it's almost like watching a family member destroy themselves with a deadly addiction. Disturbing.
To increase your enjoyment of your family's Sunday morning routine, may I suggest that you cease and desist listening to the NPR programming and reading the Sunday papers (New York Times and Washington Post, I'm guessing, given your Ground News Blindspotter bias rating)? Please don't take this pejoratively. Some of us have found a more meaningful life outside of the echo chambers we prefer to inhabit. It's that whole "itchy ears" phenomenon: We can't but want them to be scratched. This particular NPR segment obviously led to them bleeding. Listen to classical music next Sunday instead. Better on your ears.
That blindspotter site is so cool. Never new about it. I love it. I think I am a bit more centrist than it suggests (it is influenced by what I choose to tweet) but it is great. Thanks.
You're most welcome. I'll bet your are a bit more centrist than the Blindspotter algorithm suggests. I was surprised by it for you given your Substack sharing. It's not saying much, but I try I put my confirmation bias aside as well as anyone on Twitter (which Ground News is evaluating through its Blindspotter algorithm). It doesn't lie, though. It accurately describes your "diet" on Twitter. A change in diet never hurts if done rationally (read: skip the "NPR Diet Coke habit" now and then; you'll feel better after the withdrawal).
I understand your point, but don't some things start with observational studies, because otherwise more research would never be done? There are so many products we ingest and apply to our bodies, our homes and our yards that were never subject to rigorous safety studies in the first place. You clarify your concern re the reporting on the study vs. the study itself. Popular media are not medical journals, are they going to run a story saying no implications can be made? Perhaps they should have asked what prompted the study to be done (did anecdotal reports suggest a possible association)? If there is a study done and no one reports it and no one is around to hear it, would it make a sound?
Did the interviewer also ask the researcher to estimate how many of the carcinogens had been put in the hair products by Donald Trump, and did the interviewer say "Trump," "the former occupant of the White House," or "Drumpf?" ;-)
"Causation cannot be demonstrated in an observational study". Causation cannot be demonstrated in THIS observational study. Some observational studies are unbiased, for instance studies showing that mammograms cause breast cancer (Corcos & Bleyer, NEJM, 2020; and Corcos, BioRxiv, 2017 https://www.biorxiv.org/content/10.1101/238527v1.full ).
Causation cannot be demonstrated in a single observation. But when the same cause is always followed by the same effect, you can conclude that there is a causality:
I think the problem is much of medical "journalism" is not about assessing or probing the facts. And there is frequently a lack of understanding about how studies and trials are set up, the parameters that are measured, and the questions that need to be asked - a common confusion is the difference between relative and absolute risk.
Medical journalism is often more about explaining how something is supposed to work or the theories behind an idea. It doesn't provide the same level of scrutiny or examination as many other areas of journalism such as politics or business do.
So if, for example, a sugar company published a study saying we should all eat more sugar that would be rightly viewed with scepticism and questions about conflicts of interest asked. But those same questions are not asked, if a pharmaceutical company publishes a press release about a new drug. That is not to say that there is not important and valuable work published by the pharmaceutical industry or indeed many other groups, but simply that this should be held to the same level of scrutiny and it's not.
Here's a good example that I reported on a couple of months ago:
In 2008, a meta-review was published by Kirsh et al, this looked at clinical trials submitted to the US Food and Drug Administration for the licensing of four new-generation antidepressants in adults. This paper received a great deal of publicity at the time, because the researchers reported they found no meaningful difference between antidepressants and placebos. Using the standardized mean difference (SMD) to measure the effect size between antidepressants and placebos, Kirsch found an effect of 0.32 which his study claimed was too small to be clinically significant. So, it was widely reported as evidence that antidepressants don’t work.
Fast forward to 2018 and another antidepressant meta-review hits the headlines, this time the study is by Ciprani et al, this looks at 522 clinical trials of antidepressants in adults. However, now it’s reported that this paper suggests antidepressants do work after all. But the puzzling fact that was completely overlooked in the media coverage was that Ciprani actually found a slightly smaller effect than Kirsch. He reported a 0.30 SMD between the placebo and antidepressant group.
This means that, somewhat counterinituitively, the group that claimed that antidepressants do work found a marginally lower difference between the placebo and the active drug, than the group that claimed that antidepresssants didn’t!
This point went completely unreported in the media.
I think you could have found much worse examples of churnalism. But as always, your analysis is enlightening for a layperson like me. I think the worst omission in the NPR interview was the failure to explain that all the participants in the study had at least 1 sister diagnosed with breast cancer. That fact should have been mentioned. However, I give Alexandra White credit for clarifying that correlation does not imply causation. It would be interesting to give both the host and Dr. White an opportunity to respond to your criticisms.
Not a surprise as that is almost all NPR does now days from climate change to racism, politics, you name it.
I give the researcher some credit for gently restraining NPR's jump-to-conclusions reporter. At least she pointed out that the absolute risk was 1.6%, so a "doubled" risk was still not overwhelming. But the fact that everyone in the study group had a sister with breast cancer? Failure to explain that makes this a first-class example of Churnalism. I like Ayesha Rascoe, but maybe someone should invite her to sign up for a Healthcare Evidence 101 class. Can Gary Schwitzer recommend one?
My first thought was that women who used these products should be aware-but-not-terrified -- but I worried about hairdressers who worked with the stuff every day! The fact that their risk was actually lower than that of the "user" group was interesting (and also not mentioned on NPR). What precautions do hairdressers typically use -- gloves, masks, Vaseline? And could these tell us anything about how to mitigate the risk?
Of the seven sins, Doctor, I would rank "incuriosity" as the most deadly.
Oh, and thanks for the morning chuckle; your relating the exhortation "go write an article and leave us in peace" reminds me of the old "why don't you go fix the broken turnsignal switch in the car and leave us in peace" sort of urging that I remember from days gone by. The corollary being "you promised to fix the washing machine this weekend and the day isn't getting any younger."
Regarding NPR; I first began to suspect that their "oh so calmly reasonable" stylistic presentation masked an appetite for absurdity, when they started to take Krugman seriously, back in the nineties. It's been difficult to think NPR a credible source of information ever since.
Good analysis, Dr. Cifu! I used to listen to NPR quite frequently, and like you, often (too often) found myself ranting at the seeming lack of critical thinking skills by the journalist. Or perhaps NPR journalists just refused to ask probing questions for fear of challenging a guest/interviewee. Either way, I stopped listening or contributing to public radio as a result. I highly recommend it—restore your peaceful Sunday mornings! They do wonders for the soul (and your health).
Our tax dollars pay for NPR’s one sided BS! Defund bad journalism, not the police.
NPR is shite!
I understand your point and frustration. Kudos for running us through the "7 Deadly Sins" treatise. Bullseye. It was definitely worth reading. Thank you, Dr. Cifu. Perfect takedown of "churnalists" who continue to insult our intelligence and abuse those out there who lack critical thinking skills. It's sad to see what's become of NPR. It may be a false equivalency to say but it's almost like watching a family member destroy themselves with a deadly addiction. Disturbing.
To increase your enjoyment of your family's Sunday morning routine, may I suggest that you cease and desist listening to the NPR programming and reading the Sunday papers (New York Times and Washington Post, I'm guessing, given your Ground News Blindspotter bias rating)? Please don't take this pejoratively. Some of us have found a more meaningful life outside of the echo chambers we prefer to inhabit. It's that whole "itchy ears" phenomenon: We can't but want them to be scratched. This particular NPR segment obviously led to them bleeding. Listen to classical music next Sunday instead. Better on your ears.
That blindspotter site is so cool. Never new about it. I love it. I think I am a bit more centrist than it suggests (it is influenced by what I choose to tweet) but it is great. Thanks.
You're most welcome. I'll bet your are a bit more centrist than the Blindspotter algorithm suggests. I was surprised by it for you given your Substack sharing. It's not saying much, but I try I put my confirmation bias aside as well as anyone on Twitter (which Ground News is evaluating through its Blindspotter algorithm). It doesn't lie, though. It accurately describes your "diet" on Twitter. A change in diet never hurts if done rationally (read: skip the "NPR Diet Coke habit" now and then; you'll feel better after the withdrawal).
I understand your point, but don't some things start with observational studies, because otherwise more research would never be done? There are so many products we ingest and apply to our bodies, our homes and our yards that were never subject to rigorous safety studies in the first place. You clarify your concern re the reporting on the study vs. the study itself. Popular media are not medical journals, are they going to run a story saying no implications can be made? Perhaps they should have asked what prompted the study to be done (did anecdotal reports suggest a possible association)? If there is a study done and no one reports it and no one is around to hear it, would it make a sound?
Did the interviewer also ask the researcher to estimate how many of the carcinogens had been put in the hair products by Donald Trump, and did the interviewer say "Trump," "the former occupant of the White House," or "Drumpf?" ;-)
Gell-Mann Amnesia... don't listen to so much NPR.
"Causation cannot be demonstrated in an observational study". Causation cannot be demonstrated in THIS observational study. Some observational studies are unbiased, for instance studies showing that mammograms cause breast cancer (Corcos & Bleyer, NEJM, 2020; and Corcos, BioRxiv, 2017 https://www.biorxiv.org/content/10.1101/238527v1.full ).
Causation cannot be demonstrated in a single observation. But when the same cause is always followed by the same effect, you can conclude that there is a causality:
https://ourworldindata.org/smoking-big-problem-in-brief
I like the "split hairs"... very good.