Sooooooo important for clinicians to understand and interpret study results and promotional press releases with. And yet it seems so few do actually understand these fundamental different ways of interpreting data. Thanks John as always.
Great explanation of ARR, RRR, and NNT. It is true that many physicians and APPs struggle with these concepts. I agree RRR is used to sell benefits and ARR to explain away side effects in order to sell products.
I give up. You docs may understand medicine and procedures, but you know absolutely nothing about healthy living. You know nothing about life styles and how they may create the very instances you are treating. Someday, the modern stone age medical system will be out of business because it will run out of patients to treat (they mostly will be dead of various causes) and those that are living will have no need of health care except in emergencies.
There is a substantial minority of physicians who do know about healthy living. It is very difficult to change living habits, however, and less lucrative, which is a societal problem.
Notice, everyone, that the doctors and the company involved in this trial already knew ahead of time that the absolute risk of an infection was low, yet they felt it might be worthwhile to try to eliminate it because when infection does happen, it can be devastating. So, I don’t see any deception here in reporting the relative risk reduction, nor can we assume that the product is not worthwhile for some patients.
Think of it this way: suppose you were about to go on vacation to a tropical paradise, but there is a small outbreak of malaria there. Your chance of getting malaria is only 1%, but your doc suggests that you take a cheap, side-effect-free malaria prevention pill just in case, which reduces the chance to .01%. (Fiction, alas!). Wouldn’t you do it? Sometimes knowing the relative risk reduction is crucial to making smart decisions. I think both should be reported to patients, whenever possible.
Remember how Pfizer got everyone hooked on the X% effective with the first "study" they did? I think it was like 92% effective. It was literally the worst population study I've ever seen. From what I recall, all they did was monitor the vax'd population (no comparison class) and said that 92% of them didn't get COVID from the time they got the vaccine to the (arbitrary) end of study date. And with 5th-grade arithmetic they only barely took in any other controlling factors other than the most inconsequential demographics.
How people still think this passes for anything like science truly boggles the mind. I have to say it looks more like astrology. Which actually is a perfectly fine mind game -- just please don't base public health on it?
John, Good analysis, but there's a major piece still missing: One must also incorporate the harms of an intervention. I acknowledge that you've discussed the financial "harm" of the envelope. This is a valid point for nearly all new modalities, and vary much from low to high.
However, therapies (old and new) include medical/surgical harms. E.g. drug side effects, surgical/anesthesia risks for invasive interventions, and so forth. For the envelope, there may be a minuscule risk of antibiotic sensitivity that is absent in the older, non-drug devices. As a minimum, I'd check out this risk before recommending the device. The risk may be immeasurably low, but a prudent clinician addresses the question.
For all interventions, I strongly advise the reader fully use their critical thinking skills. Many drugs/devices are promoted by their RR reduction, as in this case, but the harms are reported as an absolute risk increase. Consider this statement: "Drug A reduces MI by 40%, but only entail a risk of bleeding of just 0.5%." If drug A is aspirin for low/average risk patients, then we know it is a net harm. (Even though cost is extremely modest.)
Please say it isn't so! I don't blame the docs because with managed care, they barely have time to treat their patients. Clearly the docs here at Sensible Medicine are critical thinkers! :)
I have had a pacemaker for 10 years and it is never really had to pace my heart. I had a very successful ablation. I would like to have the pacemaker removed, but I don’t know how to assess the risk. Can you give me some idea where I would go to read enough to know how to assess the rest.
I have been explaining this to my patients for 25 years. No one really gets it.
You need even an simpler explanation. The goal of relative risk statistics is to sell you something. In medicine or anywhere else. RR and HR are nearly meaningless. Even JAMA knows this. Nature magazine knows this. But they all persist.
NNT is nearly incomprehensible to everyone. That is how the mRNA vaccines were sold. That is how statins are sold. That is how Tide is sold.
I like your thinking! As a psychologist who worked in schools for years, I was always frustrated that advanced math was pushed on kids in high school, regardless of interest, aptitude, or career plans, while financial literacy and statistics were ignored. At least now the Common Core Math Standards includes some basic statistics and probability (link below). But not all states have adopted Common Core, and now 12 states are repealing them. Granted, there have been issues, but let's fix those and not throw the baby out with the bath water! I bet more parents would like their kids to understand the type of math they will need to make intelligent life decisions as adults. BTW, ratios and proportionality isn't taught until 6th-7th grade. Abstract reasoning is still pretty weak in 5th grade. I'd be happy if all 5th graders were adept at basic math calculation skills! :)
Sooooooo important for clinicians to understand and interpret study results and promotional press releases with. And yet it seems so few do actually understand these fundamental different ways of interpreting data. Thanks John as always.
Great explanation of ARR, RRR, and NNT. It is true that many physicians and APPs struggle with these concepts. I agree RRR is used to sell benefits and ARR to explain away side effects in order to sell products.
I give up. You docs may understand medicine and procedures, but you know absolutely nothing about healthy living. You know nothing about life styles and how they may create the very instances you are treating. Someday, the modern stone age medical system will be out of business because it will run out of patients to treat (they mostly will be dead of various causes) and those that are living will have no need of health care except in emergencies.
There is a substantial minority of physicians who do know about healthy living. It is very difficult to change living habits, however, and less lucrative, which is a societal problem.
I love the simplicity of your explanation. I needed to hear this. Thank you Dr. Mandrola.
Notice, everyone, that the doctors and the company involved in this trial already knew ahead of time that the absolute risk of an infection was low, yet they felt it might be worthwhile to try to eliminate it because when infection does happen, it can be devastating. So, I don’t see any deception here in reporting the relative risk reduction, nor can we assume that the product is not worthwhile for some patients.
Think of it this way: suppose you were about to go on vacation to a tropical paradise, but there is a small outbreak of malaria there. Your chance of getting malaria is only 1%, but your doc suggests that you take a cheap, side-effect-free malaria prevention pill just in case, which reduces the chance to .01%. (Fiction, alas!). Wouldn’t you do it? Sometimes knowing the relative risk reduction is crucial to making smart decisions. I think both should be reported to patients, whenever possible.
Remember how Pfizer got everyone hooked on the X% effective with the first "study" they did? I think it was like 92% effective. It was literally the worst population study I've ever seen. From what I recall, all they did was monitor the vax'd population (no comparison class) and said that 92% of them didn't get COVID from the time they got the vaccine to the (arbitrary) end of study date. And with 5th-grade arithmetic they only barely took in any other controlling factors other than the most inconsequential demographics.
How people still think this passes for anything like science truly boggles the mind. I have to say it looks more like astrology. Which actually is a perfectly fine mind game -- just please don't base public health on it?
John, Good analysis, but there's a major piece still missing: One must also incorporate the harms of an intervention. I acknowledge that you've discussed the financial "harm" of the envelope. This is a valid point for nearly all new modalities, and vary much from low to high.
However, therapies (old and new) include medical/surgical harms. E.g. drug side effects, surgical/anesthesia risks for invasive interventions, and so forth. For the envelope, there may be a minuscule risk of antibiotic sensitivity that is absent in the older, non-drug devices. As a minimum, I'd check out this risk before recommending the device. The risk may be immeasurably low, but a prudent clinician addresses the question.
For all interventions, I strongly advise the reader fully use their critical thinking skills. Many drugs/devices are promoted by their RR reduction, as in this case, but the harms are reported as an absolute risk increase. Consider this statement: "Drug A reduces MI by 40%, but only entail a risk of bleeding of just 0.5%." If drug A is aspirin for low/average risk patients, then we know it is a net harm. (Even though cost is extremely modest.)
An excellent point - there's been research showing that clinical studies do a very poor job of reporting adverse events.
Excellent article. Thank you!
Please say it isn't so! I don't blame the docs because with managed care, they barely have time to treat their patients. Clearly the docs here at Sensible Medicine are critical thinkers! :)
I have had a pacemaker for 10 years and it is never really had to pace my heart. I had a very successful ablation. I would like to have the pacemaker removed, but I don’t know how to assess the risk. Can you give me some idea where I would go to read enough to know how to assess the rest.
I have been explaining this to my patients for 25 years. No one really gets it.
You need even an simpler explanation. The goal of relative risk statistics is to sell you something. In medicine or anywhere else. RR and HR are nearly meaningless. Even JAMA knows this. Nature magazine knows this. But they all persist.
NNT is nearly incomprehensible to everyone. That is how the mRNA vaccines were sold. That is how statins are sold. That is how Tide is sold.
Your calculation of the inverse of 05% turned out to be 0.005? How can this be correct to calculate the NNT?
The confidence internal for the hazard ratio il is huge for such a large sample size and it could easily be a 0.98 HR, or a 2% protection
Every senior high school student should be taught RRR and ARR. IMO.
Super article! You are my hero.
This is pure gold.
Along with financial literacy (earlier the better) this should be introduced at grade 5 and reviewed and expanded every year after.
I like your thinking! As a psychologist who worked in schools for years, I was always frustrated that advanced math was pushed on kids in high school, regardless of interest, aptitude, or career plans, while financial literacy and statistics were ignored. At least now the Common Core Math Standards includes some basic statistics and probability (link below). But not all states have adopted Common Core, and now 12 states are repealing them. Granted, there have been issues, but let's fix those and not throw the baby out with the bath water! I bet more parents would like their kids to understand the type of math they will need to make intelligent life decisions as adults. BTW, ratios and proportionality isn't taught until 6th-7th grade. Abstract reasoning is still pretty weak in 5th grade. I'd be happy if all 5th graders were adept at basic math calculation skills! :)
https://www.mathsisfun.com/links/core-high-school-statistics-and-probability.html
Thank you great explanation Wonderful illustration