I contrast two population screening trials--DANCAVAS and NordICC. One stayed true to the principle of randomization. The other broke traditional intention-to-treat principles
“Potential differences in patients” is not data. Imagine you advertised that your product had a risk of side effects of 10 %. If only 50 % of subjects in your test actually used the product and you don’t know that, then you can only report what you can measure — that is intention to treat although we never needed a name. Now, however, if the regulatory agency found out that you actually knew that there was a 20 % risk, but didn’t report that, you would be in deep shit. The road to legal action. is paved with good intentions.
The Inter99 authors published an analysis comparing persons in the intervention group who participated or not. It demonstrates the confounding well: https://pubmed.ncbi.nlm.nih.gov/26190370/
I guess it is okay to report results from as-screened analyses, but it would be good to e.g. have several sensitivity analyses or different causal approaches at least. I think NordICC authors used two different models.
Is there a RCT that randomised patients to genuine or fake colonoscopy? The fake group does all the prep etc, is sedated, but just gets a scope inserted a few inches and gas to blow up the colon, twiddle thumbs for a few minutes, then remove the scope without any observations or polypectomies, ... I would think this is the kind of trial that would reliably show whether or not there is a benefit to colonoscopy? My understanding is that the assumption is the putative benefits are too small/negligible and that's why this trial isn't done? As it would require too many participants to show a benefit, hence no-one can be bothered attempting it?
It is simple: If you break randomization or don't follow the ITT principle, your study is just another unadjusted observational study. The participants are just self selected which comes with loads of confounders. You can call it per -protocol and such, but it will ALWAYS be biased.
I don't understand. Everybody in a study is self-selected unless compelled to comply. If you select the participants including those who don't take the drug or intervention, you cannot say that they did take the drug. It is an ethical breach. It is a lie. It is "biased" towards being in a study where you can compare the behavior to the outcome. A scientific study does not follow from your intention. Intention-to-treat has a million reductio ad absurdum examples. One hundred people sign up for a study of effect of aspirin. For whatever reason, none of them take the aspirin. If you know this and say "the effect of aspirin...." Of course, again, if you don't know whether anybody took the aspirin, you can only report the outcome but that is not a good experiment.
When I say "self-selected, I mean they self-selected the exposure in an observational study, but this is not the case in an RCT.
If someone happily enters an RCT study and don't take the pill, it could be either they don't get the intended benefits, there are adverse effects, or the benefits were too good . All these are problematic . and should be somehow accounted for. If you only include those who adhered to the pill , then it is clearly not the true effect of the drug. And then this turns out be an obs study for all practical purposes.
And If they are not doing it for some random reason not related to the outcome, then it is fine. And these random missingness would also more likely to occur in the control group and probably would cancel out and wouldn't matter
"If someone happily enters an RCT study and don't take the pill, it could be either they don't get the intended benefits, there are adverse effects, or the benefits were too good. All these are problematic. and should be somehow accounted for."
None of those effects are associated with the pill because they didn't take it.
"If you only include those who adhered to the pill , then it is clearly not the true effect of the drug."
There is no "effect" at all if you don't take the pill. In essence, you are not in the experiment.
The experiment asks: what is the effect of taking the pill.
(If you explicitly say that the experiment ask what is the effect of happily entering the RCT then you can include whatever you want).
"And then this turns out be an obs study for all practical purposes."
Call it what you want. If everybody gets better but nobody took the pill, you do know anything about the effect of the pill.
"And If they are not doing it for some random reason not related to the outcome, then it is fine. And these random missingness would also more likely to occur in the control group and probably would cancel out and wouldn't matter."
The outcome comes AFTER the performance of taking or not taking the pill.
The control group never takes the pill. Some experimenters count the people who did not take the pill as being in the control group. Formally, not recommended, but the observations are more important than statistical formalism.
ITT as stated is illogical and foolish.
Most of all if, anybody acts on the conclusion of an ITT and there is harm from doing so. They may be liable.
The road to malpractice is paved with good intentions.
The main lesson here is that, while there is substantial overlap, medicine is different from science. There is no definite procedure for establishing general practice. What I and others have written about ITT shows that it is a foolish concept and nobody can explain why not. The typical answer is to restate the principle. And the intensity with which people insist on something so clearly illogical brings out one of the differences between science and medicine. In science, doubt is an opportunity. In medicine, doubt is a threat.
NordICC style messaging must surely be approaching the threshold for “nefarious” even in Dr. JMM’s book 😉.
Cuz there are only 2 options. And 1 would require readers to believe that the investigators simply do not understand the concept of randomization and the basic step 1 of analyzing an RCT.
So I’m going with the other. It’s rhymes with “$”.
The correct response to that NordiCC author is then why didn’t you randomize later in the process. It’s completely possible to create a realistic situation where people are invited to a regular physical, informed about potential screening for CC during the physical and then randomized if they consent. Some people will still drop out before the colonoscopy but few enough that they won’t impact a clear positive result.
Of course you wouldn’t be able to claim that everyone should have a colonoscopy but you would at least have a statistically valid trial on the population that actually matters in a practical sense.
Not at all. Dr Prasad’s point was that trying to fix the design by leaving out the people that didn’t get a colonoscopy creates a potential bias and therefore the result cannot be trusted; this doesn’t mean that there is definitely no benefit. In fact he is clear that screening with sigmoidoscopy does have a benefit and that what remains unclear is whether the added cost and risk of colonoscopy is worthwhile.
In practical terms what is needed is to know whether a patient who is willing to consider screening after explaining the options should be recommended colonoscopy (or some other option). This is still a broad recommendation but can be done while minimizing the noise from those that would never agree to a colonoscopy while symptom free and are therefore not relevant to the question.
Should we screen? Yes, we know that invasive screening with sigmoidoscopy increases survival. Should we use colonoscopy as the preferred option? Unknown. From a theoretical perspective it is incredibly unlikely that it has worse efficacy compared to sigmoid, and good reasons to think it is better. It also has increased risks and costs a lot more. A responsible CDC would see the need to answer an important question and fund a properly designed trial; maybe even get the EU to pay part of the cost. What I’d really like to see is a three arm study where the randomized procedure is non-invasive testing, sigmoidoscopy, or colonoscopy.
Too bad that most of your treatments (drugs) are nothing burgers. I can see that with my wife as drugs have ruined her gut biome among other bodily functions. Trials for poisons...what a way to make a living.
Actually, this is a simple topic. It becomes complicated if you don't understand randomization.
I made it complicated (including an error) by not adequately proof-reading and, originally by making the discussion too general. This is the revised version.
The problem is easiest seen with an example.
You are studying the effect of an antidepressant pill on a cognitive test like matching words. There are two groups, one who takes he pill and a control who takes a look-alike placebo. Both are randomized to relevant variables, age, health status, etc.
100 people are assigned to the the pill group. The outcome is that 30 people show a increase in performance on the test. You would report that people in the pill group have the measured increase. That's what we always did. The measured effect of being assigned to take the pill is an increase of 30 %.
However, you suspect that not everybody took the pill for whatever reason. The experiment is repeated with TV cameras and blood test for presence of the pill. It turns out that, for some reason, only 60 people actually took the pill. They included the 30 who has decreased performance, that is, 50% of the people who took the pill showed an effect.
This is called per-protocol, that is, the subjects did what you told them to do. You report that the drug is 50% effective. In addition, you report that there is only 60% adherence. Again, this is what we always did.
Now, what's odd is the appearance of the idea of intention-to-treat which says that, even if yo know that only 60 people took the drug, you must report the data as if all 100 subjects took the drug. It doesn't make any sense. It's, in fact, false and misleading.
Where did it come from? It doesn't make sense. The answer is that frequently all we know is how many people were told to take the drug and what the outcome was. We suspect people may have lost the pill or spit out or whatever but we don't know this so we have no choice to report the outcome as a fraction of all the people in the group. As in the example above, it's. what we always did because it's all we could do and did not give a name.
Intention-to-treat answers the question: what is the effect of being assigned to an intervention?
Per-protocol answers the question: what is the effect of actually following the intervention.
Usually, we are not interested in the actual assignment to the study group. The important part is, if we follow instructions, how will it turn out. Again, as the experimenter, we frequently don't have good access to adherence to the protocol. In this case, we have no choice but reporting the data we have.
The confusion is in understanding randomization.
Randomization is over before the experiment begins. If you are randomized, the outcome cannot affect the randomization. If you find out that subjects were not randomized -- e.g. they were smokers, but did not report that during randomization -- you have to do the experiment over.
Once in the study, you cannot be un-randomized by the outcome anymore than you can become un-baptized. You can become a bad person, or you can become excommunicated, or whatever, but you were baptized.
Intention to treat asks what is the effect of HAVING BEEN TOLD to follow instructions.
Per protocol asks what is the effect of following instructions, that is, what is the effect of the intervention. We discussed this in a blog post and a publication (link in the post).
Thank you for your explanation, and to Dr. Mandrola using these two examples to highlight the issue. Describing the results in terms like “Adherence” and “effective” intuitively makes more sense (to me) than intention to treat and per protocol.
It would seem to me it matters what the goal of running the trial is. Adherence/intention to treat seems far more important with public health interventions where one must balance many more priorities (effectiveness, cost, etc.). The NordICC helps answer the public health question in Denmark’s system.
there is a huge difference between the health benefit of inviting someone to quit smoking and the health benefit of actually quitting!
Thanks again to all on this forum who increase my knowledge and understanding.
There are really four populations. (1) those invited who screened; (2) those invited who did not screen; (3) those not invited who screened anyway; (4) those not invited who did not screen. Population (3) is really the fly in the ointment. Then there are the sub-populations of (1) and (3) who detected illness (1/3 A) and those that screened that did not detect illness (1/3 B). It would be interesting to know false positives and false negatives in these two groups and associated outcomes, including adverse side effects. Why? You might discover, for example, that sending an invitation has no impact on screening rates, or maybe it does. You might also find that screening is ineffective in accurately diagnosing disease and that the side effects resulting from false positives overwhelms the advantage of finding the disease early, or the opposite. If you were actually trying to figure out what policy to adopt would this information be valuable? I think it would. Why invite people to an ineffective screening? Why invite people to a screening that is more likely to cause harm than to advance health? Of course, once you start the study you probably have little ability to modify it on the fly without injecting bias. So it is best practice to really think things through before you start and to challenge your assumptions. Maybe screening is ineffective. Maybe screening is effective. I suspect that the study designers started from the assumption that screening is effective. But that assumption may not be true. Whatever the truth was, it would impact the results of this study.
A simple way of seeing the problem: viewing from the top, we have an RCT. Embedded inside the ‘assigned to colonoscopy arm’ is the equivalent of a classical observational study. We know that observational studies show that colonoscopies appear to have a benefit, but the point of this study is to learn whether an RCT will show a benefit. So, for the people who claim a benefit by comparing the two groups inside the ‘assigned to colonoscopy arm’, they are making the argument that observational studies prove causality and that RCTs are not necessary. They may not realize they are doing this.
Remain strictly true to prepublication design was really all I meant. I am an older clinician who remembers the days when data were routinely manipulated post hoc then published to find some meaningless benefit like subgroup analysis etc. Then there were the negative trials that simply never saw the light of day ( publication). Before the Editors of the big refereed journals got together and stopped that nonsense. But that good did not go unpunished.
My comment tried to explain this more simply: as you say, release all the data. If you don't have all the data -- e.g., you suspect some people spit out the pill -- you have no choice. You do what we always did, report the data you have.
“Potential differences in patients” is not data. Imagine you advertised that your product had a risk of side effects of 10 %. If only 50 % of subjects in your test actually used the product and you don’t know that, then you can only report what you can measure — that is intention to treat although we never needed a name. Now, however, if the regulatory agency found out that you actually knew that there was a 20 % risk, but didn’t report that, you would be in deep shit. The road to legal action. is paved with good intentions.
The Inter99 authors published an analysis comparing persons in the intervention group who participated or not. It demonstrates the confounding well: https://pubmed.ncbi.nlm.nih.gov/26190370/
I guess it is okay to report results from as-screened analyses, but it would be good to e.g. have several sensitivity analyses or different causal approaches at least. I think NordICC authors used two different models.
Is there a RCT that randomised patients to genuine or fake colonoscopy? The fake group does all the prep etc, is sedated, but just gets a scope inserted a few inches and gas to blow up the colon, twiddle thumbs for a few minutes, then remove the scope without any observations or polypectomies, ... I would think this is the kind of trial that would reliably show whether or not there is a benefit to colonoscopy? My understanding is that the assumption is the putative benefits are too small/negligible and that's why this trial isn't done? As it would require too many participants to show a benefit, hence no-one can be bothered attempting it?
Great post!
It is simple: If you break randomization or don't follow the ITT principle, your study is just another unadjusted observational study. The participants are just self selected which comes with loads of confounders. You can call it per -protocol and such, but it will ALWAYS be biased.
I don't understand. Everybody in a study is self-selected unless compelled to comply. If you select the participants including those who don't take the drug or intervention, you cannot say that they did take the drug. It is an ethical breach. It is a lie. It is "biased" towards being in a study where you can compare the behavior to the outcome. A scientific study does not follow from your intention. Intention-to-treat has a million reductio ad absurdum examples. One hundred people sign up for a study of effect of aspirin. For whatever reason, none of them take the aspirin. If you know this and say "the effect of aspirin...." Of course, again, if you don't know whether anybody took the aspirin, you can only report the outcome but that is not a good experiment.
Hi Richard,
When I say "self-selected, I mean they self-selected the exposure in an observational study, but this is not the case in an RCT.
If someone happily enters an RCT study and don't take the pill, it could be either they don't get the intended benefits, there are adverse effects, or the benefits were too good . All these are problematic . and should be somehow accounted for. If you only include those who adhered to the pill , then it is clearly not the true effect of the drug. And then this turns out be an obs study for all practical purposes.
And If they are not doing it for some random reason not related to the outcome, then it is fine. And these random missingness would also more likely to occur in the control group and probably would cancel out and wouldn't matter
you wrote:
"If someone happily enters an RCT study and don't take the pill, it could be either they don't get the intended benefits, there are adverse effects, or the benefits were too good. All these are problematic. and should be somehow accounted for."
None of those effects are associated with the pill because they didn't take it.
"If you only include those who adhered to the pill , then it is clearly not the true effect of the drug."
There is no "effect" at all if you don't take the pill. In essence, you are not in the experiment.
The experiment asks: what is the effect of taking the pill.
(If you explicitly say that the experiment ask what is the effect of happily entering the RCT then you can include whatever you want).
"And then this turns out be an obs study for all practical purposes."
Call it what you want. If everybody gets better but nobody took the pill, you do know anything about the effect of the pill.
"And If they are not doing it for some random reason not related to the outcome, then it is fine. And these random missingness would also more likely to occur in the control group and probably would cancel out and wouldn't matter."
The outcome comes AFTER the performance of taking or not taking the pill.
The control group never takes the pill. Some experimenters count the people who did not take the pill as being in the control group. Formally, not recommended, but the observations are more important than statistical formalism.
ITT as stated is illogical and foolish.
Most of all if, anybody acts on the conclusion of an ITT and there is harm from doing so. They may be liable.
The road to malpractice is paved with good intentions.
The main lesson here is that, while there is substantial overlap, medicine is different from science. There is no definite procedure for establishing general practice. What I and others have written about ITT shows that it is a foolish concept and nobody can explain why not. The typical answer is to restate the principle. And the intensity with which people insist on something so clearly illogical brings out one of the differences between science and medicine. In science, doubt is an opportunity. In medicine, doubt is a threat.
NordICC style messaging must surely be approaching the threshold for “nefarious” even in Dr. JMM’s book 😉.
Cuz there are only 2 options. And 1 would require readers to believe that the investigators simply do not understand the concept of randomization and the basic step 1 of analyzing an RCT.
So I’m going with the other. It’s rhymes with “$”.
The correct response to that NordiCC author is then why didn’t you randomize later in the process. It’s completely possible to create a realistic situation where people are invited to a regular physical, informed about potential screening for CC during the physical and then randomized if they consent. Some people will still drop out before the colonoscopy but few enough that they won’t impact a clear positive result.
Of course you wouldn’t be able to claim that everyone should have a colonoscopy but you would at least have a statistically valid trial on the population that actually matters in a practical sense.
Not at all. Dr Prasad’s point was that trying to fix the design by leaving out the people that didn’t get a colonoscopy creates a potential bias and therefore the result cannot be trusted; this doesn’t mean that there is definitely no benefit. In fact he is clear that screening with sigmoidoscopy does have a benefit and that what remains unclear is whether the added cost and risk of colonoscopy is worthwhile.
In practical terms what is needed is to know whether a patient who is willing to consider screening after explaining the options should be recommended colonoscopy (or some other option). This is still a broad recommendation but can be done while minimizing the noise from those that would never agree to a colonoscopy while symptom free and are therefore not relevant to the question.
Should we screen? Yes, we know that invasive screening with sigmoidoscopy increases survival. Should we use colonoscopy as the preferred option? Unknown. From a theoretical perspective it is incredibly unlikely that it has worse efficacy compared to sigmoid, and good reasons to think it is better. It also has increased risks and costs a lot more. A responsible CDC would see the need to answer an important question and fund a properly designed trial; maybe even get the EU to pay part of the cost. What I’d really like to see is a three arm study where the randomized procedure is non-invasive testing, sigmoidoscopy, or colonoscopy.
Excellent 👍
Too bad that most of your treatments (drugs) are nothing burgers. I can see that with my wife as drugs have ruined her gut biome among other bodily functions. Trials for poisons...what a way to make a living.
Actually, this is a simple topic. It becomes complicated if you don't understand randomization.
I made it complicated (including an error) by not adequately proof-reading and, originally by making the discussion too general. This is the revised version.
The problem is easiest seen with an example.
You are studying the effect of an antidepressant pill on a cognitive test like matching words. There are two groups, one who takes he pill and a control who takes a look-alike placebo. Both are randomized to relevant variables, age, health status, etc.
100 people are assigned to the the pill group. The outcome is that 30 people show a increase in performance on the test. You would report that people in the pill group have the measured increase. That's what we always did. The measured effect of being assigned to take the pill is an increase of 30 %.
However, you suspect that not everybody took the pill for whatever reason. The experiment is repeated with TV cameras and blood test for presence of the pill. It turns out that, for some reason, only 60 people actually took the pill. They included the 30 who has decreased performance, that is, 50% of the people who took the pill showed an effect.
This is called per-protocol, that is, the subjects did what you told them to do. You report that the drug is 50% effective. In addition, you report that there is only 60% adherence. Again, this is what we always did.
Now, what's odd is the appearance of the idea of intention-to-treat which says that, even if yo know that only 60 people took the drug, you must report the data as if all 100 subjects took the drug. It doesn't make any sense. It's, in fact, false and misleading.
Where did it come from? It doesn't make sense. The answer is that frequently all we know is how many people were told to take the drug and what the outcome was. We suspect people may have lost the pill or spit out or whatever but we don't know this so we have no choice to report the outcome as a fraction of all the people in the group. As in the example above, it's. what we always did because it's all we could do and did not give a name.
Intention-to-treat answers the question: what is the effect of being assigned to an intervention?
Per-protocol answers the question: what is the effect of actually following the intervention.
Usually, we are not interested in the actual assignment to the study group. The important part is, if we follow instructions, how will it turn out. Again, as the experimenter, we frequently don't have good access to adherence to the protocol. In this case, we have no choice but reporting the data we have.
The confusion is in understanding randomization.
Randomization is over before the experiment begins. If you are randomized, the outcome cannot affect the randomization. If you find out that subjects were not randomized -- e.g. they were smokers, but did not report that during randomization -- you have to do the experiment over.
Once in the study, you cannot be un-randomized by the outcome anymore than you can become un-baptized. You can become a bad person, or you can become excommunicated, or whatever, but you were baptized.
Intention to treat asks what is the effect of HAVING BEEN TOLD to follow instructions.
Per protocol asks what is the effect of following instructions, that is, what is the effect of the intervention. We discussed this in a blog post and a publication (link in the post).
https://feinmantheother.com/2011/08/21/intention-to-treat-what-it-is-and-why-you-should-care/
Thank you for your explanation, and to Dr. Mandrola using these two examples to highlight the issue. Describing the results in terms like “Adherence” and “effective” intuitively makes more sense (to me) than intention to treat and per protocol.
It would seem to me it matters what the goal of running the trial is. Adherence/intention to treat seems far more important with public health interventions where one must balance many more priorities (effectiveness, cost, etc.). The NordICC helps answer the public health question in Denmark’s system.
there is a huge difference between the health benefit of inviting someone to quit smoking and the health benefit of actually quitting!
Thanks again to all on this forum who increase my knowledge and understanding.
There are really four populations. (1) those invited who screened; (2) those invited who did not screen; (3) those not invited who screened anyway; (4) those not invited who did not screen. Population (3) is really the fly in the ointment. Then there are the sub-populations of (1) and (3) who detected illness (1/3 A) and those that screened that did not detect illness (1/3 B). It would be interesting to know false positives and false negatives in these two groups and associated outcomes, including adverse side effects. Why? You might discover, for example, that sending an invitation has no impact on screening rates, or maybe it does. You might also find that screening is ineffective in accurately diagnosing disease and that the side effects resulting from false positives overwhelms the advantage of finding the disease early, or the opposite. If you were actually trying to figure out what policy to adopt would this information be valuable? I think it would. Why invite people to an ineffective screening? Why invite people to a screening that is more likely to cause harm than to advance health? Of course, once you start the study you probably have little ability to modify it on the fly without injecting bias. So it is best practice to really think things through before you start and to challenge your assumptions. Maybe screening is ineffective. Maybe screening is effective. I suspect that the study designers started from the assumption that screening is effective. But that assumption may not be true. Whatever the truth was, it would impact the results of this study.
A simple way of seeing the problem: viewing from the top, we have an RCT. Embedded inside the ‘assigned to colonoscopy arm’ is the equivalent of a classical observational study. We know that observational studies show that colonoscopies appear to have a benefit, but the point of this study is to learn whether an RCT will show a benefit. So, for the people who claim a benefit by comparing the two groups inside the ‘assigned to colonoscopy arm’, they are making the argument that observational studies prove causality and that RCTs are not necessary. They may not realize they are doing this.
I think that you must remain true to intention to treat. Period. Until you publish the results that is.
And then release all the raw data and let us look at perfect world as treated just to see. Because you know. We want to know.
Remain strictly true to prepublication design was really all I meant. I am an older clinician who remembers the days when data were routinely manipulated post hoc then published to find some meaningless benefit like subgroup analysis etc. Then there were the negative trials that simply never saw the light of day ( publication). Before the Editors of the big refereed journals got together and stopped that nonsense. But that good did not go unpunished.
My comment tried to explain this more simply: as you say, release all the data. If you don't have all the data -- e.g., you suspect some people spit out the pill -- you have no choice. You do what we always did, report the data you have.
English language is part of it. Re-reading an old Blog post, I see that I had considered intention to treat : https://feinmantheother.com/2011/09/05/portion-control-low-carb-diets-and-the-language-of-food/
This is a complicated topic that you presented very well. Congratulations!