A Fatally Flawed Study Enjoys a Warm Reception
This is the second of a three-part series on the importance of the control arm in medical trials.
Last week, I told a personal story of how I and other doctors were fooled into believing that a laser procedure relieved anginal chest pain.
The trickery happened because we deemed a procedure successful well before a properly placebo-controlled trial was done.
Today’s column describes a trial presented at last week’s ACC conference (and published in the New England Journal of Medicine) that tested an invasive procedure for a common heart valve condition.
Trial authors told us it was a positive trial. I felt a buzz about this positivity at the meeting. The problem: a faulty control arm.
The tricuspid valve regulates blood flow between the right atrium and right ventricle. As we age and develop conditions that cause dilation of the right ventricle, the valve often leaks. We call this TR or tricuspid regurgitation.
We measure TR with doppler echocardiography. It is an extremely common condition. Patients who have severe TR can feel terrible because blood can back flow into the liver and extremities and cause swelling.
The medical approach to TR is to use diuretic tablets and correct (as best as possible) the conditions causing strain on the RV. The surgical approach to TR is to either repair the valve or replace it. But this is really hard on patients.
Enter technology. There exists now a device that allows doctors to place a clip on the valve leaflets, essentially pulling them together and decreasing the amount the valve leaks. We call this transcatheter edge-to-edge repair (TEER).
TEER is approved for mitral regurgitation, but from an innovator’s standpoint, the potential for profit is far greater with the tricuspid valve because TR is more common.
The main problem, and I mean huge problem, is that TR almost always occurs because of other conditions. We call this secondary TR, and it is a key issue because correcting TR may not do anything to correct the underlying condition. And, as it always is in medicine, fixing a symptom or sign may actually worsen things for the patient.
The TRILUMINATE study authors randomized 350 patients with TR to either medical therapy or the TEER procedure. Medically treated patients knew they were in a trial, and they knew they weren’t getting the procedure. Procedural patients knew they got the procedure. The clip is bright white, so readers of the echocardiograms also knew who had the procedure.
The next sentence is super-important.
The authors chose as the primary endpoint to measure four outcomes: death, need for tricuspid surgery, hospitalization for heart failure, or quality of life as measured by the Kansas City Cardiomyopathy Questionnaire (KCCQ).
The results: overall, the primary endpoint favored the group that got the valve. The p-value made significance at p = 0.02.
Now to the problems. The authors reported no difference in the objective outcomes of death, need for TCV surgery or hospitalization for heart failure. The driver of the positive results were better scores on the KCCQ questionnaire. KCCQ measures how a patient assess their own well-being.
But, given last week’s discussion, and the fact that one group got a big procedure, and one group knowingly did not, a subjective assessment of well-being as an endpoint is useless. (Just like the first TMR study that I cited last week.) It is why we use placebo tablets in drug trials.
Another line of evidence that makes me believe that this trial had no substantial benefits was that the authors also reported no difference in the distance measured in the 6-minute walk test.
Comments:
It belies scientific principles to measure a subjective endpoint like KCCQ when you have NOT included a proper placebo control.
The authors had two choices to discover the truth about this procedure: a) they could easily have chosen a blinded procedure where the control group gets an access in the vein, but no valve clipping; or b) they don’t include a subjective outcome as a primary endpoint. (without KCCQ, TRILUMINATE would not have been a positive trial.)
Here is a Tweet from an author who did a placebo controlled trial:
Experts debate the exact nature of placebo effects, but the history of medicine has many examples of how placebo-controlled studies disproved the efficacy of an established procedure. Like I described with TMR last week.
This pre-market approval study will go to FDA for approval with only a subjective endpoint being positive.
I hope to have convinced you that TRILUMINATE is fatally flawed without a proper control arm. I hope that FDA saves us from ourselves.
Because if this device is approved, you can expect major increases in interventions on the tricuspid valve—because TR is so common. And because doctors like to do procedures.
Now for a harder thought. We must go there. Otherwise Sensible Medicine is no different from any other news site.
Why did the authors make this decision?
These were not rookies, who, out of inexperience, included a subjective endpoint in an unblinded trial. These are really smart people who have expertise in trials.
They made an active choice to include an endpoint that would bias the trial before the first patient was enrolled. Then, when the results came, the primary investigator of the paper said at ACC about his trial: "very clinically meaningful.”
Most of the authors are proponents of the procedure. I will link to the disclosure document. The company that makes the device and stands to profit from its sales sponsored the trial.
I will close on a broader point. It would be better to have these trials conducted by neutral funders and scientists. But that is not the current reality. Industry-sponsored studies that are conducted by proponents of the intervention is the norm.
Sometimes they lead to great innovations. (The ICD or anticoagulant drugs, for instance.) The model is not inherently nefarious, but the confluences and conflicts of interests must be considered in the interpretation of the results.
In this case, there is clear bias in the study design. A bias that precludes concluding that the procedure works.
And it tempts us to ask whether the conflicts and confluences of interests had a role in choosing this design.
Maybe we could specify that the first characteristic sensible medicine is that it be honest medicine. "Gaming the system," is something that is routine and rewarded in many areas of America's competitive life, including law, business and medicine. Unfortunately, this form of corruption is both rarely punished and dangerously corrosive. Our systems for combatting it are mostly absent, ineffective or easily circumvented. Thanks for this clear example of the problem!
This is a great discussion. Thank you. It is very disconcerting that seasoned medical researchers would choose the incorrect control group design. The list of disclosures is also disconcerting as it suggests, as you said, these authors were bought, henceforth influenced, by industry. This is not unique. I look at disclosures for most papers I read after reading the abstract and methods. It seems like half of all studies published in NEJM have significant author disclosures attached to them. Does that suggest NEJM is possibly “captured” by industry because they publish industry supported studies or is it simply a case that NEJM requires authors to disclose and publishes the disclosures along with the paper.