On yesterday’s podcast, I talked with Bobby Yeh, an academic cardiologist who made a compelling case for enhancing credibility of observational research. Please do listen. Bobby is one of the smartest people in cardiology today. After closely reviewing the cardiac literature for the past decade or so, I have become increasingly hopeless that we could glean any useful information from non-random retrospective comparisons. Mainly due to systemic bias.
Every practicing physician needs to read this and take it to heart. Far too many have allowed observational studies to sway their practice and recommendations to their patients.
Paul's comments truly are brilliant. The logical conclusion of his argument is that prospective observational studies based on intentional and well-funded data collection have a potential to inform treatment effectiveness estimation if the data collection includes all the treatment selection factors that at least 10 disinterested experts enumerate. Note the emphasis on "prospective". With relatively expensive prospective observational studies one can define time zero, collect the right baseline variables, collect the right outcomes, minimize missing data, and use objective, specific inclusion/exclusion criteria. It is very rare that retrospective observational studies using convenience samples will be reliable enough for the task at hand. The attempt to get quick and cheap answers to therapeutic effectiveness questions almost always leads to disappointment. This begs the question of why not follow the great clinical trialist Thomas Chalmer's edict "randomize early and often".
There is a role for prospective registries for quick starting clinical trials, and sometimes such registries, if they contain a rich variety of variables on which therapeutic choices are made that experts deem to be almost complete, can form the bases of prospective cohort studies.
The pandemic experience should certainly be a sobering lesson on the dangers of observational studies, especially with the politicization of public health.
Dr Dorian's concise analysis regarding the strengths and weaknesses (mostly weaknesses) of observational studies is a gem. "Interpretation creep" is well worth remembering: medical colleagues are sometimes guilty of this fallacy; the news media always.
Thank you John and Dr. Dorian for shining some light on the reality of medicine and science which only have a passing relationship with each other. This high quality thinking/writing is so very welcome! 👏
There is another side to this argument. Optimal medical therapy (OMT) for cardiometabolic disease is defined by 5 targets for diabetes and 4 for vascular disease. Blood pressure, statin therapy, sugar control, not smoking, and on an aspirin. Of course, in vascular disease without diabetes, you drop sugar control. There are medications that protect cells and organs more at the same risk factor level compared with other drugs: Lisinopril , losartan, spironolactone, and eplerenone for hypertension. Statins for cholesterol. Metformin and Jardiance for glucose. Controlled trials that don't have patients on baseline OMT show a higher impact that would be seen in patients on OMT because that is the best treatment for vascular disease.
Minnesota tracts OMT performance statewide and reports on it. That is the standard of care and we should track it and assure patients are on it just as in a research study. We have been working with a worksite clinic for 5 years on consistently producing OMT for chronic diseases. Care in the clinic costs half of that in the community. Patients seen in the clinic are hospitalized one fifth as often and ER visits are one third. They do other things like direct contracting, telemedicine, and same day visits, but OMT is an important factor. Of course, there will always be people who decide not to participate in care, but if you want to know what is possible in the real world, providing, supporting, and tracking OMT with other systemic best practices produces the best results I have seen.
To add some anecdotal evidence to the comment, I was involved in a project to discover if hapu could be predicted based on patient data. It turned out that for our data set (from a major hospital), the original major driving factor was people with major spinal surgeries who had been miscoded as in the hospital for non debilitating (for movement) reasons !
Color me impressed. Paul Dorian’s writing/analysis is thoughtful, articulate, and discerning, reflecting an intelligence and critical thinking skills that all too often seem to be lacking these days. Thanks for sharing, Dr. Mandrola!
Well put! One of the worst parts of current practice is the charting/coding involved, which is so forced, so artificial, there is no way you can make a reliable study out of chart codes. For the non doctors out there, imagine if you were required to fit every single interaction with every unique individual into a prefabricated mold to make some bureaucrat happy. So that, say, watching a stupid tik tok video your annoying coworker shows you and watching the video of your child’s dance recital both have to be officially recorded as “encounter for watching screens.” Now imagine doing stupid, inhuman coding like that all day, every day - and then having another bureaucrat who’s never met you think he can collate it all into a meaningful study! Madness.
I know we will never get rid of this stupid charting waste of time, because lawyers, but my dream is that we at least some day succeed in making lawyers’ lives as stupidly tedious and prohibit them from charging any clients for their time unless they precisely document all their legal reasoning into codes taken from a technical manual written by a robot. It won’t accomplish anything worthwhile or improve the legal profession, but it will be sweet, sweet revenge: )
This is kind of discouraging, but the good news is that, unless you're a researcher in search of hypotheses to investigate, you can cut WAY back on your medical literature reading! :)
The same old adage applies to administrative data for research: "Garbage in = garbage out"
Every practicing physician needs to read this and take it to heart. Far too many have allowed observational studies to sway their practice and recommendations to their patients.
Doesn’t it just feel GREAT to read a writer’s critical thinking? THANK. YOU.
Paul's comments truly are brilliant. The logical conclusion of his argument is that prospective observational studies based on intentional and well-funded data collection have a potential to inform treatment effectiveness estimation if the data collection includes all the treatment selection factors that at least 10 disinterested experts enumerate. Note the emphasis on "prospective". With relatively expensive prospective observational studies one can define time zero, collect the right baseline variables, collect the right outcomes, minimize missing data, and use objective, specific inclusion/exclusion criteria. It is very rare that retrospective observational studies using convenience samples will be reliable enough for the task at hand. The attempt to get quick and cheap answers to therapeutic effectiveness questions almost always leads to disappointment. This begs the question of why not follow the great clinical trialist Thomas Chalmer's edict "randomize early and often".
There is a role for prospective registries for quick starting clinical trials, and sometimes such registries, if they contain a rich variety of variables on which therapeutic choices are made that experts deem to be almost complete, can form the bases of prospective cohort studies.
The pandemic experience should certainly be a sobering lesson on the dangers of observational studies, especially with the politicization of public health.
Dr Dorian's concise analysis regarding the strengths and weaknesses (mostly weaknesses) of observational studies is a gem. "Interpretation creep" is well worth remembering: medical colleagues are sometimes guilty of this fallacy; the news media always.
Thank you John and Dr. Dorian for shining some light on the reality of medicine and science which only have a passing relationship with each other. This high quality thinking/writing is so very welcome! 👏
There is another side to this argument. Optimal medical therapy (OMT) for cardiometabolic disease is defined by 5 targets for diabetes and 4 for vascular disease. Blood pressure, statin therapy, sugar control, not smoking, and on an aspirin. Of course, in vascular disease without diabetes, you drop sugar control. There are medications that protect cells and organs more at the same risk factor level compared with other drugs: Lisinopril , losartan, spironolactone, and eplerenone for hypertension. Statins for cholesterol. Metformin and Jardiance for glucose. Controlled trials that don't have patients on baseline OMT show a higher impact that would be seen in patients on OMT because that is the best treatment for vascular disease.
Minnesota tracts OMT performance statewide and reports on it. That is the standard of care and we should track it and assure patients are on it just as in a research study. We have been working with a worksite clinic for 5 years on consistently producing OMT for chronic diseases. Care in the clinic costs half of that in the community. Patients seen in the clinic are hospitalized one fifth as often and ER visits are one third. They do other things like direct contracting, telemedicine, and same day visits, but OMT is an important factor. Of course, there will always be people who decide not to participate in care, but if you want to know what is possible in the real world, providing, supporting, and tracking OMT with other systemic best practices produces the best results I have seen.
Thank you. Just thank you for the sane, educational discourse.
To add some anecdotal evidence to the comment, I was involved in a project to discover if hapu could be predicted based on patient data. It turned out that for our data set (from a major hospital), the original major driving factor was people with major spinal surgeries who had been miscoded as in the hospital for non debilitating (for movement) reasons !
Color me impressed. Paul Dorian’s writing/analysis is thoughtful, articulate, and discerning, reflecting an intelligence and critical thinking skills that all too often seem to be lacking these days. Thanks for sharing, Dr. Mandrola!
Well put! One of the worst parts of current practice is the charting/coding involved, which is so forced, so artificial, there is no way you can make a reliable study out of chart codes. For the non doctors out there, imagine if you were required to fit every single interaction with every unique individual into a prefabricated mold to make some bureaucrat happy. So that, say, watching a stupid tik tok video your annoying coworker shows you and watching the video of your child’s dance recital both have to be officially recorded as “encounter for watching screens.” Now imagine doing stupid, inhuman coding like that all day, every day - and then having another bureaucrat who’s never met you think he can collate it all into a meaningful study! Madness.
I know we will never get rid of this stupid charting waste of time, because lawyers, but my dream is that we at least some day succeed in making lawyers’ lives as stupidly tedious and prohibit them from charging any clients for their time unless they precisely document all their legal reasoning into codes taken from a technical manual written by a robot. It won’t accomplish anything worthwhile or improve the legal profession, but it will be sweet, sweet revenge: )
This is kind of discouraging, but the good news is that, unless you're a researcher in search of hypotheses to investigate, you can cut WAY back on your medical literature reading! :)