Exercise Churnalism Part 2

Let’s discuss one more article before finishing our introduction to exercise churnalism.

No better place to spend your pregnancy than the summit of Mt Everest

This article appeared in The New York Times and was based on two academic articles. The subject is important, the message empowering, but the headline, and much of the article, is misleading and potentially dangerous. The headline read, “Vigorous Exercise, Even a Trek Up Everest, May Be Safe During Pregnancy.”

First, let’s talk about pregnancy. The human body is truly remarkable and has been able to nurture and deliver healthy babies for millennia. Women successfully carry pregnancies in jungles and deserts, at land that lies just below sea level and on plateaus that leave the newcomer gasping with any exertion. Probably the best advice for a woman with an uncomplicated pregnancy is to not do anything too stupid and the body will take care of things. Despite this, pregnant women have been the victims of a lot of well-intentioned but misguided advice for, well, probably for all of human history. From ludicrous dietary recommendations, to prohibitions against any physical activity, to medications that turned out to be dangerous to mother or child or both. Every armchair expert (many of them men) seem to have a heaping plateful of advice to offer. Most of that advice is not justified by the best available science and much of it has been contradicted by better studies later.

Next, let’s talk about Mt. Everest. Everest is the highest peak in the Himalayan range. The summit, at 29,029 ft, is the cruising altitude of passenger jets. Even the base of Mt. Everest is high, 17,900 ft. on the Nepalese side. When people say they are “trekking up Everest” that usually means they are climbing from 17,900 to 29,000 ft.

Now the article. The article claims that even trekking up Mt Everest may be safe in pregnancy. Two pieces of evidence are used to support this claim. First, an anecdote, another case study, the lowest level of “evidence” there is. There was once a woman who was 7 months pregnant who worked at the Everest base camp (not the summit). She was physically fit, active, and went on to deliver a healthy baby girl. Second, the author cites a study of maternal complications among elite athletes. This study compared the birth outcomes of elite women athletes who participated in high-impact sports, of women who participated in low-impact sports, and of women who were pregnant but not world-class athletes. The research found that athletes deliver babies as healthy as do non-athlete women.

These two academic papers tell us what we already know, that the human body has evolved to carry fetuses under almost any condition that the earth can throw at us. They do not, however, tell us that “Vigorous Exercise, Even a Trek Up Everest, May Be Safe During Pregnancy.” What are the flaws in the reasoning here? They are mostly about extrapolation, the second sin. The woman working at the Everest base camp was clearly amazing, but her experience, 12,000ft below the top of Mt Everest, is just one person’s story that does not provide any support for the suggestion that pregnant women could or should ascent Everest. As to the elite athlete study, because an elite athlete can safely train at a high level while pregnant says nothing about whether a regular, conditioned woman would be safe to get anywhere near this level of exertion. 

Going beyond the errors associated with extrapolation, there is a terrible flaw in the reasoning here as well. Concluding that this level of exercise is safe is like concluding that going to a war zone is safe by noting that everyone who returns home lives a normal lifespan. There is no way of knowing if pregnancies were lost prior to delivery in the elite athlete group. We do not mean to imply that extreme exercise is akin to going to war, just that studying complications at the time of birth is no way to judge whether it is safe during pregnancy. 

Once again, the journalist buries the brief disclaimer in the article, performing the classic disclaim and pivot maneuver (fifth sin). “These two studies obviously are small, tightly focused on athletes and, in the case of the Nepalese Sherpa, singular, so the results may not be meaningful for other pregnant women wondering how much they should exercise.” This sentence is included as a passing caveat at the end of the article. This is the most important sentence in the article and the reason that the article should not have been published in the first place.

Vibration of Effects

Underpinning much of the research that we have discussed are comparisons of dissimilar groups. Whether it is chili aficionados compared to bland food connoisseurs or tennis players compared to swimmers, we cannot separate the activity of interest from the rich complexity of genetics, exposures, and free will that makes every person unique and might correlate with the activity of interest. What we really want to know is, if you took a person who could either play tennis or swim, would they live longer doing one rather than the other? Instead, the data we are left with compare very messy groups. The group of all swimmers includes some folks who could play tennis, but have chosen to swim, but it also includes some people who simply cannot play tennis because tennis makes their knees swell, or they cannot find a partner, or they cannot afford court time for nine months of the year. At the same time, tennis players may include some people who cannot swim. They are terribly afraid of water, or never learned to swim, or cannot stand what chlorine does to their hair.

What all this means is that if you want to compare swimming and playing tennis, you have to adjust for an enormous number of differences to isolate the effect of the sport from the kinds of people who play the sport. All the research we have looked at adjust for some variables, hoping to isolate the effect of interest. Researchers adjust for age and sex; or age, sex, and socioeconomic status. They might also include history of heart disease, family history, alcohol use, race, ethnicity, and many other factors. The trouble is, there are no rules for adjusting. One researcher could make the case that adjusting for age and sex is sufficient while another might adjust for four or six additional factors. Different researchers may examine the same associations and, depending on what they adjust for, get a range of answers. Moreover, not every researcher will disseminate his or her findings. Researchers who get boring results — like maybe it really does not matter if you swim, play tennis, bicycle, or run — may not seek publication. These researchers may not even write up the results. Articles reporting research that found no difference between to exposures is almost always harder to publish than research that finds provocative associations. 

Led by Chirag Patel, one team appreciated that results depend enormously on which variables are controlled for and performed a clever experiment. They picked single exposures, for instance how much lycopene you eat, and looked at its effect on your chance of dying. Then they chose 15 common variables that people adjust for in medical studies, things like age, sex, smoking, education, family history of heart disease, and so forth. Then they ran analyses adjusting for every possible combination of these variables. They ran analyses in which they adjusted for age, and age and sex, and age, sex and race, and sex and race, and on and on.  

What is so clever about this experiment is that it simulates the possible behaviors of the entire research community. For a given “hot” topic, it is likely that dozens of researchers will perform analyses with many different adjustments over the years. Each researcher can come up with good reasons to adjust for some but not other factors. In the real world, not every study is done, or reported, but Dr. Patel and his colleagues’ results show us what we would see if everything was reported. Patel and colleagues are modeling the entire research community.

The authors produced clouds of associations. For about 1/3 of the combinations of exposures and outcomes, they found that, simply by choosing what to adjust for, they could get favorable and unfavorable associations for the same pairing. This demonstrates that depending on adjustments they could actually make the exposure appear beneficial or harmful. They also found that, for many pairs of exposures and outcomes, there was absolutely no relationship. 

How does the work of Patel and his colleagues relate to churnalism? Their research hints at a deep problem behind the sorts of studies we have discussed. Eating, drinking and exercising are daily activities that we all share. There is a small army of researchers probing these relationships. They look at people who, for a multitude of reasons, choose to eat some things and not others, or participate in some sports but not others. They then associate this choice with an outcome. Because the people who eat one thing and not something else differ in many ways other than this single choice, they adjust for factors that might be influencing the outcome. But there is no gold standard for what to adjust for, there is no accepted research playbook. Each researcher does what he or she thinks is best. The net result is a sea of investigations. A fraction of these studies are submitted for publication, a smaller fraction are accepted, and an even smaller fraction of these make the news. It is critical to remember that there is an entire universe of similar, equally valid studies, that have not been run, published, or reported everytime we read a health-related article. This is the sixth sin of churnalism. Keep testing, report just once. That does not just apply to a single researcher or team, a whole field of research can collectively engage in this activity, even as the individual researchers are oblivious to it, believing that their analysis was the sole one performed.

How should we be reading health journalism? Unless we are reading it as a human-interest story, we should be dismissing the importance of any article based on a case report. For articles that are drawing on basic science research -- for example, does iron intake inhibit absorption of lycopene -- we should remind ourselves that this may or may not have any relation to human health. For all those articles that are trying to convince us of a diet or activity that will improve our health, we should remember that it is possible to create almost any association by how data are manipulated. One researcher might perform calculations in such a way that playing tennis or eating radishes is good for you while another might find that they are useless, or maybe even harmful. We should also never forget that association does not equal causation. Eating chili peppers can be associated with a long life without causing it. And lastly, find the sentence or paragraph in the article in which the journalist notes the other possible explanation for the finding; this explanation may or may not be interesting but it is probably the truth.