Improving Your Critical Appraisal Skills #6: Decision Analysis

I’ve taken a little break from the “Improving Your Critical Appraisal Skills” series but I’m returning with a few more. These articles are a little treat for our paid subscribers. The first five are here.

I must admit, I rarely read decision analyses. That said, I should appreciate them a whole lot more. First, the whole point of a decision analysis is to determine the best course of action when faced with two or more alternatives. Isn’t this what most medical decision making is about? Decision analyses are done when data from more experimental studies do not exist. We’d all love to live in a world where all clinical questions have been addressed by RCTs. We do not, and we never will. Lastly, and what I find most attractive about decisions analyses, is that they can consider everything important in actual clinical decisions: patients’ values; potential harms and benefits; financial and opportunity costs; and the uncertainties inherent in all of these.

Clinical Question

Decision analyses begin with a clinical question. A good decision analysis will feature a good PICO question, specifying the patient, intervention, comparison, and outcome. In the schematic, decision nodes are squares and chance nodes are circles. For this post, let’s consider the following question: For a patient with three months of back pain with sciatica, from a herniated disk (documented on MRI), who has not improved with conservative therapy, should the next step be a lumbar epidural steroid injection (LESI) or lumbar discectomy? Here is what the schematic would look like.

Outcomes

The next step in a decision analysis is the researchers need to consider the consequences of the decision. This means coming up with all the possible outcomes that follow the chance nodes. These outcomes will obviously include desirable and non-desirable ones. Each branch might have different possible outcomes. (In figure 2, death is a possible outcome of surgery but not of LESI.) When reading a study, consider whether all possible outcomes have been included. These trees can become quite large. Here is a pretty abbreviated one for our question.

Most decision analyses will have multiple “subtrees” as the model might be run again and again until everybody reaches some final status -- Well or dead? Maybe just dead?

Probabilities

This next step is the one I find most illuminating. From the best available evidence, the researchers must assign a probability of reaching each endpoint. This step often makes a decision analysis worth reading even if you do not buy the results because it will give you a sense of the likelihood of various outcomes of a clinical situation dependent on various interventions. How likely is a patient, like the one we are considering, to improve with LESI or suffer neurologic injury from surgery? For each branch of a tree, the probabilities will equal 1. Below is our example with the (completely made up) probabilities included.

Utilities

Utilities are the relative value of an outcome. The value might be to a patient, a decision-maker, a payer, or society in general. Utilities can be determined in a number of ways. Commonly, patients are asked to give a value to each outcome (usually between 0 and 1). I find it easiest to think about these in terms of trade-offs – this is how they are often posed to people when they are being defined. “If you have 10 years of life remaining, how much of that would you give up to not have to take a Motrin every day? How much would you give up to not have to live with moderate back pain?” If a person says, “I would give up a month to not have to take a Motrin and a year to not have to live with back pain” then life with back pain is assigned a utility of 0.9 and life with Motrin is assigned 0.99 (119/120). Dollar amounts may also serve the same purpose as costs or cost savings can be assigned to each outcome. Below is our example with (totally made up) utilities included.

Relative Values

With all the information we now have, one can calculate a value for each strategy. The utility or cost of each outcome is multiplied by the associated probabilities. The values are then “folded back” until a value for the whole strategy is determined. In the figure above, each utility is multiplied by the associated probability. Adding the resulting numbers for each node gives you the relative value of each choice. Here, surgery clearly wins out.

Surgery: (0.7 X 0.95) + (0.1 X 0.35) + (0.15 X 0.45) + (0.05 X 0) = 0.767

LESI: (0.5 X 1) + (0.4 X 0.5) + (0.1 X 0.4) = 0.74 

Sensitivity Analysis

Although decision analyses are based on data from the literature, they are often done because there is insufficient data from experimental studies. This contradiction necessitates sensitivity analyses. Here the parameters used in the above calculations (probabilities and utilities) are varied to account for uncertainty in the data. Often, the probabilities are varied to the extremes of the confidence intervals in the underlying literature. If the outcome is “sensitive” to a variable, it means that the outcome changes as the variable changes.

This metaphor might not work but bear with me. They say you are entitled to your own opinions but not your own facts. The beauty of sensitivity analysis is that it allows you to vary your opinions (utilities) and see out how it effects a decision. Because facts are not always clear, you can also play with these by varying probabilities.

In our example, maybe we would find that the results are sensitive to the surgeon’s operative mortality.  

Are these studies actually useful?

Decision analysis can be useful when there are no studies that exactly address a particular question. They can also be helpful when “hard endpoint” data is available (decreased mortality), but quality of life benefit is unclear. They are also used when two alternatives appear to be equally efficacious but may lead to markedly different costs.

---

A few of my favorite decision analyses

Fendrik AM, et al. Alternative Strategies for Patients with Suspected Peptic Ulcer Disease.  Ann Int Med 1995;123:260-268.

Man-Son-Hing M, et al. Choosing Antithrombotic Therapy for Elderly Patients with Atrial Fibrillation Who Are at Risk for Falls. Arch Int Medicine 1999;159:657-685.

Sanders GD, et al. Cost-effectiveness of implantable cardioverter-defibrillators. NEJM 2005;353:1471-80.

Yeh JM. Routine Echocardiography Screening for Asymptomatic Left Ventricular Dysfunction in Childhood Cancer Survivors: A Model-Based Estimation of the Clinical and Economic Effects. Ann Intern Med 2014;160:661-671.

Vijan S. Effect of Patients’ Risks and Preferences on Health Gains With Plasma Glucose Level Lowering in Type 2 Diabetes Mellitus. JAMA Intern Med. 2014;174:1227-1234.

---

Users’ Guides for Decision Analysis

1. Were all the important strategies and outcomes included?

2. Was an explicit and sensible process used to identify, select and combine the evidence into probabilities?

3. Were the utilities obtained in an explicit and sensible way from credible sources?

4. Was the potential impact of any uncertainty in the evidence determined?

5. In the baseline analysis, does one strategy result in a clinically important gain for patients?  If not, is the result a toss-up?

6. How strong is the evidence in the analysis?

7. Could the uncertainty in the evidence change the result?

8. Do the probability estimates fit my patients’ clinical features?

9. Do the utilities reflect how my patients would value the outcomes of the decision?