Introduction
In the first article in this series (EBSJ, volume 3, issue 4), we described some basic terminologies related to health economic studies. In this second and final article, we describe
As with clinical studies, health economic studies vary in scope and quality. Their usefulness for decision and policy making is linked to their methodological rigor, accuracy, and generalizability. Thus, critical appraisal is important to put the results in the context of study quality.
Economic evaluations identify and compare appropriate alternatives of care, their incremental impact on health outcomes, and their incremental costs. As described in the first part of the series, there are several types of comprehensive medico-economic evaluations.
-
Cost-minimization studies consider the cost differences between alternatives of equal effectiveness. This assumes each alternative is equally effective, which is rarely true.
-
Cost-benefit studies consider both costs and benefits in monetary terms. Expressing benefits in monetary terms is, however, controversial.
-
Cost-effectiveness studies consider differences in costs and differences in effectiveness, but effectiveness is measured variably between studies. For comparisons involving the same disease, condition, or health state, this may be the most meaningful type of evaluation. It is less helpful for comparing studies involving different diseases.
-
Cost-utility studies consider differences in costs and outcomes for quality-adjusted survival, most often using the quality-adjusted life year (QALY). Cost-utility studies have the advantage of providing an incremental cost-effectiveness ratio (ICER) expressed as “cost per quality-adjusted life year” (cost per QALY) that eases comparison across multiple studies. This approach incorporates assumptions regarding how each alternative impacts patient's quality of life.
Each employs different methodologies, potentially complicating critical appraisal, but some common criteria can be assessed across studies. No standard, universally accepted method of critical appraisal of economic analyses is currently in use. Across the available methods, there is agreement on several points of methodological quality.[1]
Assessing the Quality of Economic Evaluations
Assessing the Quality of Economic Evaluations
Several organizations and research groups (e.g., Center for Health Economics, University of York [http://www.york.ac.uk/che/]; Tufts University Cost Effectiveness Registry [http://research.tufts-nemc.org/cear4/]) are dedicated to excellence in the conduct of health economic evaluation and provide detailed insights into many aspects of health economic analysis. The Tufts site contains a registry that allows one to search for cost-effectiveness analyses.
A variety of tools are available to assist new or experienced readers of economic evaluations, such as the Quality of Health Economic Studies (QHES) checklist,[2]
[3] the Consensus Health Economic Criteria list (CHEC),[4] and the British Medical Journal (BMJ) standards for quality of health economic evaluations.[5] All share emphasis on critically appraising some common domains of economic evaluations. In one formal comparison of QHES, the BMJ, and CHEC instruments,[6] the following domains were included for assessment in all three:
At least two of the instruments included assessment of the following domains:
Let us look at a few of the concepts related to these domains and how they apply to evaluation of study quality in a bit more detail.
What Is the Study Question or Objective of the Economic Evaluation?
What Is the Study Question or Objective of the Economic Evaluation?
Specification of the study question should set the stage for assessing the appropriateness of the economic evaluation design and methods. If the stated study question is to evaluate the cost-effectiveness of artificial disc replacement but only information on costs are presented and no comparison to viable alternative treatments is done, the quality of the study would immediately be suspected and the usefulness to policy makers is limited.
Does the Study Compare Costs and Outcomes of Meaningful and Appropriate Clinical Alternatives?
Does the Study Compare Costs and Outcomes of Meaningful and Appropriate Clinical Alternatives?
This is one of the first questions to consider. Studies that report only costs or do not formally compare clinical alternatives are not considered full economic evaluations. High-quality economic evaluation will clearly state all components of both intervention and comparator interventions. Meaningful incremental analysis in a full economic study is only possible if costs and outcomes for competing alternatives for care are reported. It is crucial that both intervention and comparator be realistic, appropriate reflections of actual use or care situations. A justification for why specific alternatives are evaluated should be stated. Studies that do not take such an approach or do not clearly state the alternatives being compared are not considered of high quality.
Does the Study Justify Its Perspective and Time Horizon Clearly?
Does the Study Justify Its Perspective and Time Horizon Clearly?
In a high-quality economic evaluation, the perspective is clearly stated and the rationale for its selection justified. As you may remember from the first article in this series, economic evaluations can take one of several perspectives, which have direct bearing on the types of costs and outcomes considered, such as a patient, payer, provider, health system, or society. The types and sources of cost data, measurement of benefits, and various modeling assumptions are influenced by the choice of perspective. Whatever the choice of perspective, the authors' choice of parameters should reflect it. If a societal perspective is chosen, for example, our critical appraisal should address the extent to which all long-term costs (both direct and indirect costs such as lost productivity, recovery time, and direct costs to patients) and outcomes to health systems and patients are included in the model. The perspective must be consistent with the research question or objective. The data used must be consistent with the perspective.
Economic evaluations generally employ complex modeling techniques to estimate costs and outcomes over time. A high-quality study will use a time horizon of at least 10 years to include all relevant sequelae and relapse, reoperation, or recurrence. Shorter time horizons can also provide valuable information depending on the study objective and available data. For longer time horizons, preferences for future benefits over immediate ones should be described in the study assumptions and modeled appropriately, usually using discounting of costs and benefits beyond 1 year, typically between 3 and 5%.
How Appropriate Are the Data Sources?
How Appropriate Are the Data Sources?
Economic evaluations regularly include data from disparate sources such as clinical trials, previously published scientific data, surveys collected from patients, or health care cost and utilization databases. High-quality studies provide clear descriptions for all data sources, they are complete, justify their selection, and employ data that are least subject to bias.
-
Patient-level clinical and cost data collected alongside a randomized controlled trial (RCT) or high-quality meta-analyses for RCTs are examples of data with low potential for bias. If these data are not available, then data from high-quality observational studies (e.g., methodologically rigorous prospective cohort studies) can provide useful information for modeling.
-
Existing databases can provide valuable cost estimates as well, though should be critically appraised for their generalizability and their fit with the study's perspective. For example, data from a health plan cost-and-use database used in a study taking a societal perspective might have incomplete data on indirect costs to patients.
-
Expert opinion—providers' own estimates of clinical or cost estimates—is often biased and therefore considered the least reliable source of data.
-
Data relevant to all aspects of the economic model need to be described.
How Were Outcomes Measured?
How Were Outcomes Measured?
The choice and measurement of the outcome should reflect the study objective. Instruments and/or clinical endpoints (e.g., survival) that are reliable and validated in the patient population of interest are preferred to surrogate or intermediate endpoints. There is an increasing trend toward the use of patient-important outcomes, ideally developed with input from the patient population.[7] For spinal conditions, patient-relevant conditions might include pain relief, function, and productivity.[8]
In a cost-utility analysis, quality-adjusted survival is calculated using a measure of utility combined with the health outcome in question. A high-quality economic evaluation will use validated utility measures and state clearly how they were calculated, ideally from a sample of people affected by condition or from a general community population (clinician assessment of utility would not be the ideal method of assessment). Some universal measures used to estimate utility (reported as a number between 0 and 1, where 1 represents perfect health) include the EQ-5D, the SF-6D, or through patient weighting of hypothetical scenarios such as discrete choice analysis.[9]
[10] Condition-specific measures are also available, such as the Oswestry Disability Index.[11]
Were Sensitivity Analyses Presented?
Were Sensitivity Analyses Presented?
Economic evaluations employ a variety of inputs and assumptions, all of which may induce uncertainty and variability. Thus, a high-quality evaluation will employ sensitivity analyses to assess uncertainty and evaluate factors that most influence the results. Some of these techniques can be quite complex. One-way sensitivity analyses vary individual variables (such as reoperation rate, pain reduction, or procedure cost) across a range to see how the results are changed. Probabilistic sensitivity analyses employ bootstrapping or other statistical techniques to arrive at a sense of which variables are “driving” the model results. This can be useful to decision-makers who need to understand the most important components of the cost and how they impact the cost-effectiveness.[12]
What about Bias?
Just like in clinical studies, a high-quality economic study will discuss the potential, magnitude, and direction of potential bias. For example, leaving out indirect costs to patients in a study taking a societal perspective where indirect costs are higher in the intervention compared with the alternative might alter the cost-effectiveness ratio. Failure to account for the range of adverse events can induce bias. Various assumptions about modeling may introduce bias. For example, if assumptions regarding the modeling of risk of certain adverse events are from poor quality studies or data that do not reflect current practice, the results may inaccurately favor one of the alternatives.
How Transparent Were the Authors?
How Transparent Were the Authors?
Increasingly, health economics practitioners are encouraged to provide the details of data sources as well as assumptions made regarding modeling of the data to encourage replication by other teams and so that major assumptions that influence the applicability of the model and study findings can be examined. The potential for conflict of interest has become increasingly important among policy makers. High-quality economic study will clearly state the funding sources provided to conduct the work and the role of the funder in analysis and reporting of the study.
Are the Conclusions Appropriate?
Are the Conclusions Appropriate?
As in assessing any clinical study, the assessment of its conclusions can be somewhat subjective but should be based on clinical knowledge and can be somewhat of an art. In addition to the points presented earlier, a more “gestalt” level assessment can consider how well
-
the study objective matched the study design and methods employed; as well as
-
the strength and certainty of the justification of the conclusions given the types of data used, modeling assumptions, perspective, time horizon, and potential sources of bias.
In other words, if the conclusions of a study seem too good to be true, then they just might be.
Summary
There are many opportunities for bias in economic evaluations. Thus, critical appraisal of economic studies is very important but complex. No standard methodology for appraisal has been universally adopted. However, consideration of the common elements that constitute a high-quality economic study provides the clinician with the tools to understand when one might be dealing with a high-quality study, a deeply flawed one, or one that made the best use of the data available.
Studies of economic impact have become increasingly important in health policy decisions given the finite resources available for health care. While they should not be the sole basis of decision making, such studies provide potentially valuable information if well done. It is therefore important that high-quality economic studies be conducted and the results of economic studies be placed in the context of their quality.
Learn more about economic evaluation
-
Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programs. New York, NY: Oxford University Press; 2005
-
What is an economic evaluation? Introduction to Economic Evaluation. Centers for Disease Control and Prevention. Available at: http://www.cdc.gov/owcd/EET/SeriesIntroduction/1.html
-
Health Economics Information Resources. A self-study course. National Information Center on Health Services Research and Health Care Technology (NICHSR). Available at: http://www.nlm.nih.gov/nichsr/edu/healthecon/beginningend.pdf
-
Boos N. The impact of economic evaluation on quality management in spine surgery. Eur Spine J 2009;18(Suppl 3):338–347
-
Holtz A, Nelson HD, Reid E. Our questions, our decisions: standards for patient-centered outcomes research. Preliminary Draft Methodology Report; PCORI Methodology Committee; Patient-Centered Outcomes Research Institute 2012; Available at: http://www.pcori.org/assets/Preliminary-Draft-Methodology-Report.pdf
