Keywords data visualization - clinical informatics - clinical practice guidelines - systematic
reviews - data access - integration and analysis - evidence-based medicine and nursing
- guidelines and protocols - evaluation
Background and Significance
Background and Significance
Leveraging research evidence to inform guidelines and clinical practice is essential
to improving quality and efficiency of patient care. Systematic reviews are rigorous,
peer-reviewed sources of evidence intended to support the translation from bench to
bedside, but the data often are complex and presented in dense, static formats. The
fixed structure of inquiry and extensive text-based reports may hinder uptake of these
data by persons tasked with developing evidence-based guidelines.[1 ]
[2 ]
[3 ]
[4 ]
[5 ]
[6 ] Even in well-written reviews, the structure is bound to the original analysis which
may make comparisons across groups more difficult. Alternate reporting methods exist
that allow users to interact with the data outside the original structure of the report
and informed by the local contexts of their health systems, but these have rarely
been incorporated into existing reviews. Harnessing these alternate tools to complement
the large volume of data in systematic reviews may increase accessibility and usability
of research evidence and facilitate adoption of evidence-based health care in guidelines.
The Agency for Healthcare Research and Quality (AHRQ) asked established Evidence-Based
Practice Centers (EPCs) to conduct pilot projects to investigate potential solutions.
During this process, the Pacific Northwest EPC performed a needs assessment for alternative
reporting and assessed numerous existing software solutions. Tableau[7 ] was chosen for this aspect of the project due to its ease of use, familiar Microsoft
Excel-based data structure (Microsoft Office 2016) robust customer service and community
support, and widespread adoption by many health systems.[8 ]
Objectives
The main objective of this project was to develop and evaluate a prototype report
using Tableau. This prototype allowed for a demonstration of the software's visualization
capabilities, the feasibility of creating similar reports in the future, and the overall
value to stakeholders involved in the development of clinical guidelines.
Methods
Data were extracted from the previously completed report on noninvasive nonpharmacological
treatment for chronic pain.[9 ] Information for all studies was extracted from the forest plots and the summaries
for each study in the report. As the original data were not available in an accessible
format, all data used in the project were manually extracted and stored in a spreadsheet
document employing a relational structure. A subset of the available data was used
to allow for rapid development and simple, representative interaction. Data were extracted
only for measures which were summarized using a standardized mean difference (SMD)
with a single-intervention modality. The data extracted included four types of pain
(chronic low back pain, chronic neck pain, osteoarthritis of the hip, and osteoarthritis
of the knee), eight intervention categories (acupuncture, exercise, massage, spinal
manipulation, mind–body practices, mindfulness-based stress reduction, physical modalities,
and psychological therapies), two outcome categories (effect on pain and effect on
function), three terms of follow-up (short, intermediate, and long). These data were
extracted for the individual studies, as well as the pooled estimates, for each group.
The prototype was developed using Tableau Desktop by an informatics researcher with
minimal previous experience with Tableau. The platform's support videos were the sole
source of training on database development. To facilitate interpretation of the prototype
by stakeholders accustomed to systematic review data visualizations, charts were modeled
after traditional forest plots ([Fig. 1 ]) but were slightly modified to take advantage of Tableau's dynamic reporting methods
([Fig. 2 ]).
Fig. 1 Traditional forest plot in a systematic review (reprinted from Skelly et al,[9 ] 2018). AC, attention control; CI, confidence interval; MI, minimal intervention;
N, number; NE, no exercise; ODI, Oswestry Disability Index; PDI, Pain Disability Index;
RDQ, Roland-Morris Disability Questionnaire; SD, standard deviation; SMD, standardized
mean difference; UC, usual care; WL, waitlist.
Fig. 2 Presentation of results in Tableau, simulating a traditional forest plot.
Six key stakeholders, who had previous experience with guideline development committees,
were given individual, guided demonstrations of the prototype. During the guided demonstration,
stakeholders were asked to qualitatively assess the accessibility and usability of
the prototype in guideline development, as well as provide feedback, on improving
the prototype for future development. Feedback was recorded and then analyzed by our
team to establish a consensus of key themes.
Results
The data were extracted from the PDF of the original report.[9 ] Tableau uses a relational format for its data input, where each row of data corresponds
to a single observation. For this project, we chose to use an Excel (2016) workbook,
though numerous other options would have worked as well (MySQL, Microsoft Access,
delimited text files, cloud servers, etc.) The data were extracted into an Excel (2016)
workbook, organized in a relational format with three sheets, such as Conditions,
Studies, and Outcomes ([Fig. 3 ]). The sheets of Conditions and Studies contained information about the conditions
and studies. The final sheet, Outcomes, had individual rows for each outcome and the
foreign keys for its associated condition and study. Both study-level and summary-level
outcome data were stored in the Outcomes table to allow for interactivity in a shared
visualization. Summary-level outcome records were denoted by an invalid Study ID in the Outcomes table, severing the link to the Studies table. Data extraction was
completed over the course of a month, accounting for most of the hours spent on this
project.
Fig. 3 Data from the original PDF were extracted and stored in Excel (2016) using a relational
structure.
The visualization was prototyped using Tableau Desktop. Individual visualizations
for each study (Study level) and for summary results across similar studies (Summary
level) were developed. The studies level visualization ([Fig. 4 ]) included data for six possible outcomes, including three follow-up terms (short,
intermediate, and long) for two outcome measures (pain or function). Outcomes were
grouped across Condition, Intervention Category, Comparator, Outcome, and Term. The
Summary level visualization ([Fig. 5 ]) included the summarized data by the factors used to group them in the original
report.
Fig. 4 Study- level visualization of data extracted from the original report.
Fig. 5 Summary-level visualization of aggregated data.
The Study and Summary visualizations were combined into an interactive dashboard ([Fig. 6 ]) that allowed users to select factors they wish to view or hide which subsequently
filtered the data depicted in the Summary and Studies sections. Additionally, selections
made in the Summary section filtered the data shown in the Study section, allowing
users to view the individual studies contained in a summary record. In Tableau, tooltips
are custom pop-up windows that are used to display additional information about the
data when selected or hovered over. For the Summary and Study data, these were displayed
were displayed when hovering over data. Tooltips within the Summary level ([Fig. 7 ]) included aggregate measures and summary statistics for the group of studies included,
such as the standardized mean difference (SMD), number of studies, and the strength
of evidence. Tooltips within the Studies level ([Fig. 8 ]) included the participant numbers for both intervention and control, study quality
according to published criteria, and citation information for the study, including
a link to the PubMed abstract. Conditional formatting of tooltips was accomplished
through creation of intermediate Calculated Fields. Dynamic filtering of study level
data and generation of PubMed URLs was completed using Custom Actions. Lastly, a final
tab was created to provide an example of the visualization's functionality by way
of a guided analysis ([Fig. 9 ]). This functionality, known as a Tableau story, demonstrated the functionality of
the tool to users using saved states of the report designed by the research team.
The final product was posted publicly on Tableau Public.[10 ]
Fig. 6 Tableau dashboard, including the combined Summary and Studies level visualizations;
global filters for data are available at the top of the page. Selecting specific output
in the Summary level data will filter the visible Studies level data and show only
data that were used to calculate the selected Summary output.
Fig. 7 Example of a Summary level tooltip. Conditionally formatted fields were created using
“Calculated Fields” in Tableau.
Fig. 8 Example of a Study level tooltip. Conditionally formatted fields were created using
“Calculated Fields” in Tableau. A hyperlink to the PubMed entry was created using
the “Actions” functionality, and appears after the user clicks the data bar in the
visualization.
Fig. 9 A guided “tour” of the dashboard was created using the Tableau Story functionality.
This allows the user to be guided through the potential use of the visualization.
The dashboard was created with an emphasis on flexible and customizable analysis of
the review data. The original report evaluated treatments by individual outcomes and
conditions to answer prespecified key questions, preventing any data analysis outside
the rigid report structure. When developing guidelines or investigating potential
treatments, end-users may want to “slice and dice” the data outside the scope of the
key questions, for example, comparing a specific treatment across all the types of
pain, or looking at multiple treatments in several conditions. The dashboard allowed
for novel analysis of tailored questions by using custom filtering of fields, in comparison
to the traditional ‘flat’ reporting format. For example, a user could easily assess
the long-term effectiveness of Pilates, a type of exercise, across chronic back pain
and chronic neck pain.
Overall, response from interviewed stakeholders was positive for the prototype both
aesthetically and for its potential functionality. The freedom of inquiry afforded
by the prototype was highlighted as a strong positive, though the need for a more
in-depth narrative as afforded in the original report may be necessary for some users
with less familiarity with the data.[8 ]
Discussion
Systematic reviews contain a large volume of data essential for decision-making, but
accessibility and usability may be hampered by “flat” reporting tools that mirror
prespecified questions. We developed a dynamic visualization of data from a completed
systematic review using the commercial product Tableau and assessed its potential
to permit customized inquiries beyond the original report structure. While the use
of Tableau for visualizing existing data are not novel, the adaptation of systematic
review data into a dynamic visualization shows potential for improving dissemination
of systematic review data. Stakeholders stated that the dynamic visual representation
of the data would be immensely useful for investigating novel questions rooted in
the local guideline committee needs; however, this method is only sufficient for looking
at the results in their original context. The original meta-analyses in systematic
reviews are calculated with many assumptions in mind and cannot be recalculated ad
hoc without careful analysis, typically by a biostatistician.[11 ] Given the density and nuance of scientific and medical data required to develop
guidelines, the visualization lacks the depth of detail and context described textually
in the written report. While the prototype is promising in its capacity to make complex
data accessible to informatics-naïve audience, the dynamic visualization should be
used only to supplement the traditional reporting methods, not replace them.
Limitations
The primary limitation of the project was the method in which the systematic review
data were abstracted. The template for data abstraction was developed to meet the
needs of a specific format, rather than a relational structure; data points were often
collapsed into a joint field, limiting the ability to programmatically extract the
data. This method is standard procedure in some groups, given regulatory requirements
for accessibility and presentation. If dynamic visualizations were included in the
analysis and reporting plan at the beginning of the review, the abstraction template
could be revised to facilitate a more amenable storage structure. Based on our experience
and assessment of staff proficiency, we recommend engaging an informaticist throughout
the project. This additional expertise will mitigate time- and resource-intensive
training of research staff, as well as ensure development, of an appropriate and sustainable
template for data extraction and storage protocols that can be shared with other centers.
If a project is started with creating a visualization in mind, then steps can be taken
to ensure that the data are collected in an appropriate way, reducing or eliminating
the need for reprocessing of data after the fact.
The second major limitation of this project was the limited scope. The scope of the
project was to develop potential visualizations of systematic review data that could
be used in guideline-development committees. While this provided potential value for
future work, the project did not include a rigorous analysis of the alternative reporting
method. Future studies would benefit from inclusion of quantitative and qualitative
comparisons between the traditional and alternative methods for reporting of systematic
review data, such as comparing usability ratings of the method or assessing retention
of the information by the users.
Other limitations were related to the structure of the desired visualization which
prohibited the inclusion of all data from the original report. For example, trials
that combined multiple interventions reported summary data, rather than outcomes by
individual intervention. We did not extract these data to simplify our initial use
case. Additionally, data that were not summarized using a standardized mean difference
(SMD) were excluded, as data on separate scales cannot be compared without standardizing.
For example, while most outcomes in the original report were reported using SMD, fibromyalgia
outcomes were measured using a common scale, so SMD was not required. To address this
limitation, we could recalculate these values for the visualization but chose not
to do so in the interest of time.
Conclusion
As more health systems invest in the development of evidence-based practices and incorporate
large volumes of complex data, the need for data dissemination will continue to grow.
The current reporting paradigm for systematic reviews is not conducive to dynamic,
efficient consumption of evidence. Innovative reporting tools are required to improve
accessibility and usability of data. We created a novel visualization prototype using
published data and an existing reporting tool and demonstrated genuine value in exploring
alternative reporting modalities. To facilitate further exploration and adoption of
these innovative modalities, we advocate for inclusion of informaticists on research
teams to inform alternative data extraction and storage practices.
Clinical Relevance Statement
Clinical Relevance Statement
The exploration of alternative reporting methods for systematic reviews is essential
to the translation of research to clinical guidelines. Current reporting via in-depth,
narrative “flat” documents is necessary but not sufficient, as their predefined and
rigid structure does not facilitate agile analysis of complex inquiry by subject matter
experts. Dynamic visualization tools show promise in improving accessibility and usability
of data, leading to more robust investigation during clinical guidelines development.
Multiple Choice Questions
Multiple Choice Questions
When considering the use of Tableau for visualization, data should be stored in or
be able to be represented in what format?
JSON.
Relational/worksheet.
Nonrelational.
SAS Grid.
Correct Answer: The correct answer is option b. Tableau uses a relational format to join multiple
data sources, resulting in a worksheet/data-frame format. This format is organized
in that each row corresponds to an individual or observation of many variables.
When considering a visualization of Systematic review data, the visualization should
be used to _____ the final report?
Correct Answer: The correct answer is option c. While the visualization is very powerful for analyzing
the data, it is not sufficient to replace the report entirely. The rigid structure
of the typical final report allows for a more in-depth analysis of the data based
on the key questions, along with a narrative portion that is harder to include in
a dynamic report.
