Introduction
Proper implementation of guidelines has the potential to improve health care quality
and reduce costs by promoting the use of evidence-based interventions [1]
[2]
[3]. However, these benefits vary significantly, based largely on the way guidelines
are applied in clinical practice. Physician adherence to guidelines is a determining
factor in their successful implementation [4]. It has been suggested that physicians have difficulties in memorizing and processing
the complex information contained in some guidelines.
Computer-based clinical decision support systems (CDSSs) are defined as “any software
designed to aid in clinical decision-making in which characteristics of individual
patients are matched to a computerized knowledge base for the purpose of generating
patient-specific assessment or recommendations, that are then presented to clinicians
for consideration” [5]. As a result of the reported suboptimal evidence-based health care delivered and
its critical consequences for patient health outcomes [6]
[7]
[8], different health care organizations have been promoting CDSSs, aiming to improve
care delivery [9]. These systems have been shown to facilitate prescribing practices, reduce medication
errors, enhance the delivery of preventive care services, and improve adherence to
clinical guidelines [10]
[11]
[12]
[13]
[14]
[15].
Taking into account the global use of cell phones, the development of technologies
and software integrated into a standard mobile app independent of personal computers
could provide an effective approach i to allow these decision support systems to be
implemented into other clinical information systems, such as a mobile decision support
system (MDSS) [16]
[17]
[18].
Colorectal cancer (CRC) is a major global medical and public health challenge, accounting
for 881.000 deaths annually [19]. Evidence-based CRC screening strategies have shown to be effective in reducing
CRC mortality. Hence, the majority of CRC deaths are believed to be the result of
CRC screening process breakdowns [20]. Multiple studies report overuse and underuse of colonoscopy, as well as poor physician
knowledge about and adherence to CRC screening guidelines, impairing their ability
to apply the guidelines in clinical practice. All these factors may threaten the effectiveness
of CRC screening programs.
Therefore, we developed a mobile-based ap,p named CaPtyVa CCR, that could be useful
for physicians to improve CRC screening and surveillance guideline-based decision-making
[21].
The aim of this prospective study was to evaluate whether this app improved the performance
of specialists in gastroenterology and colorectal surgery in addressing CRC screening
and surveillance clinical scenarios according to current guidelines.
Patients and methods
Study design
A prospective, binational, single-blinded, randomized study was conducted between
May and June 2020. A 10-question online survey was created with CRC screening and
surveillance clinical cases reflecting real-life daily practice scenarios. After the
questions were developed, they were pretested and discussed for content and clarity
by three board-certified gastroenterologists with more than 10 years’ experience in
practice ([Fig. 1]). Gastroenterologists and colorectal surgeons who were members of four medical societies
from Argentina and Uruguay—Endoscopistas de Buenos Aires (ENDIBA), Federación Argentina
de Endoscopía Digestiva, Sociedad Argentina de Coloproctología (SACP), and Sociedad
Uruguaya de Endoscopía Digestiva—were randomly selected and invited by email to participate
in this study and to answer the online questionnaire. One additional reminder email
was sent 1 week later to increase the response rate. Only physicians who responded
to the email and consented to enter the study were randomized and included in the
trial. All participants were asked to complete a digital informed consent explaining
the purpose of the study, the inclusion and exclusion criteria, and investigator and
participant responsibilities.
Fig. 1 a CRC screening clinical vignettes used to test gastroenterology and coloproctology
specialist performance according to current local guidelines. b Surveillance clinical vignettes used to test gastroenterology and coloproctology
specialist performance according to current local guidelines. Boldface indicates the
correct answer according to current local guideline.
The study was approved by the Institutional Review Board and Ethics Committee of the
Hospital Alemán, Buenos Aires Argentina (2632 V4). The study was also registered at
Clinical Trials (NCTNCT04389502).
Assignments and blinding
The specialists included in the study were randomly allocated in a 1:1 ratio using
a random numbers sequence created with Microsoft Excel (Microsoft Corp. 2016. Redmond,
Washington, United States) (central randomization). Randomization and allocation were
performed by an investigator who was blinded to physician specialty, experience, CRC
screening guideline knowledge, and endoscopy practice status. The participants were
divided into two groups: 1. control group (asked to complete the questionnaire based
on their current knowledge or in the same way they did it in daily practice); and
2. intervention group (received a version of the app and a tutorial video and were
asked to complete the questionnaire using the dedicated app).
Investigators were blinded to treatment allocation throughout the study until planned
statistical analysis was finished. All authors had access to the study data and reviewed
and approved the final manuscript.
Assessments
Physician demographic data, including age, sex, specialty (gastroenterology or coloproctology),
professional experience, location, place of work, and use of smartphone medical tools
were assessed.
Outcome measures
To assess physician performance, the main outcome measure was the proportion of physicians
correctly responding to at least 60 % of the clinical scenarios, according to local
current guidelines. Answers consistent with the current CRC screening and surveillance
guidelines of the Instituto Nacional de Cáncer from Argentina [22] (INC) were considered accurate. The secondary outcome was the proportion of physicians
correctly responding to at least 80 % of the clinical vignettes. The proportion of
physicians correctly responding to the clinical vignettes across their different categories’(screening
and surveillance) were also analyzed.
Physician perception about the ease of the app’s operation, as well as its usefulness
in daily practice, was also assessed in the intervention group with a five-poiknt
Likert scale questionnaire.
CaPtyVa CRC app development
The CaPtyVa Mobile app was custom developed with the support of ENDIBA and INC for
the purpose of improving knowledge about CRC screening and surveillance and of increasing
adherence to clinical practice guidelines among physicians. The app stores information
about all the current local clinical practice guidelines related to CRC screening
and surveillance, and through its decision-making algorithms, artificial intelligence
assists physicians in the screening process recommendation. In the screening function,
the app helps physicians to easily classify patients according to their individual
CRC risk and to determine how and when to start screening tests ([Fig. 2a], [Fig. 2b], [Fig. 2c]). In the surveillance function, the app assists physicians in determining when to
repeat the next colonoscopy, according to guideline recommendations ([Fig. 2 d], [Fig. 2e], [Fig. 2f]).
Fig. 2 a, b, c App screenshots showing the main menu, the screening function data loader, and the
screening query results. d, e, f The main menu, the surveillance functios data loader, and the surveillance function
query results.
Sample size calculation and statistical analysis
Based on previous publications, the proportion of gastroenterologists responding to
100 % of the clinical vignettes correctly was observed to be only 22 % to 37 % [21]. The study expert committee from the participating societies agreed that a minimum
of 60 % correct responses to the clinical vignettes was required to achieve acceptable
guidelines adherence in daily practice. Based on a US survey [23], 60 % of the gastroenterologists responded correctly to at least 60 % of the clinical
vignettes, and 31 % to at least 80 % of the vignettes. We hypothesized that the app
would increase by, at least, 20 % the proportion of physicians that responded correctly
to 60 % of the vignettes and estimated that 82 physicians in each group would provide
the study with 80 % power to detect a difference at a two-sided significance level
of 0.05. Considering a non-response rate of 35 %, 250 invitations were sent.
Statistical analysis was performed using STATA 13 (StataCorp. 2013. College Station,
Texas, United States). A chi-square test, Fischer´s exact test, student t-test, and
correlation analysis were used whenever applicable. Logistic regression analysis was
used to determine the characteristics associated with physician performance greater
than 60 %.
Results
A total of 250 physicians were initially invited, of whom, 213 answered the invitation
(85.2 %) and were included in the study and randomized 1:1 ton one of the two study
groups ([Fig. 3]). The mean age of participants was 42.15 years ± SD (9.74) with a mean time practicing
in the specialty of 9.10 years. The majority of the participants were gastroenterologists
(50 %) and 85.1 % performed colonoscopies and 76.8 % reported using smartphone medical
tools in daily practice. There were no significant differences between the two groups
regarding professional or demographic characteristics ([Table 1]).
Fig. 3 CONSORT flow diagram.
Table 1
Characteristics of the physicians included in the study by group assignment.
|
app group
|
Control group
|
|
|
|
n = 101
|
n = 112
|
|
|
|
N
|
%
|
N
|
%
|
P
|
Test
|
|
Age (yr, ±SD)
|
42.40
|
± 9.95
|
41.93
|
± 9.59
|
0.73
|
t-test
|
|
Physician specialty
|
|
|
90
|
89.11
|
93
|
83.04
|
0.24
|
Chi-square
|
|
|
11
|
10.89
|
19
|
16.96
|
|
Time practicing the specialty
|
|
|
31
|
30.69
|
36
|
32.14
|
0.19
|
Chi-square
|
|
|
21
|
20.79
|
31
|
27.68
|
|
|
26
|
25.74
|
16
|
14.29
|
|
|
33
|
22.77
|
29
|
25.89
|
|
Orders screening colonoscopy
|
101
|
100
|
112
|
100
|
0.62
|
Chi-square
|
|
Performs colonoscopy
|
90
|
89.11
|
94
|
83.93
|
0.28
|
|
Current practice setting
|
|
|
15
|
14.85
|
11
|
9.82
|
0.33
|
Chi-square
|
|
|
45
|
44.55
|
60
|
53.57
|
|
|
41
|
40.59
|
41
|
36.61
|
|
Uses of any smartphone medical tool
|
77
|
76.24
|
87
|
77.68
|
0.80
|
Chi-square
|
|
Place of residence
|
|
|
65
|
64.36
|
78
|
69.64
|
0.57
|
Chi-square
|
|
|
10
|
9.90
|
5
|
4.46
|
|
|
7
|
6.93
|
6
|
5.36
|
|
|
8
|
7.92
|
11
|
9.82
|
|
|
1
|
0.99
|
3
|
2.68
|
|
|
10
|
9.90
|
9
|
8.04
|
SD, standard deviation.
Continuous variables are expressed as mean ± SD. P < 0.05 is considered statistically significant.
Primary outcome
The proportion of physicians who correctly responded to at least 60 % of the clinical
vignettes according to current local guidelines was significantly higher in the app
group as compared to the control group (90 % versus 56 %; RR1.6, 95 % confidence interval
[CI] 1.34–1.9). Performance was also higher in the app group when analyzing both vignette
categories: CRC screening (93 % versus 75 %; RR 1.24, 95 %CI 1.01–1.40) and surveillance
(85 % versus 47 %; RR 1.80 95 %CI 1.46–2.22), respectively ([Table 2]).
Table 2
Primary and secondary outcomes.
|
Outcome
|
App group
(N = 101)
|
Control group
(N = 112)
|
Relative risk
(95 %CI)
|
P value
|
|
Primary outcome-number (%)
Physicians correctly responding to ≥ 60 % of the clinical vignettes
(CRC screening and surveillance)
|
91 (90.1)
|
63 (56.2)
|
1.60 (1.34–1.91)
|
< 0.001
|
|
Physicians correctly responding to ≥ 60 % of the screening clinical vignettes
|
94 (93.1)
|
84 (75.0)
|
1.24 (1.01–1.40)
|
< 0.001
|
|
Physicians correctly responding to ≥ 60 % of the surveillance clinical vignettes
|
86 (85.1)
|
53 (47.3)
|
1.80 (1.46–2.22)
|
< 0.001
|
|
Secondary outcome-number (%)
Physicians correctly responding to ≥ 80 % of the clinical vignettes
(CRC screening and surveillance)
|
70 (69.3)
|
16 (14.3)
|
4.85(3.03–7.78)
|
< 0.001
|
|
Physicians correctly responding to ≥ 80 % of the screening clinical vignettes
|
85 (84.2)
|
52 (46.4)
|
1.81 (1.46–2.25)
|
< 0.001
|
|
Physicians correctly responding to ≥ 80 % of the surveillance clinical vignettes
|
55 (54.5)
|
11 (9.82)
|
5.54 (3.08–9.99)
|
< 0.001
|
Physician performance in CaPtyVa app group compared with control group for primary
and secondary outcomes. Results are expressed as percentages with relative risks and
their corresponding 95 % confidence intervals (CI).
Secondary outcome
The proportion of physicians who correctly responded to at least 80 % of the vignettes
according to local current guidelines was also higher in the app group as compared
to the control group (69 % versus 14 %; RR 4.85, 95 %CI 3.03–7.78). Performance was
also higher in the app group for both vignette categories: CRC screening (84.2 % versus
46.4 %; RR 1.81(1.46–2.25) and surveillance (54.5 % versus 9.82; RR 5.54, 95 %CI 3.08–9.99),
respectively ([Table 2]).
The app group was superior to the control group for almost every clinical vignette.
[Fig. 4] summarizes the results for each clinical vignette comparing the performance of physicians
in the CaPtyVa app group with those in the control group.
Fig. 4 Physician performance (correct answers) in the app group compared with the control
group for each clinical vignette. Results for each question are expressed as relative
risks with their corresponding 95 % confidence intervals (CIs).
Independent predictors of performance according to current local guidelines
Time practicing in the specialty was the only significant independent predictor of
correctly responding to at least 60 % of the clinical vignettes according to current
local guidelines. Physicians with more than 15 years of specialty practice were significantly
less likely to achieve at least 60 % correct answers (OR 0.37 [95 %CI 0.18–0.70],
P = 0.003) in the multivariate analysis.
Physician perception about the app
Regarding physician perception about the app, 96 % of them agreed or strongly agreed
(Likert scale: 4 and 5) that the app was easy to use, while only 4 % of them strongly
disagreed, disagreed or were neutral (Likert scale: 1, 2 and 3). On the other hand,
89 % of the physicians agreed or strongly agreed (Likert scale: 4 and 5) that its
implementation in daily practice could be of great use, and only 11 % strongly disagreed,
disagreed or were neutral (Likert scale: 1, 2 and 3)
Discussion
To our knowledge, this is the first randomized controlled trial evaluating the efficacy
of a MDSS for improving physician performance in responding to CRC screening and surveillance
daily practice clinical scenarios. The present study shows that the utilization of
a CRC screening MDSS by gastroenterology and colorectal surgeons improved their performance
in responding to CRC screening and surveillance clinical scenarios based on guidelines.
In addition, the app was perceived by most of the physicians as “easy to use” and
as a tool of great utility in clinical practice.
Our study contributes several important pieces of information that add to understanding
of the potential role of MDSSs in CRC screening clinical guideline implementation.
First, this study documented substantial limitations in knowledge about and adherence
to CRC screening and surveillance guidelines among gastroenterology and colorectal
surgeons, an issue that needs to be addressed in order to optimize the allocation
of resources to CRC screening policies. Only 56 % of the specialists included in the
control group of our study were able to correctly respond to at least 60 % of the
clinical scenarios with their current knowledge, while only 14 % could respond to
at least 80 % of them. These results are congruent with those reported in previous
similar studies that also aimed to assess gastroenterologist guideline knowledge.
A US national survey of 306 gastroenterologists using 12 clinical scenarios to test
knowledge of both CRC screening and surveillance showed that only 60 % of the physicians
scored greater than 60 %, also highlighting a knowledge deficit [23]. In the largest US nationwide survey of 1432 gastroenterologists, Patell et al evaluated
guideline knowledge using four clinical vignettes and correct identification of all
factors used to determine screening policies, such as age to start or intervals of
surveillance colonoscopy. This study also showed that only 22 % of respondents were
100 % accurate about screening and only 37 % were 100 % accurate about surveillance
[21]. Thus, the poor specialist clinical practice guideline level of knowledge observed
in our study is comparable to that observed in other studies and highlights the need
for implementing new strategies to improve clinical practice guideline knowledge and
adherence. In our study, the only factor independently associated with physician performance
according to clinical practice guidelines was the number of years practicing in the
specialty. In multivariate analysis, those with more than 15 years of practice were
significantly less likely to respond correctly to at least 60 % of questions This
result is also congruent with previous studies that showed that more recent training
among specialists was associated with a greater knowledge about and adherence to CRC
screening guidelines [21], proving that the population included in our study is similar to those included
in similar studies.
The primary outcome measure of our study was the proportion of physicians correctly
responding to at least 60 % of the clinical scenarios according to local current guidelines.
This same outcome has also been used in previous studies to assess physician screening
guideline knowledge and adherence. Guideline knowledge has been shown to be associated
with better guideline recommendation adherence in clinical practice [24]. Therefore, the poor guideline knowledge observed in our study and in previous similar
studies may reflect the reality of practice in the real world and may have important
implications for healthcare expenditures. In addition to studies showing a low level
of guideline knowledge, multiples studies reporting very high rates of screening colonoscopy
overutilization and underutilization also have been published in the last decade.
A recent meta-analysis published by Djinbachian et al that included 16 studies reporting
on guidelines adherence for surveillance colonoscopy in different countries showed
an adherence rate of only 48 % (95 %CI 37.3 –60.4) [25]. This meta-analysis demonstrated a low worldwide adherence to surveillance colonoscopy
guidelines, with an overall guideline adherence rate of less than 50 %. This lack
of adherence to guidelines manifested through overutilization and underutilization
in the above-mentioned study represents a clear example of how deviation from recommendations
may threaten the effectiveness of CRC screening programs, as low-risk lesions or negative
colonoscopies were assigned to shorter than recommended intervals, whereas in contrast,
high-risk lesions were associated with delayed surveillance intervals. In fact, the
consequences of these screening process failures have been demonstrated in a recent
study by Doubeni et al, which showed that screening at inappropriate intervals or
the failure to receive adequate-follow up for abnormal results significantly increased
the risk for CRC death [21]. Based on all the above-mentioned studies, we consider that physician guideline
knowledge and adherence is a determining factor for successfully implementing CRC
screening programs. Although we are aware that there are many possible explanations
for non-adherence to guidelines, such as disagreement with the guidelines, physician
concerns about missed polyps, and perhaps financial reasons [26]
[27]
[28]
[29], the implementation of strategies that prove to to improve physician knowledge about
clinical practice guidelines will have a positive impact on guideline adherence, and
therefore, CRC screening effectiveness.
Our study shows that the use of a CRC screening MDSS is effective in improving the
performance of specialists in responding correctly to both CRC screening and surveillance
clinical daily practice scenarios, and it may, therefore, improve adherence. Physicians
assigned to the app group had a higher probability of correctly resonding to at least
60 % of the clinical vignettes as compared to controls (90 % versus 56 %; RR1.6 CI
1.34–1.91). The app group was also superior to the control group regarding secondary
outcomes and in responding to both vignette categories – screening and surveillance.
Various educational strategies have been employed in the past to improve knowledge
of CRC screening, ranging from didactic lectures and guideline checklists, to an interactive
case-based model [30]
[31]
[32]. However, these educational methods resulted in only a modest and variable improvement
in knowledge. Their widespread implementation may require a significant effort and
maintaining these educational strategies over time remains challenging. The technological
evolution in smartphone mobile computing has given rise to the development of a new
class of decision support systems, known as MDSS. These systems can be very beneficial
for a wide range of different activities in which complex decisions are made under
time pressure, decision-makers are on the move, and for which a large amount of constantly
updated information need to be managed. Physician lack of time during a consultation
adds to the large amount of complex and constantly updated information contained in
CRC screening and surveillance guidelines, making MDSS an ideal resource for improving
physician evidence-based decision-making about CRC screening. Using MDSS to guide
clinical decision-making can increase standardization, which could have a positive
impact on healthcare resource utilization, reduce risk of errors, eliminate unjustified
variation in treatment, and increase patient safety. Because MDSS recommendations
are the ones provided by clinical practice guidelines and are periodically updated
when new information becomes available, the source documents and not the MDSS should
be referred to as legal back-up whenever needed. The purpose of these kind of tools
is to assist physicians in the clinical decision-making process and they should never
replace clinical judgement.
Regarding the study design, a crossover study could have been an appropriate design
for this research, based on the need for fewer participants than parallel trials to
achieve sufficient power. Nevertheless, we chose not to use a crossover design because:
1) they usually take longer to complete, increasing the risk of attrition bias; and
2) we had some concerns about carryover effect and the adequate washout period to
answer known clinical vignettes.
Our study does have certain limitations. First, although it was a binational study,
only a particular study population participated (physicians who were smartphone users
and who responded to the invitation to participate), which limits the generalizability
of the results. Second, the main outcome measure was physician knowledge and not the
actual adherence to guidelines in clinical practice. In addition, the clinical cases
were based on a local guideline for CRC screening adopted in Argentina, which may
differ slightly from other international guidelines.
Conclusions
In conclusion, our study demonstrated that a MDSS is an effective tool to improve
the ability of colorectal surgeons and gastroenterologists to respond correctly to
CRC screening and surveillance scenarios seen in daily clinical practice. This dedicated
mobile app was shown to be user-friendly and potentially useful in clinical practice.
Digital resources may help physicians to make evidence-based decisions about CRC screening,
improving its effectiveness.
