Keywords Medical record systems, computerised - privacy - general practice - Delphi technique
- Artificial Intelligence
Introduction
Health systems around the globe are under stress due to many socio-political factors.
The delivery of health services using optimal resources without compromising patient
safety is in demand more than ever before. The rise in ageing population with multiple
chronic diseases together with the increase of healthcare spending worldwide are some
of the key factors putting strain on healthcare systems [1 ]. Primary health care (PHC) to some extent can respond to these demands at both population
and community levels [2 ]. PHC is rapidly evolving not only in terms of health policies but also technologically.
The majority of PHC providers are now digitized and use health information systems
as part of care provision. With the advances in computational and informatics technologies
it is now possible to exploit these health information systems using Artificial intelligence
(AI) concepts such as machine learning and deep learning [3 ]
[4 ].
AI is not a new concept and has been around for more than 50 years, popularized in
the 1980s and 1990s with the advent of neural networks. However, this trend did not
last long mainly due to bottleneck in computational capabilities of hardware at the
time. With the latest advances in Graphics Processing Units (GPUs), we can now overcome
these computational limitations allowing us to develop more efficient neural networks
in the form of deep learning. Deep learning is a machine learning technique where
the models are trained using artificial neural networks with many layers (sometimes
around 1,000). Deep learning has demonstrated significant results in various non-health
and health-related applications using computer vision and natural language processing
[5 ]
[6 ]. However, very few of health-related AI systems are actually incorporated into clinical
practice [7 ].
In recent years, deep learning has been used in PHC. Abramoff et al . developed an AI system, approved by the Federal Drug Administration of USA, to detect
diabetic retinopathy in PHC centres [8 ]. A similar AI system using deep learning was developed in 2016 for the same purpose
– automated diagnosis of diabetic retinopathy [9 ]. The key limitations of these two specific systems included a need for external
validation, integration into clinical workflow, and the attitudes of clinicians [10 ]
[11 ]. AI systems based on deep learning and other similar machine learning techniques
are heavily critiqued for their ‘black-box’ paradigm wherein some of the intrinsic
estimations are not clinically interpretable in biological terms. Additionally, various
ethical issues are observed in the application of AI in PHC. One such ethical issue
is the risk of introducing bias. An AI system can incorporate the biases inherent
to the training data set, and propagate them into the validation set [12 ]. Collective knowledge from clinicians might be able to avoid these biases and subsequently
help making appropriate clinical decisions. Another ethical issue is that dependence
on AI by clinicians might change the patient-clinician relationship dynamic.
The above-mentioned studies together with several others give us a glimpse into the
future on how PHC can leverage AI. However, despite all the methodological and computational
advances in AI, very few are translated into routine clinical practice. We believe
the issues and challenges surrounding the use of AI in PHC are one of the key reasons.
Additionally, there is significant variability of opinions on the use of AI in PHC
among various stakeholders - clinicians, informaticians, AI researchers, and AI practitioners.
In this context, the International Medical Informatics Association (IMIA) Primary
Health Care Informatics Working Group undertook this Delphi study to seek consensus
on the perceptions, issues, and challenges of AI in PHC.
Methods
Consensus Exercise
We recruited volunteer health informatics experts and clinicians involved in the Primary
Health Care Informatics Working Group of IMIA to conduct a three-round Delphi study.
The study was conducted during the months of October and November 2018. Each round
lasted for about two weeks.
Round 1: Identifying the global perspectives of issues and challenges associated with
using artificial intelligence in primary care – an online survey
Round 1 was an online survey which aimed to explore clinicians’ and health informatics
experts’ awareness of typical uses of AI in primary care setting. We also inquired
about the role of AI in fulfilling requirements of safety, interoperability, data
quality, and ethics. Finally, we inquired about the future potential of AI in primary
care to enhance health care. The recruitment was done mainly within the primary care
but was also extended to other related professional networks based on their interest
and exposure to the topic. The response period for the survey was two weeks with a
reminder being sent to the invitees during the last week.
Round 2: Rating statements using the RAND/UCLA appropriateness method – an online
survey
The responses from Round 1 were tabulated and analysed by the authors. Responses were
organised according to a series of themes. We created 14 consensus statements based
on the responses and across the themes identified. The 14 consensus statements were
sent to the panel of 20 experts who responded to Round 1. In addition to the consensus
statements, we also included two open ended questions to capture additional information
on specific uses of AI and the clinician’s role as a learned intermediary. The two
open ended questions were:
Are there any other AI use cases or scenarios that you would like to see included?
In the AI environment, does the clinician still have a role as the “learned intermediary”
between the system/knowledge source and the patient?
Twelve participants (60%) responded to the Round 2 survey. The list of statements
is given in [Table 1 ]. We replaced the standard terms used in the UCLA/RAND appropriateness method, “Highly
appropriate” and “Highly inappropriate”, with “Strongly agree” and “Strongly disagree”.
Round 3: Discussion of the findings by health informatics experts – an online panel
discussion
The final round of the consensus process was an online panel discussion conducted
as a video conference with a shared screen to present results. Three separate online
meetings were organised to engage panel members in different time zones. Thirteen
experts (65%) participated to this round.
Fig. 1 Distribution of the health informatics experts who participated to the Round 1.
Table 1
Examples of benefit use cases in which AI can be leveraged in a primary care setting
as suggested by the panel members.
Themes
Examples of benefit use cases of AI in primary care setting
Decision support to improve primary health care processes
a) Improving accessibility by triaging primary care patients and conduct a preliminary
analysis suggesting likely diagnosis.
b) Learning preferred prescribing patterns of clinicians that use AI-enhanced computerised
medical records
c) Assisting the prototype development of decision support tools
Pattern recognition in imaging results
a) Automatic detection of tumours using whole slide digital pathology images
Predictive modelling performed on primary care health data
a) Detection of high risk for mental health disorders/ cardiovascular disease
b) AI-driven tools for clinicians e.g. prediction of mortality
c) Assistance with diagnosis of obscure cases using iterative algorithms of accumulated
case histories
d) Assistance with management of complex cases, using iterative accumulation of outcome
data (big data repositories with complex neural networks)
e) Early diagnosis of diseases in primary care patients
Business analytics for primary care provider
a) AI applications that operate on routinely collected administrative data could provide
regular feedback to practice managers, business owners, and individual clinicians
(doctors, nurses, and others) to reduce variability and improve quality of care
b) AI modelling of administrative data could assist in finding organizational models
for an effective comparison among different countries
Results
The process involved inviting and consulting with an international panel of 20 experts
from 9 countries: Australia, Belgium, Canada, Croatia, Italy, New Zealand, Spain,
United Kingdom, and USA.
Panel Characteristics
The panel included experts from a range of professions including clinicians (7), academics
(9), informaticians (2), and researchers (2). The majority of panel members were knowledgeable
about AI although they did not have substantial hands-on experience with utilising
AI applications in practice.
Benefit Use Cases for AI in Primary Care
The panel provided a range of benefit use cases where they considered AI to be a useful
addition in the primary care setting. The responses received are generalised across
several themes in [Table 1 ].
Risks Associated with Using AI in Primary Care
The panel was asked about potential risks associated with the use of AI as an integral
part of primary health care. We have grouped the use cases to generalise situations
that could potentially be harmful to patients in [Table 2 ].
Table 2
Examples of risk use cases in which AI could result in a potential risk to patients
in primary care as suggested by the panel members.
Themes
Examples of risk use cases of AI in primary care setting
AI technology currently available to deploy in primary care is still not competent
to replace human decision making in clinical scenarios
a) Interpreting the results of an analysis using AI without an understanding of the
primary health care context
b) Overreliance on what AI can do. Using AI as a substitute for due clinical diligence
c) Missing competencies/willingness in using AI properly
d) In AI, few techniques such as deep neural networks are incapable of explaining
the underlying models completely. This makes it hard to interpret the interplay between
covariates in a model
e) Relying on AI and not using human skills to ensure it is correct
f) Going down the primrose path. One of the most dangerous aspects of black-box algorithms
is not knowing the source of the data. To take an extreme example, if the AI is built
for fever of unknown origin at a major referral hospital in the US, it will not be
applicable to a patient with fever in sub-Saharan Africa who in fact has malaria.
Risk of medical errors
a) Potential for errors in prescribing. If a doctor prescribes a medication using
adult doses for a child, and the AI doesn’t have a guideline to spot the error, the
AI could propagate the error into the child’s future and that of other children on
the same medication. This happens with humans (who are experts and specialists) and
can happen in a learning AI scenario
b) Incorrect diagnosis leading to unnecessary treatment
c) Assumed effectiveness before proper trials undertaken
Risk of bias
a) That the data behind the constructed AI knowledge model was biased, or not compatible
with the patient to whom the clinician applies the AI: e.g., a model learned in a
population with specific sub-phenotypes may not be adequate to another population,
or a model learned with past data models (ICD-9) may not be adequate/generalizable
to new data models (ICD-10)
Risk of secondary effects of utilising AI
a) Insurance providers using AI for higher premiums or even excluding certain people
for insurance
In order to enable safe use of AI applications in primary care, it was believed that
input data (for AI application), output data, and access protocols should be kept
within a secure infrastructure. For a safe processing of patient data, compliance
with data protection regulations such as the General Data Protection Regulation (GDPR)
was considered to be important.
Adoption of AI in Primary Care
The panel members unanimously agreed that using common standards in computerised medical
records such as common data models, common metadata standards, common terminologies,
and common data quality metrics would facilitate effective implementation of AI across
various primary care providers. To encourage the adoption of AI in primary care, the
panel strongly believed that AI applications need to be usable within the practitioner’s
workflow and relevant to clinical practice. Respondents expressed the desire for a
strong evidence to support AI. The panel, however, had mixed views about cost efficiency
being a factor for encouraging adoption of AI.
Ethical and Lawful Processing of Patient Data by AI Applications
The panel agreed that AI applications require close monitoring when processing patient
data. There was an agreement for the need for compliance with standards for AI applications
and the need for transparency regarding data processing. Most panel members considered
informed consent to be important for lawful processing. There was less agreement that
the principle of the “learned intermediary” applies to AI applications (the “learned
intermediary” principle holds clinicians responsible to use technology in combination
with their professional knowledge and experience providing care) [13 ]. Similarly, there was less agreement for the requirement of an ethics committee
to have well defined processes for dealing with inconsistent outputs from AI applications.
Implications of AI in Learning Health Systems
The expert panel members provided a range of implications from using AI in learning
health systems. They indicated that learning health systems should include a useful
collection of methodologies that will help reflect data back to the system to drive
quality improvement. They also suggested that AI may optimise and tailor best practice
to the local environment. This is positive at first but over time the system may begin
to overfit, meaning that the learning system reached a saturation point from which
it may be difficult to learn new changes, especially if they are contradictory to
what was previously learnt. AI systems must be open-minded over time to adopt and
perhaps even challenge contradictory rules and behaviours.
Future of AI in Primary Care
The increasing use of electronic medical record systems in the last few decades means
that there is a large volume of data available for AI applications to utilise. AI
can help by augmenting (supporting) tasks such as decision making to reduce cognitive
burden on clinicians. This would be particularly helpful for challenging diagnostic
or therapeutic decision-making. It can also do the background data analysis to enable
providers to have a more integrated record of their patient during a consultation.
AI may also have an important role in identifying populations of patients at risk.
The panel members expressed optimism that AI would be most promising to learn new
risk stratification models and rules from GP data. In particular, AI systems may help
reduce health inequalities by surfacing the most vulnerable patients. The need for
clinicians to drive care delivery will not go away, and in fact will become more critical
since various outcomes suggested by AI applications required physician validation
for the particular patient. Widespread acceptance of AI outputs requires considerable
further work to assure it a place as an additional and completely trusted source for
direct patient care. As an example, panel members speculated that as physicians learn
to validate or refute deep learning decisions which may initially appear non-plausible,
this will increase physicians’ trust in AI processes. Over time, we can either accept
these as good AI decisions, or learn when the human brain may need to override a proposition
for a final decision.
There was a good degree of consensus, as defined by the RAND/UCLA method [14 ] by the end of the final round (see [Table 3 ]). The statements for Round 3 (Table 2) had agreement on 8 out of 14 statements.
Table 3
Consensus statements generated from the analysis of Round 1’s responses (with Agreement
written in green, Equivocation in brown, and Disagreement in red according to responses
from Round 1).
Statement 1 - The most prevalent use of AI currently in primary care is for predictive
modelling (e.g. detection of high risk for mental health disorders / cardiovascular
disease) based on knowledge inferred from large clinical datasets.
Statement 2 - AI in primary care is currently needed more to manage provision of care
(e.g. triage) than for clinical decision support.
Statement 3 - AI applications can be incorporated more easily in business analytics
in primary care than analytics to support the clinical process.
Statement 4 - AI applications should be capable of assessing and adapting to the preferences
of a clinician (e.g. learning about preferred medication that a clinician prescribes
for male adult hypertensive).
Statement 5 - (Over) reliance on AI applications to make clinical decisions can be
harmful to patients.
Statement 6 - Current AI applications mainly operate as black boxes (from the perspective
of clinicians) and therefore need regular scrutiny by users (e.g. clinicians and managers)
Statement 6 - Excessive patient data will reduce the effectiveness of patients’ online
experience. [Inhibitor] [Equivocation]
Statement 7 - Current datasets used to train and testing AI applications are not representative
of patient services enhances shared decision-making. [Enabler] [Disagreement]
Statement 9 - Access to patient data such as radiology results or lab results will
not be cost-beneficial as it will not be used by the wider patient population. [Inhibitor]
[Disagreement]
(a) the real world (e.g., a patient wearing fitness monitoring devices may be healthier
than the general population (worried well)).
(b) specified population (e.g., a model learned in a population with specific sub-phenotypes
may not be applicable to other populations).
(c) the underlying terminological system (e.g., a model learned with past data
models (ICD-9) may not be adequate/generalizable to new data models (ICD-10 or SNOMED-CT)).
Statement 8 - Clinical decisions made by AI applications may lead to unnecessary treatment
which may not be those recommended by evidence-based guidelines.
Statement 9 - Ethics committees (or institutional risk management committees) should
be trained in formal processes to assess the ethical processing of data in AI applications.
Statement 10 - Data governance committees should also oversee AI applications.
Statement 11 - Data processing in AI applications needs to be monitored closely.
Statement 12 - Data output display needs to be assessed for fidelity and quality.
Statement 13 - Mechanisms to identify biases in unsupervised algorithms need to be
implemented in all AI applications.
Statement 14 - Advances in AI application in primary care will lead to improvement
of a) clinical decision making; b) risk assessment; c) care processes; d) continuity
of care; e) coordination of care; f) safety of care; and g) managerial processes in
health care.
The expert panel discussed various possible reason for the variability in agreement
levels for the statements in Round 2. The discussion section incorporates feedback
received during these meetings.
Discussion
Principal Findings
The participants suggest that AI has potential to improve primary health care but
unsupervised machine learning is currently not sufficiently mature or robust to be
confidently used without checks in place. They were mostly in agreement that advances
in AI application in primary care can lead to improvement of managerial and clinical
decisions and processes. The primary care community needs to be proactive and guide
the ethical and rigorous development of AI applications so that they will be safe
and effective in the workplace.
The most established use of AI in primary care reported is suggested to be predictive
modelling [15 ]. This is likely to be because the respondents do not have substantial clinical experience
with AI tools - their suggested use cases may be more academic and non-clinical such
as predictive modelling. Similarly, their responses to the statements are likely to
be more academic.
Participants also agreed that formal processes need to be developed and Ethics Committees
(or Institutional Risk Management Committees) be trained to assess the ethical processing
of data in AI applications. Data governance committees should contribute the oversight
of AI applications and have processes in place to monitor data processing in and outputs
from AI applications for fidelity, bias, and quality.
There was less agreement on whether AI applications should be focused on service provision
or decision support or whether it was easier to support the managerial or clinical
process. It was not agreed that AI applications should learn and adapt to clinician
preferences or behaviour or the extent of the potential for harm to patients.
Implication of Findings
The clinical and informatics community need to establish the professional rules for
the initial and on going use of AI applications to support managerial or clinical
practice. Specific legislation may be needed to address some of the more intractable
issues such as the liability for “black box” approaches of AI or even the liability
of the clinician as a learned intermediary.
There is an agreed need for regular scrutiny by users (e.g., clinicians and managers)
because the accuracy, fidelity, or relevance of the output of AI is not guaranteed,
the current training datasets for AI applications may not be representative of specific
populations or of the underlying terminological system or data models, and there is
a need for mechanisms to identify biases in unsupervised algorithms. Identification
of biases should be followed up with “unlearning” processes that increase the accurate
functionality of AI applications [16 ]
[17 ].
Caution is needed as it may be more difficult to assess the impact of AI-based applications
on continuity and coordination of care.
The panel members noted the unexpected finding that there was a lack of consensus
regarding the potential for AI to assist and adapt to clinician preferences. Neural
networks can continuously learn, which could assist primary care clinicians to define
their particular patient population as well as include PHC’s individual treatment
preferences. Yet respondents appeared to not agree with this. Perhaps this was due
to a misunderstanding of the question, or perhaps the panel was diversely versed in
the promise of AI. The findings of our study closely mirrored outcomes of a recent
qualitative survey involving a large cohort of general practitioners in the UK in
which they expressed both scepticism and optimism on the notion of replacing human
roles in health care using AI [18 ].
Limitations of the Method
We used an opportunistic sample of health informatics experts drawn from international
Primary Care Health Informatics Working Groups. While a globally representative list
of experts was invited, there was no response from the African, South Asian, and Middle
Eastern countries. Because respondents did not have substantial clinical experience
with AI tools, their suggested use cases may be more academic and non-clinical such
as predictive modelling. Similarly, their responses to the statements are likely to
be more academic. In addition, as with most self-reported methods, the phrasing of
questions may have an effect on the responses obtained.
Conclusions
PHC and informatics experts reported that AI has the potential to improve managerial
and clinical decisions and processes. However, unsupervised machine learning is currently
not sufficiently mature or robust to be used confidently without checks in place.
The primary care informatics community needs to be proactive to guide the ethical
and rigorous development of AI applications so that they will be safe and effective
in the workplace.
