A Scoping Review of Integrated Medical Devices and Clinical Decision Support in the Acute Care Setting

Keywords

integrated devices - clinical decision support - interoperability

Background and Significance

The Institute of Medicine's report “To Err is Human” was published in 2000. At that time, it was estimated that 44,000 to 98,000 people died from preventable harm in health care settings due to medical errors.[1] This report was considered by some to be the impetus of the patient safety movement,[1] resulting in changes to incident reporting requirements, financial penalties related to preventable harm events, and the coalescing of health care organizations, payers, clinicians, researchers, and others around harm reduction iniatives.[2] [3] [4] [5] Despite this focus on patient safety, hospitalized patient harm remains a persistent problem.[3] Makary and Daniel estimated medical errors as the third leading cause of death in the United States, with 251,000 people dying each year.[6] [7] [8] Clinical decision support systems (CDSS) and medical device integration are two examples of health information technology (IT) and informatics inventions designed to enhance patient safety and harm reduction during care delivery by raising the situational awareness and care coordination activities of the health care team.

Medical devices are essential to the provision of care for patients in the hospital setting. These can include point-of-care testing devices such as glucometers, monitoring devices such as automated vital sign machines, or devices that support physiological functions such as ventilators. Integrated devices that are interoperable with the electronic health record (EHR) allow for timely information exchange without impacting clinical workflow, while also attempting to ameliorate errors such as transcription inaccuracies or adding to clinician burden.[9] [10] Inputs from integrated medical devices[11] are an important source of data used in CDSS.

CDSS generate suggestions that support clinicians' decision making by matching evidence-based computerized knowledge, contextual assessments, and individual patient characteristics to generate patient-specific recommendations to the clinician.[12] There are numerous functions and advantages of CDSS, including those pertaining to patient safety, clinical management, cost containment, automation, diagnostics support, patient decision support, documentation, and clinical workflow. There are some potential hazards with CDSS, including an overreliance on the CDSS by the clinician or dismissal of CDSS guidance due to inappropriate alerts or excessive alarms.[13] CDSS guidance should be considered an adjunct to decision making, not a replacement for clinical expertise, and decision support should be optimized to avoid alert fatigue.[7] To further illustrate, how data generated from a device prevents patient harm by clinical decision support (CDS), consider the example of a diabetic patient who requires hourly glucose checks to titrate an insulin drip. In this case, the glucose measure is taken and transmitted to the EHR. The registered nurse (RN) can then adjust the infusion to be aligned with the provider's order, which is based on a glycemic range. However, if the RN has some concerns about the range based on clinical expertise, or patient presentation, they can escalate their clinical concerns to the provider team.

The objective of this scoping review is to (1) examine the existing evidence related to integrated medical devices, notably medication administration pumps, and associated CDS in acute care settings and (2) to identify how acute care clinicians may use device CDS in clinical decision making. The rationale for this review is that integrated devices are pervasive in the acute care setting, and they generate data that may help to contribute to the situational awareness of the clinical team necessary to provide individualized patient care. The Joanna Briggs Institute (JBI) Manual for Evidence Synthesis and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extensions for Scoping Review[14] [15] guidelines were used in this review. JBI is a methodology for conducting scoping reviews that includes developing an a priori protocol, inclusion and exclusion criteria, and an iterative approach to article review to ensure a systemic, transparent review process.[15] Scoping reviews differ from systematic reviews in that scoping reviews map the existing evidence and are conducted as a precursor to a systematic review.

Methods

Information Sources and Search Terms

A comprehensive search strategy is a critical step when conducting a scoping review.[11] [15] Our search strategy used truncation and Medical Subject Headings (MeSH) terms. The search strategy was developed and finalized after consultation with the informationist (J.U.) on the writing team ([Supplementary Appendix A] available in the online version). PubMed, CINAHL, IEEE Xplore, and Scopus databases were searched on December 29, 2020, for scholarly, peer-reviewed journals indexed between January 1, 2010 and December 31, 2020. January 2010 served as the inception for the search strategy with the signing the Patient Protection and Affordable Care Act and the inclusion of the Health Information Technology for Economic and Clinical Health Act in March 2010.[16]

Articles and data extraction were organized according to the initial use scenarios described in the revised Conceptual Model for Technology, Nursing and Patient Safety (TNPS)[17] ([Fig. 1]). This model has been adapted from the original to reflect “clinician” rather than “nurse.” Additionally, the TNPS conceptual model espouses the primary tenets of Donabedian's quality of care framework[18] represented as technology, clinician process, and outcomes, which we then used to consider the components of this work, namely integrated medical devices and CDSS (structure), clinician decision making (process), and clinical actions (outcomes).

Fig. 1 Revised Technology, Nursing and Patient Safety Conceptual Model. Reprinted with permission from Powell-Cope G, Nelson AL, Patterson ES. Patient care technology and safety. Note: This figure is adapted from Powell-Cope et al (2008) to reflect “clinician” rather than “nurse.”

Eligibility Criteria

A priori inclusion criteria were established ([Table 1]). Literature that described medical devices which were used by clinicians when providing care in the inpatient setting, device data that were incorporated in the workflow of providers and nurses, and literature that described clinicians interacting with devices and acting on alerts (e.g., medication alerts), were included. The definition of “integrated” included bidirectional communication loops, where programming details were sent from orders in the EHR to the device, as well as devices' data automatically being entered into the EHR by clinician verification, not transcription. Device data informed clinician-focused CDSS such as those related to intravenous insulin administration, for example. Some articles described “integrated” as with other devices in the care environment and did not explicitly state their interface with the EHR. These were not included. While consumer wearables can generate large amounts of physiological data which may be useful for remote monitoring, the current health IT infrastructure does not support the integration of these data into the EHR, thus they were not included. Imaging studies were not included as they are not exclusively hospital-based, and the results are used for diagnosis, and we were interested in learning more about the clinician use scenarios at the point-of-care.

Table 1 Study eligibility criteria

Inclusion criteria

Exclusion criteria

● Hospital setting - acute care, inpatient, critical care ● Empirical studies, published in scholarly peer-reviewed journals ● English language ● Integrated devices such as vital signs, telemetry, smart IV pumps, Holter monitors ● Clinical decision support systems (CDSS)—solicited or unsolicited ● Addresses one or more of the initial use scenarios by clinician:  1. Direct care delivery  2. Indirect care delivery  3. Patient /clinician protection from harm  4. Patient assessment, monitoring, and surveillance  5. Patient assistance  6. Remote patient monitoring  7. Communication  8. Continuous learning  9. Pattern identification ● Screening articles ● Full-text articles ● Dates: January 1, 2010, to December 31, 2020

● Devices that are not integrated with the hospital EHR system (i.e., continuous glucose monitoring) ● Wearable devices, consumer (e.g., FitBit) ● Systematic reviews ● Conference abstracts ● Guidelines ● Diagnostic and imaging studies

Abbreviations: EHR, electronic health record; IV, intravenous.

Note: Solicited CDSS: clinician actively seeks guidance from the CDSS (e.g., antibiotic selection); unsolicited CDSS: system-generated guidance which may come in the form of an alert or message to the clinician.

Data Screening and Charting

Covidence, an online software program for managing and organizing systematic and scoping reviews, was used to facilitate all steps of the screening, review, and extraction portions of the scoping review process. All authors were given access to the project in Covidence. Two authors (J.W. and K.C.) independently conducted title and abstract screening as well as the final full-text review. To ensure interrater reliability, the two screening authors reviewed and discussed approximately 15 titles and abstracts to ensure consistency during the screening and review phases. A third author (J.S.) resolved any discrepancies between the two screening authors. These discrepant articles were discussed among the three authors until consensus was reached.

Data extraction was completed by one author (J.W.) using Covidence. This was verified by the second author (K.C.). Two authors (J.W. and K.C.) defined the items to be extracted including study characteristics, geographic location of the study, data source, year, sample size, clinical setting, CDSS, integrated device, and clinician group(s) that are primarily using CDSS and/or device, and assigned an initial use case as described in Powell-Cope's conceptual framework. Our interpretation was required for charting the initial use scenario as these were not explicitly stated in any article.

Results

The JBI Scoping Review guidelines were used to ensure transparency and structure for this review intended to map the existing literature ([Table 2]). The preliminary search of the databases yielded 2,174 articles. Two hundred and fifty duplicates were removed, leaving 1,924 articles to be screened. During the title and abstract screening phase, 1,744 articles were determined to be irrelevant after they violated the exclusion criteria which included: consumer wearable devices, patient-generated data devices that were not integrated with a hospital EHR, wrong care settings, diagnostic devices, systematic reviews, or conference proceedings. This screening resulted in 38 articles being eligible for full-text review. Twenty articles were excluded after the full-text review, resulting in 18 articles for synthesis ([Fig. 2]). The primary reasons for these full-text exclusions included wrong care setting, and that the devices were used for treatment and/or diagnosis. The remaining articles were organized according the TNPS conceptual framework which grounded the articles in the TNPS clinical use scenarios to provide a basis for parity of the synthesis. The main limitations noted in these articles included generalizability,[19] [20] [21] [22] [23] [24] [25] [26] [27] [28] small samples sizes,[23] [26] [29] [30] [31] [32] potential deviations from methodological convention,[22] [33] and feasibility.[24] [34]

Fig. 2 Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA). Flow diagram of study eligibility screening.

Table 2 Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist

Section	Item	PRISMA-ScR checklist item	Reported on page no.
Title
Title	1	Identify the report as a scoping review	1
Abstract
Structured summary	2	Provide a structured summary that includes (as applicable): background, objectives, eligibility criteria, sources of evidence, charting methods, results, and conclusions that relate to the review questions and objectives	2–3
Introduction
Rationale	3	Describe the rationale for the review in the context of what is already known. Explain why the review questions/objectives lend themselves to a scoping review approach	4–5
Objectives	4	Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (e.g., population or participants, concepts, and context) or other relevant key elements used to conceptualize the review questions and/or objectives	5
Methods
Eligibility criteria	6	Specify characteristics of the sources of evidence used as eligibility criteria (e.g., years considered, language, and publication status), and provide a rationale	6–7; [Table 1]
Information sources[a]	7	Describe all information sources in the search (e.g., databases with dates of coverage and contact with authors to identify additional sources), as well as the date the most recent search was executed	6–7
Search	8	Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated	[Supplementary Appendix A]
Selection of sources of evidence[b]	9	State the process for selecting sources of evidence (i.e., screening and eligibility) included in the scoping review	8
Data charting process[c]	10	Describe the methods of charting data from the included sources of evidence (e.g., calibrated forms or forms that have been tested by the team before their use, and whether data charting was done independently or in duplicate) and any processes for obtaining and confirming data from investigators	7–8
Data items	11	List and define all variables for which data were sought and any assumptions and simplifications made	7
Synthesis of results	13	Describe the methods of handling and summarizing the data that were charted	8–11
Results
Selection of sources of evidence	14	Give numbers of sources of evidence screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally using a flow diagram	8, [Fig. 2]
Characteristics of sources of evidence	15	For each source of evidence, present characteristics for which data were charted and provide the citations	8–9; [Table 3]
Results of individual sources of evidence	17	For each included source of evidence, present the relevant data that were charted that relate to the review questions and objectives	[Table 3]
Synthesis of results	18	Summarize and/or present the charting results as they relate to the review questions and objectives	12–15
Discussion
Summary of evidence	19	Summarize the main results (including an overview of concepts, themes, and types of evidence available), link to the review questions and objectives, and consider the relevance to key groups	17
Limitations	20	Discuss the limitations of the scoping review process	16–17
Conclusions	21	Provide a general interpretation of the results with respect to the review questions and objectives, as well as potential implications and/or next steps	17–18
Funding
Funding	22	Describe sources of funding for the included sources of evidence, as well as sources of funding for the scoping review. Describe the role of the funders of the scoping review	19

Note: Table 2 is a modified version of the original from Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018,169(7):467-473. The modifications to this table are related to Items 5 (Protocol and Registration), Item 12 (Critical Appraisal of Individual Sources of Evidence), and Item 16 (Critical Appraisal Within Sources of Evidence) which were not included as these items did not apply to this manuscript. The nonmodified version of the table by Tricco and colleagues (2018) can be found here (https://www.prisma-statement.org//Extensions/ScopingReviews).

^a Where sources of evidence (see second footnote) are compiled from, such as bibliographic databases, social media platforms, and Web sites.

^b A more inclusive/heterogeneous term used to account for the different types of evidence or data sources (e.g., quantitative and/or qualitative research, expert opinion, and policy documents) that may be eligible in a scoping review as opposed to only studies. This is not to be confused with information sources (see first footnote).

^c The frameworks by Arksey and O'Malley (6) and Levac et al (7), and the JBI guidance (4, 5) refer to the process of data extraction in a scoping review as data charting.

^d The process of systematically examining research evidence to assess its validity, results, and relevance before using it to inform a decision. This term is used for items 12 and 19 instead of “risk of bias” (which is more applicable to systematic reviews of interventions) to include and acknowledge the various sources of evidence that may be used in a scoping review (e.g., quantitative and/or qualitative research, expert opinion, and policy document).

Summary of Study Characteristics

The included articles used various study designs and methodologies, with most of the studies having sample sizes less than 200. Half of the studies were conducted internationally. Half of the articles were published in 2018 and later. Most of the articles were set in intensive care units (ICUs; pediatric and adult). The target users when identified in the studies were generally described as “clinicians” rather than specifically physicians or RNs. The most prevalent CDSS were early warning systems for patient deterioration and insulin infusion management ([Table 3]; [Supplementary Appendix B], available in the online version).

Table 3 Summary of study characteristics

Author(s)year	Title	Location	Hospital setting	Study population	Sample size	Study objective/aim	Clinical decision support (CDS)	Target user	Model use scenario	Device–clinician interaction
Amrein et al[29] 2014	Glucose control in intensive care: Usability, efficacy, and safety of Space GlucoseControl in two medical European intensive care units	Austria and Switzerland	ICU	Adults	40	Investigate the efficacy of the system defined as time in target range using a broader target range of 4.4–8.3 mmol/L for glucose control in medical ICU patients	Continuous insulin infusion	RNs	Direct care	Infusion pump with alerts for timed blood glucose reading - RN measures blood glucose and enters - CDS-generated advice on insulin dose and next blood glucose measure
Baig et al[19] 2015	Multiple physical signs detection and decision support system for hospitalized older adults	New Zealand	Acute care	Older adults (65 years and older)	30	(A) To present the development of a remote/wireless vital signs monitoring and early detection system in order to detect seven key physical signs in older adults in hospital settings(B) To provide multiple combinations of extracted parameters in order to help clinicians with the detection and estimation of health conditions and/or early diagnosis	Early warning for patient deterioration	Clinicians	Patient assessment, monitoring, surveillance	Wireless V/S devices - alarms and warning are communicated to the clinician
Barasch et al[20] 2020	Automation and interoperability of a nurse-managed insulin infusion protocol as a model to improve safety and efficiency in the delivery of high-alert medications	The United States	ICU	Critical care nurses	Not stated	To describe the process used to create a prototype integrated, electronic, clinical decision support, and medication control system for a widely used nurse-managed insulin infusion protocol	Continuous insulin infusion	Critical care nurses	Direct care	Infusion pump with Smart Agent interface - recommended infusion rate is presented to RN, and once confirmed, the pump is updated
Bosque[21] 2020	Development of an alarm algorithm, with nanotechnology multimodal sensor, to predict impending infusion failure and improve safety of peripheral intravenous catheters in neonates	The United States	NICU	Neonates	Not stated	To present an invention using supportive data from pilot studies in order to develop an intravenous infusion nanotechnology monitoring system to alert the nurse to impending PIV infusion failure	Peripheral intravenous catheter status	NICU RNs	Patient assessment, monitoring, and surveillance	The pump will alarm when conditions (pressure, pH, and oxygen saturation) of impending peripheral IV failure are met - RN intervenes to minimize injury
Campion[22] et al 2011	Barriers and facilitators to the use of computer-based intensive insulin therapy	The United States	ICU	ICU RNs	25	To investigate the barriers and facilitators to using intensive insulin clinical decision support systems in clinicians in ICU settings	Intensive insulin infusion therapy	ICU clinicians	Direct care	CDSS embedded in CPOE - titration recommendation generated based on blood glucose measure
Clifton et al[35] 2013	A Large-Scale Clinical Validation of an Integrated Monitoring System in the Emergency Department	The United Kingdom	ED	Not explicitly stated	10,000	To describe a large clinical trial, undertaken to evaluate automatic methods for assessment of patients, based on an electronic patient record augmented with machine learning algorithms	Early warning for patient deterioration	Clinicians	Patient assessment, monitoring, and surveillance	Automatic bedside monitors and PDAs for manual entry capture V/S data - patient status is calculated and pushed to clinician PDAs and touch screens, providing an overview of all patients
Colopy et al[23] 2018	Bayesian Optimization of Personalized Models for Patient Vital-Sign Monitoring	The United Kingdom	Step-down unit	Not explicitly stated	169	To describe a method to build Gaussian process models with varying complexity and regularization on a patient-specific level, for the purpose of robust vital-sign forecasting	Early warning of patient deterioration	Clinicians	Patient assessment, monitoring, and surveillance	V/S monitoring device - clinicians are alerted to V/S out of range specific to patient
Flohr et al[24] 2018	Clinician-Driven Design of VitalPAD An Intelligent Monitoring and Communication Device to Improve Patient Safety in the Intensive Care Unit	Canada	Pediatric ICU	MDs, RNs, respiratory therapists (RTs)	10 (observation phase); 7 (prototype phase)	To identify requirements for the VitalPAD application and to design and evaluate application components through a participatory design process	Early warning system for patient deterioration	ICU clinicians	Patient assessment, monitoring, and surveillance	Mobile device to allow for remote monitoring and alert notification of ICU patients
Koutkias et al[34] 2014	From adverse drug event detection to prevention. A novel clinical decision support framework for medication safety	Europe (Denmark, France, Bulgaria)	Inpatient	Not explicitly stated (adults vs. pediatrics)	24,753	Presentation of a clinical decision support framework targeting medication safety	Adverse drug event prevention	Clinicians including MDs, RNs, pharmacists	Patient protection from harm	Medication alerts for MDs, RNs, and pharmacists
Li et al[30] 2012	Onboard Tagging for Real-Time Quality Assessment of Photoplethysmograms Acquired by a Wireless Reflectance Pulse Oximeter	The United States	Not explicitly stated	Adults	27	To develop a unique onboard feature detection algorithm to assess the quality of PPGs acquired with a custom reflectance mode, wireless pulse oximeter	Not explicitly
Mathioudakis et al[36] 2019	Development and Implementation of a Subcutaneous Insulin Clinical Decision Support Tool for Hospitalized Patients	Baltimore, Maryland, the United States	Medical-surgical units	Adults, including obstetrics patients	Not stated	To develop a computerized CDSS to assist hospital-based clinicians in insulin management	Subcutaneous insulin for noncritically ill	Ordering providers	Direct care delivery	CDS tool guides providers in the selection of appropriate basal-bolus insulin regimens
McGrath et al[33] 2019	Improving Patient Safety and Clinician Workflow in the General Care Setting With Enhanced Surveillance Monitoring	The United States	Medical-surgical inpatient units	General care population	200 (100 pre-/post-observations)	The three primary hypotheses were explored: the time required to perform vital signs assessment would decrease; the adoption of wireless pulse oximetry sensors would increase overall monitoring utilization; and alarm rates associated with the system would decrease, especially brief alarms	Early warning system for patient deterioration	RNs, PT/OT	Patient assessment, monitoring, and surveillance	Wireless v/s monitoring devices are integrated with the EHR
Ni et al[25] 2020	Integrating and evaluating the data quality and utility of smart pump information in detecting medication administration errors: Evaluation study	The United States	NICU	9 high-risk continuous IV medications	2,307 medication orders	To evaluate the quality and utility of smart pump records incorporated with EHR data in detecting medication administration errorsThe specific aims of this study were to (1) develop an automated algorithm that aligns SPRs with EHR data to facilitate manual review of medication administration, (2) characterize discrepancies identified from EHRs and SPRs, and (3) develop a novel assessment that measures the concordance between the ability of EHR and SPR data in detecting medication administration discrepancies	NA	NA	Patient protection from harm	Identification and measurement of discrepancies between medication ordering and smart pump administrations
Singh et al[26] 2019	Neo-Bedside Monitoring Device for Integrated Neonatal Intensive Care Unit (iNICU)	India	NICU	Neonates	92	To present, Neo, a neonatal bedside safety surveillance system (that is brand/vendor agnostic) that acquires, integrates and analyzes the data from diverse bedside medical devices in the NICU environment	Predicts morbidity in newborns	Clinicians	Patient assessment, monitoring, and surveillance	Neo organizes incoming data from physiological and biomedical devices into chronological visualization - clinician is able to evaluate overview of the health status of newborn
Stultz and Nahata[27] 2015	Preventability of Voluntarily Reported or Trigger Tool-Identified Medication Errors in a Pediatric Institution by Information Technology: A Retrospective Cohort Study	The United States	Children's hospital: ICU, ED, and outpatient settings	Adverse events at the children's hospital	1,694	The fourfold objectives include: to categorize reported or trigger tool-identified medication errors and ADEs (adverse drug events) occurring at Nationwide Children's Hospital (NCH) in Columbus, Ohio, USA; to identify whether medication errors that reached patients were preventable by the IT systems utilized at NCH; to determine why IT-preventable errors were not prevented, and to identify risk factors for the occurrence of errors that reached patients and were not preventable by IT systems utilized at NCH	NA	NA	Patient prevention of harm	CPOE and CDS to decrease prescribing errors; automated dispensing cabinets to decrease dispensing errors; BCMA and smart pumps to decrease administration errors
Subbian et al[31] 2015	Integration of New Technology for Research in the Emergency Department: Feasibility of Deploying a Robotic Assessment Tool for Mild Traumatic Brain Injury Evaluation	The United States	Emergency department	Mild TBI patients presenting to the ED	42	To demonstrate the effective deployment of a robotic assessment tool for the evaluation of mild traumatic brain injury (mTBI) patients in a busy, resource-constrained, urban emergency department (ED)	Risk stratification for development of post-concussive syndrome	ED physicians	Patient assessment, monitoring, and surveillance	KINARM robotic device provides a measure of a mTBI patient performance and provides a comparison to a normative group, allowing for risk stratification of development of PCS while presenting to the ED for initial treatment
Wang et al[32] 2018	A New Smart Mobile System for Chronic Wound Care Management	China	Hospital ward	Wound care nurses	10	To develop usable mobile wound care management systems for nurses and doctors	Wound care management	Ward nurses and doctors	Patient assessment, monitoring, and surveillance	Mobile device application prototype aids in measurement, visualization, description of wounds, documentation of treatment, and wound progress that is interoperable with EHR through barcode scanning
Yoo et a[28] 2018	Experience of emergency department patients with using the talking pole device: Prospective interventional descriptive study	Seoul, South Korea	Emergency department	Patients and caregivers	52	To evaluate the satisfaction of ED patients using a patient-friendly health information technology (HIT) device, the “Talking Pole,” and to assess the factors relevant to their satisfaction	NA	NA	Two: Communication and patient assessment, monitoring, and surveillance	Nurses can monitor patients' fluid status; it serves as a communication tool between patients and nurses, and the information for the patients' EHR can be accessed by patients via the pole (i.e., lab values, schedule)

Abbreviations: ADE, adverse drug event; BCMA, bar code medication administration; CDS, clinical decision support; CDSS, clinical decision support system; CPOE, computerized provider order entry; ED, emergency department; EHR, electronic health record; ICU, intensive care unit; IT, information technology; IV, intravenous; MD, Doctor of Medicine; NICU, neonatal intensive care unit; OT, occupational therapist; PCS, postconcussive syndrome; PDA, personal digital assistant; PIV, peripheral intravenous; PPG, photoplethysmogram; PT, physical therapist; RN, registered nurse; SPR, smart pump records; TBI, traumatic brain injury.

All of the articles that underwent full-text review were able to be sorted into various clinical use cases of the TPNS conceptual model. Ten articles described patient assessment, monitoring, or surveillance use,[19] [21] [23] [24] [26] [30] [31] [32] [33] [35] with vital signs monitoring and CDSS related to early warning detection comprising most of this subset.[19] [23] [24] [33] [35] Three articles described patient protection from harm, all of which focused on CDSS for medication safety.[25] [27] [34] Four articles described direct care use scenarios, all of which described insulin administration and a CDSS tool for insulin infusions.[20] [22] [29] [36] Finally, one article described a hybrid situation of patient communication and monitoring, with CDSS specific to fluid management[28] ([Table 3]). Beyond the TPNS categorization, the following commonalities among the articles were noted.

Medication Management Clinical Decision Support Systems

Many integrated CDSSs included in this review aimed to address medication administration errors (n = 7), a leading cause of patient harm. Harm reduction strategies for administering high-risk medications such as insulin infusions include CDSS, computerized provider order entry, and “smart pumps” that interface with the EHR in order to generate an automated dose calculation.[20] [22] [27] Understanding the existing, site-specific nursing workflow and institutional practices related to insulin infusion management was one of the first steps to designing and implementing insulin infusion CDSS in most cases,[20] [22] [36] whereas in other circumstances, the vendor-specific CDSS for insulin management was adopted.[29]

Studies that were included in this review aimed at preventing medication errors through implementation of CDSS and other IT interventions. While these strategies have decreased preventable medication errors, they have not eliminated them completely.[25] One article detailed why insulin dosing CDSS guidance was overridden. In that case, the RN did not agree with the recommendation based on the patient's clinical presentation at the time. There were also challenges of integrated glucometer readings when faced with the realities of timely medication administration in a clinical setting. Decisions were made at the point of care whether to administer or withhold the insulin, given the testing schedule and the medication's pharmacodynamics and onset of action.[22] Clinician distrust and suspicion of the CDSS guidance can result in disregard and bypassing of the systems, and a subsequent decrease in user compliance.[22] [25] [27] There is an ongoing need for staff education and awareness regarding the limits and capabilities of health IT,[27] managing alert fatigue,[22] [27] [34] and clinician workarounds (e.g., manual entries, basic infusions)[33] which can be human factors that influence clinical decision making when presented with CDSS guidance.

Early Warning for Patient Deterioration Clinical Decision Support Systems

Six studies discussed vital signs and the role of physiologic metrics in early warning CDSS. Two of these emphasized the distinction between a generalized monitoring model (thresholds ranges or standard deviation changes which are implemented based on age) versus individualized monitoring model, and how this improves reliability and decreases false alarms.[19] [23] One study notes an increase in nonclinical alarms due to system architecture (e.g., wireless signal strength).[33] Deep learning methods were noted in multiple articles along with fuzzy logic,[19] Bayesian hypothesis testing,[30] novelty detection,[35] and Gaussian process regression,[23] all with the intention of decreasing false alarms through individualization by using machine learning approaches. Wireless vital sign monitoring in addition to hardwired bedside monitors are used in the hospital setting. This wireless monitoring of vital signs is a forerunner to the future applications to support remote patient monitoring, EHR integration, and CDS in settings outside of the hospital.[19] [26] [30] [33]

Innovative Clinical Decision Support Systems

Five articles described innovative devices that aided in patient assessment, monitoring, or surveillance in the clinical setting. In all the articles, the study team included health care professionals and engineering, computer science, and/or innovators. There was a clear focus on portability using mobile applications in specific clinical scenarios. These included wound management, mild traumatic brain injury, patient fluid dynamics, and early warning detection dashboard from all integrated device inputs (vital signs, ventilator settings, intravenous pumps, and waveforms).[24] [28] [31] [32] One article described “smart” intravenous catheter technology coupled with a fuzzy logic algorithm to detect impending intravenous catheter failure in neonates and its affiliated warning system.[21] Another article described a wireless device that monitored the patient's fluid status and movement through sensors, although its primary objective was to facilitate communication between clinicians and patients. The device and the associated dashboard were designed to engage patients directly in several ways. Some of the main features included displaying individualized elements of the plan of care, allowing patients to enter their current pain rating, and providing a two-way communication between the clinician and patient.[28] All studies were conducted on a small scale (n = 15 to 200) or in simulated environments.

Discussion

The results of this scoping review highlight a few primary points for discussion. Clinical device integration in the hospital setting is uneven in its maturity, with some articles describing unidirectional communication (i.e., vital sign machine integration with EHR) to bidirectional communication (i.e., smart pump interoperability with bar code scanning for medication administration) to the incorporation of deep learning to support individualized care (i.e., patient-specific parameters rather than population parameters for vital signs monitoring). Mitigating alarm and alert fatigue in clinicians is a prominent focus and one that has important patient safety and clinician well-being connotations. Machine learning and predictive analytics are essential to moving toward individualized care delivery and personalized medicine. One topic that was repeated in several of the review articles examines the ability for continuous electrocardiogram (ECG; telemetry) monitoring to include patient-specific parameters. Finally, in considering our conceptual framework, much has been written about integrated devices and CDSS in direct patient care, assessment/monitoring/surveillance, and patient protection clinical use scenarios including medication administration, glucose management, and vital signs monitoring, with less written about the actual impact of CDSS on clinicians and their decision making. Potential future research directions include exploring integrated devices, CDSS, and clinical use scenarios (e.g., patient communication) in other care settings such as outpatient, ambulatory, and home, and examining patient outcomes tied to clinician decision making in these settings.

Continuum of maturity in CDS and application in the acute care setting: Stead[37] discussed the challenges associated with applied medical informatics research and conceptualized a framework of system development that may provide some additional context to the uneven uptake and integration of the different devices in the clinical setting. In this review, the spectrum of device integration and CDSS classification included knowledge-based and nonknowledge-based CDSS, as well as active and passive delivery. A few articles described device feedback to the EHR (e.g., vital sign interoperability) to applications of machine learning algorithms to better identify the individual physiological patterns to support patient-specific early warning CDSS. There were differences in the level of sophistication of integration across various clinical settings (e.g., acute care vs. pediatric ICU), geographic locations, and in clinical use scenarios.

Considerations for alarm and alert reduction: alarm and alert fatigue pose an ongoing safety concern in health care. Interestingly, alarms and alerts which are designed to keep patients safe may unintentionally create conditions that harm them instead.[38] [39] [40] [41] [42] [43] [44] Improving the safety of clinical alarms and alerts is a 2021 Joint Commission National Patient Safety Goal in hospital settings.[38] False alarms and alerts and human–computer interactions were frequently mentioned. There are other known contributors to alarm fatigue and alert burden in clinicians. For example, the Agency for Healthcare Research and Quality notes that ECG monitoring alarms are generally false or clinically insignificant the majority of the time.[43] Patient-specific settings rather than population-level parameters are mentioned as a useful strategy to ensure that alarms are related to physiological changes for the individual. There is also a need to optimize integrated systems (e.g., wireless) to minimize nonclinical alarms, as this can contribute to an accumulation of alert fatigue. For example, a weak wifi signal on the nursing unit repeatedly alerts staff to this signal interruption, while local monitoring is not interrupted. Reducing unnecessary alerts and clinical alarms has clear implications for care delivery in the acute care setting in the immediate term. Additionally, there are repercussions related to erroneous alarms (e.g., clinician and client communication to ascertain the legitimacy of the alarms) that will need to be managed when considering future applications of patient monitoring in alternative settings such as the home environment.

CDSS and clinician decision making: the importance of user-centered design and usability testing was mentioned in several articles in this review. In addition to stakeholder engagement, end-user feedback is necessary to understand clinician workflow considerations, and clinician decision making. One such example is the article about disregarding CDSS on insulin administration, clinician decision making, and the impact of CDSS dependent of both the clinician and the system.[22] For example, if CDSS advice is not optimized, clinicians may lose trust in the guidance, and quickly dismiss it, without fully evaluating the recommendations. In other instances, clinicians may become overreliant on CDSS guidance, allowing the CDSS to have outsized effect on patient care decisions.[45] The articles in this review discussed devices and the associated CDSS that are used by clinicians, primarily RNs, in the daily provision of care for patients. Integrated device data provide insight into user–device interactions and helps to illustrate health care processes, especially the activities when providing direct care to patients in an acute care setting. While there are CDSS designed to support the clinician while working with devices, RNs and providers may disregard this guidance, and defer to their own expertise. Additionally, if clinicians perceive CDSS as intrusive, they are at risk for alarm and alert fatigue if CDSS are not tailored to sync with the workflow of the end-user.

Potential future research opportunities with integrated devices: harnessing signals in integrated device log metadata through machine learning-based analytics can raise situational awareness about activities taking place in the clinical care setting that may not be readily apparent or quantifiable with current practices. Metadata is data that accompanies and describes the primary data and is used to give context to the primary data.[46] Analyzing the metadata (e.g., user, time of day, system configuration) from integrated devices may provide insight into behavioral, structural, or technological patterns pertaining to workflow, such as instances of clinical workarounds, dismissal of CDSS guidance, or characterizations of clinical versus nonclinical alarms.[13] Conducting a systematic review of this phenomenon is one possible way to investigate this. In the review articles by Campion et al,[22] participants described the reasons that the CDSS-generated guidance for insulin administration was dismissed, and Koutkias et al[34] noted their alarm analysis to determine an acceptable threshold for alerts. Rossetti et al described how metadata from clinician interactions with the EHR, integrated devices, and communication tools model clinician expert judgment which can be leveraged when making clinical prognostics.[47] Analyzing metadata from integrated devices alongside EHR audit log data may improve CDS by providing insights into clinical workflow and intensity. For example, a medication order for a vasopressor requiring titration to achieve a blood pressure threshold may not reveal the workload associated with the monitoring and administration of this medication for clinical staff as they adjust to achieve hemodynamic stability, although analysis of metadata of smart pumps and vital sign monitors could. Adler-Milstein et al[48] anticipate that audit logs will be a widely accepted source of data for researchers who are studying health care processes and outcomes. Analysis of these data can inform interventions that address deviations from the intended use of health IT, increase user satisfaction, and preserve patient safety. .

Artificial intelligence (AI), deep learning, and predictive analytics will be featured in CDSS and other health care applications. Davenport and Kalakota[49] posit that the greatest challenge to AI in health care is not the usefulness of these tools but rather their adoption, use in daily practice, and evaluation of effectiveness in patient outcomes. Stead[37] reinforces the premise that these technologies augment clinician judgment, intelligence, and training and do not supersede it. There is a growing body of evidence that explores the dynamics of human–AI collaboration, clinician trust, and patient perceptions of AI in health care.[50] [51] [52] [53]

Integrated device security, the procurement and maintenance of these devices, and CDSS development and deployment are resource-intensive aspects of health IT maturation that may result in uneven implementation. There are varying degrees of medical device integration with the EHR.

Limitations

While our initial search strategy yielded more than 2,000 articles, our final sample was far less, due to various violations of exclusion criteria, this included the dismissal of systematic reviews. Given that systematic reviews can be a treasure trove of references the decision to exclude them was potentially a missed opportunity. Additionally, there is the potential that there were eligible articles that were not included in the search if they were outside of the four databases queried. The choice of databases was strategic and was not restricted to health care databases only. The MeSH terms were limited, with most articles reviewed pertaining to medication administration, and this will limit generalizability. We summarized the study characteristics as a method of quality assessment for this work ([Table 3]). A quality assessment is not a component of the scoping review methodology at present,[54] and currently there is no validated tool for conducting one for scoping review. There is ongoing debate in the literature as whether incorporation of a quality assessment should be included for scoping reviews as this is an inherent limitation of this approach.[55] [56] This analysis was guided by a conceptual framework adapted from Powell-Cope et al[17] that considered various clinical use scenarios rather than the framework in its entirety. Considering more fully the various outcomes described in the model, whether at the patient, clinician, or organizational level, and the use of integrated devices is an interesting consideration and plausible next step, as this was beyond the scope of this review.

Conclusion

The articles in this review discussed devices such as medication administration pumps and the associated CDSS that are used by clinicians, primarily RNs, in the daily provision of care for patients. Integrated device data provide insight into user–device interactions and helps to illustrate health care processes especially the activities when providing direct care to patients in an acute care setting. While there are CDSSs designed to support the clinician while working with devices, RNs and providers may disregard this guidance, and defer to their own expertise. Additionally, if clinicians perceive CDSSs as intrusive, they are at risk for alarm and alert fatigue if CDSS are not tailored to sync with the workflow of the end-user.

Integrated device data and the associated CDSS that are driven by these data inputs provide insight into user–device interactions, help to illustrate health care processes, and facilitate safe care delivery. Considering universal emphasis on patient safety, cost-effective, and efficient care delivery, our results indicate that establishing closed-loop and/or bidirectional communication related to point-of-care devices and their associated CDS is an important pursuit of health care organizations. There are human factor considerations that must be considered as well as the health IT infrastructure to ensure the safe and effective integration of devices and their associated CDSS into the workflow of clinicians. Analysis of integrated device log files and metadata provide a novel way to conduct quality improvement projects and are a rich data source for research questions, and could provide insights into clinicians' workflow and clinician decision making. Further research and innovation are needed to broaden and extend the existing evidence related to integrated devices.

Clinical Relevance Statement

Integrated medical device data can provide additional insight into clinician workflow and decision making. Optimizing these devices and ensure seamless integration will facilitate safe and efficient patient care delivery in the hospital. Most of the integrated devices that are discussed in this review fall under the purview of RNs, with providers generally using the associated CDSS when ordering the interventions that will be delivered by those devices.

Multiple-Choice Questions

1. Evaluating metadata from integrated medical devices can be used to enhance an understanding of the clinical environment by:

   • Raising situational awareness of clinician workflow patterns.

   • Recording patient self-reported clinical observations and impressions.

   • Automating patient care outcome reporting.

   • Registering the hire date for clinicians using the device.

   Correct answer: the correct answer is option a. Analyzing the metadata (e.g., user, time of day, system configuration) from integrated devices may provide insight into behavioral, structural, or technological patterns pertaining to workflow, such as instances of clinical workarounds, dismissal of CDSS guidance, or characterizations of clinical versus nonclinical alarms. Harnessing signals in integrated device log metadata through machine learning-based analytics can raise situational awareness about activities taking place in the clinical care setting that may not be readily apparent or quantifiable with current practices.

2. A well-designed clinical decisions support system should:

   • Augment the end-user's decision-making about patient care decisions by providing guidance based on patient-specific data.

   • Supersede the clinician's decision-making about patient care decisions by providing guidance based on patient-specific data.

   • Be done by health information technologists and other analysts, with end-user feedback on usability after being deployed in the care environment.

   • Be considered optional.

   Correct answer: the correct answer is option c. CDSS should be considered an adjunct to a clinician's decisions making and should never supersede.

• References

• 1 Institute of Medicine (US) Committee on Quality of Health Care in America, Kohn LT, Corrigan JM, Donaldson MS, eds. To Err is Human: Building a Safer Health System. Washington, DC: National Academies Press (US); 2000
• 2 Agency for Healthcare Research and Quality. Patient safety 101. Accessed June 24, 2021, at: https://psnet.ahrq.gov/primer/patient-safety-101
• 3 Gluck PA. Patient safety: some progress and many challenges. Obstet Gynecol 2012; 120 (05) 1149-1159
• 4 Kronick R, Arnold S, Brady J. Improving safety for hospitalized patients: much progress but many challenges remain. JAMA 2016; 316 (05) 489-490
• 5 Thirukumaran CP, Glance LG, Temkin-Greener H, Rosenthal MB, Li Y. Impact of Medicare's nonpayment program on hospital-acquired conditions. Med Care 2017; 55 (05) 447-455
• 6 Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ 2016; 353: i2139
• 7 Li J, Carayon P. Health Care 4.0: a vision for smart and connected health care. IISE Trans Healthc Syst Eng 2021; 11 (03) 171-180
• 8 Marques da Rosa V, Saurin TA, Tortorella GL, Fogliatto FS, Tonetto LM, Samson D. Digital technologies: an exploratory study of their role in the resilience of healthcare services. Appl Ergon 2021; 97: 103517
• 9 Klarich A, Noonan TZ, Reichlen C, Barbara SMJ, Cullen L, Pennathur PR. Usability of smart infusion pumps: a heuristic evaluation. Appl Ergon 2022; 98: 103584
• 10 Soegaard Ballester JM, Bass GD, Urbani R. et al. A mobile, electronic health record-connected application for managing team workflows in inpatient care. Appl Clin Inform 2021; 12 (05) 1120-1134
• 11 St John A, Price CP. Existing and emerging technologies for point-of-care testing. Clin Biochem Rev 2014; 35 (03) 155-167
• 12 Sim I, Gorman P, Greenes RA. et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (06) 527-534
• 13 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
• 14 Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Scoping reviews (2020 version). In: Aromataris E, Munn Z. eds. JBI Manual for Evidence Synthesis. JBI; 2020
• 15 Tricco AC, Lillie E, Zarin W. et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018; 169 (07) 467-473
• 16 Protection P, Act AC. Patient protection and affordable care act. Public Law 2010; 111 (48) 759-762
• 17 Powell-Cope G, Nelson AL, Patterson ES. Patient Care Technology and Safety. In: Hughes RG. ed. Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality (US); 2008
• 18 Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005; 83 (04) 691-729
• 19 Baig MM, GholamHosseini H, Connolly MJ. Multiple physical signs detection and decision support system for hospitalized older adults. Physiol Meas 2015; 36 (10) 2069-2088
• 20 Barasch N, Romig MC, Demko ZO. et al. Automation and interoperability of a nurse-managed insulin infusion protocol as a model to improve safety and efficiency in the delivery of high-alert medications. J Patient Saf Risk Manag 2020; 25 (01) 5-14
• 21 Bosque EM. Development of an alarm algorithm, with nanotechnology multimodal sensor, to predict impending infusion failure and improve safety of peripheral intravenous catheters in neonates. Adv Neonatal Care 2020; 20 (03) 233-243
• 22 Campion Jr. TR, Waitman LR, Lorenzi NM, May AK, Gadd CS. Barriers and facilitators to the use of computer-based intensive insulin therapy. Int J Med Inform 2011; 80 (12) 863-871
• 23 Colopy GW, Roberts SJ, Clifton DA. Bayesian optimization of personalized models for patient vital-sign monitoring. IEEE J Biomed Health Inform 2018; 22 (02) 301-310
• 24 Flohr L, Beaudry S, Johnson KT. et al. Clinician-driven design of VitalPAD-an intelligent monitoring and communication device to improve patient safety in the intensive care unit. IEEE J Transl Eng Health Med 2018; 6: 3000114
• 25 Ni Y, Lingren T, Huth H, Timmons K, Melton K, Kirkendall E. Integrating and evaluating the data quality and utility of smart pump information in detecting medication administration errors: evaluation study. JMIR Med Inform 2020; 8 (09) e19774
• 26 Singh H, Kaur R, Gangadharan A. et al. Neo-bedside monitoring device for integrated neonatal intensive care unit (iNICU). IEEE Access 2019; 7: 7803-7813
• 27 Stultz JS, Nahata MC. Preventability of voluntarily reported or trigger tool-identified medication errors in a pediatric institution by information technology: a retrospective cohort study. Drug Saf 2015; 38 (07) 661-670
• 28 Yoo J, Soh JY, Lee WH, Chang DK, Lee SU, Cha WC. Experience of emergency department patients with using the talking pole device: prospective interventional descriptive study. JMIR Mhealth Uhealth 2018; 6 (11) e191
• 29 Amrein K, Kachel N, Fries H. et al. Glucose control in intensive care: usability, efficacy and safety of Space GlucoseControl in two medical European intensive care units. BMC Endocr Disord 2014; 14: 62
• 30 Li K, Warren S, Natarajan B. Onboard tagging for real-time quality assessment of photoplethysmograms acquired by a wireless reflectance pulse oximeter. IEEE Trans Biomed Circuits Syst 2012; 6 (01) 54-63
• 31 Subbian V, Ratcliff JJ, Meunier JM, Korfhagen JJ, Beyette Jr. FR, Shaw GJ. Integration of new technology for research in the emergency department: feasibility of deploying a robotic assessment tool for mild traumatic brain injury evaluation. IEEE J Transl Eng Health Med 2015; 3: 3200109
• 32 Wang S, Zhang Q, Huang W. et al. A new smart mobile system for chronic wound care management. IEEE Access 2018; 6: 52355-52365
• 33 McGrath SP, Perreard IM, Garland MD, Converse KA, Mackenzie TA. Improving patient safety and clinician workflow in the general care setting with enhanced surveillance monitoring. IEEE J Biomed Health Inform 2019; 23 (02) 857-866
• 34 Koutkias VG, McNair P, Kilintzis V. et al. From adverse drug event detection to prevention. A novel clinical decision support framework for medication safety. Methods Inf Med 2014; 53 (06) 482-492
• 35 Clifton DA, Wong D, Clifton L. et al. A large-scale clinical validation of an integrated monitoring system in the emergency department. IEEE J Biomed Health Inform 2013; 17 (04) 835-842
• 36 Mathioudakis N, Jeun R, Godwin G. et al. Development and implementation of a subcutaneous insulin clinical decision support tool for hospitalized patients. J Diabetes Sci Technol 2019; 13 (03) 522-532
• 37 Stead WW. Clinical implications and challenges of artificial intelligence and deep learning. JAMA 2018; 320 (11) 1107-1108
• 38 Hospital: 2023 National Patient Safety Goals. Accessed November 14, 2022, at: https://www.jointcommission.org/standards/national-patient-safety-goals/hospital-national-patient-safety-goals/
• 39 Reducing the Safety Hazards of Monitor Alert and Alarm Fatigue. Accessed November 14, 2022, at: https://psnet.ahrq.gov/perspective/reducing-safety-hazards-monitor-alert-and-alarm-fatigue#ref2
• 40 Bach TA, Berglund LM, Turk E. Managing alarm systems for quality and safety in the hospital setting. BMJ Open Qual 2018; 7 (03) e000202
• 41 Meghan W, Bacon O. Alarm fatigue. In: Hall KK, Shoemaker-Hunt S, Hoffman L. et al. Making Healthcare Safer III: A Critical Analysis of Existing and Emerging Patient Safety Practices. Rockville, MD: Agency for Healthcare Research and Quality (US); 2020
• 42 van der Sijs H. Errors related to alert fatigue. In: Safety of Health IT. Springer Cham; 2016: 41-54
• 43 Digital health and patient safety. . Accessed November 14, 2022, at: https://psnet.ahrq.gov/issue/digital-health-and-patient-safety
• 44 Agboola SO, Bates DW, Kvedar JC. Digital health and patient safety. JAMA 2016; 315 (16) 1697-1698
• 45 Hartigan S, Brooks M, Hartley S, Miller RE, Santen SA, Hemphill RR. Review of the basics of cognitive error in emergency medicine: still no easy answers. West J Emerg Med 2020; 21 (06) 125-131
• 46 Badawy R, Hameed F, Bataille L. et al. Metadata concepts for advancing the use of digital health technologies in clinical research. Digit Biomark 2019; 3 (03) 116-132
• 47 Rossetti SC, Knaplund C, Albers D. et al. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework. J Am Med Inform Assoc 2021; 28 (06) 1242-1251
• 48 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: a new goldmine for health services research?. J Biomed Inform 2020; 101: 103343
• 49 Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J 2019; 6 (02) 94-98
• 50 Asan O, Bayrak AE, Choudhury A. Artificial intelligence and human trust in healthcare: focus on clinicians. J Med Internet Res 2020; 22 (06) e15154
• 51 Habli I, Lawton T, Porter Z. Artificial intelligence in health care: accountability and safety. Bull World Health Organ 2020; 98 (04) 251-256
• 52 Nagendran M, Chen Y, Lovejoy CA. et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 2020; 368: m689
• 53 Shinners L, Aggar C, Grace S, Smith S. Exploring healthcare professionals' understanding and experiences of artificial intelligence technology use in the delivery of healthcare: an integrative review. Health Informatics J 2020; 26 (02) 1225-1236
• 54 Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005; 8 (01) 19-32
• 55 Khalil H, Peters MD, Tricco AC. et al. Conducting high quality scoping reviews-challenges and solutions. J Clin Epidemiol 2021; 130: 156-160
• 56 Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods 2014; 5 (04) 371-385
• 57 Williams PA, Woodward AJ. Cyber security vulnerabilities in medical devices: a complex environment and multifaceted problem. Med Devices (Auckl) 2015; 8: 305-316