Buy or Build: Challenges Developing Consumer Digital Health Interventions

Keywords

digital technology - hypertension - medication adherence - patient participation - consumer health informatics

Background and Significance

Over the past decade, several initiatives and technology developments have allowed patients and consumers to collect and view their personal health data. These data, often termed patient-generated health data (PGHD), are health-related data collected, owned, and shared by patients or consumers.[1] PGHD can support self-management of chronic disease, serve as an input to interactions with the formal care system, and aid transitions back to the home environment. Creating and viewing of PGHD, and integration of PGHD into clinical care, therefore has the potential to support more timely, empowering, comprehensive, personalized, and safer care.[2]

One common way patients collect and view PGHD is through mobile health (mHealth) devices (including wearables) and apps. The market for mHealth apps and devices is on an unprecedented growth trajectory. As of 2020, there were over 350,000 mHealth apps available in major app stores, and 3.87 million individuals in the United States used a health or fitness app monthly.[3] mHealth app use surged 53% in 2021, in part, driven by the coronavirus disease-2019 pandemic.[4] As of 2023, approximately 25% of U.S. adults used a wearable device, including smart watches, fitness trackers, and blood glucose and blood pressure (BP) monitors.[5]

mHealth devices and apps are becoming increasingly embedded within health care interventions. A 2020 review of manuscripts at the intersection of human factors and health informatics found that almost 40% of the 2020 studies focused on the development or evaluation of mHealth-related interventions.[6] While mHealth devices and apps are only a component of these health care interventions, if the technologies fail, the larger intervention will likely also fail. Studies have shown that patients' abilities to access, interact with, and use their own data influences their motivation and satisfaction with digital health technologies.[7] [8] It is therefore crucial that we ensure patient-facing mHealth technologies be useful and usable.

Our team designed and implemented two mHealth-enabled interventions that captured and shared PGHD using a diverse range of devices, apps, and connections to the electronic health record (EHR). Our projects included stand-alone devices (electronic BP cuff, near-field communication [NFC]-tagged medication bottles), wearables (smartwatch), a custom-developed app (medication tracker), a provider portal (graphical medication adherence summary), and a direct connection to an EHR flowsheet. While these projects spanned over a decade, used a diverse range of technologies described above, and had diverse end users (patients with hypertension and diabetes, patients with or at risk for HIV/AIDS, nurses, and physicians), we repeatedly encountered the same types of challenges. We term these “buy or build” challenges because they all stemmed from decisions about whether to purchase or use commercial technologies or custom-design technologies. While PGHD-related technologies will constantly change, those creating PGHD-related interventions will always have to make “buy or build” decisions. This paper describes and categorizes a broad range of “buy or build” decisions and their ensuing complications we encountered that we believe will frequently arise in PGHD-related interventions. Neither a recent scoping review of studies focused on PGHD and EHR integration,[9] nor a systematic review of barriers and facilitators of eHealth services (which includes PGHD-related interventions),[10] nor studies detailing PGHD-related challenges[11] [12] nor of challenges with developing secure mHealth interventions[13] mentions such “buy or build” issues. Our objective is therefore to help others developing PGHD-related interventions anticipate and mitigate similar issues they may encounter.

Digital Health Intervention Descriptions

In this section, we describe two PGHD-related digital health technology-enabled interventions we developed and deployed. From 2010 to 2015, we developed CONDUIT-HID (CONtrolling Disease Using Information Technology-Hypertension In Diabetes). CONDUIT-HID was designed to improve BP control in patients with diabetes and hypertension through home BP monitoring using an electronic cuff, automated uploading of readings, direct integration into the EHR, and changes in medication regimens based on those readings. From 2016 to 2022, we developed a system called USE-MI (Unobtrusive SEnsing of Medication Intake), aimed at developing and evaluating a system of devices and an app to capture data about medication intake and support patients and clinicians in improving HIV-related medication adherence.

CONDUIT-HID

CONDUIT-HID ([Fig. 1]) involved giving patients a home BP monitor with a computer connection and encouraging them to measure their BP and upload measurements to the EHR where those data were integrated into clinical flowsheets. The system was developed and tested in the context of a multispecialty practice setting using a common EHR (Epic Systems, Verona, Wisconsin, United States). Custom EHR-based software generated alerts to clinicians if the data identified potentially dangerous BP values and sent periodic reports of patients' mean BP values to physicians and nurses to facilitate medication adjustment when BP was not controlled. There was no direct feedback from the system to the patients about BP control, aside from patients directly viewing their own BP values. Nurses with hypertension expertise reviewed the reports and communicated with the patients, recommending medication changes per a protocol based on guidelines issued by members appointed to the Eighth Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure[14] or their clinician's recommendation.

Fig. 1 Design of the CONDUIT-HID system. CONDUIT-HID, CONtrolling Disease Using Information Technology-Hypertension In Diabetes.

USE-MI

USE-MI ([Fig. 2]) involved a research coordinator and patient jointly creating a schedule of ideal times for taking medications and inputting that schedule into a database. The system recognized medication intake using multiple mechanisms: NFC and motion sensing of pill bottle manipulation and medication ingestion gestures, Bluetooth communication between a smartphone and smartwatch, and/or manual entry of data by patients into a smartphone app. The system reminded patients if medication intake had not been detected within the scheduled time window(s) and generated medication intake summaries within the app and via a clinician-facing portal. We recruited participants nationally from multiple health care systems and could not afford to embed the system within multiple EHRs, so developed a stand-alone system.

Fig. 2 Design of the USE-MI system. USE-MI, Unobtrusive SEnsing of Medication Intake.

In both systems, we made design choices to minimize patient and provider burden by integrating the system into their existing processes and workflows.

Structured Analysis of Our “Buy or Build” Decisions

[Tables 1] and [2] describe critical needs for CONDUIT-HID and USE-MI, and the solutions we developed based on our “buy or build” decisions. We also describe the common themes that arose as we made these “buy or build” decisions within each system and resulting adjustments we made to our systems.

Table 1 CONDUIT-HID buy or build decision analysis

System component	Need	Issue/problem	Solution	Result	Buy or build	Theme(s)
Data collection	Clinically validated electronic BP monitor	The only BP cuff that would connect directly to Epic was not clinically validated	Clinically validated Omron BP cuff	Needed intermediary (Microsoft HealthVault) to transmit data from the BP cuff to the EHR	Buy	Mismatch between existing and new technologies
Data transfer	Users to independently upload data	Omron would unexpectedly move the driver repository location we provided to users	Continuously updated patient resources	Constant monitoring of Omron driver location changes and tech support	Buy	Technology stability
	Intermediary (Microsoft HealthVault) to transmit data from the BP cuff to the EHR	Apple users and those without a computer could not use HealthVault	Clinic computers to upload data from BP monitors	Clinic computers had to be designated and set up at each clinic	Buy and build	Mismatch between existing and new technologies
	Patients to upload BP data to clinic computers	HealthVault did not intend to have many users' devices connect to one computer; other patients' PHI could be exposed to the patient uploading data	Clinic staff had to upload BP data	Clinic staff followed a strict protocol to ensure BP data were sent to the correct patient record and PHI from other patients were not exposed	Buy and build	New or unintended use case
	Stable, timely BP data transfer to EHR	HealthVault updates broke connection between the BP cuff and Epic	Ongoing immediate technical fixes	Constant monitoring of HealthVault updates and tech support	Buy and build	Technology stability
	System should work long term	HealthVault was discontinued after our study	Health system replaced HealthVault with a different data store	Entire technology replacement	Build	Technology longevity
	Clinician access to BP data within their existing EHR	Epic did not support connection to Omron BP cuff or HealthVault and was not interested in our use case	Health system developed its own HL7 Observation Reporting interface to connect systems	Interface polled HealthVault for new data and pulled it into Epic clinical flowsheets, tagged as home BP readings	Build	Mismatch between existing and new technologies
Data analysis and data display	Easy and unobtrusive access to data for clinicians	Clinicians wanted home BP data analyzed and displayed in a different format than Epic supported at the time	Custom epic alerts and inbox messages for (1) out of range values and (2) BP summaries	Iterative development of scenarios that trigger alerts and summary message content	Build	New or unintended use case

Abbreviations: BP, blood pressure; CONDUIT-HID, CONtrolling Disease Using Information Technology-Hypertension In Diabetes; EHR, electronic health record; NFC, near-field communication; PHI, protected health information; RFID, radio frequency identification.

Table 2 USE-MI buy or build decision analysis

System component	Need	Issue/problem	Solution	Result	Buy or build	Theme(s)
Data collection and data transfer	Detect interactions with medication bottle	Android wear NFC, our original approach, used tags for transactions, not as a reader	Abandoned NFC	RFID tag on bottle and wrist-worn sensor[22] to passively detect manipulation of bottle	Buy then build	New or unintended use case
		No commercial wrist-worn devices were compatible with this approach	Abandoned RFID	Bluetooth motion sensor, compatible with any Android phone	Buy and build	Mismatch between existing and new technologies
		Android Bluetooth stack errors and repeated Bluetooth connectivity issues resulted in essential data loss for our use case	Ongoing system modifications, abandoned bottle motion detection	Constant monitoring and tech support; transitioned to active tap of phone on NFC-tagged bottle	Buy and build	New or unintended use case and technology stability
	Users to have one phone	Many users had iPhones, but we lacked resources to also develop for iOS	Gave participants an Android phone if they did not own one	Some users had to use two phones	Buy	Resource limitations and costs of adopting technologies
		Later, Android Bluetooth stack error affected all phones except Google Pixels	Gave participants an Android Pixel phone if they did not own one	Virtually all users had to use two phones	Buy	Technology stability; new or unintended use case
	Reliable watch for 6-axis motion-detection data	Our watches stopped automatically installing our app when the app was installed on the phone	Ongoing system modifications	Constant monitoring and tech support	Buy	Technology stability
Data collection and analysis	Platform that would allow us to collect and process novel motion detection data	Numerous Android bugs and problems, particularly with Android operating system updates	Repeated debugging and system modifications	Continued system instability until decision made to abandon Bluetooth approach	Buy and build	Technology stability; new or unintended use case
Data display	Reliable patient-facing smartphone app	Android bugs, particularly when Google released software updates	Repeated debugging and system modifications	Constant monitoring of issues and tech support	Build	New or unintended use case
	Reliable clinician-facing view of data	Could not easily integrate our data-types and visualizations into the EHR	Developed separate clinician-facing portal	Clinicians need to access data outside of their existing EHR workflow	Build	New or unintended use case

Abbreviations: EHR, electronic health record; NFC, near-field communication; RFID, radio frequency identification; USE-MI, Unobtrusive SEnsing of Medication Intake.

We wanted CONDUIT-HID ([Table 1]) to be as scalable and transportable as possible to other settings. At the time, the EHR used by the medical group we were collaborating with could receive data directly from only one BP monitor, but it was not clinically validated, which would have been unacceptable to our clinicians. Thus, we needed to create our own pipeline for transmitting BP data from validated monitors. To maintain scalability, we used a free, commercially supported personal health record system (Microsoft HealthVault) as an intermediary data store. We used standard interfaces to transmit data from this intermediary data store to the EHR. We developed custom rules within the EHR to compute mean BP values and generate clinical alerts as there was no way to carry out these tasks in a platform-independent manner. Our desire for scalability drove our choices to use commercial devices and software when possible but led to problems outside of our control as vendors made changes to their systems or discontinued services ([Table 1]).

We also wanted USE-MI to be scalable, so chose commercially available products when possible. Because our system represented a more novel use case (automated detection of medication intake) and end users who wanted the system to maintain their privacy, we needed to use more custom-developed technologies than we did in CONDUIT-HID. Inadequate technical capabilities of potential commercial technologies (e.g., NFC) and workload to develop custom components (e.g., machine learning algorithms for processing motion data) led to a series of complex “buy or build” decisions. Additionally, in part because our use case fell outside of “typical” use cases for the technologies we chose, system interactions between commercial and custom-developed components caused ongoing problems ([Table 2]).

Discussion

Both of these projects, despite being performed at different times with different technologies, use cases, and end user characteristics, required us to make and remake similar “buy or build” decisions that led to unanticipated problems. Despite the differences in aims, technologies, and timeframes, we realized there were several commonalities among these problems.

In both projects, we used commercial technology where possible, often driven by a desire to enable inexpensive dissemination of our intervention and reduce development time and cost. At times, commercial technologies were preferred for unanticipated reasons. For example, we developed and considered using wrist-worn radio frequency identification (RFID) technology in the USE-MI project but concluded it was not an appropriate solution as prototype devices might attract undesired attention for those taking HIV-related medications.

Across both projects, we found that the existing commercial technologies were often designed for use cases different from ours, essentially making our interventions nonstandard “custom” deployment of the technology. This customization became more laborious and expensive the further our use case was from the intended use case and as the functionality was deemed more critical for our intervention. For example, data latency was a key issue for USE-MI. Missing the beginning of a continuous data stream such as step counting for consumer applications is unlikely to matter, while missing a single, nonrepeating event of high importance, such as medication intake, could render a medical application useless, or even dangerous. Understandably, Android development focused on power optimization rather than capturing short duration data.

We also faced barriers due to absent functionality to bring PGHD into the EHR.[15] [16] Data must have extremely high value for clinicians to go outside of their EHR to retrieve it, making it imperative to directly integrate these data into the EHR. In our CONDUIT-HID project, we created custom-built interfaces to bring home BP data into the clinical EHR. While our project started a decade ago, uploading of home BP measurements to the EHR remains unavailable to the vast majority of patients,[15] even though out-of-office BP measurement is now recommended as the standard for diagnosing and managing hypertension.[17] [18] Reviews of publications about PGHD interventions found few successful PGHD–EHR integrations,[9] [19] of which our CONDUIT-HID project was one. In the USE-MI project, because we were generating nonstandard data types and recruiting participants covered by multiple EHRs, there was no way to bring the data into the EHR and we relied instead on a custom portal interface for clinicians, likely limiting the system's impact.

We also encountered several problems caused by unpredictable vendor actions that were outside of our control, with similar consequences in both projects. The aforementioned power optimization changes were unanticipated and fixable by our team, but a discontinued product like Microsoft HealthVault requires complete system component redesign. Issues related to vendor barriers to PGHD and EHR integration, including the possibility of breakage with updates, were also noted in a previous analysis of issues related to the dominant EHR vendor in the United States.[20]

Our experiences are certainly not unique. Developers of other PGHD-related interventions will face analogous choices about buying and building technology and should anticipate encountering similar problems. Those using features of existing commercial technologies that are “edge cases” rather than core functions should anticipate more problems. Given the need for ongoing maintenance and updating, including the ability to quickly repair unpredictable breakage caused by vendors, this means that most sustainable PGHD-related interventions will need substantial ongoing financial resources. It is very unlikely that noncommercial efforts, even if made freely available, will be sustainable. The medical literature is replete with examples of interventions that improved health outcomes yet have not been sustained or disseminated because there was no mechanism for continued support after grant funding ended. An ongoing revenue stream will be needed to keep PGHD-related interventions functional, even without upgrades. This likely will require either a financially strong sponsor, adopting a “prescription digital therapeutics”[21] model, and/or an alternative revenue stream, such as advertising or selling user data. Our experience shows that even a decision to use what appear to be vendor “core functions,” such as Google Health, Microsoft HealthVault, or Android, are vulnerable to unpredictable vendor decisions that can render an intervention inoperative. While secondary uses of these interventions may be beneficial, such as the use of USE-MI for tracking non-HIV-related medications, these secondary use cases may cause additional unintended breakages of either “bought” or “built” systems.

Conclusion

While we still believe there are significant benefits to using commercial technologies when designing PGHD-related interventions, particularly related to development cost and scalability, those considering developing PGHD-related interventions should assess the likelihood that factors beyond their control may render these commercial technologies inoperative or unreliable, thereby impacting the success of the entire intervention. The farther the intervention use case is from a core commercial technology function, or the stricter the requirements for performance are, the more likely such system “breakage” will occur. Significant challenges also exist regarding integration of PGHD directly into the EHR. Recently, vendor-agnostic standards such as Fast Healthcare Interoperability Resources have focused on consumer access to EHR data. While this is a step in the right direction, these standards do not currently support writing PGHD into EHRs, even for our common use case of uploading discreet home BP data.[15] Writing novel types of PGHD, like home medication intake detection, is likely a long way off. The seamless exchange of PGHD and clinical EHR data without expensive customization for each EHR vendor system and EHR installation is essential for PGHD-related interventions to improve health at scale.

Clinical Relevance Statement

Our experiences highlight the types of “buy or build” barriers that developers of PGHD-related health interventions should expect to encounter and have plans to address if their interventions are to be sustainable and improve health.

Multiple-Choice Questions

1. Which of these is a potential problem when using existing commercial hardware and software for digital health interventions?

   • Commercially available hardware and software is more expensive than custom-built technologies.

   • Software programming interfaces for commercial technologies are poorly designed and result in erroneous behavior.

   • Software, firmware, and operating system updates can disrupt integration with a larger system and require extensive updates over the span of a deployment.

   • Commercial hardware is not well tested and is more prone to failure than custom-built technologies.

   Correct Answer: The correct answer is option c. In both projects, we encountered numerous problems where system updates made by a vendor rendered a previously working system inoperative. In general, a is not true, although there could be occasional cases where it is, and in nearly all applications use of commercial systems will still require custom-built components. b and d are not true, as commercial software and hardware typically perform per their tested use case specifications.

2. What is the biggest risk in using commercially available backend software to store EHRs used for digital health interventions?

   • Commercial programming interfaces are too complex and implementing a fully custom solution will accelerate development time.

   • Software designed to interface with commercial devices may be discontinued as the financial goals and objectives of a corporation or company shift during the lifespan of a research project, rendering software development efforts useless.

   • A commercial backend is less secure and researchers and patients cannot trust that data privacy will be guaranteed.

   • Commercial systems typically use legacy code and one is better off implementing software from scratch to take advantage of the most recent advancements in software engineering best practices.

   Correct Answer: The correct answer is option b. While commercially available software typically has well-defined software interfaces and is typically well tested, stable, and secure, the lack of control in how and when another organization's backend updates is inherently incompatible with mHealth research. Software development in a research setting moves much more slowly than in industry and in many cases cannot be updated since these updates would potentially impact the outcomes of ongoing studies. Even more problematic, is the potential of a backend system to be completely discontinued, which we experienced in our research, such scenario requires a complete code rewrite that is not typically written into the budget for a research project.

• References

• 1 Conceptualizing a Data Infrastructure for the Capture, Use, and Sharing of Patient-Generated Health Data in Care Delivery and Research through 2024 (Office of the National Coordinator for Health Information Technology). 2018 . Accessed January 25, 2023 at https://www.healthit.gov/sites/default/files/onc_pghd_practical_guide.pdf
• 2 Integrating Patient-Generated Digital Health Data into Electronic Health Records in Ambulatory Care Settings: An Environmental Scan. 2021 . AHRQ Publication No. 21-0031. Accessed January 25, 2023 at: https://digital.ahrq.gov/sites/default/files/docs/citation/pghd-environmental-scan.pdf
• 3 Mobius MD. 11 surprising mobile health statistics. Accessed April 29, 2023 at: https://mobius.md/2021/10/25/11-mobile-health-statistics
• 4 Sensor Tower. U.S. mobile devices saw an average of 47 apps used per month last year. Accessed April 29, 2023 at: https://sensortower.com/blog/number-of-apps-used-per-device-2021/
• 5 Insider Intelligence. Latest trends in medical monitoring devices and wearable health technology (2023). Insider Intelligence. Accessed April 29, 2023 at: https://www.insiderintelligence.com/insights/wearable-technology-healthcare-medical-devices/
• 6 Marquard J. Human factors and organizational issues in health informatics: innovations and opportunities. Yearb Med Inform 2021; 30 (01) 91-99
• 7 The Office of the National Coordinator for Health Information Technology. Conceptualizing a data infrastructure for the capture, use, and sharing of patient-generated health data in care delivery and research through 2024. Accessed January 25, 2023 at: https://www.healthit.gov/sites/default/files/onc_pghd_practical_guide.pdf
• 8 Lai AM, Hsueh PS, Choi YK, Austin RR. Present and future trends in consumer health informatics and patient-generated health data. Yearb Med Inform 2017; 26 (01) 152-159
• 9 Tiase VL, Hull W, McFarland MM. et al. Patient-generated health data and electronic health record integration: a scoping review. JAMIA Open 2020; 3 (04) 619-627
• 10 Schreiweis B, Pobiruchin M, Strotbaum V, Suleder J, Wiesner M, Bergh B. Barriers and facilitators to the implementation of eHealth services: systematic literature analysis. J Med Internet Res 2019; 21 (11) e14197
• 11 Abdolkhani R, Gray K, Borda A, DeSouza R. Patient-generated health data management and quality challenges in remote patient monitoring. JAMIA Open 2019; 2 (04) 471-478
• 12 Adler-Milstein J, Nong P. Early experiences with patient generated health data: health system and patient perspectives. J Am Med Inform Assoc 2019; 26 (10) 952-959
• 13 Aljedaani B, Babar MA. Challenges with developing secure mobile health applications: systematic review. JMIR Mhealth Uhealth 2021; 9 (06) e15654
• 14 James PA, Oparil S, Carter BL. et al. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA 2014; 311 (05) 507-520
• 15 Public Health Informatics Institute. Self-measured blood pressure monitoring: key findings from a national health information technology landscape analysis. Accessed January 25, 2023 at: https://phii.org/wp-content/uploads/2021/09/PHII-Report-on-SMBP_FINAL.pdf
• 16 Lobach DF, Boxwala A, Kashyap N. et al. Integrating a patient engagement app into an electronic health record-enabled workflow using interoperability standards. Appl Clin Inform 2022; 13 (05) 1163-1171
• 17 Krist AH, Davidson KW, Mangione CM. et al; US Preventive Services Task Force. Screening for hypertension in adults: US Preventive Services Task Force reaffirmation recommendation statement. JAMA 2021; 325 (16) 1650-1656
• 18 Whelton PK, Carey RM, Aronow WS. et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Hypertension 2018; 71 (06) 1269-1324
• 19 Demiris G, Iribarren SJ, Sward K, Lee S, Yang R. Patient generated health data use in clinical practice: a systematic review. Nurs Outlook 2019; 67 (04) 311-330
• 20 Koppel R, Lehmann CU. Implications of an emerging EHR monoculture for hospitals and healthcare systems. J Am Med Inform Assoc 2015; 22 (02) 465-471
• 21 Maas B. Digital therapeutics. Accessed January 25, 2023 at: https://www.rxbenefits.com/blogs/digital-therapeutics/
• 22 Kiaghadi A, Hu P, Gummeson J, Rostaminia S, Ganesan D. Continuous measurement of interactions with the physical world with a wrist-worn backscatter reader. ACM Trans Internet Things 2020; 1 (02) 1-22