Keywords
patient-reported outcome measures - multilingualism - mobile health
Introduction
Health data are nowadays typically collected and stored electronically. The amount of these data is increasing exponentially, thus producing new opportunities for clinical research.[1] Electronic health records (EHRs) improve health care efficiency and safety,[2]
[3] as well as enable rapid methods for the secondary use of clinical records. Unfortunately, the collaboration and sharing of information between outpatient clinics and hospitals is still difficult due to legislation, security concerns, incompatibility of the systems, and lack of standards in use.
The medical informatics community is conducting several initiatives to solve the lack of interoperability between electronic health information systems. One of the most widely accepted solutions is the use of standards that enable the classification of elements in a way that several systems are compliant with, including transport protocols and security layers to preserve data integrity and privacy. The adoption of these standards is an ongoing task and not always straightforward. The large variety of standards and a lack of global decisions on which to use (especially regarding transport protocols and formats) produce a lack of consensus and interoperability problems.
Patient-reported outcomes (PROs) are defined by the Food and Drug Administration of the United States as “any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else.”[4] PROs have become a valuable tool for improving patient diagnosis and treatment, as well as the secondary use of health data for clinical research, although their selection needs to be handled carefully.[5] The validation of PROs and their translations, defined by Acquadro et al as “the ability to express and investigate equivalent concepts across all language versions,” ensures their correctness and enables their use in clinical routine and research practice.[6]
Language barriers affect quality of care around the world, hampering comprehension and adherence as well as patient and provider satisfaction.[7] With increasing migration in Europe,[8] physicians often need to treat patients who do not speak the local language(s), which hampers patient diagnosis and treatment. A solution could be the engagement of translators, but they are not always available or even requested.[9] Translated and validated PROs can be used to gather patient information regardless of the local language.
Electronic systems and mobile devices provide vital opportunities for the collection of PROs. Electronic PRO (ePRO) systems can lead to more accurate and complete data, improved protocol compliance, avoidance of secondary data entry errors, easier implementation of skip patterns, less administrative burden, high respondent acceptance, reduced sample size requirements, and potential cost savings.[10] In addition, ePROs could be used to collect patient data in any language and the results be visualized in the local language. This would support patient diagnosis for international patients when a translator is not an option and enable the easy analysis of clinical data in international clinical studies. So far, the majority of ePROs have been single purpose: they were developed for a certain study, and thus they are generally neither multilingual, multicenter, scalable, nor interoperable.
There is therefore a need for an ePRO that allows for the collection of patient data in several centers across different countries and in several languages, being able to export the data in different formats and standards. Furthermore, the system should be evaluated and accepted by both clinicians and patients.
The Institute of Medical Informatics (IMI) of the University of Münster started in 2010 the development of the ePRO MoPat,[11] which was solely implemented for use at a single center (the Center for Chronic Pruritus at the University Hospital of Münster) in a single language (German). MoPat was newly developed (MoPat2) in 2014 and is being routinely updated.[12] MoPat2 is a Web-based ePRO with a user-friendly interface that enables patients to answer medical questionnaires. Patient answers are encapsulated into encrypted messages, either in Health Level-7 or Operational Data Model (ODM) format, and securely transferred to EHRs and/or research databases.[13] MoPat2 is a generic ePRO that includes several functions such as the import of questionnaires from ODM files, creation of new questionnaires and surveys, generation of user statistics, and a score calculator based on questionnaire responses.
Objectives
The aim of this research is to report the development of a multicenter, multilingual, and interoperable ePRO and the evaluation of its user satisfaction in an international setting.
Methods
Development: Process Steps and Technologies
The development process started with the requirements acquisition and design of the software architecture. In this initial process step, existing solutions were explored and their reuse was analyzed.
The second step involved the development of the system. MoPat2 was developed using the Java 8 programming language, utilizing the frameworks jQuery Mobile 1.4.5, Spring 4, and Maven 3. The resulting functionality will be shown in detail in the “Results” section. Its development was completed with automatic and semiautomatic tests using jUnit and the Selenium plugin for Mozilla Firefox.
Evaluation
The newly developed functionality was evaluated by the European Network on Assessment of Severity and Burden of Pruritus (PruNet), an expert group funded by the European Academy of Dermatology and Venereology (EADV). The aim of the said expert group is to harmonize the assessment of itch in routine dermatological care in Europe[14] and establish a consensus among several experts on which PROs best serve this purpose. The selected PROs[14] were then translated and validated in several languages following their selection. The validation of these PROs required a multilingual functionality that allows patients from different countries to answer the same set of questions in their native language. For a better and easier analysis of the results, the data collected had to be stored in a central system and remain exportable in a format compliant with statistic tools.
Organizational Setting
The PruNet study was led by the Center for Chronic Pruritus and the IMI at the University Hospital of Münster. The multilingual ePRO MoPat2 was evaluated in eight centers in eight different countries: the Policlinico A. Gemelli, Catholic University of Sacred Heart, Rome, Italy; the Medical University of Graz, Austria; the University Hospital of Brest, France; the Wroclaw Medical University, Poland; the Adnan Menderes University, Aydın, Turkey; the Kantonsspital Aarau, Switzerland; the Hospital Sant Pau, Barcelona, Spain; and the University Hospital of Münster, Germany.
Data Collection
Data were collected in 1-month periods, from July 2015 until March 2016, in each clinic. During this time, all dermatological patients who visited these clinical institutions and clinicians working at the clinics involved in the PruNet project were asked to complete an electronic survey including a user satisfaction questionnaire using MoPat2.
The PRO validation for the PruNet study[14] included two MoPat2 patient surveys: the first survey was completed by patients following their arrival at the clinical institution, and the second survey after their visit with the physician. The first survey contained 67 questions and included the PROs requiring validation and a MoPat2 user satisfaction questionnaire.
Two questionnaires were created by the IMI team to assess the user satisfaction of MoPat2, with one addressing clinicians and the other patients. The clinician questionnaire contains seven Boolean questions (yes/no), one multiple-choice question with five possible answers, and four free-text questions. In contrast, the patient questionnaire contains eight Boolean, two multiple-choice, and four free-text questions. Although MoPat2 can be used in every tablet personal computer (PC) running on Android v4.0, iOS v7.0, or Windows Mobile v8.0 or higher; only iPad 2 tablet devices running on iOS v9.0 were used in this study for the purpose of data collection. As no instructions are provided to the users regarding the use of MoPat2, the satisfaction questionnaires contain questions concerning the general usability of the iPad. MoPat2 was not referred to, since users are unaware of its name and/or role.
The questions in the satisfaction questionnaires are meant to assess the user satisfaction of MoPat2 (questions 1, 2, 8, and 9 in the patients' questionnaire; 1, 2, 3, and 8 in the clinicians') and the acceptance of the MoPat2 PROs (questions 3–7 in the patients' questionnaire and 4–7 in the clinicians'). The analysis will be focused on the former. The user satisfaction questionnaires have been included as an additional file in the [Supplementary Appendix A].
Data Analysis
A descriptive and explorative analysis of the results was performed using R and the corrected contingency coefficient K*∈[0,1] between patient responses and age, gender, and language of the respondents calculated.
Results
Development of ePRO Multilingual Functions
Multilingual capabilities for the administration and user functions of MoPat2 have been developed. Since it was previously only possible to store a single question text, the database required enhancement. The single column storing the corresponding text was changed to a multivalue attribute that stores the combined language code according to International Organization for Standardization (ISO) 639, the specific country code according to ISO 3166, and the bidirectionally translated text for the defined language. The administration interface was extended to allow the addition of all languages available in Java 8 Locale class. To improve the simplicity of the user interface, the input fields for single languages have been made collapsible and are marked with colors to indicate whether they have been included.
This functionality allows users to create questionnaires and surveys in all known languages. The graphical user interface (GUI) was translated in seven different languages: English, French, German, Italian, Polish, Spanish, and Turkish.
The workflow for conducting a survey using MoPat2 ([Fig. 1]) is as follows: (1) The clinician selects a language for the GUI and logs into MoPat2. (2) The clinician enters patient's case number, pseudonym, or study identification number. (3) The clinician selects a questionnaire bundle (the set of questionnaires that conforms the electronic survey in MoPat2) and the patient's language. If the questionnaire bundle is in a language not yet supported by the general GUI, the survey (answers and questions) will be in the patient's language and the general GUI (buttons next, previous, etc.) will remain in the currently displayed language. (4) The patient receives the tablet PC and completes the survey, handing it back to the clinician once the survey has been completed.
Fig. 1 Process steps to be followed to perform a multilingual survey using MoPat2.
As a Web-service, MoPat2 is available through the Internet. A Web-clip with MoPat2's URL was installed in the devices used for surveying the patients. Once a patient completes a survey, responses are securely sent via a Secure Sockets Layer to x4T: an electronic data capture system located in Münster.[15] x4T was also used for subject recruitment and survey completeness tracking. All values received in x4T apart from free-text fields were displayed in the home language.
User Satisfaction
The user satisfaction evaluation included the clinicians involved in the use of MoPat2 and the patients visiting the clinic that were recruited for the study and signed the agreement form.
Clinician User Satisfaction
Twenty-eight clinicians completed the user satisfaction survey. The MoPat2-relevant results are as follows:
-
Ninety-three percent of clinicians (26 out of 28) think that patient data collection with MoPat2 is acceptable.
-
Ninety-six percent of clinicians (27 out of 28) think that MoPat2 is an appropriate tool for recording pruritus in patients.
-
On a scale from 1 (“very good”) to 5 (“very bad”), the participating clinicians rated the usefulness of the iPad system (MoPat2) with an average score of 2 (“good”).
Patient User Satisfaction
A total of 495 patients participated in the PruNet study and 468 (94.5%) completed the user satisfaction questionnaire. Participants completed the survey including the satisfaction questionnaire in 18 minutes and 2 seconds in average with a median value of 9 minutes and 20 seconds.
The age distribution by survey language selected in MoPat2 is depicted in [Fig. 2]. The participants who completed the survey in Turkish are, on average, slightly younger (45 years) than the general patient population (52 years), whereas Italian-speaking participants were found to be slightly older (58 years).
Fig. 2 Distribution of population by language and age: tr_TR (Turkish-Turkey), fr_FR (French-France), pl_PL (Polish-Poland), de_DE (German-Germany), es_ES (Spanish-Spain) and it_IT (Italian-Italy). One patient in Spain preferred to complete the survey in English (en_GB). The dashed line represents the inclusion criterion of a minimum age of 18 years. The solid blue lines represent the median and quartiles of their collective ages. The dashed line represents the inclusion criterion of a minimum age of 18 years. The solid blue lines represent the median and quartiles of their collective ages.
The results of the patient user satisfaction questionnaire are shown in [Table 1].
Table 1
Patient responses to the user satisfaction questionnaire
|
Question
|
True
|
False
|
NA
|
K* Gender[a]
|
K* Language[a]
|
|
1. Do you feel you have been given sufficient information about the purpose of the iPad survey?
|
452 (96.6%)
|
14 (3%)
|
2 (0.4%)
|
0.00
|
0.31
|
|
2. Did you get on well using the iPad to complete these questionnaires?[b]
|
439 (93.9%)
|
27 (5.8%)
|
2 (0.3%)
|
0.04
|
0.33
|
|
2a. I generally don't get on well with iPads/tablets[c]
|
20 (74%)
|
7 (26%)
|
–
|
0.34
|
0.27
|
|
2a. The display/the font is too small[c]
|
4 (15%)
|
23 (85%)
|
–
|
0.16
|
0.31
|
|
2a. I had problems selecting the answer/number on the display and had to make frequent corrections[c]
|
7 (26%)
|
20 (74%)
|
–
|
0.15
|
0.50
|
|
2a. I find it hard to enter data into the iPad because of my manual/visual limitations (e.g., osteoarthritis of the hands, rheumatism, eye disease, or similar)[c]
|
9 (33%)
|
18 (67%)
|
–
|
0.37
|
0.26
|
|
2b. Other (free text)[c]
|
0 (0%)
|
27 (100%)
|
–
|
|
|
|
3. Did you find the individual questions easy to understand?
|
451 (96.4%)
|
14 (3%)
|
3 (0.6%)
|
0.00
|
0.26
|
|
4. Did you find it helpful to have to answer the questions in the order they were given?
|
437 (93.4%)
|
27 (5.8%)
|
4 (0.8%)
|
0.17
|
0.17
|
|
5. Were there any questions where you had difficulty in choosing an answer?
|
152 (32.5%)
|
314 (67.1%)
|
2 (0.4%)
|
0.13
|
0.24
|
|
6. Did the survey on the iPad seem too long?
|
74 (15.9%)
|
392 (83.8%)
|
2 (0.3%)
|
0.03
|
0.25
|
|
7. Would you be willing to use the iPad if it was used for routine surveys in an outpatient clinic?[b]
|
416 (88.9%)
|
50 (10.7%)
|
2 (0.4%)
|
0.01
|
0.27
|
|
7a. Is answering questions on the iPad too much effort for you?[c]
|
21 (42%)
|
29 (58%)
|
–
|
0.42
|
0.82
|
|
7a. Do you have difficulty using an iPad?[c]
|
21 (42%)
|
29 (58%)
|
–
|
0.25
|
0.65
|
|
7b. Other (free text)[c]
|
11 (22%)
|
39 (78%)
|
–
|
0.05
|
0.74
|
|
8. Do you feel that the iPad is a suitable tool for collecting data on itching?
|
420 (89.8%)
|
44 (9.5%)
|
4 (0.9%)
|
0.14
|
0.32
|
|
8a. Why not? (free text)[c]
|
27 (61%)
|
17 (39%)
|
–
|
|
|
|
9. Did anything else come to mind? (free text)
|
130 (27.8%)
|
338 (72.3%)
|
–
|
|
|
a The corrected contingency coefficient K*∈[0,1] in the last two columns is used as a measure of association where higher values mean higher disassociation.
b For subquestions note that in question 2, there is no “false,” i.e., no data in subquestions for language tags “en_GB,” “es_ES,” and “pl_PL,” and in question 7, none for “en_GB.”
c The questions 2a, 2b, 7a, 7b, and 8a were shown only when the questions 2, 7, and/or 8 were answered with a “No,” respectively.
Twenty-seven patients expressed a hindered ability to use the iPad to complete the questionnaire ([Table 1]), being “I generally don't get on well with iPads” the most common reason (20 out of 27).
Fifty patients reported they would be unwilling to use the survey app (MoPat2) in an outpatient clinic ([Table 1]). These patients were then asked for the reason why in a multiple-choice format and free-text subquestion. The effort involved with using an iPad and the difficulties in using one were the most common reasons given (both with 21 out of 50 responses). Two of the 50 refused to answer when asked to provide a reason for their skepticism. Five participants had concerns regarding both the effort needed to answer questions and the difficulties using an iPad. Another 16 participants specified the effort needed to answer questions and the same number reported difficulties using an iPad as their only concern. Eleven selected “other reasons.” An analysis of the free-text responses was performed using Google Translate for the unknown languages and did not reveal any issues regarding the MoPat2.
The responses to the patient user satisfaction survey were also correlated with the language and gender of the patients. The highest disassociations were found on the correlation with language groups, especially on the answers to the subquestion 7 “Is answering questions on the iPad too much effort for you?” (K* = 0.84) and “Do you have difficulty using an iPad?” (K* = 0.65) ([Table 1]). Those associations were, in most cases, overrepresentations of participants who answered the questionnaires in Turkish (and in a few cases those speaking Italian).
The results for question 7 were also compared with the age of the respondents ([Fig. 3]). A difference of 9 years was observed among the median age of the positive and negative respondents. The median age of the group not willing to further use MoPat2 was 61 years, while the median age of the remaining participants was 52.
Fig. 3 Responses to the question “Would you be willing to use the iPad if it were used for routine surveys in an outpatient clinic?” correlated with the age of the respondents.
The form of associations was analyzed via the inspection of (extended) mosaic plots, but is not detailed in this section as the authors did not consider them significant. The complete analysis can be found in the [Supplementary Appendix B].
Discussion
Multilingual and multicenter capabilities were implemented for the ePRO MoPat2. The system was successfully evaluated by 28 clinicians and 468 patients in 8 European clinical centers and represents the first report on the use and evaluation of an electronic system for the collection of multilingual PROs.
The majority of the interviewed clinicians (96%) stated that the iPad is a valid tool for measuring pruritus and reported a high user satisfaction of the survey system (overall score of 2.0, “good”) despite being unable to perceive the benefits of using MoPat2 for their clinical practice, as they did not have access to the patient responses (the data collected was solely used for the validation of the PRO translations). This might have negatively affected the system's overall score. Among the patients, the responses were positive when asked if they got on well using the iPad and most of them would be willing to further use the system on an outpatient clinic. These findings suggest a high user satisfaction for MoPat2.
The patients unwilling to use MoPat2 for completing surveys in an outpatient care center are, in average, 9 years older than the ones willing to use it ([Fig. 3]). Other studies on this specific issue found similar results,[16] although trends could be evolving toward more acceptance of mobile-based systems by the elderly,[17] meaning that a future evaluation of the system could result in a higher MoPat2 acceptance by the elderly and the gap between the age of the users willing to use the system could be reduced.
To avoid fatigue and study “drop outs” due to too many questions in the survey[18] as well as reducing to the minimum the completion time of the survey before the clinical examination, the number of questions in the survey was limited to the PruNet study-oriented PROs and a single user satisfaction questionnaire. The use of standard tools such as the System Usability Scale[19] could enhance the scalability of the study, but would provide less specific information about the system and the iPad for the collection of PROs.
When an ePRO is needed, it must be decided whether a new one has to be implemented, as there are already several systems available and some offer functions similar to those of MoPat2.[20]
[21]
[22]
[23] LimeSurvey[24] offers an extensive spectrum of possibilities for multilingual online questionnaires but does not include EHR linkage capabilities and data protection may be an issue. REDCap is one of the best-known data capture systems, but is mainly clinical research-oriented as it does not include a connection with the EHR.[25] Other ePROs such as PatientViewpoint and Patient Care View are able to export the data to an EHR, but in a single format and language.[26] The Computer-based Health Evaluation Software[27] is a multilingual ePRO that can be used for data collection in multicenter clinical studies. The two most important mobile device operative systems also have their own initiatives to support the electronic collection of PROs: ResearchKit (Apple)[28] and ResearchStack (Android)[29] provide a framework for the easy creation of ePROs. Other initiatives such as C3-PRO try to establish a framework for integration of ePROs.[30] However, none of these ePROs has the functional capabilities and versatility of MoPat2, especially with regard to its multilingualism and interoperability.
Since the use of MoPat2 for data collection within the PruNet, its development and use has continued and its general GUI has been translated into the following languages (apart from the seven previously reported): Arabic, Dari, Farsi, Hindi, Kurdish, Dutch, Norwegian, Portuguese, Russian, Albanese, and Swedish.
MoPat2 is currently only usable at the clinical site. Remote capabilities should be implemented so that MoPat2 can be used by patients outside of a clinical setting. A future evaluation of these features should include a standard measurement tool for results scalability, as well as possible differences when using the system at home or at the clinical setting.
Conclusions
The successful implementation of multilingual functions for an ePRO and its use in a multicountry setting for the collection of anonymous data for a clinical study have been demonstrated. MoPat2 can now be used for the collection of clinical routine and study data in an international environment. The user satisfaction evaluation of the system reveals a general acceptance of the system by both clinicians and patients from several countries. Future work should include the possibility to use MoPat2 outside of a clinical setting.
