Mobile Technology in Medicine: Development and Validation of an Adapted System Usability Scale (SUS) Questionnaire and Modified Technology Acceptance Model (TAM) to Evaluate User Experience and Acceptability of a Mobile Application in MRI Safety Screening

Abstract

Background Magnetic resonance imaging (MRI) safety screening is a crucial procedure for patient preparation before entering into MRI room. Many hospitals in Malaysia are still using the MRI safety checklist printed form. Besides, clinicians will not get a definite conclusion about whether the patient is contraindicated for MRI or not. Hence, we have created one mobile application named MagnetoSafe to overcome this issue. The application will provide an instant decision on whether the patient has no contraindication, relative contraindication, or absolute contraindicated for MRI. We need to check for acceptability and user experience for any newly created mobile application.

Objective This study was designed to check the validity of the adapted Technology Acceptance Model (TAM) and System Usability Scale (SUS) Questionnaire.

Method The validity and reliability of the questionnaire were investigated. Subsequently, 52 fully completed responses were collected.

Results Face and content validity of the questionnaires are considered acceptable with only minor changes to Item 10 of SUS. The Cronbach's alpha for the SUS questionnaire (10 questions) is −0.49, which is not acceptable. The Cronbach's alpha for TAM questionnaire (3 domains; 14 questions) is acceptable, which is 0.910 for perceived usefulness, 0.843 for perceived ease of use, and 0.915 for intention to use.

Conclusion Face validity of the adapted SUS and modified TAM questionnaires is acceptable with only minor changes to Item 10 in SUS. Content validity with experts is good. However, the reliability of the SUS questionnaire is not acceptable and therefore adapted SUS will not be used for assessing user experience. The reliability of the modified TAM questionnaire with the original three-factor structure is considered acceptable and can be used to evaluate the user's acceptability of MagnetoSafe.

Keywords

magnetic resonance imaging - system usability scale - technology acceptance model - mobile application - MRI safety

Introduction

In clinical practice, magnetic resonance imaging (MRI) safety screening is a crucial procedure for patient preparation before entering into MRI room. Most of the hospitals in Malaysia are still using the MRI safety checklist printed form. This method is not environment friendly as many papers have to be used. Clinicians need to interview the patient and ask for any MRI contraindication. Most of our MRI screening forms do not have a complete list of MRI contraindications. Besides, after filling out the form, clinicians will not get a definite conclusion whether the patient is contraindicated for MRI or not. Hence, with the advancement of mobile technology and application, one mobile application named MagnetoSafe was created. Inside this application, the complete list of ferromagnetic objects was integrated into it, and at the end, the application will provide an instant decision regarding MRI safety.[1] Clinicians can know whether the patient is contraindicated or not before booking an appointment for MRI scanning. Therefore, it can be used for the early detection of MRI unsafe situations and to avoid delay in patient care by finding an alternative imaging modality for the patient's clinical condition in such situations.

The acceptability and usability of this application need to be tested. Based on the literature review, System Usability Scale (SUS) questionnaire is an excellent tool for the broad measurement of the user experience of a newly developed system.[2] This questionnaire consists of 10 questions that are easy to administer and can be used on a small sample size with reliable results.[2] [3] For this study, we adapted the SUS questionnaire by changing some wordings. Moreover, the perceived usability, ease of use, and intention of use of the application also need to be investigated. Therefore, Technology Acceptance Model (TAM) questionnaire was used as it encompasses these components. TAM questionnaire is a widely used tool to help understand and explain user behavior in a technology system.[4] [5] Many researchers have used this questionnaire to test new technology, and the results are reliable.[6] In this study, we have done minor modifications to TAM questionnaires.

Literature Review

Adapted Technology Acceptance Model

TAM is an information system theory, a robust and precise model to understand information technology usage. In 1989, Fred Davis developed and published this model in his dissertation.[7] Because of the growing demand for technology at that time, this model spread to one of the most cited models in the context of technology diffusion. Many models predict the diffusion of a system. However, the TAM is the only model that emphasizes information system.[5] According to F. Davis, there are two important goals in this model,[6] to understand the technology acceptance processes and to increase the theoretical insights for the design, which provide a basis for user acceptance testing methodology and evaluate the proposed system before their actual implementation. The acceptance of technological innovation is derived from one's attitude towards using the system, which, in turn, is determined by two beliefs, that is, perceived usefulness (PU) and perceived ease of use (PEU). The measurement scales for PU and PEU in the original TAM model are shown in [Table 1]. PU is defined as a “prospective user's subjective probability that using a specific application system will increase his or her job performance within an organisational context.” The questions on PU are shown as [Table 2]. PEU is the “degree to which the prospective user expects the target system to be free of effort.” The questions on PEU are shown in [Table 3]. In addition to this, F. Davis also states that the system usage is determined by behavioral intention to use (BI). BI is determined by the “person's attitude towards using a system (A) and perceived usefulness (U).”[6] A meta-analysis study further proved these changes, which showed that behavioral intention to use (IU) the system was found to be affected directly by the user's perception of usefulness and ease of use.[8] By definition, behavioral IU measures how likely a user employs the system.[9] Kim et al created the questions to assess the behavioral intention[10] ([Table 4]). The key concept framework of the original TAM is shown in [Fig. 1]. In summary, multiple modifications and adjustments have been made to the TAM model and the questionnaire. In 1996, the final version of TAM was found in a paper by Venkatesh and Davis 1996[11] [12] ([Fig. 2]).

Fig. 1 Original Technology Acceptance Model proposed by F. Davis.[7]

Fig. 2 Final version of Technology Acceptance Model.[12]

Table 1 Measurement scales for PU and PEU in the original Technology Acceptance Model (TAM) model by F. Davis[7] [13]

Perceived usefulness (PU)	Perceived ease of use (PEU)	Innovation technology
-Work more quickly -Job performance -Increase productivity -Effectiveness -Makes job easier -Useful	-Easy to learn -Clear and understandable -Easy to become skillful -easy to use -Controllable -Flexible	Lab experiment with email and graphics

Table 2 Six items scale for perceived usefulness worded toward CHART-MASTER[7]

Item no.	Candidate item for psychometric measures for perceived usefulness
1	Using CHART-MASTER in my job would enable me to accomplish tasks more quickly
2	Using CHART-MASTER would improve my job performance
3	Using CHART-MASTER in my job would increase my productivity
4	Using CHART-MASTER would enhance my effectiveness on the job
5	Using CHART-MASTER would make it easier to do my job
6	I would find CHART-MASTER useful in my job

Table 3 Six items scale for perceived ease of use worded towards CHART-MASTER[7]

Item no.	Candidate item for psychometric measures for perceived ease of use
1	Learning to operate CHART-MASTER would be easy for me
2	I would find it easy to get CHART-MASTER to do what I want it to do
3	My interaction with CHART-MASTER would be clear and understandable
4	I would find CHART-MASTER to be flexible to interact with
5	It would be easy for me to become skillful at using CHART-MASTER. Likely
6	I would find CHART-MASTER easy to use

Table 4 Behavioral intention by Kim et al[10]

Item no.	Candidate item for psychometric measures for behavioral intention to use
1.	I will consider using this service
2.	I plan to use this service.
3	I will continue to use this service.
4.	I will inform other of the goodness of this service

In this study, we used a modified version of TAM questionnaire, which consists of PU and PEU as the independent variables and IU as the sole dependent variable.

We also incorporated the adapted theoretical framework by Pande et al,[14] which assesses both the usability and acceptability component by combining SUS and TAM questionnaires. This is because user experience from SUS questionnaire did play an important role in determining the system's acceptability, which subsequently affects the behavioral IU ([Fig. 3]).

Fig. 3 An adapted framework from Pande et al comprising System Usability Scale and Technology Acceptance Model.[14]

System Usability Scale Questionnaire

The SUS questionnaire was originally created by John Brooke at Digital Equipment Corporation in United Kingdom in 1986.[3] This questionnaire measures three important aspects. The first aspect of the effectiveness of using this technology measures users' ability to achieve their objectives. The second aspect is efficiency, in which how much user's effort and user's resources are expended in achieving those objectives. The third aspect is satisfaction, whereby how satisfactory the user's experience is.

This questionnaire allows us to evaluate various products and services, including hardware, software, mobile devices, websites, and applications. With the advent of technology, SUS has become a widely used industry standard, with references in over 1300 articles and publications.[15] There are a few benefits of using this questionnaire; (i) it is a very easy scale to administer to participants, (ii) it can be used on small sample sizes, and (iii) it can effectively differentiate between usable and unusable systems.[2] [3] It consists of 10 items with five response options for respondents, from Strongly agree to Strongly disagree[3] ([Fig. 4]). There are two tones in this questionnaire whereby the odd-numbered questions are all considered positive tone, and even-numbered questions are all considered negative tone. Based on the Likert scale, if participants choose “Strongly Disagree,” a minimum of 1 point will be given. Whereas if participants choose “Strongly Agree,” a maximum of 5 points will be given.[10]

Fig. 4 Original question of System Usability Scale digital equipment corporation, 1986.[3]

After that, we calculate the SUS score by using the formula SUS score = (X + Y) × 2.5 [ X = Sum of the points for all odd-numbered questions – 5] [ Y = 25–Sum of the points for all even-numbered questions]. This formula will give a SUS acceptability score out of 100.

To elaborate further, 5 marks for “Strongly agree” answer in positive statements (if all are “strongly agree” answers for all 5 positive statements (odd number question), then the total marks will be 25, and X value = 25-5 =20.

However, 1 mark for “Strongly disagree” answer in negative statements (if all are “strongly disagree” answers for all 5 negative statements (even number question), then the total marks will be 5, and Y value = 25-5 = 20.

Based on the above example, SUS acceptability score is (20 + 20) × 2.5 = 100.

According to the articles, a SUS score above a 68 would be considered as good acceptability and a score below 68 is considered below average.[16] [Fig. 5] shows the interpretation of SUS scoring.[17]

Fig. 5 Standard version of the System Usability Scale and the interpretation.[15]

Methods

The study was carried out in the Faculty of Medicine, Universiti Sultan Zainal Abidin (UniSZA), Malaysia, with 52 clinicians as respondents. Approval from the Education Ministry of Malaysia and UniSZA Human Research Ethics Committee was taken. The original questionnaires, that is, TAM and SUS Questionnaire were cited. Minor modification and adaptation of the questionnaires were performed. For example, the word “system” in SUS was replaced by “app.” On the other hand, the wordings in TAM questionnaire are unchanged except the “system” is replaced by the “MagnetoSafe apps.” Nevertheless, the scoring system of these two questionnaires remains unchanged.

First, face validity was done with five subjects. After that, participants were selected based on the inclusion criteria in which the participant must have the experience using a manual MRI safety checklist form before. Participants installed the mobile application on their phones and started using it. Five case scenarios that simulated the actual clinical situation were given to them to key in. For those participants without knowledge of using medical applications, an enumerator will explain the mobile user interface to them personally. Finally, the adapted TAM and SUS questionnaires were given to them. Data were collected from May to August 2021. Data from the questionnaires were entered into SPSS v. 22.

Mobile Application

MagnetoSafe is an Android-based mobile application created to complement the MRI screening checklist form. This application can be installed on all Android-based mobile phones. For those without Android-based mobile phones, we will provide an Android-based tablet. Due to budget limitations, we unable to create the software in iOS.

Users were required to key in their username and password on the homepage. Then, they will be navigated to the second page, where they have to key in the patient's information and clinical details. Participant needs to select the suitable item from the list of dropdown menu. They need to scroll until find the item which fit the patient.

Once they have hit the submit button, the mobile application will show the result of whether the patient is safe for an MRI procedure or not. [Fig. 6] demonstrates the mobile user interface. At least five simulated case scenarios were given to the participant ([Fig. 7]).

Fig. 6 Mobile user interface of MagnetoSafe.

Fig. 7 Example of case scenario.

Questionnaires

The questionnaire used in this study is divided into three sections: the first section on background information, including the demographic data; the second section on the SUS questionnaire component; and the third section on the TAM model questionnaire.

The questionnaire is shown in [Figs. 8] to [10].

Fig. 8 Questionnaire on background information.

Fig. 9 Adapted System Usability Scale questionnaire.

Fig. 10 Modified Technology Acceptance Model questionnaire.

Result and Discussion

Demographic Data

The demographic characteristics and background information of participants were analyzed. Out of 52 participants, 23 (44.2%) are female and 29 (55.8%) are male. Ten participants (19%) are in the age group of 25 to 34, 18 (34.6%) in the age group of 35 to 44, 17 (32.7%) in the age group of 45 to 54, and 7 (3.5%) in 55 to 64. For educational level, 39 participants (75%) had completed a master's degree, 10 participants (19%) had a bachelor's degree, and 3 participants (6%) had completed a PhD. Background information showed 14 participants (27%) have experience using mobile applications for medical purposes and 38 participants (73%) have no experience. Among the 38 participants who have no experience using a mobile application for medical purposes, the majority are 45 to 54 years old. The experience status of the participants in using the nonmedical or medical mobile application was presented in [Table 5].

Table 5 The experience status of participant in using mobile application (n =52)

Items	Yes	No
Do you have experience in using any mobile application for nonmedical purpose?	47 (90.4%)	5 (9.6%)
Do you have experience in using any mobile application for medical purposes?	14 (26.9%)	38 (73.1%)

Validity of the Questionnaire

First of all, the contents of the draft questionnaire were checked by two radiologists and an expert in information technology. Then, the face validity was carried out with five participants comprising two medical officers, one house officer, and two specialists from a tertiary hospital. In item number 10 of the SUS questionnaire, “How do you rate your skill as a technology-savvy person?,” one participant was confused about the word “technology savvy,” and thus the definition of technology-savvy was added below that question. Two subjects had difficulty understanding the word “cumbersome” in SUS point 8, and this word was replaced by “awkward.” This is because we find other SUS questionnaire articles using this word as well.[17] Face validity on the modified TAM questionnaire was unremarkable.

Public health experts were asked to check the contents. Then the first draft of the questionnaire was distributed to four lecturers from UniSZA Medical Faculty to check the face validity of the questionnaire. After reshaping the questionnaire, the pretest study was conducted among 30 office staff and medical lectures from UniSZA Medical Faculty using Google Form to evaluate the construct validity and reliability of the questionnaire. As a result of the pretest study, the items in the questionnaire were revised and edited based on the respondents' comments and the public health experts' opinions

Reliability of Adapted Technology Acceptance Model Questionnaire

From the reliability statistics, the Cronbach's alpha for ten items in adapted SUS is −0.049. [Table 6] shows the item-total statistics. According to Tavakol and Dennick, Cronbach's alpha is acceptable if the value is between 0.70 and 0.80, and good if the level is above 0.80.[18] This means that these items are unrelated to each other, and this adapted questionnaire is unreliable. One possible explanation is item number 2, “I found the application unnecessarily complex,” and item number 3, “I thought the application was easy to use,” are related to each other and give the same meaning.

Table 6 SPSS output for item-total statistics of SUS

	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total correlation	Cronbach's alpha if item deleted
1. I think that I would like to use the MagnetoSafe application frequently	27.29	8.327	0.255	−0.256[a]
2. I found the application unnecessarily complex	29.69	12.217	−0.372	0.169
3. I thought the application was easy to use	27.21	9.856	−0.002	−0.052[a]
4. I think that I would need the support of a technical person to be able to use this system	28.06	8.291	−0.004	−0.059[a]
5. I found the various functions in this application were well integrated	27.29	9.699	0.086	−0.102[a]
6. I thought there was too much inconsistency in this application	29.63	10.119	−0.046	−0.022[a]
7. I would imagine that most people would learn to use this application very quickly	27.23	9.828	−0.020	–.039[a]
8. I found the application very awkward to use	29.71	10.994	−0.185	0.064
9. I felt very confident using the application	27.21	10.366	−0.068	−0.011[a]
10. I needed to learn a lot of things before I could get going with this application	28.62	7.026	0.179	−0.303[a]

Abbreviation: SUS, System Usability Scale.

^a The value is negative due to a negative average covariance among items. This violates reliability model assumptions. You may want to check item codings.

Reliability of Adapted Technology Acceptance Model Questionnaire

The adapted TAM questionnaire consists of 14 items, including PEU, PU, and IU domains. The Cronbach's alpha for 14 items is 0.926.

From the reliability statistics, the Cronbach's alpha for seven items in the PU domain is 0.910. This means that if we repeatedly measure the questionnaire response to the same participant, this will give us 91% reliability that is good.

The Cronbach's alpha for PEU is 0.843, whereas for IU, it is 0.915 ([Table 7]). Therefore, the reliability of all the respective domains in the adapted TAM questionnaire is good.

Table 7 Reliability statistics for adapted TAM questionnaire

Domains in TAM questionnaire	No of items	Cronbach's alpha
PU	7	0.910
PEU	4	0.843
IU	3	0.915

Abbreviations: IU, intention to use; PEU, perceived ease of use; PU, perceived usefulness; TAM, Technology Acceptance Model.

Reliability on the Combination of Adapted SUS and Adapted TAM Questionnaire

However, if the two sets of questionnaires are combined, the Cronbach's alpha of the 24 items is 0.797, which is acceptable. Despite this result, SUS cannot be used because of negative reliability.

Conclusion

In summary, the face validity of the adapted SUS and modified TAM questionnaires is acceptable with only minor changes to Item 10 in SUS. Content validity with experts is good. However, the reliability of SUS questionnaire is not acceptable and therefore adapted SUS will not be used for assessing user experience. The reliability of the modified TAM questionnaire with the original three-factor structure is considered acceptable and can be used to assess the user's acceptability of MagnetoSafe.

Conflict of Interest

None declared.

Acknowledgement

The authors would also like to extend our great appreciation to Faculty of Medicine, Ethics Committee of Universiti Sultan Zainal Abidin, and all research participants.

• References

• 1 Dill T. Contraindications to magnetic resonance imaging: non-invasive imaging. Heart 2008; 94 (07) 943-948
• 2 Sauro J. A Practical Guide to Measuring Usability. Denver: Measuring Usability LLC; 2010
• 3 Brooke J. SUS-A quick and dirty usability scale. Usability evaluation in industry. 1996; 189 (194) 4-7
• 4 Holden H, Rada R. Understanding the influence of perceived usability and technology self-efficacy on teachers' technology acceptance. J Res Technol Educ 2011; 43 (04) 343-367
• 5 Surendran P. Technology acceptance model: a survey of literature. Int J Business Soc Res 2012; 2 (04) 175-178
• 6 Lee Y, Kozar KA, Larsen KR. The technology acceptance model: Past, present, and future. Comm Assoc Inform Syst 2003; 12 (01) 50
• 7 Davis FD. Technology Acceptance Model: TAM. Al-Suqri, MN, Al-Aufi, AS: Information Seeking Behavior and Technology Adoption, S 1989: 205-219
• 8 Rahmi B, Birgoren B, Aktepe A. A meta-analysis of factors affecting perceived usefulness and perceived ease of use in the adoption of e-learning systems. Turk Online J Distance Educ 2018; 19 (04) 4-42
• 9 Gagnon M-P, Ngangue P, Payne-Gagnon J, Desmartis M. m-Health adoption by healthcare professionals: a systematic review. J Am Med Inform Assoc 2016; 23 (01) 212-220
• 10 Kim N, Park Y, Lee D. Differences in consumer intention to use on-demand automobile-related services in accordance with the degree of face-to-face interactions. Technol Forecast Soc Change 2019; 139: 277-286
• 11 Davis FD, Venkatesh V. A critical assessment of potential measurement biases in the technology acceptance model: three experiments. Int J Hum Comput Stud 1996; 45 (01) 19-45
• 12 Venkatesh V, Davis FD. A model of the antecedents of perceived ease of use: development and test. Decis Sci 1996; 27 (03) 451-481
• 13 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Manage Inf Syst Q 1989; 13: 319-340
• 14 Pande T, Saravu K, Temesgen Z. et al. Evaluating clinicians' user experience and acceptability of LearnTB, a smartphone application for tuberculosis in India. mHealth 2017; 3 (07) 30
• 15 Brooke J. SUS: a retrospective. J Usability Stud 2013; 8 (02) 29-40
• 16 Sauro J. . ways to interpret a SUS score,” 2018. Available at: https://measuringu com/interpret-sus-score
• 17 Lewis JR, Sauro J. Item benchmarks for the system usability scale. J Usability Stud 2018; 13 (03) 158-167
• 18 Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ 2011; 2: 53-55

Publication History

Article published online:
07 December 2022