Key words
radiation safety - safety - statistics - health policy and practice
Introduction
In times of highly fragmented healthcare services, cooperation between various providers
in the sector is regarded as one key factor to ensure high quality of care [1]. Gathering reliable data on the quality of specialists’ services as perceived by
referring physicians can yield important information making it possible to assess
and improve services as well as cooperation on an organizational level and beyond.
Concerning imaging services, a number of studies and questionnaires have proposed
the assessment of the satisfaction of referring physicians with and the opinion of
the quality of imaging services. They evaluate quality in general [2], focus on certain imaging subspecialties [3]
[4]
[5]
[6] or concentrate on reporting of results [7]
[8]
[9]. However, based on a literature search, validated questionnaires assessing the opinion
of referring physicians regarding the quality of imaging services are lacking.
On the contrary, the question as to which aspects of quality are of special importance
to referring physicians and determine the decision to choose a specialist provider
have been intensively discussed. Not surprisingly, most research reveals that a referring
physician’s perception of a specialist’s medical skills is an important criterion.
In addition to that, previous positive experiences, patient feedback as well as communication
with the specialist are very important elements for referring physicians [10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]. Communication includes talking or writing about organizational aspects such as
the scheduling of appointments, and medical aspects such as specialists’ response
by letter or phone. Institutional and medical quality appear to be tightly linked
to each other. Positive experiences are the basis for sustainable relationships between
referring physicians – and vice versa, with personal contacts providing an opportunity
to ask medical questions [18]. This is also shown by Hackl et al. [19] who report that referrals within a doctor’s personal network are more appropriate
in terms of patient outcomes than referrals outside the network, demonstrating that
personal connections reduce information asymmetry with respect to the specialists’
abilities, meaning that referring physicians are better able to evaluate the specialists’
competences and their limits. In summary, measuring the satisfaction of referring
physicians with an imaging service is crucial to improving the quality of care provided.
The aim of the present study was to develop and validate a questionnaire for referring
physicians that measures their assessment of the quality of care provided by outpatient
imaging services.
Materials and Methods
Questionnaire development
The designing of the questionnaire was part of a broader initiative. This initiative
brought together personnel of several imaging services with quality experts. In a
participatory project, standards for infrastructures, patient and referring physician
management, teamwork and quality development were compiled. The initiative and the
development of the questionnaire were organized by the EQUAM Foundation.
A survey instrument originally generated by the Professional Association of German
Surgeons, which was unpublished but distributed to physicians referring to specialists
in Germany and Switzerland by the aQua-Institut and the EQUAM Foundation was used
as the basis for the development of the questionnaire. The original instrument was
discussed and modified within a group of radiologists, radiographers, referring physicians
and experts in order to draft a first version specifically addressing physicians referring
to outpatient imaging services.
After that, the questionnaire underwent a qualitative pre-test [20] with two general practitioners and two specialists and was adapted accordingly.
After a final discussion of the pre-test results with experts and referring physicians,
a last modification of the questionnaire was executed. The instrument was then field-tested.
The questionnaire included 24 items organized in several sections. Four items were
summarized under the topic of “professional knowledge and skills”. The satisfaction
of referring physicians with the services’ contribution to integrated care as well
as radiologists’ reports was represented by five items. Another seven items assessed
the impressions of referring physicians with regard to the treatment of patients by
radiology services and the final three items addressed the topic of “service”. All
items assessing the quality of imaging services used a 5-point Likert response scale
ranging from “strongly disagree with this statement” to “strongly agree with this
statement”. Referring physicians were also asked to provide demographic data, namely
their field of specialization, the number of years since their state examination,
as well as the frequency with which they refer patients to imaging services.
Sample and Procedures
The questionnaire was field-tested as an online survey and invitations for participation
were sent to 448 physicians who refer to four radiology outpatient imaging services
in the German-speaking part of Switzerland. The referring physicians’ addresses were
provided by the participating services. Recipients were asked to complete the questionnaire
within two weeks. Participation was voluntary and anonymous. Referring physicians
who had not answered within a two-week period received a reminder.
Statistical Methods
Descriptive statistics (means, distributions, missing answers) assessed the quality
and distribution of the data. An individual mean score was calculated for each respondent
by aggregating all item ratings. Total mean scores and their corresponding distributions
were calculated as the overall mean scores and distributions of the individual mean
scores.
We examined known-groups validity [21]. Based on the evidence presented above, we hypothesized that physicians referring
with high frequency estimate the quality of radiology services to be higher than physicians
referring with low frequency. Physicians referring with high frequency are obviously
satisfied with the service provided when they have, as it is the case in Switzerland,
free choice of specialists. At the same time, frequent referrals enhance the quality
of imaging services by stabilizing communication and contacts [19]. The concept of known-groups validity expresses that a questionnaire claiming content
validity should reproduce such well-established differences (for a similar procedure
see [22]
[23]).
Differences were analyzed using a one-sided two-sample Wilcoxon test (Mann-Whitney
test) [24]
[25]. P-values < 0.05 were regarded as statistically significant.
Even though the study was not designed to have the power to show differences at the
item level, these were also assessed in order to gain insight about which quality
criteria prove to be especially discriminative.
Internal consistency and reliability were measured with Cronbach’s Alpha [26]. This measure can be viewed as the expected correlation of two tests measuring the
same construct, varying between 0 and 1. A value of > 0.7 was assumed to be sufficient.
All analyses were performed with the Open Source Software R, Version 3.4.3 from 2017
[27].
Results
In total, 148 questionnaires were returned, resulting in a return rate of 33 % (148/448).
Ten questionnaires were excluded since the respondents only filled in the demographic
data and did not proceed with the questionnaire. Thus, the corrected return rate was
31 % (138/448). [Table 1] summarizes the characteristics of the study sample. Most of the referring physicians
held a specialization in general internal medicine and had completed their state examination
more than 21 years ago. 64 % (89/138) of the participants usually refer patients to
the imaging services at least once a week, while 36 % (49/138) have a lower referral
rate to radiology services ([Table 1]).
Table 1
Summary of the study sample’s characteristics (n = 138).
Tab. 1 Zusammenfassung der Charakteristiken des Samples (n = 138).
|
characteristic
|
n (%)
|
|
imaging service
|
|
|
imaging service 1
|
13 (9)
|
|
imaging service 2
|
38 (28)
|
|
imaging service 3
|
59 (43)
|
|
imaging service 4
|
28 (20)
|
|
specialization (multiple selections possible)
|
|
|
|
100
|
|
|
10
|
|
|
5
|
|
|
5
|
|
|
5
|
|
|
4
|
|
|
3
|
|
|
3
|
|
|
2
|
|
|
2
|
|
|
2
|
|
|
2
|
|
|
2
|
|
|
7
|
|
years since completion of state examination (2 missing answers)
|
|
|
|
0 (0)
|
|
|
1 (1)
|
|
|
5 (4)
|
|
|
38 (28)
|
|
|
92 (67)
|
|
frequency of referral to imaging services
|
|
|
|
13 (9)
|
|
|
56 (41)
|
|
|
20 (14)
|
|
|
33 (24)
|
|
|
12 (9)
|
|
|
4 (3)
|
Descriptive Statistics
Six items were deleted from the final survey instrument due to excessive missing answers
and a lack of discriminatory potential. These were items asking about counselling
for choosing examinations, taking over of the right amount of responsibility and the
collaboration between radiologists and their colleagues. A further three reverse-coded
items were deleted, concerning consent of patients to examinations, the handling of
confidential data and patient’s rights, as they showed untypical distributions, pointing
to a high frequency of confusion at the lower and higher end of the scale. The mean
scores of these items ranged between 4.2 and 4.7 on a five-point Likert Scale.
The final survey instrument under evaluation thus consists of 18 items. In 92 % of
the 138 questionnaires between zero and four answers were missing.
[Table 2] summarizes the results of the final instrument. Results show a high level of satisfaction
with a total mean score over all items of 4.5 on a 5-point Likert scale. The timeliness
of reports is ranked highest with a mean of 4.7. Moreover, professional expertise,
communication with the referring physicians’ assistants and other collaborators, information
and understandability of reports as well as the possibility to quickly obtain appointments
for patients and the reachability of staff scored high with means of 4.6. On the other
hand, referring physicians were not too satisfied with the handling of healthcare
resources with this item scoring the lowest with a mean of 4.2. Likewise, patient
information on examinations, recommendations made in reports concerning additional
or future exams as well as care for vulnerable patients scored rather low with means
from 4.3 to 4.4 ([Table 2]).
Table 2
Results (mean and standard deviation (SD)) of the 18 items that were included in the
final instrument.
Tab. 2 Resultate (Mittelwerte und Standardabweichung (SD)) der 18 Items des finalen Instruments.
|
total
|
physicians referring with low frequency (n = 49)
|
physicians referring with high frequency (n = 89)
|
exact p-value, Wilcoxon test
|
|
mean
|
SD
|
mean
|
SD
|
mean
|
SD
|
|
|
total score
|
4.5
|
0.5
|
4.4
|
0.6
|
4.6
|
0.5
|
0.019
|
|
items
|
|
the service’s staff is professionally up-to-date
|
4.6
|
0.6
|
4.5
|
0.7
|
4.6
|
0.5
|
0.14
|
|
the service’s staff knows the limits of their competencies and possibilities
|
4.5
|
0.6
|
4.4
|
0.7
|
4.5
|
0.6
|
0.12
|
|
the service’s staff informs me if a request for referral exceeds their competencies
|
4.5
|
0.8
|
4.3
|
0.9
|
4.5
|
0.7
|
0.11
|
|
the service’s staff cooperates well for the care of patients with complex problems
|
4.5
|
0.7
|
4.3
|
0.8
|
4.5
|
0.6
|
0.13
|
|
the service’s staff handles resources for healthcare efficiently (e. g. elaborate
diagnostic procedures)
|
4.2
|
0.8
|
3.9
|
1.0
|
4.3
|
0.7
|
0.03
|
|
I have the impression that the service’s staff communicates appropriately with my
assistants and other collaborators
|
4.6
|
0.6
|
4.5
|
0.6
|
4.7
|
0.5
|
0.02
|
|
radiological reports contain the expected information
|
4.6
|
0.6
|
4.4
|
0.8
|
4.7
|
0.5
|
0.02
|
|
radiological reports are comprehensible and clear
|
4.6
|
0.6
|
4.5
|
0.6
|
4.7
|
0.6
|
0.02
|
|
radiological reports are clinically useful
|
4.5
|
0.8
|
4.3
|
1.0
|
4.6
|
0.6
|
0.03
|
|
radiological reports contain a clear answer to my question
|
4.5
|
0.8
|
4.3
|
0.9
|
4.6
|
0.7
|
0.02
|
|
radiological reports contain recommendations based on actual evidence for further
radiological exams
|
4.4
|
0.8
|
4.3
|
0.8
|
4.4
|
0.8
|
0.07
|
|
I have the impression that the service’s staff provides my patients with the necessary
information about the imaging exam
|
4.3
|
0.7
|
4.2
|
0.7
|
4.4
|
0.7
|
0.07
|
|
I have the impression that the service’s staff carefully questions my patients (e. g.
about allergies)
|
4.4
|
0.7
|
4.3
|
0.6
|
4.5
|
0.7
|
0.06
|
|
I have the impression that the service’s staff treats my patients with understanding
and empathy
|
4.5
|
0.7
|
4.4
|
0.7
|
4.5
|
0.7
|
0.17
|
|
I have the impression that very vulnerable patients are also treated well
|
4.4
|
0.7
|
4.3
|
0.8
|
4.4
|
0.7
|
0.17
|
|
the service’s staff can easily be reached
|
4.6
|
0.7
|
4.5
|
0.7
|
4.6
|
0.6
|
0.13
|
|
I get appointments for my patients in due time
|
4.6
|
0.7
|
4.4
|
0.8
|
4.7
|
0.6
|
0.007
|
|
I receive reports in due time
|
4.7
|
0.6
|
4.5
|
0.8
|
4.8
|
0.5
|
0.003
|
Validity, Internal Consistency and Reliability
The mean rankings of physicians referring with high frequency were higher than those
of physicians referring with low frequency with total score means of 4.6 compared
to 4.4, respectively. The one-sided Wilcoxon test for differences between rank distributions
of physicians referring with high frequency and low frequency was significant with
a p-value of 0.019 ([Fig. 1]).
Fig. 1 Comparison of the distributions of physicians referring with high and low frequency
showed significant differences.
Abb. 1 Der Vergleich der Verteilungen zwischen häufig und weniger häufig zuweisenden Ärzten
zeigt signifikante Unterschiede.
Concerning the items of the questionnaire, [Table 2] shows that physicians referring with high frequency scored higher for all items
than physicians referring with low frequency. The largest differences between means
were observed for the questions about the handling of healthcare resources (0.4 point
difference), the clinical usefulness and information of reports and whether reports
reached referring physicians in a timely manner and the timeliness of getting appointments
differed with 0.3 points between physicians referring with high frequency and low
frequency. Also, on an item level, we found significant values of the one-sided Wilcoxon
test for eight items. Most of them concerned the radiological report, but also the
item about appointment scheduling and the question about the handling of healthcare
resources showed significant differences with a test result below 0.05.
Cronbach’s Alpha was 0.96 (95 % CI 0.95–0.97), indicating a high degree of internal
consistency of items in the survey.
Discussion
The present study aimed to develop a valid, consistent and reliable questionnaire
for assessing the quality of services provided by outpatient imaging services as rated
by referring physicians. The response rate of the questionnaire of almost one third
was similar to an earlier study conducted in Switzerland within the context of radiology
services [2]. The results revealed that referring physicians evaluate the quality of imaging
services to be overall high.
Descriptive statistics showed that especially factors such as timeliness, information
and understandability of reports, coordination of appointments as well as communication
with referring physicians’ assistants and collaborators achieved high scores. It cannot
be excluded that these ceiling effects could be partially due to selection bias since
participants were already participating in a project aiming at the development of
quality indicators.
On the other hand, as answers were anonymous, there was no pressure for referring
physicians to answer positively. Concerning the items that did not yield very high
results, there seems to be room for improvement with regard to caring for vulnerable
patients, who need special attention and support. Moreover, the quality of recommendations
regarding additional or future radiological examinations and thus the contribution
to continuous and sustainable care should be increased.
The validity of the presented questionnaire was assessed by testing for differences
between physicians referring with high frequency and low frequency. Research shows
that referring physicians decide for more or less frequent referrals with consideration
of the specialists’ medical skills [10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]. The tests for differences showed that the questionnaire reproduces these known
differences between physicians referring with a high frequency and low frequency on
the level of the total score of the instrument as well as on several items, even if
the difference proves to be small.
In accordance with other reports [11]
[12]
[13]
[14]
[15]
[16]
[17]
[18], the results of the present study revealed the great importance of communication
for referring physicians’ quality assessments and decisions for or against a certain
service. Communication relates not only to the radiological report but also to organizational
aspects. Considering Hackl et al. [19] who found that referrals within personal networks positively affect patient outcomes,
it seems important to mention that these assessments are probably not only mere subjective
‘opinions’ but indeed are reliable quality assessments. For radiology services as
well as for initiatives encompassing several services, such results can be of great
interest, when it comes to planning and implementing evidence-based quality projects.
To carefully design standards for radiological reports, to pay attention to the way
a services’ staff communicates with referring physicians and their collaborators,
to keep organizational aspects up-to-date and well running can be especially important
as soon as significant differences appear in comparison between services. The fact
that the question assessing the handling of healthcare resources was significantly
discriminative could be an indication of the referring physicians’ sensitivity to
this topic.
However, questions related to patient feedback and medical skills, which were also
deemed important, did not seem to yield the same discriminatory potential. For patient
feedback, these results could be due to the fact that, in contrast to other specialists,
outpatient radiology services offering mainly diagnostic services are often visited
by a patient only once. Feedback thus probably tends to be limited to negative experiences.
Still, the fact that a number of items concerning patient feedback, like patient information
on examinations, recommendations made in reports concerning additional or future exams
as well as the care of vulnerable patients did not score that high in the overall
sample should be taken seriously.
Concerning medical skills, the present results could confirm an observation previously
made by Grüber-Grätz et al. [14], namely, that if a referring physician does not find the technical and professional
skills of a radiology service to be of good quality, patients would not be referred
to this service. The fact that only registered referring physicians participated in
this study results in a selection bias at least to a certain degree.
To the best of our knowledge, so far there has not been a validated questionnaire
assessing the quality of outpatient radiology services as assessed by referring physicians.
Consisting of 18 questions, the final survey instrument is well suited to successfully
fulfill this task in a reasonable amount of time.
However, we are aware of the following limitations of the present study: first, the
number of participants was not high enough to evaluate the questionnaire’s potential
of discrimination on an item level. This would have provided interesting insight into
particularly important aspects regarding the quality of outpatient imaging services
as evaluated by referring physicians. Second, a certain selection bias cannot be denied.
Answers were only provided by referring physicians registered as such by the services
and not by referring physicians who may no longer refer to a service, e. g. because
of a negative quality assessment. Third, radiology services participating in the study
were already part of a larger project about quality and might thus introduce a further
positive bias. Furthermore, most of the referring physicians had more than 21 years
of professional experience. Even though no significant differences could be found
in the ratings between age groups, it might well be that younger referring physicians
have different expectations with regard to radiological services. Finally, the results
of this study only apply to outpatient radiology services. Future developments might
address the fragmentation of services and foster closer integration of radiology into
other diagnostic and treatment processes. Thus, quality assessment instruments might
need to be adapted accordingly.
The present questionnaire allows comprehensively evaluation of the quality of outpatient
radiology services as perceived by referring physicians. Furthermore, results can
be used as the basis for quality improvement on an organizational level, for comparing
various services as well as for orchestrating quality initiatives encompassing several
services.
