Referring Physicians Assess the Quality of Outpatient Diagnostic Imaging Services: Development and Psychometric Evaluation of a Questionnaire

Abstract

Goal In order to ensure high-quality cooperation between referring physicians and imaging services, it is important to assess the quality of imaging services as perceived by referring physicians. The present study aimed to develop and validate a questionnaire for referring physicians to assess the quality of outpatient diagnostic imaging services.

Materials & Methods The questionnaire was developed by discussing and modifying an existing instrument by the German Association of Surgeons. After qualitative pretesting, the instrument was tested with physicians referring to four outpatient diagnostic imaging services in Switzerland. The results were first assessed using descriptive statistics. The final instrument was tested for validity using the concept of known-groups validity. The hypothesis underlying this procedure was that physicians referring frequently to services estimated the quality of these services to be higher than physicians who referred less often to services. The differences in ratings were assessed using a one-sided two-sample Wilcoxon test. The final questionnaire was tested for internal consistency and reliability using Cronbach’s Alpha.

Results Results show a high level of satisfaction of referring physicians with the relevant services but also potential for quality improvement initiatives. The psychometric evaluation of the final questionnaire shows that it is a valid instrument, showing significant differences between the ratings of physicians referring with high and low frequency. Furthermore, the instrument proves to be consistent and reliable.

Conclusion The final instrument presents a valid, consistent and reliable option for assess the quality of outpatient diagnostic imaging services as perceived by referring physicians. Results can be used as a basis for quality improvement.

Key Points: 

• A newly developed questionnaire assesses the quality of outpatient diagnostic imaging services as perceived by referring physicians. The questionnaire was developed and tested in Switzerland.

• Psychometric evaluation showed the questionnaire to be a valid, consistent and reliable instrument.

• Results are of interest for imaging services as well as for initiatives encompassing several services.

Citation Format

• Jossen M, Valeri F, Heilmaier C et al. Referring Physicians Assess the Quality of Outpatient Diagnostic Imaging Services: Development and Psychometric Evaluation of a Questionnaire. Fortschr Röntgenstr 2019; 191: 725 – 731

Zusammenfassung

Ziel Um die Qualität der Kooperation zwischen Zuweisenden und Radiologie-Instituten zu verbessern, ist die Einschätzung der Qualität der erbrachten Leistung seitens der Zuweisenden essenziell. Die vorliegende Studie hat das Ziel, die Entwicklung und Validierung eines Fragebogens, mittels dessen Zuweisende die Qualität ambulanter Radiologie-Institute einschätzen, zu beschreiben.

Material & Methoden Der Fragebogen wurde auf der Grundlage eines bestehenden Instruments der Deutschen Gesellschaft für Chirurgie entwickelt, welches diskutiert und modifiziert wurde. Der neu entstandene Fragebogen wurde einem qualitativen Pre-Test unterzogen und anschließend bei Ärzten, die Patienten an ambulante Radiologie-Institute in der Schweiz zuweisen, erstmals eingesetzt. Die Resultate wurden mittels deskriptiver Statistik analysiert. Das finale Instrument wurde bezüglich seiner Validität mit dem “Known-Groups”-Konzept getestet. Diesem Verfahren unterliegt die Annahme, dass Ärzte, die häufig Patienten an ein Institut überweisen, mit diesem Institut eher zufrieden sind, als Ärzte, die selten Patienten an dieses Institut überweisen. Differenzen in der Bewertung wurden mittels eines einseitigen two-sample-Wilcoxon-Test gemessen. Das finale Instrument wurde mittels Cronbachs-Alpha bezüglich seiner internen Konsistenz und Reliabilität gemessen.

Resultate Die Resultate zeigen, dass die Zuweisenden generell sehr zufrieden sind mit der Arbeit der Radiologie-Institute, die Antworten weisen aber auch auf Verbesserungspotenzial hin. Die psychometrische Evaluation des finalen Instruments zeigt, dass dieses valide ist, da es signifikante Differenzen zwischen den Einschätzungen von häufiger und weniger häufig zuweisenden Ärzten zeigt. Zudem ist das finale Instrument konsistent und reliabel.

Schlussfolgerung Das finale Instrument ermöglicht eine valide, reliable und konsistente Überprüfung der Einschätzung der Qualität ambulanter Radiologie-Institute durch ihre Zuweisenden. Die Resultate können als Grundlage für Qualitätsverbesserungen genutzt werden.

Kernaussagen: 

• Ein neu entwickelter Fragebogen misst die Qualität ambulanter Radiologie-Institute aus der Sicht der Zuweisenden. Der Fragebogen wurde in der Schweiz entwickelt und pilotiert.

• Die psychometrische Evaluation zeigt, dass der Fragebogen valide, konsistent und reliabel ist.

• Die Resultate sind sowohl für Radiologie-Institute als auch für Initiativen, die über einzelne Institute hinausgehen, von Interesse.

Key words

radiation safety - safety - statistics - health policy and practice

Introduction

In times of highly fragmented healthcare services, cooperation between various providers in the sector is regarded as one key factor to ensure high quality of care [1]. Gathering reliable data on the quality of specialists’ services as perceived by referring physicians can yield important information making it possible to assess and improve services as well as cooperation on an organizational level and beyond.

Concerning imaging services, a number of studies and questionnaires have proposed the assessment of the satisfaction of referring physicians with and the opinion of the quality of imaging services. They evaluate quality in general [2], focus on certain imaging subspecialties [3] [4] [5] [6] or concentrate on reporting of results [7] [8] [9]. However, based on a literature search, validated questionnaires assessing the opinion of referring physicians regarding the quality of imaging services are lacking.

On the contrary, the question as to which aspects of quality are of special importance to referring physicians and determine the decision to choose a specialist provider have been intensively discussed. Not surprisingly, most research reveals that a referring physician’s perception of a specialist’s medical skills is an important criterion. In addition to that, previous positive experiences, patient feedback as well as communication with the specialist are very important elements for referring physicians [10] [11] [12] [13] [14] [15] [16] [17]. Communication includes talking or writing about organizational aspects such as the scheduling of appointments, and medical aspects such as specialists’ response by letter or phone. Institutional and medical quality appear to be tightly linked to each other. Positive experiences are the basis for sustainable relationships between referring physicians – and vice versa, with personal contacts providing an opportunity to ask medical questions [18]. This is also shown by Hackl et al. [19] who report that referrals within a doctor’s personal network are more appropriate in terms of patient outcomes than referrals outside the network, demonstrating that personal connections reduce information asymmetry with respect to the specialists’ abilities, meaning that referring physicians are better able to evaluate the specialists’ competences and their limits. In summary, measuring the satisfaction of referring physicians with an imaging service is crucial to improving the quality of care provided.

The aim of the present study was to develop and validate a questionnaire for referring physicians that measures their assessment of the quality of care provided by outpatient imaging services.

Materials and Methods

Questionnaire development

The designing of the questionnaire was part of a broader initiative. This initiative brought together personnel of several imaging services with quality experts. In a participatory project, standards for infrastructures, patient and referring physician management, teamwork and quality development were compiled. The initiative and the development of the questionnaire were organized by the EQUAM Foundation.

A survey instrument originally generated by the Professional Association of German Surgeons, which was unpublished but distributed to physicians referring to specialists in Germany and Switzerland by the aQua-Institut and the EQUAM Foundation was used as the basis for the development of the questionnaire. The original instrument was discussed and modified within a group of radiologists, radiographers, referring physicians and experts in order to draft a first version specifically addressing physicians referring to outpatient imaging services.

After that, the questionnaire underwent a qualitative pre-test [20] with two general practitioners and two specialists and was adapted accordingly. After a final discussion of the pre-test results with experts and referring physicians, a last modification of the questionnaire was executed. The instrument was then field-tested.

The questionnaire included 24 items organized in several sections. Four items were summarized under the topic of “professional knowledge and skills”. The satisfaction of referring physicians with the services’ contribution to integrated care as well as radiologists’ reports was represented by five items. Another seven items assessed the impressions of referring physicians with regard to the treatment of patients by radiology services and the final three items addressed the topic of “service”. All items assessing the quality of imaging services used a 5-point Likert response scale ranging from “strongly disagree with this statement” to “strongly agree with this statement”. Referring physicians were also asked to provide demographic data, namely their field of specialization, the number of years since their state examination, as well as the frequency with which they refer patients to imaging services.

Sample and Procedures

The questionnaire was field-tested as an online survey and invitations for participation were sent to 448 physicians who refer to four radiology outpatient imaging services in the German-speaking part of Switzerland. The referring physicians’ addresses were provided by the participating services. Recipients were asked to complete the questionnaire within two weeks. Participation was voluntary and anonymous. Referring physicians who had not answered within a two-week period received a reminder.

Statistical Methods

Descriptive statistics (means, distributions, missing answers) assessed the quality and distribution of the data. An individual mean score was calculated for each respondent by aggregating all item ratings. Total mean scores and their corresponding distributions were calculated as the overall mean scores and distributions of the individual mean scores.

We examined known-groups validity [21]. Based on the evidence presented above, we hypothesized that physicians referring with high frequency estimate the quality of radiology services to be higher than physicians referring with low frequency. Physicians referring with high frequency are obviously satisfied with the service provided when they have, as it is the case in Switzerland, free choice of specialists. At the same time, frequent referrals enhance the quality of imaging services by stabilizing communication and contacts [19]. The concept of known-groups validity expresses that a questionnaire claiming content validity should reproduce such well-established differences (for a similar procedure see [22] [23]).

Differences were analyzed using a one-sided two-sample Wilcoxon test (Mann-Whitney test) [24] [25]. P-values < 0.05 were regarded as statistically significant.

Even though the study was not designed to have the power to show differences at the item level, these were also assessed in order to gain insight about which quality criteria prove to be especially discriminative.

Internal consistency and reliability were measured with Cronbach’s Alpha [26]. This measure can be viewed as the expected correlation of two tests measuring the same construct, varying between 0 and 1. A value of > 0.7 was assumed to be sufficient. All analyses were performed with the Open Source Software R, Version 3.4.3 from 2017 [27].

Results

In total, 148 questionnaires were returned, resulting in a return rate of 33 % (148/448). Ten questionnaires were excluded since the respondents only filled in the demographic data and did not proceed with the questionnaire. Thus, the corrected return rate was 31 % (138/448). [Table 1] summarizes the characteristics of the study sample. Most of the referring physicians held a specialization in general internal medicine and had completed their state examination more than 21 years ago. 64 % (89/138) of the participants usually refer patients to the imaging services at least once a week, while 36 % (49/138) have a lower referral rate to radiology services ([Table 1]).

Table 1 Summary of the study sample’s characteristics (n = 138).
Tab. 1 Zusammenfassung der Charakteristiken des Samples (n = 138).

characteristic	n (%)
imaging service
imaging service 1	13 (9)
imaging service 2	38 (28)
imaging service 3	59 (43)
imaging service 4	28 (20)
specialization (multiple selections possible)
general internal medicine	100
gynecology/obstetrics	10
orthopedic surgery/traumatology of musculoskeletal system	5
otorhinolaryngology	5
rheumatology	5
pediatrics	4
gastroenterology	3
physical medicine/rehabilitation	3
anesthesiology	2
neurology	2
ophthalmology	2
pneumology	2
psychiatry and psychotherapy	2
other	7
years since completion of state examination (2 missing answers)
less than a year	0 (0)
1–5 years	1 (1)
6–10 years	5 (4)
11–20 years	38 (28)
more than 21 years	92 (67)
frequency of referral to imaging services
more than five times a week (high frequency)	13 (9)
2–5 times a week (high frequency)	56 (41)
once a week (high frequency)	20 (14)
1–3 times a month (low frequency)	33 (24)
1–3 times a quarter (low frequency)	12 (9)
less than once a quarter (low frequency)	4 (3)

Descriptive Statistics

Six items were deleted from the final survey instrument due to excessive missing answers and a lack of discriminatory potential. These were items asking about counselling for choosing examinations, taking over of the right amount of responsibility and the collaboration between radiologists and their colleagues. A further three reverse-coded items were deleted, concerning consent of patients to examinations, the handling of confidential data and patient’s rights, as they showed untypical distributions, pointing to a high frequency of confusion at the lower and higher end of the scale. The mean scores of these items ranged between 4.2 and 4.7 on a five-point Likert Scale.

The final survey instrument under evaluation thus consists of 18 items. In 92 % of the 138 questionnaires between zero and four answers were missing.

[Table 2] summarizes the results of the final instrument. Results show a high level of satisfaction with a total mean score over all items of 4.5 on a 5-point Likert scale. The timeliness of reports is ranked highest with a mean of 4.7. Moreover, professional expertise, communication with the referring physicians’ assistants and other collaborators, information and understandability of reports as well as the possibility to quickly obtain appointments for patients and the reachability of staff scored high with means of 4.6. On the other hand, referring physicians were not too satisfied with the handling of healthcare resources with this item scoring the lowest with a mean of 4.2. Likewise, patient information on examinations, recommendations made in reports concerning additional or future exams as well as care for vulnerable patients scored rather low with means from 4.3 to 4.4 ([Table 2]).

Table 2 Results (mean and standard deviation (SD)) of the 18 items that were included in the final instrument.
Tab. 2 Resultate (Mittelwerte und Standardabweichung (SD)) der 18 Items des finalen Instruments.

	total		physicians referring with low frequency (n = 49)		physicians referring with high frequency (n = 89)		exact p-value, Wilcoxon test
	mean	SD	mean	SD	mean	SD
total score	4.5	0.5	4.4	0.6	4.6	0.5	0.019
items
the service’s staff is professionally up-to-date	4.6	0.6	4.5	0.7	4.6	0.5	0.14
the service’s staff knows the limits of their competencies and possibilities	4.5	0.6	4.4	0.7	4.5	0.6	0.12
the service’s staff informs me if a request for referral exceeds their competencies	4.5	0.8	4.3	0.9	4.5	0.7	0.11
the service’s staff cooperates well for the care of patients with complex problems	4.5	0.7	4.3	0.8	4.5	0.6	0.13
the service’s staff handles resources for healthcare efficiently (e. g. elaborate diagnostic procedures)	4.2	0.8	3.9	1.0	4.3	0.7	0.03
I have the impression that the service’s staff communicates appropriately with my assistants and other collaborators	4.6	0.6	4.5	0.6	4.7	0.5	0.02
radiological reports contain the expected information	4.6	0.6	4.4	0.8	4.7	0.5	0.02
radiological reports are comprehensible and clear	4.6	0.6	4.5	0.6	4.7	0.6	0.02
radiological reports are clinically useful	4.5	0.8	4.3	1.0	4.6	0.6	0.03
radiological reports contain a clear answer to my question	4.5	0.8	4.3	0.9	4.6	0.7	0.02
radiological reports contain recommendations based on actual evidence for further radiological exams	4.4	0.8	4.3	0.8	4.4	0.8	0.07
I have the impression that the service’s staff provides my patients with the necessary information about the imaging exam	4.3	0.7	4.2	0.7	4.4	0.7	0.07
I have the impression that the service’s staff carefully questions my patients (e. g. about allergies)	4.4	0.7	4.3	0.6	4.5	0.7	0.06
I have the impression that the service’s staff treats my patients with understanding and empathy	4.5	0.7	4.4	0.7	4.5	0.7	0.17
I have the impression that very vulnerable patients are also treated well	4.4	0.7	4.3	0.8	4.4	0.7	0.17
the service’s staff can easily be reached	4.6	0.7	4.5	0.7	4.6	0.6	0.13
I get appointments for my patients in due time	4.6	0.7	4.4	0.8	4.7	0.6	0.007
I receive reports in due time	4.7	0.6	4.5	0.8	4.8	0.5	0.003

Validity, Internal Consistency and Reliability

The mean rankings of physicians referring with high frequency were higher than those of physicians referring with low frequency with total score means of 4.6 compared to 4.4, respectively. The one-sided Wilcoxon test for differences between rank distributions of physicians referring with high frequency and low frequency was significant with a p-value of 0.019 ([Fig. 1]).

Fig. 1  Comparison of the distributions of physicians referring with high and low frequency showed significant differences.

Abb. 1 Der Vergleich der Verteilungen zwischen häufig und weniger häufig zuweisenden Ärzten zeigt signifikante Unterschiede.

Concerning the items of the questionnaire, [Table 2] shows that physicians referring with high frequency scored higher for all items than physicians referring with low frequency. The largest differences between means were observed for the questions about the handling of healthcare resources (0.4 point difference), the clinical usefulness and information of reports and whether reports reached referring physicians in a timely manner and the timeliness of getting appointments differed with 0.3 points between physicians referring with high frequency and low frequency. Also, on an item level, we found significant values of the one-sided Wilcoxon test for eight items. Most of them concerned the radiological report, but also the item about appointment scheduling and the question about the handling of healthcare resources showed significant differences with a test result below 0.05.

Cronbach’s Alpha was 0.96 (95 % CI 0.95–0.97), indicating a high degree of internal consistency of items in the survey.

Discussion

The present study aimed to develop a valid, consistent and reliable questionnaire for assessing the quality of services provided by outpatient imaging services as rated by referring physicians. The response rate of the questionnaire of almost one third was similar to an earlier study conducted in Switzerland within the context of radiology services [2]. The results revealed that referring physicians evaluate the quality of imaging services to be overall high.

Descriptive statistics showed that especially factors such as timeliness, information and understandability of reports, coordination of appointments as well as communication with referring physicians’ assistants and collaborators achieved high scores. It cannot be excluded that these ceiling effects could be partially due to selection bias since participants were already participating in a project aiming at the development of quality indicators.

On the other hand, as answers were anonymous, there was no pressure for referring physicians to answer positively. Concerning the items that did not yield very high results, there seems to be room for improvement with regard to caring for vulnerable patients, who need special attention and support. Moreover, the quality of recommendations regarding additional or future radiological examinations and thus the contribution to continuous and sustainable care should be increased.

The validity of the presented questionnaire was assessed by testing for differences between physicians referring with high frequency and low frequency. Research shows that referring physicians decide for more or less frequent referrals with consideration of the specialists’ medical skills [10] [11] [12] [13] [14] [15] [16] [17]. The tests for differences showed that the questionnaire reproduces these known differences between physicians referring with a high frequency and low frequency on the level of the total score of the instrument as well as on several items, even if the difference proves to be small.

In accordance with other reports [11] [12] [13] [14] [15] [16] [17] [18], the results of the present study revealed the great importance of communication for referring physicians’ quality assessments and decisions for or against a certain service. Communication relates not only to the radiological report but also to organizational aspects. Considering Hackl et al. [19] who found that referrals within personal networks positively affect patient outcomes, it seems important to mention that these assessments are probably not only mere subjective ‘opinions’ but indeed are reliable quality assessments. For radiology services as well as for initiatives encompassing several services, such results can be of great interest, when it comes to planning and implementing evidence-based quality projects. To carefully design standards for radiological reports, to pay attention to the way a services’ staff communicates with referring physicians and their collaborators, to keep organizational aspects up-to-date and well running can be especially important as soon as significant differences appear in comparison between services. The fact that the question assessing the handling of healthcare resources was significantly discriminative could be an indication of the referring physicians’ sensitivity to this topic.

However, questions related to patient feedback and medical skills, which were also deemed important, did not seem to yield the same discriminatory potential. For patient feedback, these results could be due to the fact that, in contrast to other specialists, outpatient radiology services offering mainly diagnostic services are often visited by a patient only once. Feedback thus probably tends to be limited to negative experiences. Still, the fact that a number of items concerning patient feedback, like patient information on examinations, recommendations made in reports concerning additional or future exams as well as the care of vulnerable patients did not score that high in the overall sample should be taken seriously.

Concerning medical skills, the present results could confirm an observation previously made by Grüber-Grätz et al. [14], namely, that if a referring physician does not find the technical and professional skills of a radiology service to be of good quality, patients would not be referred to this service. The fact that only registered referring physicians participated in this study results in a selection bias at least to a certain degree.

To the best of our knowledge, so far there has not been a validated questionnaire assessing the quality of outpatient radiology services as assessed by referring physicians. Consisting of 18 questions, the final survey instrument is well suited to successfully fulfill this task in a reasonable amount of time.

However, we are aware of the following limitations of the present study: first, the number of participants was not high enough to evaluate the questionnaire’s potential of discrimination on an item level. This would have provided interesting insight into particularly important aspects regarding the quality of outpatient imaging services as evaluated by referring physicians. Second, a certain selection bias cannot be denied. Answers were only provided by referring physicians registered as such by the services and not by referring physicians who may no longer refer to a service, e. g. because of a negative quality assessment. Third, radiology services participating in the study were already part of a larger project about quality and might thus introduce a further positive bias. Furthermore, most of the referring physicians had more than 21 years of professional experience. Even though no significant differences could be found in the ratings between age groups, it might well be that younger referring physicians have different expectations with regard to radiological services. Finally, the results of this study only apply to outpatient radiology services. Future developments might address the fragmentation of services and foster closer integration of radiology into other diagnostic and treatment processes. Thus, quality assessment instruments might need to be adapted accordingly.

The present questionnaire allows comprehensively evaluation of the quality of outpatient radiology services as perceived by referring physicians. Furthermore, results can be used as the basis for quality improvement on an organizational level, for comparing various services as well as for orchestrating quality initiatives encompassing several services.

Conflict of Interest

Marianne Jossen was a member of the EQUAM Foundation from 2014 – 2017.

• References

• 1 Kvamme OJ, Olesen F, Samuelsson M. Improving the interface between primary and Secondary Care: A Statement from the European Party on Quality in Family Practice (EQuiP). Qual Health Care 2001; 10: 33-39
• 2 Kubik-Huch R, Rexroth M, Porst R. et al. How satisfied are clinicians with radiological institutions? Development and testing of a questionnaire. RöFo 2005; 177: 119-123
• 3 Lachter J, Feldman R, Krief I. et al. Satisfaction of the referring physician. A quality control study focusing on EUS. J Clin Gastroenterol 2007; 41: 889-893
• 4 Vicente GAM, Castrejon SA, Delagado MC. et al. Study of physician satisfaction as a quality criterion in nuclear medicine. Rev Esp Med Nucl Imagen Mol 2008; 27: 22-28
• 5 McMenamy J, Rosenkrantz AB, Jacobs J. et al. Use of a referring physician survey to direct and evaluate department-wide radiology quality improvement efforts. J Am Coll Radiol 2015; 12: 1223-1225
• 6 Maurer MH, Zimmermann E, Hamm B. et al. CT coronary angiography versus conventional invasive coronary angiography – the view of the referring physician. Röfo 2014; 186: 1102-1110
• 7 Clinger NJ, Hunter TB, Hillman BJ. Radiology reporting: Attitudes of reffering physicians. Radiology 1988; 169: 825-826
• 8 Grieve FM, Plumb AA, Khan SH. Radiology reporting: A general practitioner’s perspective. Br J Radiol 2010; 83: 17-22
• 9 Bosmans JML, Joost JW, Arthur MS. et al. The radiology report as seen by radiologists and referring clinicians: Results of the COVER and ROVER surveys. Radiology 2011; 259: 184-195
• 10 Javalgi R, Joseph WB, Gombeski WR. et al. How physicians make referrals. J Health Care Mark Summer 1993; 6-17
• 11 Forrest CB, Nutting PA, Starfield B. et al. Family physicians’ referral decisions: Results from the ASON referral study. J Fam Pract 2002; 51: 215-223
• 12 Kinchen KS, Cooper LA, Levine D. et al. Referral of patients to specialists: Factors affecting choice of specialists by primary care physicians. Ann Fam Med 2004; 2: 145-252
• 13 Rosen R, Florin D, Hutt R. An anatomy of GP referral decisons. A qualitative study of GP’s views on their role in supporting patient choice. King’s Fund, London: 2007 Available at: www.kingsfund.org.uk/publications . Accessed 15 March 2018
• 14 Gröber-Grätz D, Mossshammer D, Bölter R. et al. Which criteria affect general practitioners when referring patients to specialists in out-patient care? A qualitative study on general practitioners’ opinion. Z Evid Fortbild Qual Gesundhwesen 2011; 105: 446-451
• 15 Barnett ML, Keating NL, Christakis NA. et al. Reasons for choice of referral physician among primary care and specialist physicians. J Gen Intern Med 2011; 27: 506-512
• 16 Ikkersheim D, Koolman X. The use of quality information by general practitioners: Does it alter choices? A randomnized clustered study. BMC Fam Pract 2013; 14: 95
• 17 Hermeling P, de Cruppé W, Geraedts M. Need for information of out-patient physicians for referrals to hospitals and specialists. Gesundheitswesen 2013; 75: 448-455
• 18 Ketelaar NABM, Faber MJ, Elwyn G. et al. Comparative performance information plays no role in the referral behaviour of GP’s. BMC Fam Pract 2014; 15: 146
• 19 Hackl F, Hummer M, Pruckner GJ. Old boy’s network in general practitioners’ referral behavior?. Journal of Health Economics 2015; 43: 56-73
• 20 Häder M. Empiric social research – An introduction. Wiesbaden: Verlag für Sozialforschung; 2006
• 21 Davidson M. Known-groups validity. In: Michalos AC. Encyclopedia of quality of life and well-being research. Dodrecht: Springer; 2014
• 22 Weiner BJ, Lewis CC, Stanick C. et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12: 108
• 23 Richard A, Pfeiffer Y, Schwappach D. Development and Psychometric Evaluation of the Speaking Up About Patient Safety Questionnaire. J Patient Saf 2017; DOI: 10.1097/PTS.0000000000000415.
• 24 Bauer DF. Constructing confidence sets using rank statistics. J Am Stat Assoc 1972; 67: 687-690
• 25 Hollander M, Douglas AW. Nonparametric statistical methods. New York: John Wiley & Sons; 1973
• 26 Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951; 16: 297-334
• 27 R Development Core Team R: A language and environment for statistical. 2008