Transfer of a telemedicine intervention for mental disorders: a comparison between RCT results and regional routine care

Abstract

Introduction

Telemedicine for the treatment of depression and anxiety disorders was found to be successful in a randomised controlled trial (RCT); this intervention was then implemented in routine care in the Western Pomerania region in Germany. This made it possible to investigate the effectiveness of the intervention under routine care conditions and compare it with the results of the RCT.

Methods

For this retrospective controlled analysis, data from routine care (2011–2022) were analysed together with data from the previous RCT (2009–2010). A three-arm comparison (routine care, previous RCT intervention group, previous RCT control group) on the primary outcome of symptom severity (BSI-18) and a longitudinal analysis of the routine care data were conducted. The telemedical intervention was conducted in the university hospital’s psychiatric outpatient clinic in north-eastern Germany. All adult patients with an ICD-10 diagnosis of depression, anxiety or somatoform disorders could participate after discharge from the hospital. The telemedicine sessions included structured verbal questionnaires and conversational therapy concerning treatment goals and tasks. Repeated measures Welch ANOVA with the BSI-18 Global Severity Index and subscales (depression, anxiety and somatisation) was performed. A multivariate regression was conducted on the longitudinal regular care data.

Results

The n=254 subjects in the telemedical care in routine care arm (181 women, mean [95%CI] age 45.5 [44.0–47.1] years; 6-month follow-up) showed a BSI-18 score improvement M=− 4.1 [−5.3,−2.9], F(2)=3.50, p<0.05 compared to the preceding RCT intervention arms (61 women, mean [95%CI] age 44.7 [41.7–47.6] years. Telemedical care showed a significant difference in BSI-18 scores over a 10-year follow-up: M=− 3.9 [−5.4,−2.5], p<0.0001.

Conclusion

The positive results of the 2009 RCT were replicable in routine care. The more patient-centred approach resulted in improved outcomes in this telemedical intervention.

Keywords

Telemedicine - Routine care - Depression - Anxiety - Somatoform disorders

Introduction

Telemedicine has become a serious treatment option in mental healthcare over the past 20 years[1] [2] [3]. Especially during the COVID-19 pandemic, it turned out to be a lifeline to mental healthcare services for many patients[4] [5]. Telemedicine makes it possible to offer patients diagnostic, treatment, and consultation services using audio-visual communication technologies despite spatial separation[6]. Telemedicine works especially well in rural regions, where it takes patients longer to reach service providers[7] [8]. In rural regions such as Western Pomerania in Germany, waiting times for mental healthcare treatment are over 20 weeks, compared to 16 weeks in urban areas[9].

Long waiting lists after discharge from inpatient clinics impede continuous therapy. Telemedicine can bridge the waiting time gap for patients and thus help prevent relapses. Providers and patients have accepted telemedicine as a low-threshold means of care that supports timely identification of crises and needed interventions[1] [10]. The effectiveness of telemedical interventions was shown in multiple randomised controlled trials (RCTs), but the long-term effects after implementation in regular healthcare have not been extensively monitored[1] [2] [11].

Between 2009 and 2010, van den Berg et al.[11] [12] conducted an RCT where telemedicine (telephone calls and text messages) bridged the waiting period after discharge from a psychiatric day clinic for patients with anxiety, depressive, and somatoform disorders[8]. The intervention showed a reduction in psychopathological symptoms measured by the Brief Symptom Inventory-18 (BSI-18;−2,4 anxiety scale,−1.73 depression scale). A sensitivity analysis revealed floor effects: excluding the participants with the lowest BSI-18 scores led to a larger reduction in symptom scoring [11] The intervention was well received by both participants and staff. Because of its positive results and acceptance, the telemedical intervention was transferred into routine care in 2010 in the study region (Western Pomerania in north-eastern Germany).

The continued documentation of psychopathological outcomes from the telemedical intervention allowed for an evaluation of the intervention’s effectiveness using real-world evidence. Comparisons between RCTs and real-world observational studies aim to confirm an evidence-based intervention’s effectiveness in a real-world situation[13]. By minimising confounders, RCTs provide a controlled environment that supports the internal validity of the data, thus giving an intervention the best chance for success[14]. The health care situation in routine care seldom meets patients and diseases that match the ideal conditions of a RCT. A comparison between RCTs and data gathered from treatments based on RCT interventions in routine healthcare services has, to the best of our knowledge, not been done before.

The current evaluation aimed to identify whether the telemedical intervention was effective outside of controlled settings.

Methods

Design

This was a nonrandomised retrospective controlled analysis. To analyse the telemedical intervention in routine care, we performed a 3-armed retrospective analysis. One study arm included patients receiving telemedical care in routine care (data assessed in 2011–2022; 6-month observation period). The second study arm included the combined intervention arms (intervention group 1: telephone calls; intervention group 2: telephone calls and text messages) of the RCT from van den Berg et al.[11]. The third study arm was the control group (6-month observation period; data assessed in 2009–2010) of the RCT from van den Berg et al.[11]. Patients were individually assigned and no blinding took place. The RCT from van den Berg et al.[11] will be furthermore referred to as the “preceding RCT”.

To assess the further effectiveness of telemedical care in routine care, a retrospective longitudinal analysis of patient outcomes over the entire treatment time (assessed in 2011–2022; observation period up to 10 years) was performed.

Telemedical treatment procedure

Both in the preceding RCT and for the telemedical care patients, telemedicine services were conducted by trained nurses in delegation for the outpatient psychiatric clinic following discharge from associated psychiatric day clinics. A structured handover protocol with therapy goals and tasks for the patients during the telemedicine sessions was coordinated between the patient and psychotherapist prior to discharge from the day clinic. The baseline individual telephone session consisted of two parts: 1) a structured verbal questionnaire to assess sociodemographic data, psychopathological outcomes and health services use; and 2) conversational therapy, where the telemedicine nurses followed the plan of the handover protocol. In the weekly (month 1) and monthly (months 2 to 6) follow-up sessions of the RCT conversational therapy was conducted. Psychopathological outcomes were collected in month 6. The difference for the telemedical care in routine care arm was that the frequency of the telemedical sessions was defined patient-individually, and psychopathological outcomes were assessed through a telephone interview quarterly. Additionally, text messages were used by both nurses and patients to stay in touch. An overview of the different treatment and data collection steps is presented in [Fig. 1].

Fig. 1 Description of the treatment and data collection protocol for the patients from the telemedical care in routine care and the patients from the preceding RCT groups. Created with Lucid lucid.co.

Patient sample

All adult patients with an International Classification of Diseases-10 (ICD-10 [15]) diagnosis in the broad spectrum of depression and anxiety, who leave cooperating day clinics of the region Western Pomerania, could be referred to the outpatient clinic and telemedicine centre. In addition to the telemedical treatment, most patients only had occasional on-site appointments in the outpatient clinic with the psychiatrist or psychotherapist, e. g., for the prescription of drugs.

Outcome Measures

Personal and socioeconomic data of the patients were collected during the baseline session. The patients’ relevant medical information was transmitted from the inpatient psychiatrist or psychotherapist with the referral.

The German version of the Brief Symptom Inventory-18 (BSI-18) was used as a primary outcome measure to assess clinical outcomes. This is a commonly used, validated questionnaire in psychiatric treatment settings[16] [17]. The BSI-18 assesses the clinical severity of depressive symptoms, anxiety and somatization by applying a 5-point Likert scale, ranging from not at all (0) to extremely (4). The BSI-18 is divided into three 6-item subscales for depression, anxiety and somatization. Each scale has a maximum score of 24 points. The overall score of the BSI-18 is used to indicate the clinical relevance of the scores and ranges from 0 to 72, where higher scores indicate more symptoms[17] [18].

Data management

Data were collected using eCRFS and are documented with the IT-supported MedicalNformationCApture (MINCA) system[19]. The data are saved in a central database according to the current standards for data security and data privacy[20], which are documented in the data protection concept of the University Medicine Greifswald.

Data analysis

The ICD-10 codes were clustered into disease categories (depressive, anxiety, psychosomatic, and personality disorders, dependencies, schizophrenia; see online Supplementary Material 1). Symptom severity groups were created based on the severity level of the BSI-18 score at the start and the last reported BSI-18 score. A cut-off value of 12.1 for the BISI-18 score was used (two standard deviations above the mean in a normal German population[17]); if the BSI-18 score was above 12.1, it indicated that the symptom severity was clinically relevant[16].

Three-arm comparison of effectiveness

The routine care patient data were made comparable with the 2009 RCT data. Since the RCT trial observation period was 6 months, data were analysed from routine care patients for 180 days from baseline. To account for individualised appointment planning, a 20-day buffer was added, making the total timeframe 200 days. Patients who participated in the preceding RCT were excluded from the telemedical care in routine care arm.

For all arms patient characteristics were descriptively analysed and checked for statistical assumptions. Due to violations of normality, Welch’s analysis of variance (ANOVA) was used – as it is robust in situations of unequal variances and unequal group sizes [21] – to identify any significant differences in BSI-18 total and subscale scores between the arms. A chi-square test was computed to identify changes in the occurrence of clinically severe BSI-18 scores between start and finish.

Retrospective longitudinal analysis of telemedical care in routine care (up to 10 years)

Using the actual length of treatment for the patients with telemedical care in routine care, descriptive statistics were calculated for the mean BSI-18 score and its subscales for the first and last data assessment and their difference. Statistical assumptions for the sample were checked. Due to nonnormality, nonparametric comparisons of means tests and significance testing for the mean change were performed. A chi-square test on severity scores was conducted. Regression analyses were conducted to identify demographic, diagnostic, symptom-based and treatment-based influences on the end point score of the BSI-18. A sensitivity analysis excluding patients with a BSI-18 score below 12.1 was performed to explore flooring effects.

All analyses were conducted using SAS Software, Version 9.4 of the SAS System for Windows.

Ethical approval was granted by the institutional ethics committee of the board of physicians of Mecklenburg - Western Pomerania at the University of Greifswald (reg. No. BB50/09).

Results

A flow chart with patient allocation is reported in [Fig. 2].

Fig. 2 Patient allocation to treatment and comparator groups.

Three-arm comparison of effectiveness

No significant differences in patient characteristics were found between the three study arms ([Table 1]). The average BSI-18 starting score was lower for the RCT control arm (M=15.2, 95% CI [11.1, 19.4]) than for the RCT intervention arm (M=20.4, 95% CI [18.0, 22.8]) and the telemedical care in routine care arm (6 months) (M=19.6, 95% CI [18.2, 21.0]). Conversely, the control arm had fewer patients with clinically relevant BSI scores at start (51.2%) compared to the telemedical care routine care arm (6 months; 56.6%) and the RCT intervention arm (58%). Subscale and end-point scores can be found in online Supplementary Material 2.

Table 1 Patient characteristics of the participants of the three-arm comparison and of the telemedical care in routine care (up to 10-years).

	Telemedical care in routine care (data collected between 2011–2022)		Preceding RCT intervention (data collected 2009–2010)		Preceding RCT control (data collected 2009–2010)		Comparison between groups
	n=254		n=82		n=41
Variable	n	% / Mean [95% CI]	n	% / Mean [95% CI]	n	% /Mean [95% CI]	p
Women	181	71.3%	61	74.4%	27	65.9%	0.61†
Age at baseline in years		45.5 [44.0–47.1]		44.7 [41.7–47.6]		42.5[39.1–45.9]	0.34†
Number of telemedicine contacts		9.7 [8.9–10.5]		7.4 [6.8–8.0]
Diagnoses groupsa
Depression	222	87.4%	74	90%	37	90%	0.72i
Anxiety	116	45.7%	34	41%	14	34%	0.35i
Somatoform	45	17%	11	13%	3	7%	0.19i
Personality disorder	51	20%	15	18%	4	9%	0.29i
Schizoaffective disorders	9	3%	0	0%	0	0%	0.15^
Substance Dependencies	30	11%	6	7%	1	2%	0.12i
Clinically relevant BSI-18 score at start	254		82		41
Yes	180	70.9%	57	70%	21	51%
No	74	29.1%	25	30%	20	49%
Clinically relevant BSI-18 score at 6-month follow-up /*end	182 /*203		78		35
Yes	103/*109	56.6% /*53.7%	45	58%	19	54%
No	79/*94	43.4%/*46.3%	33	42%	16	46%

Note: ^a Patients can have multiple diagnoses groups (see Supplementary Material 1)

†ANOVA. ⁱChi-square. ^Fisher’s exact test.

A significant group difference was found in the mean difference score between the start and 6-month follow-up F(2)=3.50, p<0.05. Based on the mean difference scores reported in [Table 2], the results of the ANOVA could indicate that the telemedical care in routine care arm (6 months) had the largest improvement.

Table 2 Descriptive statistics on the BSI-18 mean differences for the telemedical care in routine care arm (6-months), preceding RCT intervention arm and preceding RCT control arm.

	Telemedical care in routine care 6-months (data collected between 2011–2022)			Preceding RCT intervention (data collected 2009–2010)			Preceding RCT control (data collected 2009–2010)
	n=182			n=78			n=35
Variable	M [95%CI]	MDN(IQR)	Mode	M [95% CI]	MDN(IQR)	Mode	M [95% CI]	MDN(IQR)	Mode
BSI-18 difference	−4.1 [−5.3,−2.9]	−4.0 (10.0)	−5.0	−3.0[−5.3,−0.7]	−3.5(52.0)	−4.0	0.9[−2.7, 4.6]	0.0 (9.0)	0.0
Depression	−1.9 [−2.6,−1.2]	−1.0 (4.0)	0.0	1.6 [0.5, 2.8]	1.0 (24.0)	0.0	−0.5[−1.8, 0.9]	0.0 (4.0)	0.0
Anxiety	−1.5 [-2.1,−1.0]	−1.0 (5.0)	−1.0	1.3 [0.3, 2.2]	1.0 (20.0)	0.0	−0.8[−2.5, 0.9]	0.0 (5.0)	0.0
Somatoform	−0.6 [−1.1,−0.1]	0.0 (3.0)	0.0	0.1 [−0.8, 1.0]	0.0 (19.0)	0.0	0.3 [−1.1, 1.8]	0.0 (3.0)	0.0

Using McNemar’s test, the RCT-intervention arm χ(1)=3.86, p<0.05 and the telemedical care in routine care arm (6 months) showed significant differences in severity occurrence χ(1)=23.1, p<0.0001. Fewer people in the telemedical care in routine care arm (6 months) and the RCT intervention arm had a clinically relevant BSI-18 diagnosis after 6 months than was statistically expected.

Retrospective longitudinal analysis of telemedical care in routine care (up to 10 years)

[Table 1] shows patient characteristics for telemedical care in routine care. A signed rank test showed significant score improvements on the total BSI-18 and its subscales between the first and last measurements (see [Table 3]). This result was confirmed using McNemar’s test χ (1)=23.3, p<0.0001: fewer participants had a clinically relevant diagnosis at the endpoint (see [Table 1]).

Table 3 Descriptive statistics, signed rank test results, multivariable regression model for telemedical care in routine care (10 years).

Variable	n	M [95% CI) /%	MDN (IQR)	Mode	S	p
BSI-18 difference	203	−3.9 [−5.4,−2.5]	−3.0 (12.0)	−6.0	−4157	<0.0001***
Depression		−1.9 [−2.7,−1.2]	−2.0 (6.0)	0.0	−3287	<0.0001***
Anxiety		−1.4 [−2.1,−0.8]	−1.0 (5.0)	−2.0	−3055.5	<0.0001***
Somatoform		−0.6 [−1.1,−0.1]	0.0 (3.0)	0.0	−1480	0.0124
	Adjusted R2		B	SE B	Β	p
Model of best fit	0.41		12190	1741		<0.0001**
Constant					−4.2	0.15
BSI-18 score at start					0.59	<0.0001**
Gender					0.31	0.83
Age					0.08	0.15
Number of diagnoses					2.36	<0.0001**
Clinically relevant symptoms					−1.00	0.63
Number of telemedicine contacts					−0.01	0.99
Length of treatment					0.01	0.69

Multiple linear regression using the backwards elimination method showed that two factors influenced the BSI-18 score at the end-point. The model revealed that the BSI-18 score at the end is negatively influenced by the BSI-18 score at the start and for patients with multiple psychological diagnoses (see [Table 3]). The model has moderate explanatory power and accounts for 41% of the variance.

The effects of treatment start with not clinically relevant symptoms were investigated by excluding the patients who scored a BSI-18 score at start of less than 12.1. This showed an improvement in the difference scores between start and finish, with the mean difference in the overall score: M=− 5.6, t(151)=− 6.27, p<0.0001.

Discussion

Significant differences between the telemedical care in routine care patient arm (6 months) and the in 2009 collected RCT arm were found. Telemedical care in routine care arm (6 months) had the largest average improvement on the BSI-18. This could be explained by the needs-based scheduling of appointments during routine care or by the slightly larger average number of telemedicine contacts compared to contacts during the RCT. More personalised care can have a positive effect on patient wellbeing[22]. Similar positive effects driven by personalised treatment were found to contribute to the usefulness of telemedical interventions[10]. Our results showed that in a routine care setting, telemedicine was more effective as a treatment method than it was in the rigid context of an RCT.

The telemedical care in routine care (6 months) results showed that telemedicine gave patients an improved score on the BSI-18 at the end-point. This positive result was supported by the difference in the number of participants with clinically relevant symptoms at the beginning and the end (−15%). These results indicated that the programme continued to be effective in treating symptoms of depression, anxiety and somatoform disorders. The results of the RCT by van den Berg et al. [11] indicated that a significant improvement was found only on the anxiety scale. The adaptation of the programme to a real-world setting led to significant improvements on all subscales. Similarly, floor effects were identified. After removal of the bottom scores, the average improvement on the BSI-18 scale increased. Rating scales such as the BSI-18 capture a patient’s health state or symptoms in a single moment. They generalise a patient’s mood, emotions and thoughts in a way that could be utilised for screening and monitoring[23] [24]. Based on this, only general inferences on treatment results could be made. It is possible that by using rating scales, the effects of standardised patient-centred telemedicine contacts that lie outside of symptom rating were missed. Patients may feel better because regular contact provides consistency and comfort. Regardless, the similarities between the RCT results and the outcomes from telemedical care in routine care arm supported that the intervention was successfully transformed into regional routine care services.

In telemedical care in routine care (up to 10 years), multiple regression analysis revealed that a high BSI-18 at the beginning negatively influenced the score at the end. In other words, users starting with more symptoms did not improve as much as users with fewer symptoms. Comorbidities similarly affected the end score. It would be advised to offer more intensive therapeutic support to patients with these characteristics. This was underlined by a change in utilisation of telemedicine. Initially, because of the prevailing length of waiting times, the intervention was conceptualised to support the transition between inpatient and outpatient treatment. The identified average participation in telemedical care in routine care of more than two years is longer than the average waiting time of 20 weeks for an outpatient appointment[9]. The role of telemedicine expanded from a ‘bridging-the-waiting-time-gap’ solution to a supporting service next to or after other therapies. As a consequence of the real healthcare situation and the associated non-standardised data assessment, no reliable data were available on the utilisation of further healthcare services. However, the on-site psychiatrists and psychotherapists indicated that most patients receiving telemedicine have a chronic psychiatric disorder and receive appropriate healthcare, including both telemedicine and occasional face-to-face appointments, but no formal face-to-face therapy. The lack of data transparency for previous and current therapies needs to be addressed, as these treatments might influence the effectiveness of the telemedicine intervention.

A further limitation was that no data for a control group was collected simultaneously. As this was a retrospective analysis of data from a noncontrolled intervention in a real-world healthcare setting, the changes in the BSI-18 score could not be fully attributed to telemedicine. However, since the original data from the RCT trial were available, these original data were used to improve explanatory power. Additionally, the RCT control arm had lower BSI-18 scores at the start than the other arms and could therefore not improve as much as the telemedical RCT intervention and the telemedical care in routine care arm, which started with higher scores and thus with worse clinical situations. This could have caused larger group differences between the treated and nontreated patients.

To summarise, the results of a RCT of a telemedical intervention in mental healthcare were successfully translated into a real-world healthcare setting. This evaluation of telemedical care in routine care used patient data collected over 10 years. These data indicated that telemedical treatment was an effective method to support patients with mental disorders in their transition from inpatient to outpatient services and potentially thereafter. The long-term follow-up of patients showed trends in the use of telemedicine that have not been extensively discussed in previous literature. Namely, telemedicine is used longer than would be expected if it was solely employed as a bridge to the waiting time gap solution. Our findings are generalisable for a broad group of depression and anxiety patients, as it took place in a normal healthcare setting.

Future research could employ time-varying models of symptom development to pinpoint the optimal length of telemedical treatment. A broader scope should analyse the synergies between telemedicine and normal treatment in rural areas such as Western Pomerania in Germany.

This article is part of the DNVF supplement “Health Care Research and Implementation”

Conflict of Interest

Hans Jörgen Grabe has received travel grants and speaker’s honoraria from Fresenius Medical Care, Neuraxpharm, Servier and Janssen Cilag as well as research funding from Fresenius Medical Care. Wolfgang Hoffmann is vice-editor of this supplement, but is not involved in the editing or processing of this submission. The other authors have no other conflicts of interest to declare.

Acknowledgements

Die Autoren möchten den Krankenschwestern des Telemedizinischen Zentrums für ihre fortwährende Arbeit danken. Außerdem möchten die Autoren Niklas Weber für die Erstellung eines Gruppierungsrahmens für die ICD-10-Codes danken.

• References

• 1 Guaiana G, Mastrangelo J, Hendrikx S. et al. A Systematic Review of the Use of Telepsychiatry in Depression. Community Mental Health Journal 2021; 57: 93-100
• 2 Lawes-Wickwar S, McBain H, Mulligan K. Application and Effectiveness of Telehealth to Support Severe Mental Illness Management: Systematic Review. JMIR Ment Health 2018; 5: e62
• 3 van den Berg N, Schumann M, Kraft K. et al. Telemedicine and telecare for older patients – a systematic review. Maturitas 2012; 73: 94-114
• 4 Colle R, Ait Tayeb AEK, Larminat Dde. et al. Telepsychiatry in the Post-COVID-19 Era: Moving Backwards or Forwards. Psychother Psychosom 2021; 90: 69-70
• 5 Lutz W, Edelbluth S, Deisenhofer A-K. et al. The Impact of Switching from Face-to-Face to Remote Psychological Therapy during the COVID-19 Pandemic. Psychother Psychosom 2021; 90: 285-286
• 6 German federal ministry for health. Telemedizin [Telemedicine]. Im Internet: https://www.bundesgesundheitsministerium.de/service/begriffe-von-a-z/t/telemedizin.html; Stand: 17.02.2023
• 7 Weinzimmer LG, Dalstrom MD, Klein CJ. et al. The relationship between access to mental health counseling and interest in rural telehealth. Journal of Rural Mental Health 2021; 45: 219-228
• 8 van den Berg N, Stentzel U, Hoffmann W. et al. Entlassmanagement in der Psychiatrie unter Einsatz telemedizinischer Leistungen [Discharge management in psychiatry using telemedical services]. 2016
• 9 Federal Chamber of Psychotherapists in Germany. Ein Jahr nach der Reform der Psychotherapie-Richtlinie [One year after the reform of the psychotherapy guidelines]. Wartezeiten 2018 [Waiting times 2018]; 2018
• 10 Patel S, Akhtar A, Malins S. et al. The acceptability and usability of digital health interventions for adults with depression, anxiety, and somatoform disorders: qualitative systematic review and meta-synthesis. Journal of medical Internet research 2020; 22: e16228
• 11 van den Berg N, Grabe H-J, Baumeister SE. et al. A telephone- and text message-based telemedicine concept for patients with mental health disorders: results of a randomized controlled trial. Psychother Psychosom 2015; 84: 82-89
• 12 van den Berg N, Grabe H-J, Freyberger HJ. et al. A telephone- and text-message based telemedical care concept for patients with mental health disorders -study protocol for a randomized, controlled study design. BMC Psychiatry 2011; 11: 30
• 13 Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev 2014; MR000034
• 14 Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. Trials 2010; 11: 32
• 15 World Health Organisation: International statistical classification of diseases and related health problems, 10^th revision. 2^nd ed Geneva, World Health Organisation. 2004
• 16 Franke GH, Ankerhold A, Haase M. et al. Der Einsatz des Brief Symptom Inventory 18 (BSI-18) bei Psychotherapiepatienten [The usefulness of the Brief Symptom Inventory 18 (BSI-18) in psychotherapeutic patients]. Psychother Psych Med 2011; 61: 82-86
• 17 Franke GH, Jaeger S, Glaesmer H. et al. Psychometric analysis of the brief symptom inventory 18 (BSI-18) in a representative German sample. BMC Med Res Methodol 2017; 17: 14
• 18 Spitzer C, Hammer S, Löwe B. et al. Die Kurzform des Brief Symptom Inventory (BSI -18): erste Befunde zu den psychometrischen Kennwerten der deutschen Version [The usefulness of the Brief Symptom Inventory 18 (BSI-18) in psychotherapeutic patients]. Fortschritte der Neurologie·Psychiatrie 2011; 79: 517-523
• 19 Meyer J, Fredrich D, Piegsa J. et al. A mobile and asynchronous electronic data capture system for epidemiologic studies. Computer methods and programs in biomedicine 2013; 110: 369-379
• 20 Bialke M, Penndorf P, Wegner T. et al. A workflow-driven approach to integrate generic software modules in a Trusted Third Party. Journal of Translational Medicine 2015; 13: 176
• 21 Tomarken AJ, Serlin RC. Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures. Psychological Bulletin 1986; 99: 90-99
• 22 Stewart M, Brown JB, Donner A. et al. The impact of patient-centered care on outcomes. The Journal of family practice 2000; 49: 796-804
• 23 Holloway F. Outcome measurement in mental health – welcome to the revolution. Br J Psychiatry 2002; 181: 1-2
• 24 Stryer D, Tunis S, Hubbard H. et al. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res 2000; 35: 977-993

Publication History

Received: 31 July 2024

Accepted after revision: 13 December 2024

Article published online:
10 April 2025