Download PDF
Methods Inf Med 2018; 57(03): 111-119
DOI: 10.3414/ME17-01-0021
Original Article
Schattauer GmbH

The Evaluation of Bivariate Mixed Models in Meta-analyses of Diagnostic Accuracy Studies with SAS, Stata and R

Felicitas Vogelgesang
1   Department of Radiology, Charité-Universitätsmedizin Berlin, Berlin, Germany
2   Department of Medical Statistics, Informatics and Documentation, University Hospital of Friedrich Schiller University Jena, Jena, Germany
,
Peter Schlattmann   **
2   Department of Medical Statistics, Informatics and Documentation, University Hospital of Friedrich Schiller University Jena, Jena, Germany
,
Marc Dewey   **
1   Department of Radiology, Charité-Universitätsmedizin Berlin, Berlin, Germany
› Author AffiliationsFinancial support for this study was provided in part by a grant from the Joint Program from the German Research Foundation (DFG) and the German Federal Ministry of Education and Research (BMBF) for meta-analyses for Felicitas Vogelgesang. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.
Further Information

Publication History

received: 21 February 2017

accepted: 19 September 2017

Publication Date:
02 May 2018 (online)

    Permissions and Reprints

Summary

Background: Meta-analyses require a thoroughly planned procedure to obtain unbiased overall estimates. From a statistical point of view not only model selection but also model implementation in the software affects the results.

Objectives: The present simulation study investigates the accuracy of different implementations of general and generalized bivariate mixed models in SAS (using proc mixed, proc glimmix and proc nlmixed), Stata (using gllamm, xtmelogit and midas) and R (using reitsma from package mada and glmer from package lme4). Both models incorporate the relationship between sensitivity and specificity – the two outcomes of interest in meta-analyses of diagnostic accuracy studies – utilizing random effects.

Methods: Model performance is compared in nine meta-analytic scenarios reflecting the combination of three sizes for meta-analyses (89, 30 and 10 studies) with three pairs of sensitivity/specificity values (97%/87%; 85%/75%; 90%/93%).

Results: The evaluation of accuracy in terms of bias, standard error and mean squared error reveals that all implementations of the generalized bivariate model calculate sensitivity and specificity estimates with deviations less than two percentage points. proc mixed which together with reitsma implements the general bivariate mixed model proposed by Reitsma rather shows convergence problems. The random effect parameters are in general underestimated.

Conclusions: This study shows that flexibility and simplicity of model specification together with convergence robustness should influence implementation recommendations, as the accuracy in terms of bias was acceptable in all implementations using the generalized approach.

Keywords

Bivariate mixed models - meta-analysis - diagnostic accuracy studies - SAS - Stata

** These authors contributed equally to this work.


 

  • References

  • 1 Leeflang MMG, Deeks JJ, Gatsonis C, Bossuyt PMM. Systematic Reviews of Diagnostic Test Accuracy. Annals of Internal Medicine 2008; 149 (12) 889-897.
    CrossrefPubMedGoogle Scholar
  • 2 Chu H, Cole SR. Bivariate meta-analysis with sparse data a generalized linear mixed model approach. Journal of Clinical Epidemiology 2006; 59 (12) 1331-1332.
    CrossrefPubMedGoogle Scholar
  • 3 Chu H, Guo H, Zhou Y. Bivariate Random Effects Meta-analysis of Diagnostic Studies Using Generalized Linear Mixed Models. Medical Decision Making 2010; 30 (04) 499-508.
    CrossrefPubMedGoogle Scholar
  • 4 Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology 2005; 58 (10) 982-990.
    CrossrefPubMedGoogle Scholar
  • 5 Dahabreh IJ, Trikalinos TA, Lau J, Schmid C. An Empirical Assessment of Bivariate Methods for Meta-Analysis of Test Accuracy. Method Research Report (prepared by Tufts Evidence-based Practice Center. 2012 (12(13)-EHC136-EF2) [cited 2013 May 22]. Available from: www.effectivehealthcare.ahrq.gov/reports/final/cfm
    PubMedGoogle Scholar
  • 6 Menke J. Bivariate Random-effects Meta-analysis of Sensitivity and Specificity with SAS PROC GLIMMIX. Methods of Information in Medicine 2010; 49 (01) 54-64.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine 2002; 21 (04) 589-624.
    CrossrefPubMedGoogle Scholar
  • 8 Harbord R, Whiting P, Sterne JAC, Egger M, Deeks JJ, Shang A. et al. An empirical comparison of methods for meta-analysis of diagnostic accuracy showed hierarchical models are necessary. Journal of Clinical Epidemiology 2008; 61 (11) 1095-1103.
    CrossrefPubMedGoogle Scholar
  • 9 Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR. Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Medical Research Methodology 2007; 07 (01) 1-15.
    CrossrefPubMedGoogle Scholar
  • 10 Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. Statistics in Medicine 2004; 23 (09) 1351-1375.
    CrossrefPubMedGoogle Scholar
  • 11 Schuetz GM, Zacharopoulou NM, Schlattmann P, Dewey M. Meta-analysis: Noninvasive Coronary Angiography Using Computed Tomography Versus Magnetic Resonance Imaging. Annals of Internal Medicine 2010; 152 (03) 167-177.
    CrossrefPubMedGoogle Scholar
  • 12 Burton A, Altman DG, Royston P, Holder RL. The design of simulation studies in medical statistics. Statistics in Medicine 2006; 25 (24) 4279-4292.
    CrossrefPubMedGoogle Scholar
  • 13 Arends LR, Hamza TH, Van Houwelingen JC, Heijenbrok-Kal MH, Hunink MGM, Stijnen T. Bivariate Random Effects Meta-Analysis of ROC Curves. Medical Decision Making 2008; 28 (05) 621-638.
    CrossrefPubMedGoogle Scholar
  • 14 StataCorp L. Stata Longitudinal-Data/Panel-Data Reference Manual: Release 12. StataCorp LP: 2011
    PubMedGoogle Scholar
  • 15 Dwamena BA. midas: A program for Meta-analytical Integration of Diagnostic Accuracy Studies in Stata. 2007. Division of Nuclear Medicine, Department of Radiology, University of Michigan Medical School; Ann Arbor, Michigan.:
    Google Scholar
  • 16 Rabe-Hersketh S, Skrondal A, Pickles A. GLLAMM Manual. UC Berkeley Division of Biostatistics Working Paper Series. 2004 (Working Paper 160) [cited 2017 Nov 22]. Available from: http://biostats.bepress.com/ucbbiostat/paper160
    PubMedGoogle Scholar
  • 17 bSas Institute. SAS/Stat 9.2 User’s Guide. Cary, North Carolina: SAS Institute; 2008 Chap. 38 – The GLIMMIX Procedure; chap. 56 – The MIXED Procedure; chap. 61 – The NLMIXED Procedure..
    PubMed
  • 18 Bates D, Maechler M, Bolker B. Package lme4. 2017 Version 1.1–13..
    PubMedGoogle Scholar
  • 19 Doebler P. Package mada. 2015 Version 0.5.7..
    PubMedGoogle Scholar
  • 20 Rabe-Hesketh S, Skrondal A, Pickles A. Reliable estimation of generalized linear mixed models using adaptive quadrature. The Stata Journal 2002; 02 (01) 1-21.
    PubMedGoogle Scholar
  • 21 Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary roc curve: Data-analytic approaches and some additional considerations. Statistics in Medicine 1993; 12 (14) 1293-1316.
    CrossrefPubMedGoogle Scholar
  • 22 Zapf A, Hoyer A, Kramer K, Kuss O. Nonparametric meta-analysis for diagnostic accuracy studies. Statistics in Medicine 2015; 034 (29) 3831-3841.
    CrossrefPubMedGoogle Scholar
  • 23 Schuetz GM, Schlattmann P, Dewey M. Use of 3x2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: meta-analytical evaluation of coronary CT angiography studies. BMJ 2012; 10 (345) e6717.
    PubMedGoogle Scholar
  • 24 Hamza T, Arends L, van Houwelingen H, Stijnen T. Multivariate randome effects meta-analysis of diagnostic tests with multiple thresholds. BMC Medical Research Methodology 2009; 09 (01) 73.
    CrossrefPubMedGoogle Scholar
  • 25 Putter H, Fiocco M, Stijnen T. Meta-Analysis of Diagnostic Test Accuracy Studies with Multiple Thresholds using Survival Methods. Biometrical Journal 2010; 52 (01) 95-110.
    PubMedGoogle Scholar
  • 26 Dwamena BA. Evidence-based Radiology: Step 3 – Diagnostic Systematic Review and Meta-analysis (Critical Appraisal). Seminars in Roentgenology 2009; 44 (03) 170-179.
    CrossrefPubMedGoogle Scholar
  • 27 Kiernan K, Tao J, Gibbs P. Tips and strategies for mixed modeling with SAS/STAT procedures. Proceedings SAS Global Forum 2012 (Paper 332–2012). 2012
    PubMedGoogle Scholar
  • 28 Diaz M. Performance measures of the bivariate random effects model for meta-analyses of diagnostic accuracy. Computational Statistics and Data Analysis 2015; 83: 82-90.
    CrossrefPubMedGoogle Scholar
  • 29 Riley RD, Ahmed I, Debray T, Willis BH, Noordzij JP, Higgins J. et al. Summarising and validating test accuracy results across multiple studies for use in clinical practice. Statistics in Medicine 2015; 34 (13) 2081-2103.
    CrossrefPubMedGoogle Scholar