Statistical Methods for the Validation of Questionnaires

Summary

Objectives: Questionnaires used in epidemiological studies should be validated. However, unclarity exists about the appropriate statistical methods and interpretation of validation studies. Thus, we investigated the theory and practice of statistical evaluation approaches.

Methods: Using three platforms, a literature review, own simulations, and a validation study performed by ourselves, we worked out relevant limitations, advantages, and new important aspects of evaluation methods.

Results: Our systematic literature review, based on physical activity questionnaires, revealed that correlation coefficients are still the common approach in validation studies, found in 41 of 46 reviewed publications (89.1%). This practice has been criticized in the theoretically oriented literature for more than 20 years. Appropriate evaluation methods as recommended by Bland and Altman were found in only ten publications (21.7 %).

We showed that serious bias in questionnaires can be revealed by Bland-Altman plots but may remain undetected by correlation coefficients. With our simulations we refuted the argument that correlation coefficients properly investigate whether a questionnaire ranks the subjects sufficiently well. Further, with Bland-Altman analyses we could evaluate differential errors with respect to case-control status in our validation study. Yet, this was not possible with correlation coefficients, because they generally do not identify systematic bias. In addition, we show a potential pitfall in the interpretation of Bland-Altman plots that might occur in specific rare instances.

Conclusions: The commonly used correlation approach can yield misleading conclusions in validation studies. A more frequent and proper use of the Bland-Altman methods would be desirable to improve epidemiological data quality.

Keywords

Validation - questionnaire - epidemiological methods - Bland-Altman plot - correlation

• References

• 1 Hoffmann W, Latza U, Terschuren C. Guidelines and recommendations for ensuring Good Epidemiological Practice (GEP) - revised version after evaluation. Gesundheitswesen 2005; 67: 217-25.
• 2 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307-10.
• 3 Delcourt C. et al Limitations of the correlation coefficient in the validation of diet assessment methods. CODIAB-INSERM-ZENECA Pharma Study Group. Epidemiology 1994; 5: 518-24.
• 4 Hebert JR, Miller DR. The inappropriateness of conventional use of the correlation coefficient in assessing validity and reliability of dietary assessment methods. Eur J Epidemiol 1991; 7: 339-43.
• 5 Bellach B. Remarks on the use of Pearson’s correlation coefficient and other association measures in assessing validity and reliability of dietary assessment methods. Eur J Clin Nutr 1993; 47 Suppl 2 S42-5.
• 6 Washburn RA. et al The validity of the Stanford Seven-Day Physical Activity Recall in young adults. Med Sci Sports Exerc 2003; 35: 1374-80.
• 7 Slinde F. et al Minnesota leisure time activity questionnaire and doubly labeled water in adolescents. Med Sci Sports Exerc 2003; 35: 1923-8.
• 8 Conway JM, Irwin ML, Ainsworth BE. Estimating energy expenditure from the Minnesota Leisure Time Physical Activity and Tecumseh Occupational Activity questionnaires - a doubly labeled water validation. J Clin Epidemiol 2002; 55: 392-9.
• 9 Arvidsson D, Slinde F, Hulthen L. Physical activity questionnaire for adolescents validated against doubly labelled water. Eur J Clin Nutr 2005; 59: 376-83.
• 10 Lof M, Hannestad U, Forsum E. Assessing physical activity of women of childbearing age. Ongoing work to develop and evaluate simple methods. Food Nutr Bull 2002; 23: 30-3.
• 11 Liu B. et al Assessment of total energy expenditure in a Chinese population by a physical activity questionnaire: examination of validity. Int J Food Sci Nutr 2001; 52: 269-82.
• 12 Staten LK. et al Validation of the Arizona Activity Frequency Questionnaire using doubly labeled water. Med Sci Sports Exerc 2001; 33: 1959-67.
• 13 Aadahl M, Jorgensen T. Validation of a new selfreport instrument for measuring physical activity. Med Sci Sports Exerc 2003; 35: 1196-202.
• 14 Allor KM, Pivarnik JM. Stability and convergent validity of three physical activity assessments. Med Sci Sports Exerc 2001; 33: 671-6.
• 15 Bonnefoy M. et al Simultaneous validation of ten physical activity questionnaires in older men: a doubly labeled water study. J Am Geriatr Soc 2001; 49: 28-35.
• 16 Altman DG, Bland JM. Measurement in Medicine: the Analysis of Method Comparison Studies. The Statistician 1983; 32: 307-17.
• 17 Atkinson G, Nevill AM. Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Med 1998; 26: 217-38.
• 18 Bland JM, Altman DG. Applying the right statistics: analyses of measurement studies. Ultrasound Obstet Gynecol 2003; 22: 85-93.
• 19 Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8: 135-60.
• 20 Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet 1995; 346: 1085-7.
• 21 Cade J. et al Development, validation and utilisation of food-frequency questionnaires - a review. Public Health Nutr 2002; 5: 567-87.
• 22 Tremblay MS, Inman JW, Willms JD. Preliminary evaluation of a video questionnaire to assess activity levels of children. Med Sci Sports Exerc 2001; 33: 2139-44.
• 23 Treuth MS. et al Validity and reliability of activity measures in African-American girls for GEMS. Med Sci Sports Exerc 2003; 35: 532-9.
• 24 Dinger MK. et al Stability and convergent validity of the Physical Activity Scale for the Elderly (PASE). J Sports Med Phys Fitness 2004; 44: 186-92.
• 25 Harada ND. et al An evaluation of three self-report physical activity instruments for older adults. Med Sci Sports Exerc 2001; 33: 962-70.
• 26 Luiz RR, Szklo M. More than one statistical strategy to assess agreement of quantitative measurements may usefully be reported. J Clin Epidemiol 2005; 58: 215-6.