Die Replikationskrise in der Psychologie

ZUSAMMENFASSUNG

Während des letzten Jahrzehnts war die Replikationskrise eines der am meisten diskutierten Themen in der Psychologie. Doch wie konnte es überhaupt zu dieser Krise kommen, welche Studien sind von der Replikationskrise betroffen und welche Maßnahmen wurden bisher ergriffen, um dieser Krise zu begegnen? In dieser Übersicht werden eine Reihe von Problemen der bislang üblichen Methodik wissenschaftlichen Arbeitens beschrieben, klassische Beispiele von bekannten Studien, die nicht repliziert werden konnten, aufgelistet die Verwurzelung der Probleme in der gängigen Wissenschaftspraxis, wie sie sich in unserer Gesellschaft darstellt aufgezeigt und Änderungen im Wissenschaftsbetrieb der letzten Jahre beschrieben, die Abhilfe schaffen sollen. Die Weiterentwicklung der Wissenschaftskultur ist Teil des reflektierten wissenschaftlichen Arbeitens, wird sie doch von niemand anderem getragen als von angehenden und bereits praktizierenden Wissenschaftlern selbst.

ABSTRACT

During the past decade the replication crisis has been one of the most discussed topics in psychology. But how could this crisis have come about in the first place? Which studies have been affected by the replication crisis? And what measures have been taken up so far to counter this crisis? This overview describes a number of methods related problems in empirical science studies. We list classic examples of known studies that could not be replicated. We show that the problems are rooted in how science is practiced in developed societies. Finally, we describe remedies for the crisis, i. e., changes of scientific conduct that have been proposed and/or already implemented in recent years. Science is always performed in a societal and cultural context, which is why the consideration of society and culture is part of any well-reflected scientific work. New scientific ideas and developments always come from practicing scientists themselves. Who else should, and could, find ways to establish what is true and what is not true in science?

Schlüsselwörter

Statistik - Methodenkritik - Replikation - Sozialpsychologie - Verhaltensökonomie - kognitive Psychologie - Wissenschaftstheorie

Key words

Statistics - methods in psychosocial sciences - replication - behavioral economics - cognitive psychology - social psychology - philosophy of science

² Genau genommen gibt es auch noch 2 weitere Szenarien. Ein Ergebnis wird nicht signifikant und man weiß nicht, ob es wirklich keinen Unterschied gibt, oder ob es diesen gibt, er aber nicht in den erhobenen Daten zu finden ist.

• Literatur

• 1 John LK, Loewenstein G, Prelec D.. Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychol Sci 2012; 23: 524-532
• 2 Carter EC, Kofler LM, Forster DE. et al A Series of Meta-Analytic Tests of the Depletion Effect: Self-Control Does Not Seem to Rely on a Limited Resource. 2015: 144
• 3 Vazire S. Implications of the Credibility Revolution for Productivity, Creativity, and Progress. Perspect Psychol Sci 2018; 13: 411-417
• 4 Sterling TD. Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance – or Vice Versa. J Am Stat Assoc 1959; 54: 30-34
• 5 Lykken DT. Statistical significance in psychical research. Psychol Bull 1968; 70: 151-159
• 6 Meehl PE. Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. J Consult Clin Psychol 1978; 46: 806-834
• 7 Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull 1979; 86: 638-641
• 8 Rosnow RL, Rosenthal R. Statistical Procedures and the Justification of Knowledge in Psychological Science. Am Psychol 1989; 44: 1276-1284
• 9 Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychol Sci 2011; 22: 1359-1366
• 10 Ritchie SJ, Wiseman R, French CC. Failing the future: Three unsuccessful attempts to replicate bem’s „retroactive facilitation of recall“ effect. PLoS One 2012; 7: 1-5
• 11 Makel MC, Plucker JA, Hegarty B. Replications in Psychology Research: How Often Do They Really Occur?. Perspect Psychol Sci 2012; 7: 537-542
• 12 Ioannidis JPA. Why Science Is Not Necessarily Self-Correcting. Perspect Psychol Sci 2012; 7: 645-654
• 13 Pashler H, Harris CR. Is the Replicability Crisis Overblown? Three Arguments Examined. Perspect Psychol Sci 2012; 7: 531-536
• 14 Frank MC, Saxe R. Teaching Replication. Perspect Psychol Sci 2012; 7: 600-604
• 15 Giner-Sorolla R.. Science or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science. Perspect Psychol Sci 2012; 7: 562-571
• 16 Bakker M, van Dijk A, Wicherts JM. The Rules of the Game Called Psychological Science. Perspect Psychol Sci 2012; 7: 543-554
• 17 Open Science Collaboration. Estimating the reproducibility of psychological science. Science 2015: 349
• 18 Camerer CF, Dreber A, Forsell E. et al Evaluating replicability of laboratory experiments in economics. Science 2016; 351: 1433-1436
• 19 Ebersole CR, Atherton OE, Belanger AL. et al Many Labs 3: Evaluating participant pool quality across the academic semester via replication. J Exp Soc Psychol 2016; 67: 68-82
• 20 Klein Ratliff KA, Vianello M. et al Data from Investigating Variation in Replicability: A “Many Labs” Replication Project. J Open Psychol Data 2014; 2: e4
• 21 Benjamin DJ, Berger JO, Johannesson M. et al Redefine statistical significance. Nat Hum Behav 2018; 2: 6-10 dx.doi.org/10.1038/s41562-017-0189-z
• 22 Fidler F, Wilcox J. Reproducibility of scientific results. Stanford Encycl Philos Stanford: CA Cent Study Lang Inf; 2018
• 23 Masicampo E, Baumeister RF. Toward a physiology of dual-process reasoning and judgment: Lemonade, willpower, and expensive rule-based analysis. Psychol Sci 2008; 19: 255-260
• 24 Zwaan RA, Taylor LJ. Seeing, acting, understanding: Motor resonance in language comprehension. J Exp Psychol Gen 2006; 135: 1-11
• 25 Vohs KD, Schooler JW. The Value of Believing in Free Will. Psychol Sci 2008; 19: 49-54
• 26 Zhong C, Liljenquist K. Washing Away Your Sins. Science 2006; 313: 1451-1452
• 27 Zhong CB, Leonardelli GJ. Cold and lonely: Does social exclusion ­literally feel cold?. Psychol Sci 2008; 19: 838-842
• 28 Sanna LJ, Chang EC, Miceli PM. et al Rising up to higher virtues: Experiencing elevated physical height uplifts prosocial actions. J Exp Soc Psychol 2011; 47: 472-476 dx.doi.org/10.1016/j.jesp.2010.12.013
• 29 Strack F, Martin LL, Strepper S. Inhibiting and facilitating conditions of facial expressions: A non-obstrusive test of the facial feedback hypothesis. J Personal Soc Psychol 1988; 54: 768-776
• 30 Dutton DG, Aron AP. Some evidence for heightened sexual attraction under conditions of high anxiety. J Pers Soc Psychol 1974; 30: 510-517
• 31 Bargh JA, Chen M, Burrows L.. Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action. J Pers Soc Psychol 1996; 71: 230-244 doi.apa.org/journals/psp/71/2/230.html
• 32 Datley JM, Lantane B. Bystander Intervention in Emergencies: Diffusion of Responsibility. J Pers Soc Psychol 1968; 8: 377-383
• 33 Philpot R, Liebst LS, Levine M. et al Would I Be Helped? Cross-National CCTV Footage Shows That Intervention Is the Norm in Public Conflicts. Am Psychol 2019: 377-383
• 34 Gailliot MT, Baumeister RF, Dewall CN. et al Self-Control Relies on Glucose as a Limited Energy Source: Willpower Is More Than a Metaphor. 2007 92. 325-336
• 35 Gelman A, Loken E. The Statistical Crisis in Science. Am Sci 2014: 102
• 36 Doyen S, Klein O, Pichon CL. et al Behavioral priming: It’s all in the mind, but whose mind?. PLoS One 2012: 7
• 37 Szczucka K. Attraction at first fright? What Datton & Aron really demonstrated almost 40 years ago. Polish Psychol Bull 2012; 43: 191-198
• 38 Pashler H, Wagenmakers EJ. Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?. Perspect Psychol Sci 2012; 7: 528-530
• 39 Wagenmakers EJ, Beek T, Dijkhoff L. et al Registered Replication Report: Strack, Martin, Stepper (1988). Perspect Psychol Sci 2016; 11: 917-928
• 40 Fayard J V, Bassi AK, Bernstein DM. et al Is cleanliness next to godliness? Dispelling old wives’ tales: Failure to replicate Zhong and Liljenquist (2006). J Artic Support Null Hypothesis 2009; 6: 21-30 www.jasnh.com/pdf/Vol6-No2.pdf
• 41 Papesh MH. Just out of reach: On the reliability of the action-sentence compatibility effect. J Exp Psychol Gen 2015; 144: e116-e141
• 42 Beall AT, Tracy JL, Beall AT. et al Women Are More Likely to Wear Red or Pink at Peak Fertility. 2013 pss.sagepub.com/content/early/2013/07/10/0956797613476045.full
• 43 Farris C, Treat TA, Viken RJ. et al Perceptual mechanisms that characterize gender differences in decoding women’s sexual intent: Research article. Psychol Sci 2008; 19: 348-354
• 44 Cox CR, Arndt J, Pyszczynski T. et al Terror Management and Adults’ Attachment to Their Parents: The Safe Haven Remains. J Pers Soc Psychol 2008; 94: 696-717
• 45 Ersner-Hershfield H, Mikels JA, Sullivan SJ. et al Poingnancy: Mixed Emotions in Meaningful Endings. J Pers 2010; 94: 158-167
• 46 Alter AL, Oppenheimer DM. Effects of fluency on psychological distance and mental construal (or why New York is a large city, but New York is a civilized jungle): Research article. Psychol Sci 2008; 19: 161-167
• 47 Lau GP, Kay AC, Spencer SJ. Loving Those Who Justify Inequality. Psychol Sci 2008; 19: 20-21
• 48 Monin B, Sawyer PJ, Marquez MJ. The Rejection of Moral Rebels: Resenting Those Who Do the Right Thing. J Pers Soc Psychol 2008; 95: 76-93
• 49 Camerer CF, Dreber A, Holzmeister F. et al Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2018; 2: 637-644
• 50 Wilson TD, Reinhard DA, Westgate EC. et al Just think: The challenges of the disengaged mind. Science 2014: 345
• 51 Sparrow B, Liu J, Wegner DM. Google effects on memory: Cognitive consequences of having information at our fingertips. Science 2011; 333: 776-778
• 52 Duncan K, Sadanand A, Davachi L. Memory’s Penumbra: Episodic memory decisions induce lingering mnemonic biases. Science 2012; 337: 485-487
• 53 Gervais WM, Norenzayan A. Analytic thinking promotes religious disbelief. Science 2012; 336: 493-496
• 54 Gneezy U, Keenan EA, Gneezy A. Avoiding overhead aversion in charity. Science 2014; 346: 632-635
• 55 Hauser OP, Rand DG, Peysakhovich A. et al Cooperating with the future. Nature 2014; 511: 220-223 dx.doi.org/10.1038/nature13530
• 56 Janssen MA, Holahan R, Lee A. et al Lab experiments for the study of social-ecological systems. Science 2010; 328: 613-617
• 57 Karpicke JD, Blunt JR. Retrieval practice produces more learning than elaborative studying with concept mapping. Science 2011; 331: 772-775
• 58 Kidd DC, Castano E. Reading literary fiction improves theory of mind. Science 2013; 342: 377-380
• 59 Kovács ÁM, Téglás E, Endress AD. The Social Sense: Susceptibility to Others’ Beliefs in Human Infants and Adults. Science 2010; 330: 1830-1835
• 60 Lee SWS, Schwarz N.. Washing away postdecisional dissonance. Science 2010; 328: 709
• 61 Morewedge CK, Huh YE, Vosgerau J. Thought for food: Imagined consumption reduces actual consumption. Science 2010; 330: 1530-1533
• 62 Rand DG, Greene JD, Nowak MA. Spontaneous giving and calculated greed. Nature 2012; 489: 427-430
• 63 Shah AK, Mullainathan S, Shafir E.. Some Consequences of Having Too Little. Science 2012; 338: 682-686
• 64 Ramirez G, Beilock SL. Writing About Testing Worries Boosts Exam Performance in the Classroom. Science 2011; 331: 211-214
• 65 Pyc MA, Rawson KA. Why testing improves memory: Mediator effectiveness hypothesis. Science 2010; 330: 335
• 66 Nishi A, Shirado H, Rand DG. et al Inequality and visibility of wealth in experimental social networks. Nature 2015; 526: 426-429
• 67 Ackerman JM, Nocera CC, Bargh JA. Incidental haptic sensations influence social judgments and decisions. Science 2010; 328: 1712-1715
• 68 Aviezer H, Trope Y, Todorov A. Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 2012; 338: 1225-1229
• 69 Balafoutas L, Sutter M. Affirmative action policies promote women and do not harm efficiency in the laboratory. Science 2012; 335: 579-582
• 70 Derex M, Beugin MP, Godelle B. et al Experimental evidence for the influence of group size on cultural complexity. Nature 2013; 503: 389-391 dx.doi.org/10.1038/nature12774
• 71 Klein Vianello M, Hasselman F. et al Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Adv Methods Pract Psychol Sci 2018; 1: 443-490
• 72 Vadillo MA, Gold N, Osman M.. Searching for the bottom of the ego well: Failure to uncover ego depletion in Many Labs 3. R Soc Open Sci 2018: 5
• 73 ManyBabies Consortium. Quantifying sources of variability in infancy research using the infant-directed speech preference. Adv Methods Pract Psychol Sci 2019
• 74 Amrhein V, Greenland S. Remove, rather than redefine, statistical significance. Nat Hum Behav 2018; 2: 4 dx.doi.org/10.1038/s41562-017-0224-0
• 75 Lakens D, Adolfi FG, Albers CJ. et al Justify your alpha. Nat Hum Behav 2018; 2: 168-171
• 76 Mcshane BB, Gal D, Gelman A. et al Abandon Statistical Significanc. 2017: 1-12
• 77 Wasserstein RL, Lazar NA. The ASA’s Statement on p-Values: Context, Process, and Purpose. Am Stat 2016; 70: 129-133 dx.doi.org/10.1080/00031305.2016.1154108
• 78 Silberzahn R, Uhlmann EL, Martin DP. et al Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Adv Methods Pract Psychol Sci 2018; 1: 337-356
• 79 Wicherts JM, Borsboom D, Kats J. et al The poor availability of psychological research data for reanalysis. Am Psychol 2006; 61: 726-728
• 80 Wicherts JM, Bakker M, Molenaar D.. Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One 2011; 6: 1-7
• 81 Nosek BA, Ebersole CR, Dehaven AC. et al The preregistration revolution. 2017 2017. 1-7
• 82 Simons DJ, Holcombe AO, Spellman BA. An Introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspect Psychol Sci 2014; 9: 552-555
• 83 Held L, Ott M. On p-Values and Bayes Factors. Annu Rev Stat Its Appl 2018; 5: 393-419
• 84 Benjamin DJ, Berger JO. Three Recommendations for Improving the Use of p-Values. Am Stat 2019; 73: 186-191
• 85 Nosek BA, Spies JR, Motyl M. Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability 2012
• 86 Hagger MS, Chatzisarantis NLD, Alberts H. et al A Multilab Preregistered Replication of the Ego-Depletion Effect. Perspect Psychol Sci 2016; 11: 546-573
• 87 Lin H, Blair S, Malte F. et al Strong Effort Manipulations Reduce Response Caution: A Preregistered Reinvention of the EgoDepletion Paradigm. psyarchiv.com 2019
• 88 Gelman A, Carlin J. Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspect Psychol Sci 2014; 9: 641-651
• 89 Loken BE, Gelman A. Measurement error and the replication crisis. Science 2017; 355: 584-586
• 90 Schad DJ, Vasishth S. The posterior probability of a null hypothesis given a statistically significant result. 2019 arxiv.org/abs/1901.06889
• 91 Wilson BM, Wixted JT. The Prior Odds of Testing a True Effect in Cognitive and Social Psychology. Adv Methods Pract Psychol Sci 2018; 1: 186-197
• 92 Harris M, Taylor G. Medical statistics made easy. London: Scion Banbury; 2003
• 93 Vasishth S, Nicenboim B. Statistical methods for linguistic research: Foundational Ideas – Part I. 2016
• 94 Wagenmakers EJ, Wetzels R, Borsboom D. et al Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi: Comment on Bem. J Pers Soc Psychol 2011; 100: 426-432
• 95 Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med 2005; 2: e124