Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Empirische wetenschap onder vuur

397 views

Published on

Empirische wetenschap onder vuur
Alumni dag voor de faculteit Sociale Wetenschappen @ KU Leuven

Published in: Education
  • Be the first to comment

  • Be the first to like this

Empirische wetenschap onder vuur

  1. 1. Empirische wetenschap onder vuur SW Alumni dag March 22nd 2014 @TimSmitsTim http://www.slideshare.net/timsmitstim
  2. 2. Vuur. Of toch veel rook.
  3. 3. Diederik Stapel Dirk Smeesters Yoshitaka Fuji
  4. 4. Diederik Stapel Dirk Smeesters Yoshitaka Fuji
  5. 5. Heel veel rook dus. Maar wat brand er eigenlijk?
  6. 6. FRAUD THE POPE ; JESUS Diederik Stapel (50) Yoshitaka Fuji (183) Dirk Smeesters We? (communication sciences; social sciences, KULeuven) Data fabrication Paper duplication Plagiarism Lack of IRB approval P-hacking File drawer One-sided lit. review Biased content analysis Biased interviews Other questionable research practices (e.g., not understanding science)
  7. 7. OPINIE… … Het is geen probleem van individuele fraudeurs. Die zullen er altijd zijn en de kans dat het ongestraft gebeurt, is kleiner dan ooit. … Het is geen discipline-specifiek probleem (sociale psychologie) of een bepaalde onderzoeksmethode (experimenteel; kwantitatief) (Stroebe, Postmes & Spears, 2012)
  8. 8. DUS … IS HET SYSTEMISCH - Publish or perish. Met beoordeling op ondermaatse metingen: aantal publicaties.
  9. 9. DUS … IS HET SYSTEMISCH (2) - Publicatiebias en significantie-fetish (maar dat is slechts deels systemisch)
  10. 10. (SYSTEMISCHE) OPLOSSINGEN of EVOLUTIES: -Retractions van frauduleuze of questionable papers; mega-correcties van gepubliceerde artikels -Onderzoek over fraude en QRP (questionable research practices) -Oproep tot replicatie studies; pre-registratie van onderzoek & pre-acceptatie van artikels -Open science networks: bv. publication/replication depositories – open science framework (osf.org) -Post-publication review: blogs, Twitter, etc.; HIBAR (Had I Been A Reviewer) -Sancties voor fraudeurs -…
  11. 11. From retractionwatch.wordpress.com
  12. 12. DRINGEND GEZOCHT: GEZOND VERSTAND Soms is onderzoek gewoon “TOO GOOD TO BE TRUE” Extreem voorbeeld: Greg Francis’ research (ook onder kritiek a.o. Uri Simonsohn) ***For the following slides: ALL CREDITS to Greg’s presentation on Febr 5th 2013 in Brussels***
  13. 13. Experimental methods • Suppose you hear about two sets of experiments that investigate phenomena A and B • Which effect is more believable? Effect A Effect B Number of experiments 10 19 Number of experiments that reject H0 9 10 Replication rate 0.9 0.53
  14. 14. • Effect A is Bem’s (2011) precognition study that reported evidence of people’s ability to get information from the future – I do not know any scientist who believes this effect is real • Effect B is from a meta-analysis of a version of the bystander effect, where people tend to not help someone in need if others are around – I do not know any scientist who does not believe this is a real effect • So why are we running experiments? Effect A Effect B Number of experiments 10 19 Number of experiments that reject H0 9 10 Replication rate 0.9 0.53
  15. 15. Hypothesis testing (for means) • We start with a null hypothesis: no effect, H0 • Identify a sampling distribution that describes variability in a test statistic t = X1 - X2 sX1 -X2
  16. 16. Hypothesis testing (for two means) • We can identify rare test statistic values as those in the tail of the sampling distribution • If we get a test statistic in either tail, we say it is so rare (usually 0.05) that we should consider the null hypothesis to be unlikely • We reject the null t = X1 - X2 sX1 -X2 H0
  17. 17. Alternative hypothesis • If the null hypothesis is not true, then the data came from some other sampling distribution (Ha) H0 Ha
  18. 18. Power • If the alternative hypothesis is true • Power is the probability you will reject H0 • If you repeated the experiment many times, you would expect to reject H0 with a proportion that reflects the power H0 Ha
  19. 19. Power • Use the pooled effect size to compute the pooled power of each experiment (probability this experiment would reject the null hypothesis) • Pooled effect size – g*=0.1855 • The sum of the power values (E=6.27) is the expected number of times these experiments would reject the null hypothesis (Ioannidis & Trikalinos, 2007) Sample size Effect size (g) Power Exp. 1 100 0.249 0.578 Exp. 2 150 0.194 0.731 Exp. 3 97 0.248 0.567 Exp. 4 99 0.202 0.575 Exp. 5 100 0.221 0.578 Exp. 6 Negative 150 0.146 0.731 Exp. 6 Erotic 150 0.144 0.731 Exp. 7 200 0.092 0.834 Exp. 8 100 0.191 0.578 Exp. 9 50 0.412 0.363
  20. 20. Take-home-message of Greg’s studies -The file drawer phenomenon might be immense. Don’t put your money on published studies -Think not only about the p of your failed studies, but also their power. -For most studies in our discipline, there is about a 50% chance to discover an true phenomenon (since many studies are underpowered) -Increase your N per hypothesis! It increases your “power” to discover an effect (Ha= true) and (a bit) to refute an effect’s existence (H0= true) Note: To “detect” that men weigh more than women at an adequate power of .8, you need to have n=46!!! (Simmons et al., 2013). Are we studying effects that are stronger than men outweighing women??
  21. 21. Fout begrip van wetenschap Bij publiek, journalistiek én wetenschap
  22. 22. Wetenschap… … zal een stuk saaier moeten worden. Minder nieuwigheden, meer repliceren en synthetiseren … zal een stuk minder duidelijk moeten worden. Gewicht van conclusies in functie van effectgrootte, aantal studies, omstandigheden

×