Download to read offline
![There’s No RewardReviewers […] didn't feel replication was necessary even though the original study was specific to a single company.“”](https://image.slidesharecdn.com/11-05-12-replichi-110515145731-phpapp01/75/RepliCHI-Graduate-Student-Perspectives-10-2048.jpg)
Uploaded byMichael Bernstein
RepliCHI: Graduate Student Perspectives
The document discusses graduate student perspectives on replicating studies in the field of computer science and artificial intelligence. It summarizes the results of a survey of 93 graduate students which found that 17% had replicated a study before but 83% had not, and 38% said they would replicate a study in the future but 62% said they would not. It explores some of the reasons students gave for not replicating studies, such as it not being seen as creative or novel work, reviewers criticizing submissions for not being novel if they are replications, and there being no reward or incentive structure for replicating previous work.
RepliCHI: Graduate Student Perspectives
- 1. RepliCan’tGraduate Student PerspectivesMichaelBernsteinMIT Computer Science and Artificial Intelligence Laboratorymsbernst@mit.edu | @msbernstmit human-computer interaction
- 2. Speaking for everygraduate student in SIGCHI, I can say one thing:
- 3. I can’t speakfor everygraduate studentin SIGCHI.
- 4. My Unassailable, ExtremelyScientificData Collection Protocol:Survey of CHI student volunteers, CHI-students ACM listserv, and snowballed recruitment through Facebook and Twitter.N=93 responses
- 5. Have you everreplicateda study or system?17% yes, 83% no
- 6. Do you everplan to replicatea study or system?38% “Hell yes”, 62% “Hell no”
- 7. Set Yourself Apart“Thepoint of research is to come up with exciting ideas that solve problems. Not copy others’ work.”
- 8. Set Yourself Apart“”I'mmore creative than that.
- 9. There’s No Reward“Newstudies confirming old studies have no chance of publication.”
- 10. There’s No RewardReviewers[…] didn't feel replication was necessary even though the original study was specific to a single company.“”
- 11. There’s No RewardIvery frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.“”
- 12. There’s No RewardIvery frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”.“”
- 13. Responding to IncentivesOpenaccess and replication. A true scientist’s ideals, but see:The grad student must conform. “”
- 14. Haikusstudies should breakground replication wastes our timelet's find new problems “”
- 15. Haikusthink analyzingCMC istough? try it reproducibly!“”
- 16. Haikusrepeat to besurewe stand on giants’ shouldersbut do so on faith“”
- 17. 83% have not62%will not(But we’ll need to replicate the study to be sure.)
- 19. Why?“repeat to besurewe stand on giant's shouldersbut do so on faith”
- 20. Why?“Replication is acritical component of scientific research, and it should be encouraged and rewarded. The lack of it is detrimental to the scientific soundness of our discipline. ”
- 21. Why?“we lack thetime forreplication of studiesjust review strictly”
- 22. Why?“think analyzingCMC istough? try it reproducibly!”
- 23. Why?“It seems likethe best outcome for a replication, rather than success, is actually a refutation of the original study.”
- 24. Why?“Because its's awaste of time: HCI studies are so small, I know they surely WON'T replicate, so why bother!”
- 25. Why?“CHI is toocutting edge for things like replication, or good science, or careful analysis, or the humility to accept that other topics besides Fitts' Law deserve dozens of nearly-identical studies.”
- 27. Why?“I do notintend on taking the risk of replicating some of my favorite works unless I see evidence that the CHI community supports such a thing.”
- 28. There’s No RewardCaseA. Confirmation of the earlier results (very boring)Case B. Conflict with earlier results (unpublishable problem) “”
Editor's Notes
- #2 Interfaces that embed crowd contributions to support complex and novel interactive systems.-------------Bring in the simplex? Thread through, come back to it from time to time.Soylent. Emphasize that we’re not throwing away AI, not throwing away the user, …Explain FeedMe better via the simplex?Insights from designing two different families of interfaces --- generalize lessons? Motivations. Design differences. Commonalities: small bits of effort? At end of talk.- small bits of effortMost of this has examined moving things toward crowd. But, we need a better sense of when to do that, and when not to. Feed back into traditional We need to understand in order to build. Not just the user interface, but the crowd.Embiggen the gray from data mining. Move data mining to future work?Introduce points one and two, these are the systems I’m going to talk about in this talk. Then, at the end, say there are two avenues for future work. 1) realtime, 2) data thing thing embiggen thing.Data-driven?Less cadence at the end of Soylent.Transition between Collabio and FeedMe needs work.Second half was too fast-paced? Video especially?Stupid PhD student.25 min soylent, 15 min Don’t make crazy animation w/ ShortnTry to save the Soylent demo for later. Figure out a way to convey the concept without the early handwaved demo.“Pa-nah-cea”Finish the transition. There are lots of applications where you can't use crowds. One of those is when you need socially sourced information. We have to develop techniques for this