Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this presentation? Why not share!

1,174 views

Published on

Abstract: In 2013, 392 research papers and notes were published in the CHI conference (The ACM CHI Conference on Human Factors in Computing Systems) and even more papers in the domain of Human-Computer Interaction (HCI) are constantly published in various conferences and journals. It is quite arduous, if not impossible, to read all of these papers. One approach to deal with this information deluge is to focus on skimming through lots of papers in a short period of time, so that one can more wisely choose what to read before investing time in them. In order to teach such a skimming technique, I have taught a technique, called "Quick and Dirty Review (QnDReview)," in a graduate-level HCI course. The method has been employed in the course for five semesters, and students' responses were collected and analyzed. Results showed that students spent, on average, 4.3 minutes per paper and believed that they got the gist of each paper. However, the largest benefit I noticed is that students get the overall pictures of the fields while exposing themselves to various new ideas through this approach.

Published in:
Education

No Downloads

Total views

1,174

On SlideShare

0

From Embeds

0

Number of Embeds

9

Shares

0

Downloads

34

Comments

0

Likes

4

No embeds

No notes for slide

- 1. QnDReview: Read 100 CHI Papers in 7 Hours Ji Soo Yi HIVE Lab Industrial Engineering, Purdue University 717AB, 9am, 4/30/2014
- 2. Disclaimer • Comprehensive reading a research paper requires lots of effort / time and sufficient background, so this talk is not about making you have magical, comprehensive understanding of 100 papers in 7 hours. • I also have not created any fancy tools.
- 3. A Story in 2004 “Hey, Ji Soo, for the book chapter that we are working, can you survey what kinds of research methods were used in the papers of two journals, Ergonomics and Human Factors in the past 5 years?”
- 4. Pleasantly Surprised! I was able to • skim through papers quicker and quicker; • see the overall trends in the field; • generate interesting ideas; and • actually enjoy the process. Leonard, V. K., Jacko, J. A., Yi, J. S., and Sainfort, F. Human factors and ergonomic methods. In Handbook of Human Factors and Ergonomics, G. Salvendy, Ed., 3rd ed. John Wiley & Sons, Feb. 2006, 292–321.
- 5. In 2009, I started to teach a graduate-level HCI methodology course.
- 6. What if
- 7. What if graduate students are forced
- 8. What if graduate students are forced asked
- 9. What if graduate students are forced asked to read 100 papers?
- 10. Quick and Dirty Review
- 11. QnDReview
- 12. Homework 1. Select 100 papers to read. Visit the ACM Digital Library to search for any papers that are seemingly interesting to you. I recommend you to start with CHI full papers because CHI is known to be the premier venue. ver. 2013
- 13. Homework 2. Review them all very quickly. Use the template to make your own summary review. An Excel spread sheet for the quick and dirty review should contain the following information: Column 1: Paper citation Column 2: Interest rating: 1 (Worst) - 2 (Bad) - 3 (Neither good nor bad) - 4 (Good) - 5 (Best) Column 3: A single sentence that describes the core idea of the paper. Column 4: (Optional) Any new ideas that you have after skimming through this paper. ver. 2013
- 14. Homework 3. Email your summary Email the complete review spreadsheet to the instructor by the due date with the following information in the body of the email. • What is the overall experience? • What is the notable trends in the papers? • Which are the two most interesting papers you would like to read more deeply? (Provide the full citation with DOI number, so that I can find the paper later) • How much time did you spend on this homework? • What are the most difficult part of this homework? • How to make the quick and dirty review better? ver. 2013
- 15. Participants • N = 57 over 5 years • 27 F + 30 M • 2 U + 21 MS + 34 PhD • 11 non-IE1 + 46 IE • IRBs • 1012010281 (retrospective one) • 1302013324 1 Engineering Education and Electrical and Computer Engineering.
- 16. Some Variations 2009: self-selected 100 HCI-related papers1 2010: preselected 50 CHI 2010 papers 2011: preselected 100 CHI 2011 papers 2012: preselected 120 papers • 30 from CHI 2012 • 30 from IEEE TVCG • 30 from Decision Support Systems 2013: self-selected 100 CHI 2013 papers 1 CHI, HCI, IJHCI, and IJHCS.
- 17. Results • Time Spent • Overall experiences
- 18. Total Time Spent Total Time Spent (hours) Frequency 0 5 10 15 20 051015 μ = 6.77 h σ = 3.11 h
- 19. Average Time Spent Per Paper μ = 4.3 min σ = 1.8 min Average Time Spent Per Paper (minutes) Frequency 0 2 4 6 8 10 12 14 05101520
- 20. Overall Impression • Open coding • Avoid codify any general remarks (e.g., “It was a good experience.”)
- 21. It’s so tiring N = 26 • At first I did not mind the reading at all, but I definitely became less interested as time went on. (P2012-3) • This was a pretty tiring experience. I found myself taking multiple breaks in order to regain focus and energy [...] (P2012-9) • Interesting idea, but I have done this before for my lit. review for my masters; Too many papers. The exercise became tedious after a while; [...] (P2011-8)
- 22. But, it gets better. N = 13 • Well, I need to say that this hw is kind of tough experience for me especially in the beginning. I spent much more time than I expected. But the more I did, the faster I read. [...] (P2009-4) • I have never read so many papers. Its definitely a big challenge for me. At the beginning, its hard for me to skim each paper in 4 minutes. However, I think I gradually know how to get the core idea of the paper in 4 minutes. [...] (P2010-3)
- 23. It turned out good. N = 44 big picture or trend (N = 20) useful for future research (N = 15) quite enjoyable (N = 15)
- 24. Big Picture (N = 20) • At the beginning, I thought the task was impossible to complete in such a short time. I had to force myself to stop reading within 4 minutes. Gradually I became comfortable with this kind of quick review method. I began to enjoy reviewing so many interesting topics and was surprised by the diversity of the research. This activity helped me get familiar with the HCI area. Some research topics and design were pretty insightful. (P2010-2)
- 25. Useful for Future Research (N = 15) • I think the overall experience of this project was good because it helped me learn how to quickly understand an area of research and to survey different types of research ideas within a particular field. I think I will definitely use this approach in my own research studies to especially before I write a literature review. I also really liked learning how to use Zotero and I think that is a great app to write papers with. (P2013-4)
- 26. Quite enjoyable (N = 15) • I have to say it is unexpectedly pleasant process to do the quick and dirty review though I’ve exposed to positive comments from former students. […] (P2011-13)
- 27. Quite enjoyable (N = 15) • The overall experience of the QnD review is surprisingly fantastic. […] I think it is because I skimmed the articles so quickly that only essence of each paper remained and hence the review process tended to yield understandings of breadth rather than depth. […] (P2010-10)
- 28. Some Observations • Students were able to do it. • Students are different. • For some, it is quite painful. • e.g., a student spent 20 hours. • For others, it is enjoyable. • Why? • Language? • Personality? (N vs. S in MBTI)
- 29. Some Observations • Pre-selected vs. self-selected • CHI vs. other journals • 50 vs. 100 vs. 120? • QnDReview should not replace comprehensive reading. • One highly rated paper in QnDReview turned out to be quite poor.
- 30. Limitations • It’s not a controlled study. • The sample size is small. • Students’ self reports could be biased. • They don’t want to offend me.
- 31. Future Work • How to understand overall trends? • How to generate new ideas from literature? • How to read? vs. What to read?
- 32. Acknowledgements • I appreciate all the students who have participated in this homework, and I also thank Aditya Srinath and Sung-Hee Kim for reviewing this paper and provided constructive feedback. In addition, I appreciate Dr. V. Kathlene Leonard for giving me an assignment of reviewing all the papers in Human Factors.
- 33. CHI 1982 75 CHI 1983 59 CHI 1985 35 CHI 1986 47 CHI 1987 46 CHI 1988 39 CHI 1989 54 CHI 1990 47 CHI 1991 56 CHI 1992 67 CHI 1993 62 CHI 1994 70 CHI 1995 66 CHI 1996 55 CHI 1997 55 CHI 1998 81 CHI 1999 78 CHI 2000 72 CHI 2001 69 CHI 2002 61 CHI 2003 75 CHI 2004 93 CHI 2005 93 CHI 2006 151 CHI 2007 182 CHI 2008 157 CHI 2009 277 CHI 2010 302 CHI 2011 410 CHI 2013 392 3,326 CHI papers × 1-2 hours / paper ÷ 8 hours / day = 416 - 832 days

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment