QnDReview:
Read 100 CHI Papers in 7 Hours
Ji Soo Yi
HIVE Lab
Industrial Engineering, Purdue University
717AB, 9am, 4/30/20...
Disclaimer
• Comprehensive reading a research paper
requires lots of effort / time and sufficient
background, so this talk...
A Story in 2004
“Hey, Ji Soo, for the book chapter that we
are working, can you survey what kinds of
research methods were...
Pleasantly Surprised!
I was able to
• skim through papers quicker and quicker;
• see the overall trends in the field;
• ge...
In 2009, I started to teach a graduate-level
HCI methodology course.
What if
What if graduate students
are forced
What if graduate students
are forced asked
What if graduate students
are forced asked to read
100 papers?
Quick and Dirty Review
QnDReview
Homework
1. Select 100 papers to read.
Visit the ACM Digital Library to search for any papers that are
seemingly interesti...
Homework
2. Review them all very quickly.
Use the template to make your own summary review. An Excel spread
sheet for the ...
Homework
3. Email your summary
Email the complete review spreadsheet to the instructor by the due
date with the following ...
Participants
• N = 57 over 5 years
• 27 F + 30 M
• 2 U + 21 MS + 34 PhD
• 11 non-IE1 + 46 IE
• IRBs
• 1012010281 (retrospe...
Some Variations
2009: self-selected 100 HCI-related papers1
2010: preselected 50 CHI 2010 papers
2011: preselected 100 CHI...
Results
• Time Spent
• Overall experiences
Total Time Spent
Total Time Spent (hours)
Frequency
0 5 10 15 20
051015
μ = 6.77 h
σ = 3.11 h
Average Time Spent Per Paper
μ = 4.3 min
σ = 1.8 min
Average Time Spent Per Paper (minutes)
Frequency
0 2 4 6 8 10 12 14
0...
Overall Impression
• Open coding
• Avoid codify any general remarks (e.g., “It
was a good experience.”)
It’s so tiring
N = 26
• At first I did not mind the reading at all, but I
definitely became less interested as time
went o...
But, it gets better.
N = 13
• Well, I need to say that this hw is kind of tough
experience for me especially in the beginn...
It turned out good.
N = 44
big picture or trend (N = 20)
useful for future research (N = 15)
quite enjoyable (N = 15)
Big Picture (N = 20)
• At the beginning, I thought the task was
impossible to complete in such a short time. I
had to forc...
Useful for Future Research (N = 15)
• I think the overall experience of this project
was good because it helped me learn h...
Quite enjoyable (N = 15)
• I have to say it is unexpectedly pleasant
process to do the quick and dirty review
though I’ve ...
Quite enjoyable (N = 15)
• The overall experience of the QnD review is
surprisingly fantastic. […] I think it is because
I...
Some Observations
• Students were able to do it.
• Students are different.
• For some, it is quite painful.
• e.g., a stud...
Some Observations
• Pre-selected vs. self-selected
• CHI vs. other journals
• 50 vs. 100 vs. 120?
• QnDReview should not r...
Limitations
• It’s not a controlled study.
• The sample size is small.
• Students’ self reports could be biased.
• They do...
Future Work
• How to understand overall trends?
• How to generate new ideas from
literature?
• How to read? vs. What to re...
Acknowledgements
• I appreciate all the students who have
participated in this homework, and I also
thank Aditya Srinath a...
CHI 1982 75
CHI 1983 59
CHI 1985 35
CHI 1986 47
CHI 1987 46
CHI 1988 39
CHI 1989 54
CHI 1990 47
CHI 1991 56
CHI 1992 67
CH...
QnDReview: Read 100 CHI Papers in 7 Hours
QnDReview: Read 100 CHI Papers in 7 Hours
QnDReview: Read 100 CHI Papers in 7 Hours
QnDReview: Read 100 CHI Papers in 7 Hours
QnDReview: Read 100 CHI Papers in 7 Hours
Upcoming SlideShare
Loading in …5
×

QnDReview: Read 100 CHI Papers in 7 Hours

1,174 views

Published on

The slides were used when I present an alt.chi paper, title "QnDReview: Read 100 CHI Papers in 7 Hours" in CHI 2014. The contents will show what kinds of benefits that you will have if you are forced to read papers in a short period of time. More details could be found in the following URL: http://dl.acm.org/citation.cfm?id=2578884

Abstract: In 2013, 392 research papers and notes were published in the CHI conference (The ACM CHI Conference on Human Factors in Computing Systems) and even more papers in the domain of Human-Computer Interaction (HCI) are constantly published in various conferences and journals. It is quite arduous, if not impossible, to read all of these papers. One approach to deal with this information deluge is to focus on skimming through lots of papers in a short period of time, so that one can more wisely choose what to read before investing time in them. In order to teach such a skimming technique, I have taught a technique, called "Quick and Dirty Review (QnDReview)," in a graduate-level HCI course. The method has been employed in the course for five semesters, and students' responses were collected and analyzed. Results showed that students spent, on average, 4.3 minutes per paper and believed that they got the gist of each paper. However, the largest benefit I noticed is that students get the overall pictures of the fields while exposing themselves to various new ideas through this approach.

Published in: Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,174
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
34
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

QnDReview: Read 100 CHI Papers in 7 Hours

  1. 1. QnDReview: Read 100 CHI Papers in 7 Hours Ji Soo Yi HIVE Lab Industrial Engineering, Purdue University 717AB, 9am, 4/30/2014
  2. 2. Disclaimer • Comprehensive reading a research paper requires lots of effort / time and sufficient background, so this talk is not about making you have magical, comprehensive understanding of 100 papers in 7 hours. • I also have not created any fancy tools.
  3. 3. A Story in 2004 “Hey, Ji Soo, for the book chapter that we are working, can you survey what kinds of research methods were used in the papers of two journals, Ergonomics and Human Factors in the past 5 years?”
  4. 4. Pleasantly Surprised! I was able to • skim through papers quicker and quicker; • see the overall trends in the field; • generate interesting ideas; and • actually enjoy the process. Leonard, V. K., Jacko, J. A., Yi, J. S., and Sainfort, F. Human factors and ergonomic methods. In Handbook of Human Factors and Ergonomics, G. Salvendy, Ed., 3rd ed. John Wiley & Sons, Feb. 2006, 292–321.
  5. 5. In 2009, I started to teach a graduate-level HCI methodology course.
  6. 6. What if
  7. 7. What if graduate students are forced
  8. 8. What if graduate students are forced asked
  9. 9. What if graduate students are forced asked to read 100 papers?
  10. 10. Quick and Dirty Review
  11. 11. QnDReview
  12. 12. Homework 1. Select 100 papers to read. Visit the ACM Digital Library to search for any papers that are seemingly interesting to you. I recommend you to start with CHI full papers because CHI is known to be the premier venue. ver. 2013
  13. 13. Homework 2. Review them all very quickly. Use the template to make your own summary review. An Excel spread sheet for the quick and dirty review should contain the following information: Column 1: Paper citation Column 2: Interest rating: 1 (Worst) - 2 (Bad) - 3 (Neither good nor bad) - 4 (Good) - 5 (Best) Column 3: A single sentence that describes the core idea of the paper. Column 4: (Optional) Any new ideas that you have after skimming through this paper. ver. 2013
  14. 14. Homework 3. Email your summary Email the complete review spreadsheet to the instructor by the due date with the following information in the body of the email. • What is the overall experience? • What is the notable trends in the papers? • Which are the two most interesting papers you would like to read more deeply? (Provide the full citation with DOI number, so that I can find the paper later) • How much time did you spend on this homework? • What are the most difficult part of this homework? • How to make the quick and dirty review better? ver. 2013
  15. 15. Participants • N = 57 over 5 years • 27 F + 30 M • 2 U + 21 MS + 34 PhD • 11 non-IE1 + 46 IE • IRBs • 1012010281 (retrospective one) • 1302013324 1 Engineering Education and Electrical and Computer Engineering.
  16. 16. Some Variations 2009: self-selected 100 HCI-related papers1 2010: preselected 50 CHI 2010 papers 2011: preselected 100 CHI 2011 papers 2012: preselected 120 papers • 30 from CHI 2012 • 30 from IEEE TVCG • 30 from Decision Support Systems 2013: self-selected 100 CHI 2013 papers 1 CHI, HCI, IJHCI, and IJHCS.
  17. 17. Results • Time Spent • Overall experiences
  18. 18. Total Time Spent Total Time Spent (hours) Frequency 0 5 10 15 20 051015 μ = 6.77 h σ = 3.11 h
  19. 19. Average Time Spent Per Paper μ = 4.3 min σ = 1.8 min Average Time Spent Per Paper (minutes) Frequency 0 2 4 6 8 10 12 14 05101520
  20. 20. Overall Impression • Open coding • Avoid codify any general remarks (e.g., “It was a good experience.”)
  21. 21. It’s so tiring N = 26 • At first I did not mind the reading at all, but I definitely became less interested as time went on. (P2012-3) • This was a pretty tiring experience. I found myself taking multiple breaks in order to regain focus and energy [...] (P2012-9) • Interesting idea, but I have done this before for my lit. review for my masters; Too many papers. The exercise became tedious after a while; [...] (P2011-8)
  22. 22. But, it gets better. N = 13 • Well, I need to say that this hw is kind of tough experience for me especially in the beginning. I spent much more time than I expected. But the more I did, the faster I read. [...] (P2009-4) • I have never read so many papers. Its definitely a big challenge for me. At the beginning, its hard for me to skim each paper in 4 minutes. However, I think I gradually know how to get the core idea of the paper in 4 minutes. [...] (P2010-3)
  23. 23. It turned out good. N = 44 big picture or trend (N = 20) useful for future research (N = 15) quite enjoyable (N = 15)
  24. 24. Big Picture (N = 20) • At the beginning, I thought the task was impossible to complete in such a short time. I had to force myself to stop reading within 4 minutes. Gradually I became comfortable with this kind of quick review method. I began to enjoy reviewing so many interesting topics and was surprised by the diversity of the research. This activity helped me get familiar with the HCI area. Some research topics and design were pretty insightful. (P2010-2)
  25. 25. Useful for Future Research (N = 15) • I think the overall experience of this project was good because it helped me learn how to quickly understand an area of research and to survey different types of research ideas within a particular field. I think I will definitely use this approach in my own research studies to especially before I write a literature review. I also really liked learning how to use Zotero and I think that is a great app to write papers with. (P2013-4)
  26. 26. Quite enjoyable (N = 15) • I have to say it is unexpectedly pleasant process to do the quick and dirty review though I’ve exposed to positive comments from former students. […] (P2011-13)
  27. 27. Quite enjoyable (N = 15) • The overall experience of the QnD review is surprisingly fantastic. […] I think it is because I skimmed the articles so quickly that only essence of each paper remained and hence the review process tended to yield understandings of breadth rather than depth. […] (P2010-10)
  28. 28. Some Observations • Students were able to do it. • Students are different. • For some, it is quite painful. • e.g., a student spent 20 hours. • For others, it is enjoyable. • Why? • Language? • Personality? (N vs. S in MBTI)
  29. 29. Some Observations • Pre-selected vs. self-selected • CHI vs. other journals • 50 vs. 100 vs. 120? • QnDReview should not replace comprehensive reading. • One highly rated paper in QnDReview turned out to be quite poor.
  30. 30. Limitations • It’s not a controlled study. • The sample size is small. • Students’ self reports could be biased. • They don’t want to offend me.
  31. 31. Future Work • How to understand overall trends? • How to generate new ideas from literature? • How to read? vs. What to read?
  32. 32. Acknowledgements • I appreciate all the students who have participated in this homework, and I also thank Aditya Srinath and Sung-Hee Kim for reviewing this paper and provided constructive feedback. In addition, I appreciate Dr. V. Kathlene Leonard for giving me an assignment of reviewing all the papers in Human Factors.
  33. 33. CHI 1982 75 CHI 1983 59 CHI 1985 35 CHI 1986 47 CHI 1987 46 CHI 1988 39 CHI 1989 54 CHI 1990 47 CHI 1991 56 CHI 1992 67 CHI 1993 62 CHI 1994 70 CHI 1995 66 CHI 1996 55 CHI 1997 55 CHI 1998 81 CHI 1999 78 CHI 2000 72 CHI 2001 69 CHI 2002 61 CHI 2003 75 CHI 2004 93 CHI 2005 93 CHI 2006 151 CHI 2007 182 CHI 2008 157 CHI 2009 277 CHI 2010 302 CHI 2011 410 CHI 2013 392 3,326 CHI papers × 1-2 hours / paper ÷ 8 hours / day = 416 - 832 days

×