![Students with novel answers
sometimes penalized unfairly
“damn peer review - it was a bunch of
[students] just making things fit into a rubric -
checking off a check sheet - like talking about
dog poop. what is this world coming to?”
-A student in a peer-assessed class](https://image.slidesharecdn.com/klemmer-presentation-150330115127-conversion-gate01/85/Design-at-Large-Integrating-Teaching-and-Experiments-Online-featuring-Scott-Klemmer-34-320.jpg)
![Why diversity?
Different professional
knowledge, educational
systems, and cultural
values
Information
[Tudge ’08]
Cognition
[Gurin et al. ’02]
[Nemeth ’86]
[Schwartz et al ’04]
From passivity to active,
effortful, conscious
thinking](https://image.slidesharecdn.com/klemmer-presentation-150330115127-conversion-gate01/85/Design-at-Large-Integrating-Teaching-and-Experiments-Online-featuring-Scott-Klemmer-45-320.jpg)
Uploaded byCenter for Online Innovation in Learning
Design at Large: Integrating Teaching and Experiments Online featuring Scott Klemmer
The document discusses the challenges and potential of scaling design education, emphasizing the need for practical theory and empirical approaches. It explores the concepts of peer assessment, diversity in learning, and the role of prototyping in enhancing creative outcomes. Various studies and examples illustrate how engagement and immediate feedback can improve learning experiences in large-scale settings.
More Related Content
Similar to Design at Large: Integrating Teaching and Experiments Online featuring Scott Klemmer
Design at Large: Integrating Teaching and Experiments Online featuring Scott Klemmer
- 1. DESIGN AT LARGE Scalingthe Studio and the Lab to the Globe Scott Klemmer
- 2. from the Labto the Wild
- 3.
- 4. …but the failurerate is high. The challenge: Design is often faith-based rather than research-based.
- 5. “Nothing is as practicalas a good theory” –Kurt Lewin
- 6. Norman & Klemmer(2014) How design education must change Why a shortfall of design principles? • Engineering excels at practical theory …from the physical sciences. • The human world is different • Introspection is valuable …but often misleading • Industry is empirical …but product focused
- 7. www.solveforx.com • Build practicaltheory with real-world experiments • Bake that theory into software that transforms <X> DESIGN AT LARGE
- 8. Is this possible? Klemmer& Carroll (2014) HCI Special Issue: Understanding Design Thinking
- 9. “There are norules of composition in photography, there are only good photographs” -Ansel Adams
- 11. Smith et al.1993 Examples can increase conformity...
- 12. Will nothing new everbe created?
- 13.
- 14. —E.W. Dijkstra, Onthe Cruelty of Really Teaching Computer Science Just for Small Innovations? “By ... metaphors and analogies we try to link the new to the old, the novel to the familiar. Under sufficiently slow and gradual change, it works reasonably well; in the case of a sharp discontinuity, however, the method breaks down ... our past experience is no longer relevant, the analogies become too shallow, and the metaphors become more misleading than illuminating. This is the situation ... for radical novelty.”
- 15. Les Demoiselles d'Avignon JohnRichardson, A Life of Picasso:The Cubist Rebel, 1907-1916
- 17. “Good artists borrow,great artists steal” —Pablo Picasso 19th century Fang sculptureLes Demoiselles d'Avignon John Richardson, A Life of Picasso:The Cubist Rebel, 1907-1916
- 18. The Power ofStudio Learning
- 19. Design Learning atLarge Chinmay Kulkarni et al. Peer and Self Assessment in Massive Online Classes, Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, Scott R. Klemmer. TOCHI: ACM Transactions on Computer-Human Interaction, 2013 The identify-verify pattern scales short-answer grading by combining peer assessment with algorithmic scoring, Chinmay Kulkarni, Richard Socher, Michael S. Bernstein, Scott R. Klemmer. Learning at Scale, 2014 Talkabout: Making distance matter with small groups in massive classes, Chinmay Kulkarni, Julia Cambre, Yasmine Kotturi, Michael S. Bernstein, Scott Klemmer, CSCW: ACM Conference on Computer Supported Cooperative Work, 2015 Structure and messaging techniques for online peer learning systems that increase stickiness, Yasmine Kotturi, Chinmay Kulkarni, Michael Bernstein, Scott Klemmer, ACM Learning at Scale, 2015 PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance, Chinmay Kulkarni, Michael S Bernstein, Scott R Klemmer, ACM Learning at Scale, 2015
- 20.
- 21. Beyond Being There Hollan,Jim, and Scott Stornetta. "Beyond being there.”ACM, 1992.
- 22. 3 ingredients centralto learning, but hard to scale 22
- 23. 1. Feedback onopen-ended work Schön, D. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions.
- 24. 2. Engaging DiversePerspectives Model UN Design Crit Gurin, P. et al. (2002) Diversity and higher education: Theory and impact on educational outcomes, Harvard Educational Review
- 25. 3. Revision formastery Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Image Courtesy IDEO
- 26. Global peer assessment Interfacesfor accurate assessment of open- ended work Talkabout Diversity as a design opportunity: Small group discussions in massive classes PeerStudio Scale as a design opportunity: Immediate feedback for mastery
- 27. The paradox ofpeer processes Non-experts performing expert work
- 28. Our approach: Calibrated peerreview Chinmay Kulkarni, et al. Peer and Self Assessment in Massive Online Classes, TOCHI 2013 3) Reflect (Assess: Self) 2) Assess: Peers1) Train: calibrate ✓
- 29.
- 30. Large scale peerassessment Human-computer Interaction Design Teaching character Management Constitutional law Arguments Introduction to Philosophy Essays Social Psychology Essays Programming in Python Code Child Nutrition Recipes World Music Music used by 100,000+ students
- 31. Assessment training iscrucial 0 25 50 75 100 0 25 50 75 100 Self grade (%) Peergrade(%) No Training r=0.58 0 20 40 60 80 100 0 20 40 60 80 100 Self grade (%) Peergrade(%) With Training r=0.73
- 32. How well dopeer and staff assessments correlate? 3) Reflect (Assess: Self) 2) Assess: Peers staff-graded 1) Assess: calibrate ✓ Dataset: 99 submissions with ~160 peer assessments each.
- 33. Grading for apass-fail class Extrapolated results from a bootstrapped simulation Earn certificate if staff-graded Certificate awarded No certificate 97.8% 2.2% No certificate Certificate awarded 99.3% 0.7% No certificate if staff-graded
- 34. Students with novelanswers sometimes penalized unfairly “damn peer review - it was a bunch of [students] just making things fit into a rubric - checking off a check sheet - like talking about dog poop. what is this world coming to?” -A student in a peer-assessed class
- 35. “I've never seensomething like that!” Introduction to Art “Treasure Cage” from Canada“Magical lights” from Norway
- 36. The return ofthe novices-as-experts paradox “fully interactive, page flow is complete… make it clearer what people should do next” Experts: capture the structure of rubric Peers: Focus on superficial features, even when asked not to “unpolished…Try to make UI less coloured.”
- 37. Fortune cookies forqualitative, personalized feedback • Peers can recognize errors from a list of patterns, even if they can’t articulate them • Most errors are variations on a theme + “...because _____________________” Cue Variation
- 38. Students Made itTheirs • Sharing cool interfaces, resources, articles • Collating reading lists, creating assignment aids • Doing really creative work • Helping other students • heuristic evaluation feedback • answering forum questions • extra peer assessment
- 43. I am ChandramouliSharma, a junior year undergraduate in Computer Science from the National Institute of Technology Karnataka, India. I am one of those thousands of students who took the HCI class on Coursera in October 2012. I had timing clashes, so I had to finish the course during December in vacations. Here is my amazing journey from a small project in HCI class to a platform that will now be used by thousands of schools in 47 countries and the awards I won along the way. I have illustrated it through pictures. This is a tribute to you for the great you class took at no fee. Note: Images might take some time to load. What I worked on.. I worked on a web application which could display complex environmental pollution data sets into interactive visualizations. This could be used by school students to understand environmental issues. Below is one of the paper prototypes that I developed during the class. After a few iterations I came up with a digital prototype. It looked something like this.
- 44. Global peer assessment Interfacesfor accurate assessment of open- ended work Talkabout Diversity as a design opportunity: Small group discussions in massive classes PeerStudio Scale as a design opportunity: Immediate feedback for mastery
- 45. Why diversity? Different professional knowledge,educational systems, and cultural values Information [Tudge ’08] Cognition [Gurin et al. ’02] [Nemeth ’86] [Schwartz et al ’04] From passivity to active, effortful, conscious thinking
- 46. Students are oftenhomophilic Hurtado, S. et al. (1998) The Climate for Diversity: Key Issues for Institutional Self-Study.
- 47. Talkabout: video discussions withglobal peers Kulkarni, C, et. al. “Talkabout: Making distance matter with small groups in massive classes”, CSCW 2015
- 48. Group assignment algorithm •Talkabout assigns to one of many parallel groups. • Assignment is greedy, constrained by preferred group size • balances gender • improves geographic diversity
- 49.
- 50. Discussants as farapart as New York and London Median pair-wise distance 4,100 mi (6,600 km) 0 50 100 2 3 4 5 6 Number of countries in discussion NumberofDiscussions
- 51. Students discuss twiceas long as instructors asked them to Discussion duration (minutes) Number of Students Median duration 0 100 200 30 60 90 120 150 180 Recommended duration
- 52. Do diverse, small-groupdiscussions improve learning outcomes? 1. Does participation help? 2. Does diversity amplify participation benefits? IRB #30319
- 53. Study: Benefits of Participation •n=934, Irrational Behavior • Dependent measure: total course grade (%) • Between-subjects Wait list No talkabout for first half of class Discussion Talkabout throughout class
- 54. Course grades higherin discussion condition Irrational Behavior (p<0.05) Total grade (%) 0 10 20 30 40 50 Discussion Wait list (control) 6% of total grade
- 55. Do diverse, small-groupdiscussions improve learning outcomes? 1. Does participation help? 2. Does diversity amplify participation benefits?
- 56. Study: Benefits ofDiversity • n=2,422, Social Psychology • Quasi-experiment: discussants assigned to first available group • Result: natural variation in diversity • Measure: performance on final exam • OLS regression controls for prior performance
- 57. Diverse discussions leadto higher final scores 0% 2.5% 5% Social Psychology Organizational Analysis 3.6% 2.4% Grade difference (most-least diverse) ior
- 58. Evaluation Goals Do diverse,small-group discussions improve learning outcomes? 1. Does participation help? 2. Does diversity amplify participation benefits?
- 59. Still a longway to go
- 60. Talkabout as aspringboard to global friendships “We shared emails because we are discussing issues that require a strong, networked group to change the status quo… the impact would be far greater if participants could connect and engage outside of the course” -Student in International Women’s Health and Human Rights class Average (9 classes) International Women’s Health and Human Rights 0% 25% 50% 75% 100% 92% 47.2% “Shared contact info with group”
- 61. 5,000+ students from134 countries Social Psychology International Women’s Health & Human Rights Learning How to Learn How to Change the World Understanding Research Methods Irrational Behavior Critical Perspectives on Management Organizational Analysis Think Again: How to Reason and Argue translated by students into French & Spanish
- 62. Global peer assessment Interfacesfor accurate assessment of open- ended work Talkabout Diversity as a design opportunity: Small group discussions in massive classes PeerStudio Scale as a design opportunity: Immediate feedback for mastery
- 63. PeerStudio scales interactive peerfeedback Kulkarni C., Bernstein M., Klemmer S. (2015) “PeerStudio: Rapid Feedback Emphasizes Revision and Improves Performance”, Learning@Scale Submit for feedback Give feedback to two peers Submit for grades Read feedback & revise
- 64. How might welower the training burden? 0.0 2.5 5.0 7.5 10.0 Evaluation Submission 10.5 hours 1.9 hours Training Creating own work Median hours in activity Training 1.9 hours
- 65. Solution: contrasting casesfor training-free micro-expertise Thompson, Gentner, Loewenstein (2000), “Analogical Training More Powerful Than Individual Case Training” Average Peer-majority/Staff difference: 5.7%
- 66. Time to firstfeedback: Learning How to Learn 0 50 100 <10 min <1 hr <2 hr <6 hr <12 hr <24 hr > 24 hr Time to first review Numberofsubmissions native the plot
- 67. Problem: Accurate feedback isnot always actionable
- 68. Solution: Real-time tipsfor actionable feedback • Correctness and velocity feedback leads to large improvements • Specific, topic-relevant feedback more useful • Logistic regression with bag-of-words features predicts relevance
- 69. Solution: Real-time tipsfor actionable feedback 1 Calculate an internal score for each rubric dimension 2 Generate tips for reviewer Overall, 81% of students received actionable comments
- 70. Without hints, studentsfocus on author, and what’s good I think you are, I wish you, I hope you… With hints, students focus on work and what could be better I think you should, you need to, your work could…
- 71. N=104 in “MedicalEducation in the New Millennium” (edX) Study: Does fast feedback improve final performance? Early feedback, fast (<1 hr) grades 4.4% higher than No early feedback Early feedback, delayed 24 hours No early feedback grades same as
- 72. Global peer assessment Interfacesfor accurate assessment of open- ended work Talkabout Diversity as a design opportunity: Small group discussions in massive classes PeerStudio Scale as a design opportunity: Immediate feedback for mastery
- 73. • Build practicaltheory with real-world experiments • Bake pedagogy into software that transforms learning SCALING THE STUDIO http://d.ucsd.edu/peer
- 74. This is amultidisciplinary effort.
- 75.
- 76. Pasteur’s Quadrant fundamentalunderstanding practical impact Stokes(1997) Pasteur's Quadrant: Basic Science and Technological Innovation
- 77. The Design Labcreates postcards from the future
- 78. scale personalized mastery-learning experiences? Howmight we… http://d.ucsd.edu/peer
- 79. Let’s match thisenthusiasm with insight Be the thermostat, not the thermometer
- 80.
- 81. Thomke (2000) Experimentationmatters: unlocking the potential of new technologies for innovation Learning through Prototyping “Never go to a meeting without a prototype...” —Boyle’s Law
- 82. Design Process atLarge Steven Dow Asst Prof, CMU Early and Repeated Exposure to Examples Improves Creative Work, Chinmay Kulkarni, Steven P Dow, Scott R Klemmer. Cognitive Science, 2012. Prototyping Dynamics: Sharing Multiple Designs Improves Exploration, Group Rapport, and Results, Steven P Dow, Julie Fortuna, Dan Schwartz, Beth Altringer, Daniel L Schwartz, Scott R Klemmer. CHI: ACM Conference on Human Factors in Computing Systems, 2011. Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-Efficacy, Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel Schwartz, Scott R Klemmer. ACM Transactions on Computer-Human Interaction, 2010 The Efficacy of Prototyping Under Time Constraints, Steven P. Dow, Kate Heddleston, Scott R Klemmer. Creativity & Cognition, 2009
- 84. “I went withthe whole parachute idea and what I had from the beginning...” “This is the best approach for such a design...” “I am not a very good outside-the-box thinker, so I kinda just had one idea and I was going to try to make it work...” “No... for some reason... this seems to be the only idea. There needs to be a platform and then as good of cushion as possible... I don’t see any other way.” Participants picked their concept early
- 85.
- 86. Can process offer afixation antidote? Prototype Prototype Prototype SERIAL DESIGN AT LARGE Feedback Feedback
- 87. Web-scale experiments as a researchplatform DESIGN AT LARGE
- 88. Task: Design aWeb Ad (N=33) parallel prototyping condition FINAL serial prototyping condition
- 90. Parallel design ->more clicks Parallel Clicks per million impressions Serial F(1,30)=4.227 p<.05 0 60 120 180 240 300 360 420 480 398 445
- 91. ...and more timeon the site Parallel condition Average time on client site per visitor (seconds) Serial condition F(1,493)=3.172 p=0.076 0 5 10 15 20 25 30 35 40 12.9 31.3
- 92. ...and higher expertratings Parallel condition Likert-scale rating (0-50) Serial condition F(1,5)=7.948 p<0.05 0 4 8 12 16 20 24 28 21.7 24.4
- 93. ...and more diversedesigns Parallel Serial F=182, p<0.001 0 0.5 1 1.5 2 2.5 3 3.5 3.18 2.78 7=highly similar 0=not at all similar
- 94. Gentner, Loewenstein, &Thomson, 2003 learning outcome Comparison aids learning training session “Describe the solution.” CASE#1 CASE#2 CASE#1 CASE#2 “Describe the parallels of these solutions” “Describe the solution.” SEPARATE CASES COMPARISON CASES Solutions to a landlord-renter lease ~ 3x
- 95. Sharing Multiple Benefits •User engagement • Expert rating • Individual exploration • Feature sharing • Conversational turns • Consensus • Rapport Share Multiple clicks/M 0 250 500 750 1000 1250 774.6734.9 1072.1 Share Best Share One χ2=4.72, p<0.05 self other self other … … self other