Critique is the primary method of assessment used in design education, yet is not well understood apart from traditional structures of institutional power and faculty initiation. In this study, we analyze the classroom presentations and critiques of eleven teams in a design-focused human-computer interaction graduate program, focusing on an emergent instructional design for technologically-mediated critique created by experienced students serving as peer mentors. Initial analysis suggests complex interaction between multiple modes of critique beyond the “traditional” critique: 1) public oral critique led by faculty, 2) a critique document authored in Google Docs by experienced students, and 3) backchannel chat in Google Docs by experienced students. These interactions indicate instructional affordances for including many simultaneous users within an existing critique infrastructure. Implications of this instructional design for expanding the capacity of physical critique events and the role of participation in student learning are considered.
Student-Generated Critique in Physical and Digital Spaces
1. INVERTING CRITIQUE:
EMERGENT TECHNOLOGICALLY-MEDIATED CRITIQUE
PRACTICES OF DEVELOPING DESIGN STUDENTS
COLIN M. GRAY
Purdue University
CRAIG D. HOWARD
University of Tennessee—Knoxville
2. Gray, C. M., & Howard, C. D. (2015, June). “Why are
they not responding to critique?”: A student-centered
construction of the crit. In LearnxDesign: The 3rd
International Conference for Design Education
Researchers and PreK-16 Design Educators.
Chicago, IL: School of the Art Institute of Chicago.
PRIOR WORK
3. CRITIQUE
is central to design education
[Anthony, 1991; Dannels & Martin, 2008; Klebesadel &
Kornetsky, 2009; Parnell, Sara, Doidge, & Parsons,
2012; Schön, 1987; Shulman, 2005]
5. [Anthony, 1991; Blythman, Orr, & Blair,
2007; Dutton, 1991; Gray, 2013, 2014;
Gray & Howard 2014; Webster, 2007;
Willenbrock, 1991]
CRITICAL
FRAMING
what if we looked at
critique in a
6. [Anthony, 1991; Blythman, Orr, & Blair,
2007; Dutton, 1991; Gray, 2013, 2014;
Gray & Howard 2014; Webster, 2007;
Willenbrock, 1991]
CRITICAL
FRAMING
what if we looked at
critique in a
contextualized
with
PEER ASSESSMENT,
DEVELOPMENT OF
DESIGN EXPERTISE &
IDENTITY, CSCL, ETC.
[Easterday et al., 2014; Freeman &
McKenzie, 2001; Lawson & Dorst, 2009;
Schön, 1990; Smith, 2015; Topping, 1998;
Xu & Bailey, 2013]
8. CRITIQUE &
CRITICAL PEDAGOGY
FROM
student as tabula rasa
TO
student as capable,
emerging proto-professional
[Boling, Gray, & Smith, 2015;
Gray, 2014; Freire, 1970]
12. =
Physical Space
[~60 student capacity; 8 screens; decentralized layout]
Virtual Space
[Facebook; Google Docs; SMS]
+CLASSROOM
13. MULTIMODAL
CRITIQUE
multiple critique conversations occur
concurrently around a single designed
artifact (or presentation of that artifact), in
both physical and virtual modes, with
multiple classes or groupings of interlocutors
18. 1. What volume of critique did this instructional design support?
2. How did the volume of critique generated via digital means
compare to the volume of critique in the physical classroom?
3. What was the nature of critique content that the advanced
students generated?
RESEARCH QUESTIONS
19. What volume of critique did this
instructional design support?
1
20. Team Collaborative Google Doc (GD) Backchannel (BC)
#
Turns [CL*]
Average Words
per turn (SD) Total words
#
Turns
Average Words
per turn (SD) Total words
Team A 37 [10] 46.4054 (45.6369) 1717 13 5.1538 (4.4695) 67
Team B 26 [4] 25.8462 (21.6079) 672 69 7.3043 (5.7441) 504
Team C 38 [8] 37.1579 (28.9496) 1412 135 7.1852 (6.6968) 970
Team D 9 [1] 36.8889 (21.1332) 332 9 12.6667 (10.3923) 114
Team E 28 [5] 27.2857 (19.0879) 764 4 5.0000 (2.7386) 20
Team F 27 [1] 39.9259 (35.1683) 1078 116 10.3966 (9.1622) 1206
Team G 16 [1] 47.6875 (34.7234) 763 69 7.4058 (7.1431) 511
Team H 30 [4] 28.2667 (28.5668) 848 49 8.8367 (7.3105) 433
Team I 20 [1] 50.3000 (44.6946) 1006
Team J 19 [3] 31.6316 (20.0846) 601 60 10.6667 (8.3280) 640
Team K 16 [4] 50.4375 (49.0446) 807 61 6.8525 (5.7054) 418
TOTAL 266 [42] 37.594 10000 585 8.3470 4883
21. Team Collaborative Google Doc (GD) Backchannel (BC)
#
Turns [CL*]
Average Words
per turn (SD) Total words
#
Turns
Average Words
per turn (SD) Total words
Team A 37 [10] 46.4054 (45.6369) 1717 13 5.1538 (4.4695) 67
Team B 26 [4] 25.8462 (21.6079) 672 69 7.3043 (5.7441) 504
Team C 38 [8] 37.1579 (28.9496) 1412 135 7.1852 (6.6968) 970
Team D 9 [1] 36.8889 (21.1332) 332 9 12.6667 (10.3923) 114
Team E 28 [5] 27.2857 (19.0879) 764 4 5.0000 (2.7386) 20
Team F 27 [1] 39.9259 (35.1683) 1078 116 10.3966 (9.1622) 1206
Team G 16 [1] 47.6875 (34.7234) 763 69 7.4058 (7.1431) 511
Team H 30 [4] 28.2667 (28.5668) 848 49 8.8367 (7.3105) 433
Team I 20 [1] 50.3000 (44.6946) 1006
Team J 19 [3] 31.6316 (20.0846) 601 60 10.6667 (8.3280) 640
Team K 16 [4] 50.4375 (49.0446) 807 61 6.8525 (5.7054) 418
TOTAL 266 [42] 37.594 10000 585 8.3470 4883
Average Team Critique
8.0 interlocutors
(SD=3.16; min=3; max=14)
15.53 critique acts
(SD=11.51; min=1; max=34)
22. HOW MANY
INTERLOCUTORS
DO YOU NEED?
the top five commenters
accounted for 81.70% of all
unique critique acts in the
Google Docs
(top 3 = 60.71%)
23. How did the volume of critique
generated via digital means
compare to the volume of critique
in the physical classroom?
2
24. Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
25. Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
26. Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
Google Docs Critique Acts
224
(M=20.36; SD=6.62)
27. Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
M = 1 minute, 14 seconds
Google Docs Critique Acts
224
(M=20.36; SD=6.62)
28. Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
M = 1 minute, 14 seconds
Google Docs Critique Acts
224
(M=20.36; SD=6.62)
152.38% increase in
critique capacity
29. What was the nature of
critique content that the
advanced students generated?
3
31. Team C: “I think the “presentation preview” slide at the beginning was brilliant. You
could have very easily leveraged this to better include your research early on. Make
your research the equivalent of the “turn off your cell phones” announcement.
Whatever it may be, but briefly explaining that “the tech you are about to see is
fantastical, but be assured - it is feasible.” (TM: super awesome wicked great
suggestion)”
Team I: “Your presentation style (pace, confidence, etc.) is night and day between
the last time I saw you present. Nice job. (+1 UK, +1XE +1 DK)”
Team H: “Be careful about telling your stakeholders “I guess you guys aren’t aware
of this…” You might have a boss who gets upset at their assumed ignorance (+1
TQ, you don’t want to make them feel dumb or inadequate, you need buy in!)”
PRESENTATION CRITIQUE
32. Team C: "ok, textures? I am so confused, how on earth is this happening? Is this 3d glasses? I
may not have heard how this happens, but is this sort of like Disney that you have “robots” and
existing props that add this texture that you are talking about? ok, you talked about the
textured screens at the end. I would have liked to hear this as you go through your story
because it completely distracted me..."
Team F: “I’m concerned that your design is a bit magical. Your scenario is kind of perfect;
stressful person puts on magical jacket and suddenly everything is better! Stress and anxiety
doesn’t always arise, but can arise often from actual things going on in people’s life. Your jacket
doesn’t make less things due, or make people have less work, or make the trip suddenly over.
Just because you can pat people on the back doesn’t mean you’re going to make them less
stressed. (BX: I think of [other professor] and her issues with turbulence. She needs more than a
fake back pat)”
Team I: “The in-store has been handled decently enough, but my concerns are with the system.
How does this fit into my ecosystem? The stores? Food Companies? You say it is out of the
scope, but it is something that needs addressing as you are adding a brand new infrastructure
into stores.”
DESIGN CRITIQUE
33. NEGOTIATION
THROUGH
BACKCHANNELING
“unfortunately I don't think it was stellar because
since it was backwards, the whole time I kept thinking
NO FUCKING WAY. WTF ARE YOU GUYS DOING
ARE OYU ON CRACK?”
“The message should be about the presentation
order. and i think the issues with scalability or how it
doesnt quite fit the body data stuff.”
35. DISCUSSION
mentors demonstrated an ability to manage
anonymity and ownership effectively in digitally
mediated spaces
an expansion of critique allows for increased
interaction and opportunity for crosstalk, even when
the physical space does not allow for it
multimodal critique encouraged socialization around
design topics and negotiation of meaning—mirroring
the discursive quality of physically-mediated critique
mentors built skills as assessors and design
professionals, clarifying their concerns in order to
reach consensus