1.
INVERTING CRITIQUE:
EMERGENT TECHNOLOGICALLY-MEDIATED CRITIQUE
PRACTICES OF DEVELOPING DESIGN STUDENTS
COLIN M. GRAY
Purdue University
CRAIG D. HOWARD
University of Tennessee—Knoxville

2.
Gray, C. M., & Howard, C. D. (2015, June). “Why are
they not responding to critique?”: A student-centered
construction of the crit. In LearnxDesign: The 3rd
International Conference for Design Education
Researchers and PreK-16 Design Educators.
Chicago, IL: School of the Art Institute of Chicago.
PRIOR WORK

3.
CRITIQUE
is central to design education
[Anthony, 1991; Dannels & Martin, 2008; Klebesadel &
Kornetsky, 2009; Parnell, Sara, Doidge, & Parsons,
2012; Schön, 1987; Shulman, 2005]

4.
instructor-centric
curriculum-centric
power-laden
CRITIQUE
is central to design education
[Anthony, 1991; Dannels & Martin, 2008; Klebesadel &
Kornetsky, 2009; Parnell, Sara, Doidge, & Parsons,
2012; Schön, 1987; Shulman, 2005]

5.
[Anthony, 1991; Blythman, Orr, & Blair,
2007; Dutton, 1991; Gray, 2013, 2014;
Gray & Howard 2014; Webster, 2007;
Willenbrock, 1991]
CRITICAL
FRAMING
what if we looked at
critique in a

6.
[Anthony, 1991; Blythman, Orr, & Blair,
2007; Dutton, 1991; Gray, 2013, 2014;
Gray & Howard 2014; Webster, 2007;
Willenbrock, 1991]
CRITICAL
FRAMING
what if we looked at
critique in a
contextualized
with
PEER ASSESSMENT,
DEVELOPMENT OF
DESIGN EXPERTISE &
IDENTITY, CSCL, ETC.
[Easterday et al., 2014; Freeman &
McKenzie, 2001; Lawson & Dorst, 2009;
Schön, 1990; Smith, 2015; Topping, 1998;
Xu & Bailey, 2013]

7.
CRITIQUE &
CRITICAL PEDAGOGY

8.
CRITIQUE &
CRITICAL PEDAGOGY
FROM
student as tabula rasa
TO
student as capable,
emerging proto-professional
[Boling, Gray, & Smith, 2015;
Gray, 2014; Freire, 1970]

9.
CONTEXT

10.
CONTEXT
Human-Computer
Interaction Master’s
program
Introductory project-
based design course
Final project was
based on the 2014
ACM SIGCHI Student
Design Competition

11.
CLASSROOM

12.
=
Physical Space
[~60 student capacity; 8 screens; decentralized layout]
Virtual Space
[Facebook; Google Docs; SMS]
+CLASSROOM

13.
MULTIMODAL
CRITIQUE
multiple critique conversations occur
concurrently around a single designed
artifact (or presentation of that artifact), in
both physical and virtual modes, with
multiple classes or groupings of interlocutors

14.
FIRST-YEAR STUDENT
FIRST-YEAR STUDENT (PRESENTING)
INSTRUCTOR
PROJECTOR/SCREEN
WHITEBOARD
MENTOR (SECOND-YEAR STUDENT)

15.
LIVE MENTOR CRITIQUE IN GOOGLE DOC

16.
verbally
virtually
COMMUNICATION DURING
PRESENTATION
virtually
verbally
verbally
COMMUNICATION DURING
CRITIQUE
verbally
COMMUNICATION PATTERNS

17.
PRESENTATION STUDENT CRITIQUE
MENTOR
CRITIQUE
INSTRUCTOR
CRITIQUE
GOOGLE DOC
CRITIQUE STRUCTURE

18.
1. What volume of critique did this instructional design support?
2. How did the volume of critique generated via digital means
compare to the volume of critique in the physical classroom?
3. What was the nature of critique content that the advanced
students generated?
RESEARCH QUESTIONS

19.
What volume of critique did this
instructional design support?
1

20.
Team Collaborative Google Doc (GD) Backchannel (BC)
#
Turns [CL*]
Average Words
per turn (SD) Total words
#
Turns
Average Words
per turn (SD) Total words
Team A 37 [10] 46.4054 (45.6369) 1717 13 5.1538 (4.4695) 67
Team B 26 [4] 25.8462 (21.6079) 672 69 7.3043 (5.7441) 504
Team C 38 [8] 37.1579 (28.9496) 1412 135 7.1852 (6.6968) 970
Team D 9 [1] 36.8889 (21.1332) 332 9 12.6667 (10.3923) 114
Team E 28 [5] 27.2857 (19.0879) 764 4 5.0000 (2.7386) 20
Team F 27 [1] 39.9259 (35.1683) 1078 116 10.3966 (9.1622) 1206
Team G 16 [1] 47.6875 (34.7234) 763 69 7.4058 (7.1431) 511
Team H 30 [4] 28.2667 (28.5668) 848 49 8.8367 (7.3105) 433
Team I 20 [1] 50.3000 (44.6946) 1006
Team J 19 [3] 31.6316 (20.0846) 601 60 10.6667 (8.3280) 640
Team K 16 [4] 50.4375 (49.0446) 807 61 6.8525 (5.7054) 418
TOTAL 266 [42] 37.594 10000 585 8.3470 4883

21.
Team Collaborative Google Doc (GD) Backchannel (BC)
#
Turns [CL*]
Average Words
per turn (SD) Total words
#
Turns
Average Words
per turn (SD) Total words
Team A 37 [10] 46.4054 (45.6369) 1717 13 5.1538 (4.4695) 67
Team B 26 [4] 25.8462 (21.6079) 672 69 7.3043 (5.7441) 504
Team C 38 [8] 37.1579 (28.9496) 1412 135 7.1852 (6.6968) 970
Team D 9 [1] 36.8889 (21.1332) 332 9 12.6667 (10.3923) 114
Team E 28 [5] 27.2857 (19.0879) 764 4 5.0000 (2.7386) 20
Team F 27 [1] 39.9259 (35.1683) 1078 116 10.3966 (9.1622) 1206
Team G 16 [1] 47.6875 (34.7234) 763 69 7.4058 (7.1431) 511
Team H 30 [4] 28.2667 (28.5668) 848 49 8.8367 (7.3105) 433
Team I 20 [1] 50.3000 (44.6946) 1006
Team J 19 [3] 31.6316 (20.0846) 601 60 10.6667 (8.3280) 640
Team K 16 [4] 50.4375 (49.0446) 807 61 6.8525 (5.7054) 418
TOTAL 266 [42] 37.594 10000 585 8.3470 4883
Average Team Critique
8.0 interlocutors
(SD=3.16; min=3; max=14)
15.53 critique acts
(SD=11.51; min=1; max=34)

22.
HOW MANY
INTERLOCUTORS
DO YOU NEED?
the top five commenters
accounted for 81.70% of all
unique critique acts in the
Google Docs
(top 3 = 60.71%)

23.
How did the volume of critique
generated via digital means
compare to the volume of critique
in the physical classroom?
2

24.
Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)

25.
Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)

26.
Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
Google Docs Critique Acts
224
(M=20.36; SD=6.62)

27.
Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
M = 1 minute, 14 seconds
Google Docs Critique Acts
224
(M=20.36; SD=6.62)

28.
Team
Presentation
Length Critique Length
# CL
Critique Turns
(1Y, 2Y, Inst.)
Average CL
Critique Turn Length
Team A 00:05:35 0:23:54 20 (18/1/1)* 00:01:12
Team B 00:08:44 0:12:06 10 (8/1/1) 00:01:13
Team C 00:07:20 0:20:42 14 (12/1/1) 00:01:29
Team D 00:05:59 0:09:41 14 (8/3/3) 00:00:42
Team E 00:07:26 0:19:57 11 (6/2/3) 00:01:49
Team F 00:06:03 0:17:38 17 (15/1/1) 00:01:02
Team G 00:08:06 0:15:01 17 (11/3/3) 00:00:53
Team H 00:07:56 0:15:18 15 (10/2/3) 00:01:01
Team I 00:06:18 0:12:52 8 (6/1/1) 00:01:37
Team J 00:05:46 0:11:41 12 (9/2/1) 00:00:58
Team K 00:06:17 0:14:09 9 (7/1/1) 00:01:34
TOTAL
01:15:30
(M=00:06:52;
SD=00:01:05)
2:52:59
(M=00:15:44;
SD=00:04:22)
147
(M=13.36;
SD=3.57)
(M=00:01:14;
SD=00:00:20)
Classroom Critique Acts
147
(M=13.36; SD=3.57)
M = 1 minute, 14 seconds
Google Docs Critique Acts
224
(M=20.36; SD=6.62)
152.38% increase in
critique capacity

29.
What was the nature of
critique content that the
advanced students generated?
3

30.
GD CRITIQUE TURNS
M=37.59 words (SD=34.65)
135 design critiques (M=12.27; SD=5.56)
98 presentation critiques (M=8.91; SD=4.12)

31.
Team C: “I think the “presentation preview” slide at the beginning was brilliant. You
could have very easily leveraged this to better include your research early on. Make
your research the equivalent of the “turn off your cell phones” announcement.
Whatever it may be, but briefly explaining that “the tech you are about to see is
fantastical, but be assured - it is feasible.” (TM: super awesome wicked great
suggestion)”
Team I: “Your presentation style (pace, confidence, etc.) is night and day between
the last time I saw you present. Nice job. (+1 UK, +1XE +1 DK)”
Team H: “Be careful about telling your stakeholders “I guess you guys aren’t aware
of this…” You might have a boss who gets upset at their assumed ignorance (+1
TQ, you don’t want to make them feel dumb or inadequate, you need buy in!)”
PRESENTATION CRITIQUE

32.
Team C: "ok, textures? I am so confused, how on earth is this happening? Is this 3d glasses? I
may not have heard how this happens, but is this sort of like Disney that you have “robots” and
existing props that add this texture that you are talking about? ok, you talked about the
textured screens at the end. I would have liked to hear this as you go through your story
because it completely distracted me..."
Team F: “I’m concerned that your design is a bit magical. Your scenario is kind of perfect;
stressful person puts on magical jacket and suddenly everything is better! Stress and anxiety
doesn’t always arise, but can arise often from actual things going on in people’s life. Your jacket
doesn’t make less things due, or make people have less work, or make the trip suddenly over.
Just because you can pat people on the back doesn’t mean you’re going to make them less
stressed. (BX: I think of [other professor] and her issues with turbulence. She needs more than a
fake back pat)”
Team I: “The in-store has been handled decently enough, but my concerns are with the system.
How does this fit into my ecosystem? The stores? Food Companies? You say it is out of the
scope, but it is something that needs addressing as you are adding a brand new infrastructure
into stores.”
DESIGN CRITIQUE

33.
NEGOTIATION
THROUGH
BACKCHANNELING
“unfortunately I don't think it was stellar because
since it was backwards, the whole time I kept thinking
NO FUCKING WAY. WTF ARE YOU GUYS DOING
ARE OYU ON CRACK?”
“The message should be about the presentation
order. and i think the issues with scalability or how it
doesnt quite fit the body data stuff.”

34.
DISCUSSION

35.
DISCUSSION
mentors demonstrated an ability to manage
anonymity and ownership effectively in digitally
mediated spaces
an expansion of critique allows for increased
interaction and opportunity for crosstalk, even when
the physical space does not allow for it
multimodal critique encouraged socialization around
design topics and negotiation of meaning—mirroring
the discursive quality of physically-mediated critique
mentors built skills as assessors and design
professionals, clarifying their concerns in order to
reach consensus

36.
colingray.me
THANK YOU