Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From gut feeling to a structured, summative assessment of design competencies

1,911 views

Published on

This presentation goes with the paper 'From gut feeling to a structured, summative assessment of design competencies', presented at the 14th International Conference on Engineering and Product Design Education (E&PDE2012).

bibliographic reference:
Schelling, J.A., Leurs, B.L.F., Best, S.E. and Mulder, I., From Gut Feeling to a Structured, Summative Assessment of Design Competencies. In the 14th International Conference on Engineering and Product Design Education (E&PDE2012), Antwerp, 2012.

Published in: Education, Technology
  • Be the first to comment

From gut feeling to a structured, summative assessment of design competencies

  1. 1. Research Center Creating 010Rotterdam University of Applied SciencesFrom Gut Feeling To AStructured, SummativeAssessment of DesignCompetenciesJasper Schelling, Bas Leurs,Saskia Best and Ingrid MulderAntwerp, September 6 2012 Photography by Roland Pastoor
  2. 2. We introduce a summative method forassessing the whole set of students’ designcompetencies demonstrated in their finaldesign project when graduating an interactivemedia design course, and reflect on the useof this assessment method.
  3. 3. A final design project is often used to givestudents the opportunity to demonstratewhether they have obtained the right levelof expertise to graduate.
  4. 4. The level of expertise that should be acquiredby the student is often formulated as acompetency, a combination of specificprofessional capacities: Attitude Skills Knowledge (Bakerman, 2005)
  5. 5. In our own observations, however, we haveseen many assessors struggling with lists ofambiguous criteria used to assess the students’competencies; it seemed that experiencedassessors often rely on their intuition or‘gut feeling’ to grade the students, and arenot stimulated to make their choices explicit.
  6. 6. At the same time, recent developments inthe accreditation of higher education inThe Netherlands demand a more transparentassessment and grading system.
  7. 7. “Zijlstra makes hardintervention in higher education” (May 20, 2011)
  8. 8. “Stricter supervision in higher education” (February 3, 2012)
  9. 9. The final design project consists of two elements: Investigating state of the art theory and knowledge. Provides Thesis the foundation and rationale for the design project. A real life situation at a company in which the students Designproject have to demonstrate their competencies.
  10. 10. The final assessment is a one hour sessionthat consists of four specific stages:Presentation Interview Deliberation Feedback (10 min) (20 min) (20 min) (10 min)
  11. 11. The students are graded on eleven criteria,and are scored on four levels: Research State-of-the-art Knowledge Excellent Thesis Argumentation Writing Skills and Presentation Concept Good Visualization Product Delivery Satisfactory Designrationale & Emotionale ‘Human Centered’ Design Professionalism Multidisciplinary Working Unsatisfactory & Presentation Presentation & Reflection
  12. 12. The assessment tool consists of four instruments: Calculation Aid Assessment Form Question List Manual
  13. 13. -".$%.,#/012#33%/,&.,#/1415%6,3*$,.17*+,8/ Institute for Communication, Media and Information Technology:%$*//%3<*"01 =.3*0 7.*0 study programme: Communication and Multimedia Design()*+,+G*+*."&) 9 : - ; 2#33*/+0@A3"B,+C43D"03)($((+$"(%"*43&,E"A3",3%3&,*A"FG3%(+$"(%"H,+G$030"($"*+$3DB+,&,-"(%%G3%E"A&%"&"*43&,")+*G%"&$0"(%",343I&$E"A3",3%3&,*A"($+"4(3,&G,3"(%"0(I3,%3E"A3"D3A+0+4+H-"(%"*43&,4-"03%*,(C30E"A3",3%3&,*A"A&%"430"+"&"*+$*3BG&4"D+034E"A3",3%3&,*A"FG3%(+$"(%"&$%J3,30"*43&,4-:.*H#IH)*H."1F/#J6*$8* 9 : - ;@A3"%G03$"A&%"G%30"%&36+)6A36&,"K$+J430H3"&$0"03D+$%,&3%"&$"G$03,%&$0($H"+)"A3"0(%*(B4($3L%"C+0-"+)"K$+J430H3E"A&%"J3446)+G$030"+B($(+$%"*+$*3,$($H"(E"&$0"A&%"*&B(&4(%30"+$"+BB+,G$((3%"&$0",3$0%ML"8%3*/.,#/ 9 : - ;@A3"&,HGD3$&(+$"A&%"&"4+H(*&4"%,G*G,3"&$0"%A+J%"B,+)+G$0$3%%E"&$0"%&$0B+($%"+)"+A3,%"&,3",3*+H$(%30E"32B,3%%30"&$0"*,((*&44-"32&D($30MK",,/81+F,66+1./$1>"*+*/.,#/ 9 : - ;.,G*G,3E"%-43E"%B344($HE")++$+3%94(3,&G,3",3)3,3$*3%E"FG&4(-E"4&-+G"&$0")($(%A($HM!"#$%&2#/&*> 9 : - ; 2#33*/+0 V3e@A3"*+$*3B"(%"+,(H($&4"&$0"($$+I&(I3E"A3,3"(%"&"*43&,"*+A3,3$*3"C3J33$"A3"A3%(%"&$0"A3"B,+0G*"A&"*&$"C3",&*30"C&*K"+",3%3&,*A"($+"4(3,&G,39,3)3,3$*3%"&$0"03%(H$",3%3&,*AE"&$0"A3" BF03.GS-BLS.NP-14jan2012B,+0G*",3%B+$0%"+"A3"J(%A3%"+)"C+A"A3"&,H3"H,+GB"&$0"A3"*4(3$MM,+%.6,+.,#/ 9 : - ;@A3"*+$*3B"(%"D&03"N&$H(C43OE"),+D"&$"($((&4",+GHA"%K3*A"+"&",3)($30"03%(H$M"P+$I($*($H"I(%G&4(%&(+$%E"%K3*A3%"&$0"+A3,"3*A$(FG3%"&,3"G%30"+",3B,3%3$"&$0"%A&,3"(03&%E"B,+C43D%"&$0",3%G4%M7*6,?*"C 9 : - ;@A3"B,+*3%%"A&%"430"+"A3"*,3&(+$"+)"&"B,++-B3"+)"&$"($3,&*(I3"B,+0G*"A&"A&%"C33$"3%30"($"B,&*(*3M"@3%",3%G4%"A&I3"430"+"&0&B&(+$%"&$0",3*+DD3$0&(+$%MG.,#/.6141;3#,#/.6 9 : - ;@A3"%G03$"*&$"%GC%&$(&3"03%(H$"03*(%(+$%E"*43&,4-"03D+$%,&3"A3"*+A3,3$*3"C3J33$",3%3&,*A"&$0"B,+0G*E"&$0",3I(3J"A3"*+$%3FG3$*3%"+)"B3,%+$&4"03%(H$"03*(%(+$%MNO%3./12*/*"*$N1$*+,8/ 9 : - ;@A3"%G03$"%A+J%"&C(4(-"+"3DB&A(Q3E"A&%"BG"A(D"+,"A3,%34)"($"A3"B+%((+$"+)"A3"&,H3"H,+GB"&$0"+A3,"%&K3A+403,%E"A&%"&*(I34-"($I+4I30"A3"&,H3"H,+GB"&$0"+A3,"%&K3A+403,%"($"A3"03%(H$"B,+*3%%E"A&%"+C%3,I30"J344"&$0"0,&J$"*+$*4G%(+$%"),+D"A(%M!"#I*++,#/.6,+3141!"*+*/.,#/5%6,$,+&,>6,/."C1J#"F,/8 9 : - ; 2#33*/+0@A3"%G03$"(%"&C43"+"*+44&C+,&3E"+,H&$(%3E"*+DDG$(*&3"&$0"B&,(*(B&3E"&$0"03D+$%,&3%"3))+,")+,"&$0"*+44&C+,&(+$"J(A"*+443&HG3%E"A3"*4(3$"&$0"+A3,"%&K3A+403,%M!"*+*/.,#/141G*I6*&,#/ 9 : - ;@A3"B,3%3$&(+$"A&%"&"*43&,"%,G*G,3"&$0"(%"%GBB+,30"I(%G&44-E"A3"FG&4(-"+)"A3"&$%J3,%"(%"*+$I($*($HE"A3"%G03$"(%"&C43"+",3)43*"+$"A(%"+,"A3,"B3,%+$&4"03I34+BD3$E"A&%"+B3,&30"($036B3$03$4-E"A&%"(03$()(30"&$0"G(4(%30"43&,$($H"+BB+,G$((3%E"&$0"A&%"J+,K30"+$"B3,%+$&4"03I34+B6D3$"&$0"A3"&*FG(%((+$"+)"$3J"K$+J430H3"&$0"G$03,%&$0($HM:%>*"?,+#"1@&).,"A :%>*"?,+#"1@$*.&)*$A :%>*"?,+#"1@&#3>./CA ;D*"/.61*D>*" E,/.613."F >565=?=.3*1B1:,8/.%"* =.3*1B1:,8/.%"* =.3*1B1:,8/.%"* =.3*1B1:,8/.%"*!"#"!$%&(%)&*+,-"."#".&(%)&*+,-"/"#"/++0"1"#"12*3443$""""""""567"#"!$%&(%)&*+,-"89:"#".&(%)&*+,-";"#"/++0"<95="#"12*3443$
  14. 14. -".$%.,#/012#33%/,&.,#/1415%6,3*$,.17*+,8/ Institute for Communication, Media and Information Technology:%$*//%3<*"01 =.3*0 7.*0 study programme: Communication and077234 Jeroen Stappenmaecker June 16, 2011 Multimedia Design()*+,+ ;D.3>6*1#I1&.6&%6.,#/G*+*."&) 9 : - ; 2#33*/+0@A3"B,+C43D"03)($((+$"(%"*43&,E"A3",3%3&,*A"FG3%(+$"(%"H,+G$030"($"*+$3DB+,&,-"(%%G3%E"A&%"&"*43&,")+*G%"&$0"(%",343I&$E"A3",3%3&,*A"($+"4(3,&G,3"(%"0(I3,%3E"A3"D3A+0+4+H-"(%"*43&,4-" ()*+,+ If an item has been graded with03%*,(C30E"A3",3%3&,*A"A&%"430"+"&"*+$*3BG&4"D+034E"A3",3%3&,*A"FG3%(+$"(%"&$%J3,30"*43&,4- ‘unsatisfactory’, the student 9 9/+.,+I.&#"C does not pass!:.*H#IH)*H."1F/#J6*$8* 9 : - ;@A3"%G03$"A&%"G%30"%&36+)6A36&,"K$+J430H3"&$0"03D+$%,&3%"&$"G$03,%&$0($H"+)"A3"0(%*(B4($3L%"C+0-"+)"K$+J430H3E"A&%"J3446)+G$030"+B($(+$%"*+$*3,$($H"(E"&$0"A&%"*&B(&4(%30"+$" : :.,+I.&#"C 2 P1QR11S1 3 0+BB+,G$((3%"&$0",3$0%M - -##$ 2 P1TU11S1 4 0L"8%3*/.,#/ 9 : - ;@A3"&,HGD3$&(+$"A&%"&"4+H(*&4"%,G*G,3"&$0"%A+J%"B,+)+G$0$3%%E"&$0"%&$0B+($%"+)"+A3,%"&,3" ; ;D&*66*/ P1TR11S1 +,3*+H$(%30E"32B,3%%30"&$0"*,((*&44-"32&D($30MK",,/81+F,66+1./$1>"*+*/.,#/.,G*G,3E"%-43E"%B344($HE")++$+3%94(3,&G,3",3)3,3$*3%E"FG&4(-E"4&-+G"&$0")($(%A($HM 9 : - ; 7 0 P1V11S 2 8 0 B !"#$%&!"#$%& If an item has been graded with2#/&*> 9 : - ; 9 9/+.,+I.&#"C 2#33*/+0 ‘unsatisfactory’, the student V3e does not pass!@A3"*+$*3B"(%"+,(H($&4"&$0"($$+I&(I3E"A3,3"(%"&"*43&,"*+A3,3$*3"C3J33$"A3"A3%(%"&$0"A3"B,+0G*"A&"*&$"C3",&*30"C&*K"+",3%3&,*A"($+"4(3,&G,39,3)3,3$*3%"&$0"03%(H$",3%3&,*AE"&$0"A3" : :.,+I.&#"C 2 P1QT11S1 2 4 BF03.GS-BLS.NP-14jan2012B,+0G*",3%B+$0%"+"A3"J(%A3%"+)"C+A"A3"&,H3"H,+GB"&$0"A3"*4(3$MM,+%.6,+.,#/ 9 : - ; - -##$ 2 P1QW11S1 3 2@A3"*+$*3B"(%"D&03"N&$H(C43OE"),+D"&$"($((&4",+GHA"%K3*A"+"&",3)($30"03%(H$M"P+$I($*($H"I(%G&4(%&(+$%E"%K3*A3%"&$0"+A3,"3*A$(FG3%"&,3"G%30"+",3B,3%3$"&$0"%A&,3"(03&%E"B,+C43D%"&$0" ; ;D&*66*/ 1 P1TU11S1 2 0 +,3%G4%M7*6,?*"C@A3"B,+*3%%"A&%"430"+"A3"*,3&(+$"+)"&"B,++-B3"+)"&$"($3,&*(I3"B,+0G*"A&"A&%"C33$"3%30"($" 9 : - ; 7 6 P1V11S 3 0 4 BB,&*(*3M"@3%",3%G4%"A&I3"430"+"&0&B&(+$%"&$0",3*+DD3$0&(+$%M !"#I*++,#/.6,+3141!"*+*/.,#/G.,#/.6141;3#,#/.6 9 : - ; If an item has been graded with@A3"%G03$"*&$"%GC%&$(&3"03%(H$"03*(%(+$%E"*43&,4-"03D+$%,&3"A3"*+A3,3$*3"C3J33$" 9 9/+.,+I.&#"C ‘unsatisfactory’, the student,3%3&,*A"&$0"B,+0G*E"&$0",3I(3J"A3"*+$%3FG3$*3%"+)"B3,%+$&4"03%(H$"03*(%(+$%M does not pass!NO%3./12*/*"*$N1$*+,8/ 9 : - ; : :.,+I.&#"C P1XU11S1@A3"%G03$"%A+J%"&C(4(-"+"3DB&A(Q3E"A&%"BG"A(D"+,"A3,%34)"($"A3"B+%((+$"+)"A3"&,H3"H,+GB"&$0"+A3,"%&K3A+403,%E"A&%"&*(I34-"($I+4I30"A3"&,H3"H,+GB"&$0"+A3,"%&K3A+403,%"($"A3"03%(H$" - -##$ 1 P1VU11S1 4 0B,+*3%%E"A&%"+C%3,I30"J344"&$0"0,&J$"*+$*4G%(+$%"),+D"A(%M ; ;D&*66*/ 1 P1RU11S1 5 0 +!"#I*++,#/.6,+3141!"*+*/.,#/5%6,$,+&,>6,/."C1J#"F,/8 9 : - ; 2#33*/+0 9 0 P1T11S 1 8 0 B@A3"%G03$"(%"&C43"+"*+44&C+,&3E"+,H&$(%3E"*+DDG$(*&3"&$0"B&,(*(B&3E"&$0"03D+$%,&3%"3))+,")+,"&$0"*+44&C+,&(+$"J(A"*+443&HG3%E"A3"*4(3$"&$0"+A3,"%&K3A+403,%M!"*+*/.,#/141G*I6*&,#/@A3"B,3%3$&(+$"A&%"&"*43&,"%,G*G,3"&$0"(%"%GBB+,30"I(%G&44-E"A3"FG&4(-"+)"A3"&$%J3,%"(%" 9 : - ; E,/.613."F 7 6 4 / 100 = 7, 6 4*+$I($*($HE"A3"%G03$"(%"&C43"+",3)43*"+$"A(%"+,"A3,"B3,%+$&4"03I34+BD3$E"A&%"+B3,&30"($036B3$03$4-E"A&%"(03$()(30"&$0"G(4(%30"43&,$($H"+BB+,G$((3%E"&$0"A&%"J+,K30"+$"B3,%+$&4"03I34+B6D3$"&$0"A3"&*FG(%((+$"+)"$3J"K$+J430H3"&$0"G$03,%&$0($HM E,/.613."F1@"#%/$*$A:%>*"?,+#"1@&).,"A :%>*"?,+#"1@$*.&)*$A :%>*"?,+#"1@&#3>./CA ;D*"/.61*D>*" 8 E,/.613."F >565=? Els Karreman Niels de Worp Bernadette Delfgauw=.3*1B1:,8/.%"* =.3*1B1:,8/.%"* =.3*1B1:,8/.%"* =.3*1B1:,8/.%"*!"#"!$%&(%)&*+,-"."#".&(%)&*+,-"/"#"/++0"1"#"12*3443$""""""""567"#"!$%&(%)&*+,-"89:"#".&(%)&*+,-";"#"/++0"<95="#"12*3443$
  15. 15. During the assessments of the finalprojects we observed assessor teamswhen they deliberated on the assessmentof competencies and discussed theresulting final grade.
  16. 16. After the assessment, the use of theinstruments was evaluated with assessors fromthe teaching staff. They were asked to givefeedback on how the instruments supportedthem in grading the graduation work andwhat their experiences were using this tool.
  17. 17. Observations during the deliberation phaseof the final assessment showed that assessorswere less likely to rely on their generalimpression of the work and the presentation.Instead, they were actively discussing thestudents’ results on the criteria set forth inthe assessment form.
  18. 18. When the assessors finished their discussionand asked the student to reappear before thecommittee to receive the final result, thecommittee was better prepared to linktheir feedback to the criteria set forth in theassessment and seemed to be more confidentmotivating the resulting final grade.
  19. 19. In general, the assessors mentioned that thediscussion during the grading was about thecriteria set forth in the assessment form andnot about the final grade.
  20. 20. Although the assessment instrument washelpful in assessing the design competencies,some assessors indicated that the final gradedid not match with their gut feeling.
  21. 21. A piece of visualization software was writtento visualize the scores of the assessments foreach student. This visualization was createdfor each of the 58 students.
  22. 22. These visualizations in turn provide afertile ground for continuing discussion onassessment procedures, for individual studentsas well as for the curriculum in general.
  23. 23. Being able to visually compare the outcomesof various assessments allows us to gaininsight into the different assessments andto find patterns as well as points of interest,which inform further course development.
  24. 24. The current study demonstrated that structuredassessment instruments have dual purposes: Assessing the quality of the work of the students in a Hollistic Assessment manner fitting with competence based education. Improving the Supporting the explicit motivation discussion among of the nuances of the specific design competencies. assessors
  25. 25. A side effect of revealing the tacit expertiseof experienced assessors was that the juniorassessors felt they improved theirunderstanding of the competencies andassociated criteria and proficiency level.
  26. 26. This seems to be beneficial for the studentsas well. Usually students complained aboutperceived subjectivity of the final results;interestingly, the current cohort hardlydiscussed their final grades.
  27. 27. All in all, it can be concluded that structuredassessment instruments, when properlydesigned, help assessors to distinguishbetween a ‘gut feeling’ and their intuition,as well show the relative merit of a specificcompetency and matching proficiency level.
  28. 28. Questions?Contact Information Jasper SchellingResearch Center Creating 010 j.a.schelling@hr.nlRotterdam University of Bas LeursApplied Sciences b.l.f.leurs@hr.nlcreating010@hr.nl

×