WCCE 2009 – World Conference on Computers in Education Bento Golçalves-RS, Brazil, July 27-31, 2009, www.wcce2009.org Growing a Peer Review Culture among Graduate Students Vinícius M. Kern 1,2 , Osmar Possamai 1 , Paulo M. Selig 1 , Roberto C.S. Pacheco 1 , Gilberto Corrêa de Souza 1 , Sandro Rautenberg 1,3 , and Renata Tavares da Silva Lemos 1 1 Graduate Program in Knowledge Engineering and Management, Federal University of Santa Catarina, Brasil kern@egc.ufsc.br (contact author), www.egc.ufsc.br 2 Instituto Stela, Brasil, www.stela.org.br 3 Universidade Estadual do Centro-Oeste (Unicentro), Brasil, www.unicentro.br Outline of the presentation Introduction A Systemic View of Graduate Learning Approach to Growing a Peer Review Culture Results and Discussion Concluding Remarks

Outline of the presentation

Introduction

A Systemic View of Graduate Learning

Approach to Growing a Peer Review Culture

Introduction Common processes for pursuing graduate education excellence: candidate selection teaching and coursework research (conducted by students under supervision) thesis defense Our complementary approach: the growing of a peer review culture among graduate students Grad. Prog. Knowledge Eng. & Mgmt (KEM or EGC in Portuguese), Federal U. of Santa Catarina Started in 07/2004, multidisciplinary (Eng., IT, Mgmt, Media) Research object: “knowledge as a production factor”. High candidate rates: 322 to 423 candidates for about 60 no-scholarship annual openings, since 2004.

Common processes for pursuing graduate education excellence:

candidate selection

teaching and coursework

research (conducted by students under supervision)

thesis defense

Our complementary approach: the growing of a peer review culture among graduate students

Grad. Prog. Knowledge Eng. & Mgmt (KEM or EGC in Portuguese), Federal U. of Santa Catarina

Started in 07/2004, multidisciplinary (Eng., IT, Mgmt, Media)

Research object: “knowledge as a production factor”.

High candidate rates: 322 to 423 candidates for about 60 no-scholarship annual openings, since 2004.

Introduction (II) EGC/UFSC (cont.) Faculty: ~40 from 10 departments, different worldviews/cultures (multidisciplinary); interdisciplinary character of the research object Risk of being “multidisciplinary, hence potentially dispersive, rather than interdisciplinary, hence cohesive” (Bunge, Emergence & Convergence) Interdisciplinarity requires excellent communication Need for enhanced communication >> motivation for our first annual workshop in 2004 Students presented proposals; template provided 2005 on: Research Seminars, mandatory, no credits, then the workshop Proposals with peer review: 49 (2005), 67 (2006), 54 (2007), 62 (2008) Next: systemic analysis and description of our peer review approach to growing a peer review culture; future work

EGC/UFSC (cont.)

Faculty: ~40 from 10 departments, different worldviews/cultures (multidisciplinary); interdisciplinary character of the research object

Risk of being “multidisciplinary, hence potentially dispersive, rather than interdisciplinary, hence cohesive” (Bunge, Emergence & Convergence)

Interdisciplinarity requires excellent communication

Need for enhanced communication >> motivation for our first annual workshop in 2004

Students presented proposals; template provided

2005 on: Research Seminars, mandatory, no credits, then the workshop

Proposals with peer review: 49 (2005), 67 (2006), 54 (2007), 62 (2008)

Next: systemic analysis and description of our peer review approach to growing a peer review culture; future work

Our Systemic View of Graduate Learning - Concepts DISCLAIMER: No holism . “When rigorous contemporary social scientists hear the word 'system,' they are likely to draw their intellectual guns” “ Systems have systemic (emergent) features that their components lack” Everything is a system or an actual or potential component of a system / There are no permanent strays or isolates Any concrete system σ can be modeled as μ ( σ ) = < C ( σ ), E ( σ ), S ( σ ), M ( σ ) > Composition : collection of all the parts of σ Environment : collection of (other) items that act on or are acted upon by some or all components of σ Everything has an environment except for the universe as a whole. Structure : collection of relations, in particular bonds, among components of σ (endostructure) or among these and items of the environment (exostructure) Mechanism : collection of processes in σ that make it behave the way it does – that generate qualitative novelty, emergence. Abstract systems have no mechanism SOURCES: Bunge’s E&C (2003) and ‘Mechanism and explanation”, Phil Soc Sci (1997)

DISCLAIMER: No holism . “When rigorous contemporary social scientists hear the word 'system,' they are likely to draw their intellectual guns”

“ Systems have systemic (emergent) features that their components lack”

Everything is a system or an actual or potential component of a system / There are no permanent strays or isolates

Any concrete system σ can be modeled as μ ( σ ) = < C ( σ ), E ( σ ), S ( σ ), M ( σ ) >

Composition : collection of all the parts of σ

Environment : collection of (other) items that act on or are acted upon by some or all components of σ

Everything has an environment except for the universe as a whole.

Structure : collection of relations, in particular bonds, among components of σ (endostructure) or among these and items of the environment (exostructure)

Mechanism : collection of processes in σ that make it behave the way it does – that generate qualitative novelty, emergence.

Abstract systems have no mechanism

SOURCES: Bunge’s E&C (2003) and ‘Mechanism and explanation”, Phil Soc Sci (1997)

Our Systemic View of Graduate Learning – Systems… Size Bond strength Metabolism and accompanying processes that allow for maintenance and self-repair, expression and repression of genes etc. All the bonds (direct/indirect, physical/chemical, covalent/non-cov.) that keep  together plus all the ties - phys., chem., biol. - with environmental items . Medium rich in nutrients and energy fluxes, with variables (pressure, temperature etc.) within narrow intervals . Physical and chemical micro- and mesosystems, in partic. water, carbohydrates, lipids, proteins, and nucleic acids . Organism Activities that end up in the firm's products . Work relations among the firm's members;member- environ. item . Market and government. Personnel and management. Firm Production, transmission, and reception of symbols . Collection of linguistic communication relations . Culture(s) where the language is used . People who speak the same language . Linguistic community M Mechanism S Structure E Environment C Composition  System

Structure (bonds of…) : [Endostr.] collaboration , communication , co-authorship/cooperation in projects , advising , teaching , peer pressure , [Exostr.] submission to rules, funding, refereeing , cultural influence, affiliation to UFSC, reputation-networking-partnership-affiliation with other organizations Mechanism : study+research, advising, project/publication cooperation, communication* (scientific, objective criticism and argumentation) Our Systemic View of Graduate Learning – Finally Funding agencies UFSC (host institution), staff Norms, regula- tions, laws faculty students Organizations (firms, research institutes, univ., NGOs, govern-mental, inter-national) Culture, Brazilian graduate education community

Structure (bonds of…) : [Endostr.] collaboration , communication , co-authorship/cooperation in projects , advising , teaching , peer pressure , [Exostr.] submission to rules, funding, refereeing , cultural influence, affiliation to UFSC, reputation-networking-partnership-affiliation with other organizations

Mechanism : study+research, advising, project/publication cooperation, communication* (scientific, objective criticism and argumentation)

Approach to Growing a Peer Review Culture among Students Peer review in education by first author 1994 as student (Virginia Tech), then since 1997 as professor More than 1,000 ‘victims’ http://kern.ispeople.org/par_en.html Reasons to undertake: cooperation, written expression, critical thinking, professional responsibility (working on bonds and skills…) Initiatives by several other scholars around the world (see refs) Our research seminars since 2005 (mandatory, no grade) About 8 4-h encounters: proposal preparation, live presentations and professors’ feedback, advice for refereeing (including sample referee forms) Usually 2+ professors in class Feedback on form and on internal and relational coherence of proposal sections; no criticism on merit (because of methodology) Single-round, double-blind peer review (OJS) Each student gives and receives 3 feedbacks (ideally) Full feedback to all students and advisors (spreadsheet summary) December: annual workshop (all freshmen present proposals)

Peer review in education by first author

1994 as student (Virginia Tech), then since 1997 as professor

More than 1,000 ‘victims’ http://kern.ispeople.org/par_en.html

Reasons to undertake: cooperation, written expression, critical thinking, professional responsibility (working on bonds and skills…)

Initiatives by several other scholars around the world (see refs)

Our research seminars since 2005 (mandatory, no grade)

About 8 4-h encounters: proposal preparation, live presentations and professors’ feedback, advice for refereeing (including sample referee forms)

Usually 2+ professors in class

Feedback on form and on internal and relational coherence of proposal sections; no criticism on merit (because of methodology)

Single-round, double-blind peer review (OJS)

Each student gives and receives 3 feedbacks (ideally)

Full feedback to all students and advisors (spreadsheet summary)

December: annual workshop (all freshmen present proposals)

Approach to Growing a Peer Review Culture among Students (II) Objectives For the students… Experience, acquaintedness : To know the essential parts of a thesis proposal, then re-elaborate the proposal presented for entrance in the graduate program and participate in a peer review round. Maturing : To start developing competence to give and receive professional, objective critique of scientific work. For the graduate program Culture : To create and disseminate a culture of objective, interdisciplinary scientific criticism. Spur thruput : To serve as catalyst of the advising process. Interdisciplinarity : To stimulate interdisciplinary scientific interchange.

Objectives

For the students…

Experience, acquaintedness : To know the essential parts of a thesis proposal, then re-elaborate the proposal presented for entrance in the graduate program and participate in a peer review round.

Maturing : To start developing competence to give and receive professional, objective critique of scientific work.

For the graduate program

Culture : To create and disseminate a culture of objective, interdisciplinary scientific criticism.

Spur thruput : To serve as catalyst of the advising process.

Interdisciplinarity : To stimulate interdisciplinary scientific interchange.

Approach to Growing a Peer Review Culture among Students (III) Threats to the process Without strong guidance, it becomes an exchange of opinions (at best) Upper management support desperately needed (ok) Because we are so multi disciplinary (not inter -), Method is out (I know, that’s awful…) What we did to face the threats Adopted one successfully-in-use set of guidelines (2nd author, positivist approach, engineering method) GP dean is always one of the professors Required advisor involvement Started a collaboration to tame methodology 2007: 2 collaborators gave lectures 2008: 5 professors in 2 diff Method courses (elective quanti, quali) 2009: Intro to research methodology: 7 professors plus another one who attended all classes (then 3 complementary courses) Worldviews/main classes of paradigms; Methodological tour

Threats to the process

Without strong guidance, it becomes an exchange of opinions (at best)

Upper management support desperately needed (ok)

Because we are so multi disciplinary (not inter -), Method is out (I know, that’s awful…)

What we did to face the threats

Adopted one successfully-in-use set of guidelines (2nd author, positivist approach, engineering method)

GP dean is always one of the professors

Required advisor involvement

Started a collaboration to tame methodology

2007: 2 collaborators gave lectures

2008: 5 professors in 2 diff Method courses (elective quanti, quali)

2009: Intro to research methodology: 7 professors plus another one who attended all classes (then 3 complementary courses)

Worldviews/main classes of paradigms; Methodological tour

Approach to Growing a Peer Review Culture among Students (IV) Template for proposals – usual sections Abstract Motivation Problem statement and/or question Objectives [might be split into general and specifics] Relevance Scope Main references General appraisal Software choices Conference management choices: either costly/unknown or too limited Open Journal System: free, widespread, allows attachment Attachment: Referee form spreadsheet (Excel, controlled content, easy to import to MySQL) Referee form (next)

Template for proposals – usual sections

Abstract

Motivation

Problem statement and/or question

Objectives [might be split into general and specifics]

Relevance

Scope

Main references

General appraisal

Software choices

Conference management choices: either costly/unknown or too limited

Open Journal System: free, widespread, allows attachment

Attachment: Referee form spreadsheet (Excel, controlled content, easy to import to MySQL)

Referee form (next)

Approach to Growing a Peer Review Culture among Students (V – referee form) A b Referee research area (KE/KMa/KMe), level (Dr/MSc), and self-declaration of expertise Proposal # and title, author area and level (all proposals from same level of referee’s, 2 out of 3 from same area) 1-paragraph summary of proposal Grades and comments for each proposal section Grades and comments for proposal as a whole

Results and Discussion Mostly anedoctal (and some exploratory research) Proposals processed : 2005-49, 2006-67, 2007-54, 2008-62 Additional students allowed for refereeing only Numbers for 2008 (4 extra referees  198 allocations, 192 delivered) Commitment and depth of feedback vary While it may be related to “mandatory, no credits, no grade”, strategy for improvement has been: raising awareness (related to communication bonds)

Mostly anedoctal (and some exploratory research)

Proposals processed : 2005-49, 2006-67, 2007-54, 2008-62

Additional students allowed for refereeing only

Numbers for 2008 (4 extra referees  198 allocations, 192 delivered)

Commitment and depth of feedback vary

While it may be related to “mandatory, no credits, no grade”, strategy for improvement has been: raising awareness (related to communication bonds)

Results and Discussion (II) Publication of full and aggregate results started in 2008 (Why: very limited resources; Before 2008: “here are your forms”) Aggregates: by [area, level, gender—not in the paper] , percents of self-declaration of expertise For each proposal For each referee report (typically 3 for proposal) Research area of the referee Referee’s self-declaration of expertise Grades (0-10) and comments on the topics reviewed Report allows students to… See all referee reports about own proposal Analyze, compare reports, reflect about quality of own work. See what colleagues said about the same proposals refereed Reflect about own skills as referee and about general quality of communication* bonds

Publication of full and aggregate results started in 2008 (Why: very limited resources; Before 2008: “here are your forms”)

Aggregates: by [area, level, gender—not in the paper] , percents of self-declaration of expertise

For each proposal

For each referee report (typically 3 for proposal)

Research area of the referee

Referee’s self-declaration of expertise

Grades (0-10) and comments on the topics reviewed

Report allows students to…

See all referee reports about own proposal

Analyze, compare reports, reflect about quality of own work.

See what colleagues said about the same proposals refereed

Reflect about own skills as referee and about general quality of communication* bonds

Results and Discussion (III-Aggregates 2008)

Results and Discussion (IV-Aggregates) Grades 0-3 (reject), 4-5 (weak reject), 6-7 (weak accept), 8-10 (accept) Given: From 6.67 (Scope, average) to 7.37 (General objective, avg) Self-declaration of expertise Even though students meet in 3+ mandatory courses plus the Seminars, they have a poor appraisal of their comparative expertise in the proposals’ topics… … or maybe this is a cultural issue (you can’t claim to be so good)

Grades

0-3 (reject), 4-5 (weak reject), 6-7 (weak accept), 8-10 (accept)

Given: From 6.67 (Scope, average) to 7.37 (General objective, avg)

Self-declaration of expertise

Even though students meet in 3+ mandatory courses plus the Seminars, they have a poor appraisal of their comparative expertise in the proposals’ topics…

… or maybe this is a cultural issue (you can’t claim to be so good)

Results and Discussion (V) Opportunities for Improvement and R&D Issues Instrumental issues Better processing of peer review bureaucracy (Semi-)automation of knowledge-intensive tasks, e.g. referee allocation referee rating process reliability and validity evaluation with nonlinear dynamics (Losada: connectivity) Methodological issues Articulation of Seminars with new mandatory course on Methodology (toward interdisciplinarity) Now we have some preparation to discuss paradigm and method choice Investigate mechanisms (Bunge) that create emergent properties in our graduate learning system Multilevel analysis (macro-micro systems)

Opportunities for Improvement and R&D Issues

Instrumental issues

Better processing of peer review bureaucracy

(Semi-)automation of knowledge-intensive tasks, e.g.

referee allocation

referee rating

process reliability and validity

evaluation with nonlinear dynamics (Losada: connectivity)

Methodological issues

Articulation of Seminars with new mandatory course on Methodology (toward interdisciplinarity)

Now we have some preparation to discuss paradigm and method choice

Investigate mechanisms (Bunge) that create emergent properties in our graduate learning system

Multilevel analysis (macro-micro systems)

Concluding Remarks Experience report (mostly anedoctal; empirical research to come) Computer use still very marginal Students experience the main scientific method for quality control and have an opportunity to sharpen their knowledge and strengthen their (scientific, rigorous, objective) communication bonds with their peers and professors, adviser included Part of our quest for interdisciplinarity through strengthening communication bonds Aim at contributing, as well, to establish peer review as a replicable, scalable educational approach (Kern et al., 2007)

Experience report (mostly anedoctal; empirical research to come)

Computer use still very marginal

Students experience the main scientific method for quality control and have an opportunity to sharpen their knowledge and strengthen their (scientific, rigorous, objective) communication bonds with their peers and professors, adviser included

Part of our quest for interdisciplinarity through strengthening communication bonds

Aim at contributing, as well, to establish peer review as a replicable, scalable educational approach (Kern et al., 2007)

References Angelov, C., Melnik, R.V.N., Buur, J.: The synergistic integration of mathematics, software engineering, and user-centered design: exploring new trends in education. Fut. Gen. Comp. Sys. 19(8), 1299-1307 (2003) Araújo, L.H.L.: Uma aplicação da dinâmica não-linear para a avaliação de desempenho de comunidades virtuais de aprendizagem [An application of nonlinear dynamics to performance evaluation of virtual learning communities]. Masters dissertation, Universidade Católica de Brasília, Brasília-DF, Brasil (2004) Bunge, M.: Emergence and convergence : Qualitative novelty and the unity of knowledge. Uni. Toronto Press, Toronto (2003) Bunge, M.: Mechanism and explanation. Phil. Soc. Sci. 27(4), 410-465 (1997) Bunge, M.: Systemism: the alternative to individualism and holism. J. Socio-Econ. 29(2), 147-157 (2000) Cunningham, S.J.: Using a computer conferencing system to support writing and research skill development. ACM SIGCSE Bul. 26(4), 5-8 (1994) Davies, R., Berrow, T.: An evaluation of the use of computer supported peer review for developing higher-level skills, Comput. Educ. 30(1/2), 111-115 (1998) Denning, T., Kelly, M., Lindquist, D., Malani, R., Griswold, W.G., and Simon, B.: Lightweight preliminary peer review: does in-class peer review make sense? ACM SIGCSE Bul. 39(1), 266-270 (2007) Gehringer, E.F.: Electronic peer review and peer grading in Computer-Science courses. ACM SIGCSE Bul. 33(1), 139-143 (2001) Hafen, M.: Developing writing skills in computer science students. ACM SIGCSE Bul. 26(1), 268-270 (1994) Hartman, J.: Writing to learn and communicate in a data structures course. In: 20th SIGCSE Technical Symposium on Computer Science Education , pp. 32-36 (1989) Kern, V.M., Pacheco, R.C.S., Saraiva, L.M., Pernigotti, J.M.: Peer review in Computer Science education: Requirements for continuous, large scale application. In: F.M. Mendes Neto and F.V. Brasileiro (Eds.), Advances in computer supported learning , pp. 46-65, Idea Group Publishing, Hershey (2007) Kern, V.M., Saraiva, L.M., Pacheco, R.C.S.: Peer review in education: promoting collaboration, written expression, critical thinking, and professional responsibility. Educ. Inf. Technol. 8(1), 37-46 (2003) Liu, E.Z., Lin, S.S.J., Chiu, C., Yuan, S.: Web-based peer review: the learner as both adapter and reviewer. IEEE Trans. Educ. 44(3), 246-251 (2001) Losada, M., Heaphy, E.: The role of positivity and connectivity in the performance of business teams: A nonlinear dynamics model. Am. Behav. Scient. 47(6), 740-765 (2004) Losada, M.: The complex dynamics of high performance teams. Math. Comput. Model. 30(9-10), 179-192 (1999) Moreira, D.A., Silva, E.Q.: A method to increase student interaction using student groups and peer review over the internet. Educ. Inf. Technol. 8(1), 47-54 (2003) Riggs, T., Wilensky, R.: An algorithm for automated rating of reviewers. In: 1st ACM/IEEE-CS joint conference on digital libraries , pp. 381—387. Roanoke, (2001) Schreiber, G. et al.: Knowledge engineering and management : the CommonKADS methodology‎. MIT Press, Cambridge MA (2000) Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer programming, In: 3rd IEEE International Conference on Advanced Learning Technologies (ICALT’03), (2003) Smith, A.J.: The task of the referee. IEEE Computer , 23(4), 46-51 (1990)

Thank you! Acknowledgments Authors GCS, SR, RTSL (volunteer students) Instituto Stela (network and systems administration services) Questions? Contact author: Vinícius Medina Kern, kern.ispeople.org (see menu ‘Presentations…’ for these slides) Researcher at Instituto Stela , www.stela.org.br Professor at EGC/UFSC , www.egc.ufsc.br (Both in Florianópolis-SC, Santa Catarina island)