Growing a Peer Review Culture among Graduate Students (WCCE 2009)


Published on

Preprint @, World Conference on Computers in Education
Abstract: Usual processes for pursuing education excellence in a graduate program are candidate selection, coursework, research, and thesis defense. In this paper, we present a complementary approach: the growing of a peer review culture among graduate students. We instruct first-year masters’ and doctoral students on principles for preparing a thesis proposal. Students present their proposals in collective discussion sessions with feedback from professors. The students then submit their proposals through a web interface and are instructed on the role they will play next – of anonymous referees of their peers’ proposals. The referee reports and general statistics are made available to all participating students and advisors. Updated proposals are submitted to an annual workshop open to all participating students and advisors. About 60 students take part in this annual series of seminars with peer review and workshop, generating 60 theses proposals and about 180 referee reports, 3 for each proposal. Students and their advisors receive detailed feedback on individual participation as author and referee. The main strength of the experience is the opportunity to assimilate the techniques of objective criticism and to reflect about the quality of own and others’ work. The paper also outlines research and development issues related to our effort to enhance the peer review culture among graduate students.

Published in: Education, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Os professores que estão envolvidos nas disciplinas de métodos estão discutindo juntos os detalhes de uma disciplina introdutória. A idéia central é a construção conjunta e interfacetada com o seminário.
  • O acordo que fazemos com os alunos é de que cada um revisa 2 propostas do mesmo nível e área e outra do mesmo nível e outra área (e consequentemente tem 2 revisões de colegas de área e uma de outra área) O revisor se autodeclara mais ou menos perito. Os itens avaliados são os acima, com um grau e espaço para críticas detalhadas. Há orientação intensa sobre como revisar uma proposta. Apresento, inclusive, os pareceres que recebi em minha primeira submissão internacional, para mostrar idiossincrasias do processo (pareceres discordantes, crítica sem fundamento, pareceres negativos quanto à aprovação mas muito úteis por darem críticas precisas, objetivas e acertadas etc.)
  • Growing a Peer Review Culture among Graduate Students (WCCE 2009)

    1. 1. WCCE 2009 – World Conference on Computers in Education Bento Golçalves-RS, Brazil, July 27-31, 2009, Growing a Peer Review Culture among Graduate Students Vinícius M. Kern 1,2 , Osmar Possamai 1 , Paulo M. Selig 1 , Roberto C.S. Pacheco 1 , Gilberto Corrêa de Souza 1 , Sandro Rautenberg 1,3 , and Renata Tavares da Silva Lemos 1 1 Graduate Program in Knowledge Engineering and Management, Federal University of Santa Catarina, Brasil (contact author), 2 Instituto Stela, Brasil, 3 Universidade Estadual do Centro-Oeste (Unicentro), Brasil, <ul><li>Outline of the presentation </li></ul><ul><ul><li>Introduction </li></ul></ul><ul><ul><li>A Systemic View of Graduate Learning </li></ul></ul><ul><ul><li>Approach to Growing a Peer Review Culture </li></ul></ul>Results and Discussion Concluding Remarks
    2. 2. Introduction <ul><li>Common processes for pursuing graduate education excellence: </li></ul><ul><ul><li>candidate selection </li></ul></ul><ul><ul><li>teaching and coursework </li></ul></ul><ul><ul><li>research (conducted by students under supervision) </li></ul></ul><ul><ul><li>thesis defense </li></ul></ul><ul><li>Our complementary approach: the growing of a peer review culture among graduate students </li></ul><ul><li>Grad. Prog. Knowledge Eng. & Mgmt (KEM or EGC in Portuguese), Federal U. of Santa Catarina </li></ul><ul><ul><li>Started in 07/2004, multidisciplinary (Eng., IT, Mgmt, Media) </li></ul></ul><ul><ul><li>Research object: “knowledge as a production factor”. </li></ul></ul><ul><ul><li>High candidate rates: 322 to 423 candidates for about 60 no-scholarship annual openings, since 2004. </li></ul></ul>
    3. 3. Introduction (II) <ul><li>EGC/UFSC (cont.) </li></ul><ul><ul><li>Faculty: ~40 from 10 departments, different worldviews/cultures (multidisciplinary); interdisciplinary character of the research object </li></ul></ul><ul><ul><li>Risk of being “multidisciplinary, hence potentially dispersive, rather than interdisciplinary, hence cohesive” (Bunge, Emergence & Convergence) </li></ul></ul><ul><ul><li>Interdisciplinarity requires excellent communication </li></ul></ul><ul><li>Need for enhanced communication >> motivation for our first annual workshop in 2004 </li></ul><ul><ul><li>Students presented proposals; template provided </li></ul></ul><ul><ul><li>2005 on: Research Seminars, mandatory, no credits, then the workshop </li></ul></ul><ul><ul><ul><li>Proposals with peer review: 49 (2005), 67 (2006), 54 (2007), 62 (2008) </li></ul></ul></ul><ul><li>Next: systemic analysis and description of our peer review approach to growing a peer review culture; future work </li></ul>
    4. 4. Our Systemic View of Graduate Learning - Concepts <ul><li>DISCLAIMER: No holism . “When rigorous contemporary social scientists hear the word 'system,' they are likely to draw their intellectual guns” </li></ul><ul><li>“ Systems have systemic (emergent) features that their components lack” </li></ul><ul><li>Everything is a system or an actual or potential component of a system / There are no permanent strays or isolates </li></ul><ul><li>Any concrete system σ can be modeled as μ ( σ ) = < C ( σ ), E ( σ ), S ( σ ), M ( σ ) > </li></ul><ul><ul><li>Composition : collection of all the parts of σ </li></ul></ul><ul><ul><li>Environment : collection of (other) items that act on or are acted upon by some or all components of σ </li></ul></ul><ul><ul><ul><li>Everything has an environment except for the universe as a whole. </li></ul></ul></ul><ul><ul><li>Structure : collection of relations, in particular bonds, among components of σ (endostructure) or among these and items of the environment (exostructure) </li></ul></ul><ul><ul><li>Mechanism : collection of processes in σ that make it behave the way it does – that generate qualitative novelty, emergence. </li></ul></ul><ul><ul><ul><li>Abstract systems have no mechanism </li></ul></ul></ul><ul><li>SOURCES: Bunge’s E&C (2003) and ‘Mechanism and explanation”, Phil Soc Sci (1997) </li></ul>
    5. 5. Our Systemic View of Graduate Learning – Systems… Size Bond strength Metabolism and accompanying processes that allow for maintenance and self-repair, expression and repression of genes etc. All the bonds (direct/indirect, physical/chemical, covalent/non-cov.) that keep  together plus all the ties - phys., chem., biol. - with environmental items . Medium rich in nutrients and energy fluxes, with variables (pressure, temperature etc.) within narrow intervals . Physical and chemical micro- and mesosystems, in partic. water, carbohydrates, lipids, proteins, and nucleic acids . Organism Activities that end up in the firm's products . Work relations among the firm's members;member- environ. item . Market and government. Personnel and management. Firm Production, transmission, and reception of symbols . Collection of linguistic communication relations . Culture(s) where the language is used . People who speak the same language . Linguistic community M Mechanism S Structure E Environment C Composition  System
    6. 6. <ul><ul><li>Structure (bonds of…) : [Endostr.] collaboration , communication , co-authorship/cooperation in projects , advising , teaching , peer pressure , [Exostr.] submission to rules, funding, refereeing , cultural influence, affiliation to UFSC, reputation-networking-partnership-affiliation with other organizations </li></ul></ul><ul><ul><li>Mechanism : study+research, advising, project/publication cooperation, communication* (scientific, objective criticism and argumentation) </li></ul></ul>Our Systemic View of Graduate Learning – Finally Funding agencies UFSC (host institution), staff Norms, regula- tions, laws faculty students Organizations (firms, research institutes, univ., NGOs, govern-mental, inter-national) Culture, Brazilian graduate education community
    7. 7. Approach to Growing a Peer Review Culture among Students <ul><li>Peer review in education by first author </li></ul><ul><ul><li>1994 as student (Virginia Tech), then since 1997 as professor </li></ul></ul><ul><ul><li>More than 1,000 ‘victims’ </li></ul></ul><ul><ul><li>Reasons to undertake: cooperation, written expression, critical thinking, professional responsibility (working on bonds and skills…) </li></ul></ul><ul><ul><li>Initiatives by several other scholars around the world (see refs) </li></ul></ul><ul><li>Our research seminars since 2005 (mandatory, no grade) </li></ul><ul><ul><li>About 8 4-h encounters: proposal preparation, live presentations and professors’ feedback, advice for refereeing (including sample referee forms) </li></ul></ul><ul><ul><li>Usually 2+ professors in class </li></ul></ul><ul><ul><ul><li>Feedback on form and on internal and relational coherence of proposal sections; no criticism on merit (because of methodology) </li></ul></ul></ul><ul><ul><li>Single-round, double-blind peer review (OJS) </li></ul></ul><ul><ul><ul><li>Each student gives and receives 3 feedbacks (ideally) </li></ul></ul></ul><ul><ul><ul><li>Full feedback to all students and advisors (spreadsheet summary) </li></ul></ul></ul><ul><ul><li>December: annual workshop (all freshmen present proposals) </li></ul></ul>
    8. 8. Approach to Growing a Peer Review Culture among Students (II) <ul><li>Objectives </li></ul><ul><li>For the students… </li></ul><ul><ul><li>Experience, acquaintedness : To know the essential parts of a thesis proposal, then re-elaborate the proposal presented for entrance in the graduate program and participate in a peer review round. </li></ul></ul><ul><ul><li>Maturing : To start developing competence to give and receive professional, objective critique of scientific work. </li></ul></ul><ul><li>For the graduate program </li></ul><ul><ul><li>Culture : To create and disseminate a culture of objective, interdisciplinary scientific criticism. </li></ul></ul><ul><ul><li>Spur thruput : To serve as catalyst of the advising process. </li></ul></ul><ul><ul><li>Interdisciplinarity : To stimulate interdisciplinary scientific interchange. </li></ul></ul>
    9. 9. Approach to Growing a Peer Review Culture among Students (III) <ul><li>Threats to the process </li></ul><ul><ul><li>Without strong guidance, it becomes an exchange of opinions (at best) </li></ul></ul><ul><ul><li>Upper management support desperately needed (ok) </li></ul></ul><ul><ul><li>Because we are so multi disciplinary (not inter -), Method is out (I know, that’s awful…) </li></ul></ul><ul><li>What we did to face the threats </li></ul><ul><ul><li>Adopted one successfully-in-use set of guidelines (2nd author, positivist approach, engineering method) </li></ul></ul><ul><ul><li>GP dean is always one of the professors </li></ul></ul><ul><ul><li>Required advisor involvement </li></ul></ul><ul><ul><li>Started a collaboration to tame methodology </li></ul></ul><ul><ul><ul><li>2007: 2 collaborators gave lectures </li></ul></ul></ul><ul><ul><ul><li>2008: 5 professors in 2 diff Method courses (elective quanti, quali) </li></ul></ul></ul><ul><ul><ul><li>2009: Intro to research methodology: 7 professors plus another one who attended all classes (then 3 complementary courses) </li></ul></ul></ul><ul><ul><ul><ul><li>Worldviews/main classes of paradigms; Methodological tour </li></ul></ul></ul></ul>
    10. 10. Approach to Growing a Peer Review Culture among Students (IV) <ul><li>Template for proposals – usual sections </li></ul><ul><ul><li>Abstract </li></ul></ul><ul><ul><li>Motivation </li></ul></ul><ul><ul><li>Problem statement and/or question </li></ul></ul><ul><ul><li>Objectives [might be split into general and specifics] </li></ul></ul><ul><ul><li>Relevance </li></ul></ul><ul><ul><li>Scope </li></ul></ul><ul><ul><li>Main references </li></ul></ul><ul><ul><li>General appraisal </li></ul></ul><ul><li>Software choices </li></ul><ul><ul><li>Conference management choices: either costly/unknown or too limited </li></ul></ul><ul><ul><li>Open Journal System: free, widespread, allows attachment </li></ul></ul><ul><ul><li>Attachment: Referee form spreadsheet (Excel, controlled content, easy to import to MySQL) </li></ul></ul><ul><ul><li>Referee form (next) </li></ul></ul>
    11. 11. Approach to Growing a Peer Review Culture among Students (V – referee form) A b Referee research area (KE/KMa/KMe), level (Dr/MSc), and self-declaration of expertise Proposal # and title, author area and level (all proposals from same level of referee’s, 2 out of 3 from same area) 1-paragraph summary of proposal Grades and comments for each proposal section Grades and comments for proposal as a whole
    12. 12. Results and Discussion <ul><li>Mostly anedoctal (and some exploratory research) </li></ul><ul><li>Proposals processed : 2005-49, 2006-67, 2007-54, 2008-62 </li></ul><ul><li>Additional students allowed for refereeing only </li></ul><ul><li>Numbers for 2008 (4 extra referees  198 allocations, 192 delivered) </li></ul><ul><li>Commitment and depth of feedback vary </li></ul><ul><ul><li>While it may be related to “mandatory, no credits, no grade”, strategy for improvement has been: raising awareness (related to communication bonds) </li></ul></ul>
    13. 13. Results and Discussion (II) <ul><li>Publication of full and aggregate results started in 2008 (Why: very limited resources; Before 2008: “here are your forms”) </li></ul><ul><ul><li>Aggregates: by [area, level, gender—not in the paper] , percents of self-declaration of expertise </li></ul></ul><ul><ul><li>For each proposal </li></ul></ul><ul><ul><ul><li>For each referee report (typically 3 for proposal) </li></ul></ul></ul><ul><ul><ul><ul><li>Research area of the referee </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Referee’s self-declaration of expertise </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Grades (0-10) and comments on the topics reviewed </li></ul></ul></ul></ul><ul><li>Report allows students to… </li></ul><ul><ul><li>See all referee reports about own proposal </li></ul></ul><ul><ul><li>Analyze, compare reports, reflect about quality of own work. </li></ul></ul><ul><ul><li>See what colleagues said about the same proposals refereed </li></ul></ul><ul><ul><li>Reflect about own skills as referee and about general quality of communication* bonds </li></ul></ul>
    14. 14. Results and Discussion (III-Aggregates 2008)
    15. 15. Results and Discussion (IV-Aggregates) <ul><li>Grades </li></ul><ul><ul><li>0-3 (reject), 4-5 (weak reject), 6-7 (weak accept), 8-10 (accept) </li></ul></ul><ul><ul><li>Given: From 6.67 (Scope, average) to 7.37 (General objective, avg) </li></ul></ul><ul><li>Self-declaration of expertise </li></ul><ul><ul><li>Even though students meet in 3+ mandatory courses plus the Seminars, they have a poor appraisal of their comparative expertise in the proposals’ topics… </li></ul></ul><ul><ul><li>… or maybe this is a cultural issue (you can’t claim to be so good) </li></ul></ul>
    16. 16. Results and Discussion (V) <ul><li>Opportunities for Improvement and R&D Issues </li></ul><ul><ul><li>Instrumental issues </li></ul></ul><ul><ul><ul><li>Better processing of peer review bureaucracy </li></ul></ul></ul><ul><ul><ul><li>(Semi-)automation of knowledge-intensive tasks, e.g. </li></ul></ul></ul><ul><ul><ul><ul><li>referee allocation </li></ul></ul></ul></ul><ul><ul><ul><ul><li>referee rating </li></ul></ul></ul></ul><ul><ul><ul><ul><li>process reliability and validity </li></ul></ul></ul></ul><ul><ul><ul><ul><li>evaluation with nonlinear dynamics (Losada: connectivity) </li></ul></ul></ul></ul><ul><ul><li>Methodological issues </li></ul></ul><ul><ul><ul><li>Articulation of Seminars with new mandatory course on Methodology (toward interdisciplinarity) </li></ul></ul></ul><ul><ul><ul><ul><li>Now we have some preparation to discuss paradigm and method choice </li></ul></ul></ul></ul><ul><ul><ul><li>Investigate mechanisms (Bunge) that create emergent properties in our graduate learning system </li></ul></ul></ul><ul><ul><ul><ul><li>Multilevel analysis (macro-micro systems) </li></ul></ul></ul></ul>
    17. 17. Concluding Remarks <ul><li>Experience report (mostly anedoctal; empirical research to come) </li></ul><ul><li>Computer use still very marginal </li></ul><ul><li>Students experience the main scientific method for quality control and have an opportunity to sharpen their knowledge and strengthen their (scientific, rigorous, objective) communication bonds with their peers and professors, adviser included </li></ul><ul><li>Part of our quest for interdisciplinarity through strengthening communication bonds </li></ul><ul><li>Aim at contributing, as well, to establish peer review as a replicable, scalable educational approach (Kern et al., 2007) </li></ul>
    18. 18. References Angelov, C., Melnik, R.V.N., Buur, J.: The synergistic integration of mathematics, software engineering, and user-centered design: exploring new trends in education. Fut. Gen. Comp. Sys. 19(8), 1299-1307 (2003) Araújo, L.H.L.: Uma aplicação da dinâmica não-linear para a avaliação de desempenho de comunidades virtuais de aprendizagem [An application of nonlinear dynamics to performance evaluation of virtual learning communities]. Masters dissertation, Universidade Católica de Brasília, Brasília-DF, Brasil (2004) Bunge, M.: Emergence and convergence : Qualitative novelty and the unity of knowledge. Uni. Toronto Press, Toronto (2003) Bunge, M.: Mechanism and explanation. Phil. Soc. Sci. 27(4), 410-465 (1997) Bunge, M.: Systemism: the alternative to individualism and holism. J. Socio-Econ. 29(2), 147-157 (2000) Cunningham, S.J.: Using a computer conferencing system to support writing and research skill development. ACM SIGCSE Bul. 26(4), 5-8 (1994) Davies, R., Berrow, T.: An evaluation of the use of computer supported peer review for developing higher-level skills, Comput. Educ. 30(1/2), 111-115 (1998) Denning, T., Kelly, M., Lindquist, D., Malani, R., Griswold, W.G., and Simon, B.: Lightweight preliminary peer review: does in-class peer review make sense? ACM SIGCSE Bul. 39(1), 266-270 (2007) Gehringer, E.F.: Electronic peer review and peer grading in Computer-Science courses. ACM SIGCSE Bul. 33(1), 139-143 (2001) Hafen, M.: Developing writing skills in computer science students. ACM SIGCSE Bul. 26(1), 268-270 (1994) Hartman, J.: Writing to learn and communicate in a data structures course. In: 20th SIGCSE Technical Symposium on Computer Science Education , pp. 32-36 (1989) Kern, V.M., Pacheco, R.C.S., Saraiva, L.M., Pernigotti, J.M.: Peer review in Computer Science education: Requirements for continuous, large scale application. In: F.M. Mendes Neto and F.V. Brasileiro (Eds.), Advances in computer supported learning , pp. 46-65, Idea Group Publishing, Hershey (2007) Kern, V.M., Saraiva, L.M., Pacheco, R.C.S.: Peer review in education: promoting collaboration, written expression, critical thinking, and professional responsibility. Educ. Inf. Technol. 8(1), 37-46 (2003) Liu, E.Z., Lin, S.S.J., Chiu, C., Yuan, S.: Web-based peer review: the learner as both adapter and reviewer. IEEE Trans. Educ. 44(3), 246-251 (2001) Losada, M., Heaphy, E.: The role of positivity and connectivity in the performance of business teams: A nonlinear dynamics model. Am. Behav. Scient. 47(6), 740-765 (2004) Losada, M.: The complex dynamics of high performance teams. Math. Comput. Model. 30(9-10), 179-192 (1999) Moreira, D.A., Silva, E.Q.: A method to increase student interaction using student groups and peer review over the internet. Educ. Inf. Technol. 8(1), 47-54 (2003) Riggs, T., Wilensky, R.: An algorithm for automated rating of reviewers. In: 1st ACM/IEEE-CS joint conference on digital libraries , pp. 381—387. Roanoke, (2001) Schreiber, G. et al.: Knowledge engineering and management : the CommonKADS methodology‎. MIT Press, Cambridge MA (2000) Sitthiworachart, J., Joy, M.: Web-based peer assessment in learning computer programming, In: 3rd IEEE International Conference on Advanced Learning Technologies (ICALT’03), (2003) Smith, A.J.: The task of the referee. IEEE Computer , 23(4), 46-51 (1990)
    19. 19. Thank you! Acknowledgments Authors GCS, SR, RTSL (volunteer students) Instituto Stela (network and systems administration services) Questions? Contact author: Vinícius Medina Kern, (see menu ‘Presentations…’ for these slides) Researcher at Instituto Stela , Professor at EGC/UFSC , (Both in Florianópolis-SC, Santa Catarina island)