This document summarizes a pilot study examining how students improve their writing over multiple essays through peer and instructor feedback. The study tracked feedback on 10 essays from 13 students. Both peer and instructor feedback improved linearly over the essays. Peer feedback ratings increased more slowly than instructor ratings. Positive instructor feedback on argument strength and style/mechanics correlated with improved essay quality. More analysis is needed to understand how students apply feedback to different essays and whether the quality of peer feedback improves over time with more sessions. The pilot showed this type of longitudinal study is feasible with a larger sample size.
Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Assessment Strand by Dr Gwyneth Hughes, Institute of Education and Dr Megan Crawford, Oxford Brookes University.
More details at www.cde.london.ac.uk.
Peer Feedback On Writing: Is More Better? A Pilot Study in Progress (poster)Christina Hendricks
This is a slide of a poster that was presented at the Society for Teaching and Learning in Higher Education conference in Vancouver, BC, Canada, in June of 2015. Download to view as Power Point slide and enlarge to see it all!
A presentation given at the BCcampus Symposium on Scholarly Inquiry into Teaching and Learning, Nov. 2014. I discuss a pilot research project on gauging the impact of peer feedback on writing over the course of multiple peer feedback sessions.
Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Assessment Strand by Dr Gwyneth Hughes, Institute of Education and Dr Megan Crawford, Oxford Brookes University.
More details at www.cde.london.ac.uk.
Peer Feedback On Writing: Is More Better? A Pilot Study in Progress (poster)Christina Hendricks
This is a slide of a poster that was presented at the Society for Teaching and Learning in Higher Education conference in Vancouver, BC, Canada, in June of 2015. Download to view as Power Point slide and enlarge to see it all!
A presentation given at the BCcampus Symposium on Scholarly Inquiry into Teaching and Learning, Nov. 2014. I discuss a pilot research project on gauging the impact of peer feedback on writing over the course of multiple peer feedback sessions.
Authentic Assessment Methods Workshop delivered at Yunlin Elementary School in Douliu, Yunlin County, Taiwan.
Link to Appendices: http://www.slideshare.net/slideshow/embed_code/29651700
An evidence-based model to enhance programme-wide assessment using technology: TESTA to FASTECH . Presented by Tansy Jessop and Yaz El-Hakim (University of Winchester) and Paul Hyland (Bath Spa University). Facilitated by Mark Russell (University of Hertfordshire).
Jisc conference 2011
Peer Evaluation Strategy for Improving Group Participation at Brightspace Ten...D2L Barry
"Peer Evaluation Strategy for Improving Group Participation;" at Brightspace Tennessee Ignite on February 13, 2015. Presenters: Brenda Kerr and Cary Greenwood, Middle Tennessee State University
Assessment in language learning classrooms:More questions than answersRobert Dickey
presentation for the 2015 TESOL Asia / Asian EFL Journal 14th Annual International Conference at Clark Freeport Zone, Pampanga, Philippines August 21-23, 2015
These slides are from a journal club discussion at the Pedagogic Research in the Biosciences group at the University of Leicester (UK). The meeting was reflecting on the paper "Oral versus written assessments: a test of student performance and attitudes" by Mark Huxham and colleagues from Napier University, Edinburgh. The paper is due to appear in the February 2012 edition of Assessment and Evaluation in Higher Education
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AV...Niki Lambropoulos PhD
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR
Niki Lambropoulos and Fintan Culwin presented at the Euro-CAT workshop in Barcelona 05/02/2010
Charlotte Perkins Gilman, "The Yellow Wall-Paper" and related writingsChristina Hendricks
Slides for a lecture for the Arts One program (http://artsone.arts.ubc.ca) at the University of British Columbia in Vancouver, BC, Canada. This lecture is about Gilman's "The Yellow Wall-Paper" as well as her book Our Androcentric Culture, or The Man-Made World. It also discusses historical and personal context to these writings, including neurasthenia and the "rest cure" proposed by Dr. S. Weir Mitchell.
Authentic Assessment Methods Workshop delivered at Yunlin Elementary School in Douliu, Yunlin County, Taiwan.
Link to Appendices: http://www.slideshare.net/slideshow/embed_code/29651700
An evidence-based model to enhance programme-wide assessment using technology: TESTA to FASTECH . Presented by Tansy Jessop and Yaz El-Hakim (University of Winchester) and Paul Hyland (Bath Spa University). Facilitated by Mark Russell (University of Hertfordshire).
Jisc conference 2011
Peer Evaluation Strategy for Improving Group Participation at Brightspace Ten...D2L Barry
"Peer Evaluation Strategy for Improving Group Participation;" at Brightspace Tennessee Ignite on February 13, 2015. Presenters: Brenda Kerr and Cary Greenwood, Middle Tennessee State University
Assessment in language learning classrooms:More questions than answersRobert Dickey
presentation for the 2015 TESOL Asia / Asian EFL Journal 14th Annual International Conference at Clark Freeport Zone, Pampanga, Philippines August 21-23, 2015
These slides are from a journal club discussion at the Pedagogic Research in the Biosciences group at the University of Leicester (UK). The meeting was reflecting on the paper "Oral versus written assessments: a test of student performance and attitudes" by Mark Huxham and colleagues from Napier University, Edinburgh. The paper is due to appear in the February 2012 edition of Assessment and Evaluation in Higher Education
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AV...Niki Lambropoulos PhD
Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR
Niki Lambropoulos and Fintan Culwin presented at the Euro-CAT workshop in Barcelona 05/02/2010
Charlotte Perkins Gilman, "The Yellow Wall-Paper" and related writingsChristina Hendricks
Slides for a lecture for the Arts One program (http://artsone.arts.ubc.ca) at the University of British Columbia in Vancouver, BC, Canada. This lecture is about Gilman's "The Yellow Wall-Paper" as well as her book Our Androcentric Culture, or The Man-Made World. It also discusses historical and personal context to these writings, including neurasthenia and the "rest cure" proposed by Dr. S. Weir Mitchell.
These slides are for a video, so they don't have a lot of information on them by themselves. The link to the video will be posted here as soon as the video is online.
A presentation given at Open UBC week at the University of British Columbia in Vancouver, Oct. 23, 2013. Much of the second half of the presentation was spent browsing the linked websites, so there isn't much on the slides for the second half!
A presentation given to the CTLT Institute (Centre for Teaching, Learning and Technology) at the University of British Columbia in May, 2013. In it I introduce open education, MOOCs, xMOOCs vs cMOOCs, and discuss ETMOOC--a cMOOC I participated in in 2013--as an example of a cMOOC to better explain what (some) cMOOCs are like.
Training Session I ran on AFL in the classroom using peer and self assessment.
More resources can be found at tes.co.uk
search 'dominic penney' in the resource search bar...
A presentation on various ways one might try to evaluate the effectiveness of cMOOCs, and some questions and concerns about each one, ending with a question: how best should we do this?
A lecture on Freud's case history about Dora (1905) as well as his lecture entitled "Femininity" (1933) for Arts One (a first-year, interdisciplinary course) at the University of British Columbia in Vancouver, BC, Canada.
These slides are for an introduction to philosophy course at the University of British Columbia. They focus on Thomson's "bystander at the switch," "loop," and "fat man" trolley cases.
These slides are for an Introduction to Philosophy course at the University of British Columbia in Vancouver, BC, Canada. They cover chapters 1, 2 and 5 of Mill's text called Utilitarianism. There is also a slide towards the end distinguishing act and rule utilitarianism.
These slides are for a discussion of Camus' "The Myth of Sisyphus" in an Introduction to Philosophy course at the University of British Columbia in Vancouver, BC, Canada. There are three animated gifs embedded in it, which may not play correctly here. This is most of the slides--there may be one or two more later.
Dose-response curve for peer feedback on writing: A pilot studyChristina Hendricks
These slides are for a presentation at ISSOTL 2016 (the International Society for the Scholarship of Teaching and Learning. Please direct any questions to me at the contact information provided at the end!
Pilot study: Longitudinal analysis of peer feedback in a writing-intensive co...Christina Hendricks
Results of research on students' use of peer comments for improving later essays (rather than drafts of the same essay). Presented at the Festival of Learning in Burnaby, BC, Canada in June 2016.
"Discussion boards don’t work": Evaluation of a course blog for teaching with...Chris Willmott
Slides from a presentation given at the Horizons in STEM Higher Education (Virtual) Conference, 30th June 2021. I discussed an initiative in which students had been asked to contribute to a "Shared Resource Collection" instead of a terminal exam paper. The trial was only partially successful, as demonstrated by the data in the presentation (and additional data after the final "Any Questions" slide, which was not shared at the event.
It's Not Just About the Money: Open Educational Resources and PracticesChristina Hendricks
Slides for a presentation at an event called Open Art Histories at Langara College in Vancouver, BC, Canada in January 2020. They are meant to explain the what, how and why of OER and OEP. Editable power point slides: https://osf.io/x9s5n/.
Slides from a workshop on the Scholarship of Teaching and Learning at Lakehead University in November 2019. They include an introduction to SoTL and information/activities on getting started with a research question and thinking about which data one might collect to fit that question.
Slides that introduce SoTL: what it is, some examples, and why one might do it. Presented to a few groups at Lakehead University in November 2019. Slides available to download w/o slideshare account: https://osf.io/xkw4g/
Slides for a talk at the Justice Institute of British Columbia in November 2019, designed to introduce open educational resources. PowerPoint slides available: https://is.gd/oerjibc2019
Downloadable & editable files: https://osf.io/nstbq
Slides for a presentation at the BCCAT (British Columbia Council on Admission and Transfer) articulation meeting for Philosophy in May, 2019. Discusses what OER are and how to find some OER and open textbooks to use for philosophy courses.
Download and edit here: https://osf.io/zvnqy/
Presentation at Vanderbilt University February 22, 2019. Discusses open educational practices, open pedagogy, and the values, benefits, challenges and risks of these.
Downloadable/editable slides: https://osf.io/5gf3n/
Presentation for a workshop at the Student Union Development Summit at the University of British Columbia in Vancouver, BC, Canada.
Slides for a 2-day workshop at Davidson College in North Carolina, USA. See the site I created for the workshop for more info and to download slides in power point format: https://chendricks.org/oep2018/
Here are the day 2 slides for this workshop: https://www.slideshare.net/clhendricksbc/open-educational-practices-davidson-college-day-2
Slides for a 2-day workshop at Davidson College in North Carolina, USA. See the site I created for the workshop for more info and to download slides in power point format: https://chendricks.org/oep2018/
Here are the day 1 slides for this workshop: https://www.slideshare.net/clhendricksbc/open-educational-practices-davidson-college-day-1-109408680
Students and Open Education: From the What to the How and Why (and When Not)Christina Hendricks
A keynote given at the eCampus Ontario Technology-Enhanced Seminar and Showcase 2017. https://tess17.ecampusontario.ca/home
Slides are available in an editable (PPTX) format at the Open Science Framework: https://osf.io/fcz5x/
Slides for a talk I gave at Douglas College in the Vancouver, BC (Canada) area, during open access week 2017. You can download the slides as power point on my blog: http://blogs.ubc.ca/chendricks/2017/11/11/presentation-whats-open-about-open-pedagogy/
The slides talk about what "open pedagogy" might be, showing how some people have defined it and then coming up with a list of six categories of things that are common to more than one definition of open pedagogy. They then ask what it is that these definitions share that relates to openness: what's "open" about open pedagogy?
Beyond Cost Savings: The Value of OER and Open Pedagogy for Student LearningChristina Hendricks
Slides from a workshop at Mt. Royal University March 9, 2018, for Open Education Week. These slides discuss Open Educational Practices and Open Pedagogy, and examples of each.
These slides are downloadable in Power Point format on my Open Science Framework repository: https://osf.io/kctf3
Slides for an Introduction to Philosophy course at the University of British Columbia in Vancouver, BC, Canada. These slides talk about Singer's articles: "Famine, Affluence & Morality," and "The Singer Solution to World Poverty"
O'Neill on Kant's second form of the Categorical ImperativeChristina Hendricks
Slides for an Introduction to Philosophy course at the University of British Columbia in Vancouver, BC, Canada. This is the first set of slides for O'Neill's text, "Kantian Approaches to Some Famine Problems"; there will be more slides added later.
These are the final versions of slides for a talk I gave at Douglas College in the Vancouver, BC area for Open Access Week in October 2017 (an earlier version is also posted here on SlideShare because I gave that URL out before, and SlideShare no longer allows replacing old files with new ones at the same URL).
The slides talk about what "open pedagogy" might be, showing how some people have defined it and then coming up with a list of six categories of things that are common to more than one definition of open pedagogy. They then ask what it is that these definitions share that relates to openness: what's "open" about open pedagogy?
Slides for a talk at Douglas College in the Vancouver area, British Columbia, Canada, during Open Access Week 2017. The talk was about what "open pedagogy" means, and whether and why the word "open" fits it.
These are not the latest versions of the slides, but SlideShare no longer allows replacing slides with a new file at the same URL, so I'm keeping these here because I shared this URL with others previously. Here is the URL for the final version of these slides: https://www.slideshare.net/clhendricksbc/whats-open-about-open-pedagogy-final-version
Nozick, "The Experience Machine" and Wolf, "The Meanings of Lives"Christina Hendricks
These slides are for an introduction to philosophy course at the University of British Columbia in Vancouver, BC, Canada. The first half of the lecture on Wolf's article was done by a guest lecturer so those slides are not here.
These slides are for an Introduction to Philosophy course at the University of British Columbia in Vancouver, BC, Canada. They discuss a couple of Nagel's purported "bad arguments" for saying life is absurd, then his view of why human life is absurd, and how we should respond to that.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasn’t one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Palestine last event orientationfvgnh .pptxRaedMohamed3
An EFL lesson about the current events in Palestine. It is intended to be for intermediate students who wish to increase their listening skills through a short lesson in power point.
1. Tracking a Dose-Response Curve
in Peer Feedback on Writing
A Work in Progress
Christina Hendricks
Co-Investigator: Jeremy Biesanz
University of British Columbia-Vancouver
SoTL Symposium, November 2015
Slides available here: http://is.gd/PFBwriting2015
Slides licensed CC-BY 4.0
2. Literature on peer feedback
Receiving peer
feedback improves
writing
(Paulus, 1999; Cho & Schunn,
2007; Cho & MacArthur, 2010;
Crossman & Kite, 2012)
Giving peer feedback
improves writing
(Cho & Cho, 2011; Li, Liu &
Steckelberg, 2010)
3. GAPS:
Most studies look at revisions to a single
essay, not changes across different essays
Draft 1 Draft 2 Draft 3
Essay 1 Essay 2 Essay 3 Essay 4 Essay …n
PFB
PF
B
PF
B
PFB PF
B
PFB
Few studies look at “dose-response curve”
4. Pilot study research questions
1. How do students use peer comments given and
received for improving different essays rather than
drafts of the same essay?
1. Are students more likely to use peer comments
given and received for improving their writing
after more than one or two peer feedback
sessions? How many sessions are optimal?
2. Does the quality of peer comments improve over
time?
5. • Interdisciplinary, full year course for first-years
• 18 credits (6 each in 1st year English, History,
Philosophy)
• Students write 10-12 essays (1500-2000
words)
• Peer feedback tutorials every week (4 stdnts)
http://artsone.arts.ubc.ca
Toni Morrison, Wikimedia Commons,
licensed CC BY-SA 2.0
Osamu Tezuka, public domain
on Wikimedia Commons
Jane Austen, public domain on
Wikimedia Commons
Friedrich Nietzsche, public
domain, Wikimedia Commons
6. Data for pilot study 2013-2014
• 10 essays by 13 participants (130 essays)
• Comments by students in tutorial group (4 in
group) on all essays (n=1219)
• Comments by instructor on all essays
(n=3331)
• All essays and comments coded according to
a common rubric
7. Coding Rubric
Categories
(plus
subcategories, for
11 options)
• Strength of argument
• Organization
• Insight
• Style & Mechanics
Numerical
value
1: Significant problem
2: Moderate problem
3: Positive comment/praise
E.g., STREV 2: could use more textual
evidence to support your claims
8. Progress so far
All student and instructor comments coded
60 of 130 essays
coded
-- (10 essays
each by 6 of the
13 participants)
9. Inter-coder reliability
Fleiss’ Kappa Intra-class
correlation
Student
comments
(n=141)
All categories: 0.61 (moderate)
Most used categories: 0.8
(excellent)
0.96
(excellent)
Essays (n=60) 0.68
(adequate)
3 coders:
• Daniel Munro & Kosta Prodanovich
(undergrads, former Arts One)
• Jessica Stewart (author, editor)
11. 0 2 4 6 8 10 12
024681012
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
INSTRUCTOR
Comments
-
.28**Strength
Style
Organiz.
Insight
-.04*
Number of 2 comments in each categ.
12. 0 2 4 6 8 10 12
01234
Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
STUDENT
comments
Strength
Style
Organiz.
Insight
-.16**
Number of 2 comments in each categ.
13. 0 2 4 6 8 10 12
012345
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
INSTRUCTOR
Comments .31***
Strength
Style
Organiz.
Insight
.08**
.19**
.11**
Number of 3 comments
in each category
14. 0 2 4 6 8 10 12
0.00.51.01.52.02.53.0
Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
STUDENT
Comments
Strength
Style
Organiz.
Insight
Number of 3 comments in each categ.
15. Student/instructor
agreement on comments
• Average numerical ratings for comments
across all categories agree strongly
between student and instructor (.48****)
• However, this agreement
goes down across essays (-.04**)
o This is because student ratings increase
over time at only half the rate that instructor
ratings do
*p < .05, **p< .01, ***p< .001, ****p < .0001
16. HOW DOES ESSAY QUALITY
CHANGE OVER TIME?
From this slide onwards, we are looking only at
60 essays coded so far, out of the set of 130
(essays from 6 of the 13 participants)
19. Cross-lagged panel design with
auto-regressive structure
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
20. Path A: Student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects:
• Ratings of 2 in Insight (-.53*)
• Ratings of 3 in Strength (.25*)
21. Path C: instructor comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects
• Ratings of “2” in Strength (-.31**)
• Ratings of “3” in Strength (.51***) and Style/Mechanics
(.34**)
22. Path C: student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects
• Ratings of “2” in Style (.28*) and Insight (.15*)
• Ratings of “3” in Strength (.30*)
23. Path D: Student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects:
• Ratings of 2 in Organization (.16*)
• Ratings of 3 in Style (.16*)
24. Research question 2
How do students use peer comments given
and received for improving different essays
rather than drafts of the same essay?
o Not enough evidence yet to say much about
path D
o Haven’t yet looked at differences in
comments given vs. received
25. Research question 2
Are students more likely to use peer
comments given and received for improving
their writing after more than one or two peer
feedback sessions? How many sessions are
optimal?
o No evidence yet that there is any change
over time in path D
26. Research question 3
Does the quality of peer comments improve
over time?
o No evidence yet that there is any change over
time in path A
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
27. Some conclusions so far
• Pilot study: is this sort of study feasible for
larger sample?
o Yes, but probably more so if instructors code
essay quality rather than coders
• Facilitating easy collection of student &
instructor comments is difficult
28. References
• Cho, K., & MacArthur, C. (2010). Student revision with peer
and expert reviewing, Learning and Instruction. 20, 328-338.
• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving
comments. Instructional Science, 39, 629-643.
• Cho, K. & Schunn, C. D. (2007). Scaffolded writing and
rewriting in the discipline: A web-based reciprocal peer review
system. Computers & Education, 48, 409–426
• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved
writing among students through directed peer review, Active
Learning in Higher Education, 13, 219-229.
• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or
assessee: How student learning improves by giving and
receiving peer feedback. British Journal of Educational
Technology, 41(3), 525–536.
• Paulus, T. M. (1999). The effect of peer and teacher feedback
on student writing. Journal of Second Language Writing, 8,
265-289.
29. Thank you!
Christina Hendricks
University of British Columbia-Vancouver
Website: http://blogs.ubc.ca/christinahendricks
Blog: http://blogs.ubc.ca/chendricks
Twitter: @clhendricksbc
Slides available: http://is.gd/PFBwriting2015
Slides licensed CC-BY 4.0
Editor's Notes
How much agreement do we observe relative to how much we would expect to see by chance?
-- takes into account the frequency of the type of code occurring in the data
-- some codes are more frequent, so you’d expect those to have more apparent agreement
-1 to +1
0 = amount of agreement we’d expect to see by chance
-1 is complete disagreement
0.6 is moderate agreement; 0.8 is substantial
-- Kappa includes just the category
Many of the mostly used categories have agreement in 0.8 range
Reliability on degree: intra class correlation (ICC) of 0.96
-- to what extent is the average across the three raters reliable: average of all the numbers each gave—how does this correlate with the average of everyone who could possibly do this—get no benefit for adding more people
-- average is 2.5
-- 1’s are pretty infrequent
-- people agree on whether a 2 or a 3 (40% are 2s, 60% are 3s)
Refined coding rubric
Added, subtracted, condensed dimensions acc. to student comments
Added examples of each sub-dimension
Out of 242 peer comments:
All 3 coders agree on value (1-3), regardless of dimension: 90%
2 or 3 coders agree on dimension and final decision (after meeting) is same as that: 82%
2 coders agree on dimension & final is different: 12%
3 coders agree on dimension & final is same: 56%
Just last set of 70 comments
Single meaning units
Had several comments that could be given more than one code; needed to split them up so each comment had one code so as to better do analysis