This document summarizes a presentation on assessing online writing. It finds that current practices for assessing online discussions are inconsistent and often assess participation over academic quality. Rubrics varied widely in expression and criteria, with most focusing on non-academic factors like participation rather than critical thinking or learning objectives. The document concludes current assessment lacks fidelity and recommendations designing rubrics that embrace online writing's affordances and assess learner contributions with high fidelity.

1. Assessing Online WritingCAA Conference, 201020-21 July, 2010bobby.elliott@sqa.org.uk

2. http://www.scribd.com/bobbyelliott

4. QuestionDo you feel confident about assessingstudents’ online writing?

5. Text messageText keycode to 07624806527SmartphoneGo to poll4.com and enter keywordTwitterTweet keyword with comment to @pollWebhttp://www.polleverywhere.com/multiple_choice_polls/MTA0ODQwOTAwOA/webHow confident are you when assessing students’ online writing?Very: 24697Somewhat: 25157Not at all: 25192

6. Summary of presentationResearch questionMethodologyFindingsRecommendations

7. Some background stuffDefinition of assessmentCharacteristics of assessmentNew affordances

8. AssessmentLatin root: assidere: “to sit beside”“Observing learning”

9. Definition of assessment“Assessment is an on-going process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analysing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance.”

10. Characteristics of assessmentValidityReliabilityFidelity

11. Fidelity“Fidelity is the extent to which elements that contribute to a course grade are correctly identified as academic achievement”.“Many academics cannot help but be impressed by the prodigious time and persistence that’s some students apparently invest in producing responses to an assessment task. However, effort is clearly an input variable and therefore does not fall within the definition of academic achievement”.“Assessment in which early understandings are assessed, recorded and counted, misrepresents the level of achievement reached at the end of the course.”

12. New AffordancesLarge-scale projectsDeveloped co-operatively and collaborativelyImproved authenticityReal life valueLarge audienceRicher contentMore reflectiveMore transparentPeer reviewed/assessed

13. Transforming Assessment ProjectAn Australian Learning and Teaching Council Fellowship specifically looking at the use of e-assessment within online learning environments, particularly those using one or more Web 2.0 or virtual world technologies.http://www.transformingassessment.com/geoffrey.crisp@adelaide.edu.au

14. Challenges“Most of its advocates [of writing using Web 2.0 tools] offer no guidance on how to conduct assessment that comes to grips with its unique features, its difference from previous forms of student writing, and staff marking or its academic administration.”“The few extant examples appear to encourage and assess superficial learning, and to gloss over the assessment opportunities and implications of Web 2.0’s distinguishing features.”

15. Research QuestionWhat is the current practice in the construction of rubrics to assess online discussions?

16. Methodology20 papers128 criteria/statements33 different criteria10 categories of criterion

17. Small PrintSmall sampleNot statistically robustNot generalisableNo claims about validity or reliability“However, a clear pattern emerged from the rubrics to give confidence that a more systematic approach may not have produced a significantly different outcome. The pattern of rubrics in the literature was established early in the review and was re-inforced as the sample was increased.”

18. Findings: Expressing CriteriaExpression varied considerably from rubric to rubric.Some were expressed as criteria, some were written as statements, and some were simply examples of contribution at a particular performance point.Little consistency in terminologyFor example, the quality of academic discourse was expressed in terms of “depth of insight”, “intellectual challenge”, “quality of knowledge”, and “much thought”.Subjective language was commonWords such as “good”, “substantive” and “regular” (without further definition) were often used.Language used varied in formalityWhile the majority of rubrics used formal English, some used a conversational style of language including such phrases as “Your messages didn’t leave me jazzed”.

19. Findings:Frequency of criteria

20. Findings:Criteria categorised

21. Findings:Most common criteriaParticipationAcademic discourseEtiquetteLearning objectivesCritical thinking

22. Findings: CommentaryOf the 10 categories, five broadly relate to academic standards and five relate to such things as the number of messages posted, how often students visited the forum or the attitude of the student (such as her “being positive”).Eight rubrics made direct or indirect reference to the associated learning objectives. 12 rubrics made no reference to the learning outcomes whatsoever.Eighteen rubrics rewarded participation, which was the single most common category.Four rubrics did not include any criteria relating to academic discourse whatsoever, preferring to focus solely on participation or etiquette or other non-academic criteria.Only one rubric made explicit reference to “learning objectives”.

23. Case studiesRovai: Online and traditional assessment: what is the difference? (2000)14 criteria for the assessment of online discussions, aligned against three grades (A-C)10 relate to the number of messages posted or the style of the message (“caring”, “compassionate”, “rude”) or other non-achievement variables.4 criteria relate to academic standards (including one that relates to the standard of spelling and grammar).Hazari: Strategy for Assessment of Online Course Discussions, (2004)20 criteria (for awarding points ranging from 1 to 5).10 relate to non-achievements such as “Demonstrates leadership in discussions” and “Posted regularly throughout the week”.Some criteria were crude: “Time between posting indicated student had read and considered substantial number of student postings before responding”

24. ConclusionsWe’re not sure what we’re assessingWe’re assessing the wrong thingsWe don’t know how to assess online writingWe’re not sure what online writing isWe’re unclear about how it’s differentWe don’t use rubricsWe write rubrics differentlyThe rubrics we use are pretty poorWe’re being unfair to students

25. Characteristics of a good rubric

26. RecommendationsAssess learner contributions to online discussions.Use a rubric to carry out the assessment.Design rubrics to have high fidelity.Incorporate the unique affordances of online writing into rubrics.

27. ChallengesEmbracing online writingAcknowledging the new affordancesReconsidering how we assess studentsCreating new assessment activitiesDesigning new marking rubricsBetter qualityImproved fidelityEmbedding the new affordances

28. bobby.elliott@sqa.org.uk