E assessment taylor2011
Upcoming SlideShare
Loading in...5

E assessment taylor2011



Assessment, technology and learning: who is in the driving seat?...

Assessment, technology and learning: who is in the driving seat?
Prof Josie Taylor, Director, Institute of Educational Technology, The Open University
This presentation examines assessment in the context of open educational resources and informal learning. I examine the concept of assessment 2.0, and the emancipatory effect of new forms of e- assessment that put students in the driving seat.



Total Views
Views on SlideShare
Embed Views



1 Embed 1

http://www.linkedin.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • One of the challenges for tertiary education will be the drive toward openness resulting from web 2.0/3.0 technologies. These are becoming a more and more familiar part of our lives now, and will increasingly be so in the future. This is not technological determinism – the key point is that social media are optional. You can choose not to use them if you wish, and many other means will still be available to people to achieve their goals. However institutions will be challenged by populations of students who want to use these media to serve both formal and informal learning.
  • An example of things to come …
  • What sorts of people are using this system?
  • If the people using the system are not registered with an institution, how are they defining their learning goals?
  • This is not a comprehensive view – just a few random points on a trajectory. But who is driving this progression, and how? If the institution gets involved, is it compromising learner autonomy? Or can we support that autonomy in appropriate ways?
  • But the positioning of the learner with regard to the academy, or the institution, is interestingly balanced. Who has the last say in what is valid for study? This issue is picked up in the literature around digital literacies.
  • Learners do not come as empty vessels – constructivist approaches. Texts have traditionally been the means by which institutional authority has been sustained, coupled with assessment processes. However, not only are learners breaking free from the academy, so are texts.
  • Lea and Jones highlight the fact that although we have studies of students’ position with regard to technology, we have very little in the way of detailed study of engagement.
  • The ecological approach to media use can help to an extent. At the OU we have been looking at the cultural communities that grow around our technologies (OpenLearn, iSpot, Cloudworks), and observing the activities they support. We are now doing it now for ITunesU and UTube
  • The nature of the community and the flattening of authority
  • Consistent with affinity spaces, a concept developed by Gee.
  • Informal learning in an affinity space is much more optional, much more dependent on self motivation. Students may be equally keen on some form of assessment to keep themselves motivated.
  • Things good assessment practice supports! Lets push a bit deeper into that pedagogy
  • Whitelock points us to the work of Elliott, who identifies the characteristics of Web 2.0 assessment. These are very consistent with the work of Gee and Jenkins mentioned earlier. Note the collaborative, personalised nature of the activity.
  • Bartlett-Bragg elaborates a pedagogy for self-publishing as a means of assessment focused around reflection and peer review. Whitelock advises that frameworks such as this require a supportive infrastructure because its not just a case of developing tools and promoting collaboration. Staff need to know how to capitalise on the affordancies of web 2.0. to reconceptualise their assessment practice. Students also need to know what to do next.
  • Hence the 4Ts pyramid. As tutors realise how to transform assessment to take advantage of web 2.0, so learners will be able to development their independence. However, we need to support that transfer.
  • Whitelock’s notion of ‘Advice for Action’ emphasises that if students don’t know what to do next, the value of assessment feedback/feedforward is limited. We can be more explicit about this. E.g. Open Comment
  • Whitelock and Watt have developed both Open Mentor (which supports tutors in their assessment activity) and Open Comment (which assists students in submitting better assessed pieces of writing).
  • This approach of providing open feedback is being trialled in the Arts faculty – students are able to benefit from comments prior to formal submission of assessed work.
  • In this context, motivation should not be a problem.
  • Across many of the social networking applications, we can now support a much wider range of audiences on the long tail – who were increasingly discovering, recommending, and linking the OU on a range of platforms and media outlets. Its not all about what the institution can do by itself…

E assessment taylor2011 E assessment taylor2011 Presentation Transcript

  • Assessment, technology and learning: who is in the driving seat? Prof Josie Taylor Director, Institute of Educational Technology, The Open University Middlesex, 2011
  • The Open University, UKHigher education needs toembrace a more openfuture. This will entailchanges that are likely tobe profoundWe haven’t yet fullyunderstood what thesechanges are, or what theimpact will be onorganisations, staff orstudents
  • OpenLearn at The Open University2006 – William and FloraHewlett foundationprovided the OU withfunds to investigatesharing educationalresources and moreopen approaches www.open.ac.uk/openlearn
  • Open Educational ResourcesOur definition of OER:“The open provision ofeducational resources, enabledby information andcommunication technologies,for consultation, use andadaptation by a community ofusers for non-commercialpurposes.” www.open.ac.uk/openlearn
  • OpenLearn Designed on a model analogous to the open source software movement >14 million unique visitors have used OpenLearn since 2006 Gradual build of user base Olnet.org
  • Studies by OLNet: Patrick McAndrewUndertaking analysis ofuser behaviour.The results from one ofthese studies (n = 2,011)highlighted two distinctclusters of learners:"volunteer" students"social" learners McAndrew, Scanlon & Clow (2010)
  • Volunteer studentsVolunteer students sought the content they wanted to learn from, and they expected to work through it. These learners were most interested in:  more content  tools for self-assessment  ways to reflect on their individual learning. McAndrew, Scanlon & Clow (2010)
  • Social LearnersSocial learners were less motivated to work through the content. Rather, they seem to see learning as a way to meet people with shared interests. These learners were most interested in:  communication tools  advanced features on the website. McAndrew, Scanlon & Clow (2010)
  • What are these informal learnerstrying to do? How might they frame their tasks? How will they know when they have succeeded? i.e. what ‘counts’ as success? What will be the quality of the experience? How can we best support them?
  • Process of emancipation for new learnersas confidence builds Lots of other stops along the way...
  • Learner emancipation & institutionalauthority For learners:  Not ‘just’ skill/meta-skill acquisition  Profound developmental stages for the individual For the academy:  what is a university for?  In an open world, who determines what is (or should be) of value?  Who holds the power to say ‘this is worthy’ or ‘this is valid’?
  • Digital LiteraciesMary Lea & Robin Goodfellow Learners bring a wealth of experience to bear – some appropriate, some not Learners are engaged in meaning-making Recognition of the central role of texts in construction of knowledge and practice of learning Potential shifts of power between learners, communities and institutions Role of the institution is critically important Boundaries of ‘texts’ are more fluid and unstable than in previous times
  • Improving our understanding of student behaviour? ‘Rich accounts in the literature of students’ use of technology’ ‘No detailed or in depth examination of what students actually do in contexts when using different applications, or how meanings are being made from, and through, engagement with digital technology’ ‘Recognition of the central nature of texts both in the construction of knowledge and the practice of learning’ Lea and Jones (2011)
  • Ecological approach Interrelationship among all the different communication technologies and  the cultural communities that grow up around them  the activities they support. ‘Interactivity is a property of the technology, while participation is a property of culture.’ Jenkins (2004)
  • Ecological Space in which learning happens MembersContribute feel somewhen you connection want – they care
  • Affinity Spaces Gee (2009)
  • Distinctions between formal andinformal learning spaces Formal education system Informal affinity space  Conservative  Experimental  Static  Innovative  Structures to sustain are  Structures to sustain are institutional provisional  Remain little changed over  Can respond to short-term long periods of time needs and temporary interests  Communities are bureaucratic  Communities are ad hoc and and often national localised  Does not allow for easy  Allows for easy moves in and movement in and out out of informal learning communities Gee (2009)
  • Web 2.0 Technology supports open pedagogy:peer support, communication and sharingIssues for Institutions Wider sociological issues Blurring boundaries  Need to prepare learners between formal and for future information informal learning overload as the Web Ways of learning are grows exponentially coming out of the  Need to sharpen critical academy awareness, critical skills, Emancipation is in the and concepts of authorial hands (and feet!) of voice learners
  • The e-Assessment Challenge…Push of Constructivist Learning …Pull of institutional reliability and accountability Slide courtesy of D.Whitelock, 2011
  • Assessment 2.0Denise WhitelockCharacteristic DescriptorAuthentic Involving real-world knowledge and skillsPersonalised Tailored to the knowledge, skills and interests of each studentNegotiated Agreed between the learner and the teacherEngaging Involving the personal interests of the studentsRecognition of existing skills Willing to accredit the student’s existing workDeep Assessing deep knowledge – not memorizationProblem orientated Original tasks requiring genuine problem solving skillsCollaboratively produced Produced in partnership with fellow studentsPeer and self assessed Involving self reflection and peer reviewTool supported Encouraging the use of ICT Characteristics of Assessment 2.0, Elliott (2008) in Whitelock & Watt (2008)
  • Pedagogical Framework for self-publishing withsocial software1. EstablishmentLearners actively create personalised learning environments with socialsoftware e.g. weblogs, wikis, social bookmarking and aggregation.2. InterpretationLearners develop a structure and adapt it to their perceived needs.3. Reflective MonologuesLearners publish to their software platform and establish their identity4. Reflective DialoguesLearners extend their learning environment by developing social networks.5. Distributed knowledge artefactsLearners collaborate with others, distribute their work, and gatherartefacts for review and reflection. Bartlett-Bragg (2007) in Whitelock & Watt (2008)
  • The 4Ts pyramid to facilitate moving forward withAssessment Frameworks and Web 2.0 ToolsTool development Transfer of learning from assessment tasks which include Advice for Learning Transformation of Assessment tasks Training of Staff Tool development Adapted from Whitelock (2010)
  • Advice for Action The role of socio-emotive content in feedback is critically important and cannot be ignored (e.g. Draper, 2009b). Assessment practices that focus on self assessment and peer feedback need to develop towards ‘Advice for Action’, i.e. stimulus advice for transformational change in students to get them to:  think differently  to reconceptualise the way they respond  to engage actively in the discourse Whitelock & Watt (2008)
  • Open Comment Automated formative assessment tool Free text entry for students Automated feedback and guidance Open questions, divergent assessment No marks awarded For use by Arts Faculty Whitelock & Watt (2008)
  • Stages of analysis of students’ free textentry for Open Comment: Advice with respect to content (socio-emotional support stylised example): STAGE 1a: DETECT ERRORS E.g. Incorrect dates, facts. (Incorrect inferences and causality is dealt with below)  Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go  You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y Whitelock & Watt (2008)
  • Computer analysis continued STAGE 2a: REVEAL FIRST OMISSION  Consider the role of Z in your answer Praise what is correct and point out what is missing  Good but now consider the role X plays in your answer STAGE 2b: REVEAL SECOND OMISSION  Consider the role of P in your answer Praise what is correct and point out what is missing  Yes but also consider P. Would it have produced the same result if P is neglected? … and so on Several other stages of analysis not discussed here Whitelock & Watt (2008)
  • Role of technologyAdrian Kirkwood and Linda Price Student behaviour is not driven by technology per se, but by the way in which technology is used to support learning and teaching.  ‘If academic staff genuinely want their students to be analytical and critical thinkers, and able to apply their learning to novel situations and transfer their learning to solve real problems … then their assessment methods should firstly, encourage the development of such abilities; and secondly, provide students with the opportunity to demonstrate that they have developed these higher order abilities.’ Scouller (1998) This is as true for Web 2.0 as it was in 1998 Kirkwood & Price (2008)
  • Student expectations Säljö (1979): what do you understand learning to be?  Learning as the increase in knowledge.  Learning as memorisation.  Learning as the acquisition of facts, procedures, and so on, that can be retained and/or utilised in practice.  Learning as the abstraction of meaning.  Learning as an interpretive process aimed at the understanding of reality. Beaty, Dall’Alba, and Marton (1997) added:  Learning as personal development. Kirkwood & Price (2008)
  • Quantitative change (passive):Learning as the increase in knowledge.Learning as memorisation.Learning as the acquisition of facts, procedures, and soon, that can be retained and/or utilised in practice.Qualitative change (active):Learning as the abstraction of meaning.Learning as an interpretive process aimed at theunderstanding of reality.Learning as personal development. Kirkwood & Price (2008)
  • Learners are emancipating themselves  Web 2.0 technology has the potential to enable many activities to be mediated on-line, thus making the establishment of communities much easier, much cheaper and possibly more rewarding for participants  Affinity spaces can be spontaneously formed around any topic, so can be rapid, responsive and flexible means to effective learning
  • Global News KNOWLEDGE World Serviceopenlearn
  • Can assessment practice keep up? Appropriate socio-emotive feedback Automated, but highly relevant, feedback/forward Focus on meta-level skills and their assessment Showing students how to value their own work and that of peers – peer critique is highly valuable both as a giver and receiver The academy no longer holds sway
  • References Beaty, E., Dall’Alba, G., & Marton, F. (1997). The personal experience of learning in higher education: Changing views and enduring perspectives. In P. Sutherland (Ed.), Adult learning: A reader (pp. 150–165). London: Kogan Page. Draper, S. (2009). Catalytic assessment: understanding how MCQs and EVS can foster deep learning. British Journal of Educational Technology, 40(2), 285-293. Elliott, B. (2008). Assessment 2.0: Modernising assessment in the age of Web 2.0. Scottish Qualifications Authority. Retrieved June 22, 2009, from http://www.scribd.com/doc/461041/Assessment-20 James Gee, Situated Language and Learning: A Critique of Traditional Schooling. New York: Routledge, 2004. Henry Jenkins’ blog (accessed 21 December 2009) http://henryjenkins.org/2006/10/confronting_the_challenges_of.html Lea, M., & Jones, S., (2011) Digital Literacies in Higher Education: exploring textual and technological practice, Studies in Higher Education, 36 (3)
  • References 2 McAndrew, P., Scanlon, E. and Clow, D., (2010). An Open Future for Higher Education. Educause Quarterly, 33(1) Säljö, R. (1979). Learning about learning. Higher Education, 8(4), 443–451. Henry Jenkins’ blog (accessed 21 December 2009) http://henryjenkins.org/2006/10/confronting_the_challenges_of.html Scouller, K. (1998). The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35(4), 453–472. Whitelock, D. and Watt, S. (2008). Reframing e-assessment: adopting new media and adapting old frameworks. Learning, Media and Technology, Vol. 33, No. 3, September 2008, pp.153–156 Routledge, Taylor & Francis Group. ISSN 1743-9884