Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
From proof of concept to evidence of impact:  towards a principled approach to evaluating learning design tools   Liz Mast...
Evaluating the impact of  the LDSE: conceptual foundations <ul><li>Achieve an impact on lecturers’ practice in designing t...
What constitutes  evidence of impact? <ul><li>Innovating in relation to one’s current practice </li></ul><ul><li>Giving du...
Delimiting the field of vision Organisational penetration; After Koballa (1988) (from Fishbein & Ajzen, 1975); also Ertmer...
Qualitative measurement criteria <ul><li>Awareness Recognising the potential for enhancing practice through engaging in th...
Qualitative measurement criteria <ul><li>Awareness </li></ul><ul><li>Reactions </li></ul><ul><li>Engagement </li></ul><ul>...
An example <ul><li>“ I got taken along to see what reusable learning objects were and got this lovely example of fulcrum, ...
Methodological issues <ul><li>Reliance on self-reports: e.g. from interviews and surveys </li></ul><ul><li>Constraints in ...
Upcoming SlideShare
Loading in …5
×

From proof of concept to evidence of impact: evaluating learning design tools

1,020 views

Published on

Presentation to accompany paper for the Art & Science of Learning Design workshop 2011

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

From proof of concept to evidence of impact: evaluating learning design tools

  1. 1. From proof of concept to evidence of impact: towards a principled approach to evaluating learning design tools Liz Masterman University of Oxford ASLD Workshop 13 th -14 th October 2011
  2. 2. Evaluating the impact of the LDSE: conceptual foundations <ul><li>Achieve an impact on lecturers’ practice in designing teaching and learning, particularly in relation to their use of TEL where it is relevant and appropriate </li></ul><ul><li>Strong effect on character, development or behaviour </li></ul><ul><li>Act and think differently about teaching (Biggs, 2003) </li></ul><ul><li>Qualitative transformation , not incremental or quantitative change (Wertsch, 2002 </li></ul>
  3. 3. What constitutes evidence of impact? <ul><li>Innovating in relation to one’s current practice </li></ul><ul><li>Giving due weight to students’ needs </li></ul><ul><li>Espousing appropriate theories of learning and teaching </li></ul><ul><li>Engaging in critical reflection on one’s practice </li></ul><ul><li>Building personal professional knowledge </li></ul><ul><li>Participating in a community of teachers and contributing to the development of collective professional knowledge </li></ul>
  4. 4. Delimiting the field of vision Organisational penetration; After Koballa (1988) (from Fishbein & Ajzen, 1975); also Ertmer (2007); Kaufman et al. (1996) Time Beliefs a) General pedagogic beliefs (active construction of knowledge) b) Beliefs about digital technologies in T&L (potential to benefit learning) Attitude +ve Values (Intends to use PRS to increase student participation in lectures) (Uses PRS once or twice only) Intentions re TEL 1 st -order change (reversible) Behaviours re TEL 2 nd -order change (irreversible) (PRS embedded in T&L practice) Confronts and strengthens both belief a) and belief b)
  5. 5. Qualitative measurement criteria <ul><li>Awareness Recognising the potential for enhancing practice through engaging in this activity </li></ul><ul><li>Reactions Feeling positive about enhancing practice in this way </li></ul><ul><li>Engagement Engaging with enhancing practice in this way… </li></ul>HEA Evaluation and Impact Assessment Approach (2009)
  6. 6. Qualitative measurement criteria <ul><li>Awareness </li></ul><ul><li>Reactions </li></ul><ul><li>Engagement </li></ul><ul><li>Learning from Learning or develop ideas relevant to their practice </li></ul><ul><li>Applying the learning Applying what they have learned or developed to their practice </li></ul><ul><li>Effects on student learning Identifying instances of students learning better as a result of enhanced practice </li></ul>HEA Evaluation and Impact Assessment Approach (2009)
  7. 7. An example <ul><li>“ I got taken along to see what reusable learning objects were and got this lovely example of fulcrum, load and effort and a car crashing into a wall. And […] I thought, ‘Well that’s not what I do because I don’t teach a concept that can be grasped like that.’ And there was this moment, […] I had an epiphany because I suddenly went, ‘Oh, so when I’m teaching that means I could do this!’” </li></ul>
  8. 8. Methodological issues <ul><li>Reliance on self-reports: e.g. from interviews and surveys </li></ul><ul><li>Constraints in time and resources  first-order changes in behavior or emergent shifts in attitude </li></ul><ul><li>Describe impact in the process of happening , rather than measure impact as outcome </li></ul>

×