Measuring the impact of social and informal learning

6,012 views

Published on

Measuring the impact of social and informal learning - Questionmark Chairman John Kleeman's presentation at Learning Technologies 2012, London

1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
6,012
On SlideShare
0
From Embeds
0
Number of Embeds
2,139
Actions
Shares
0
Downloads
70
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide
  • Questions before learning create intrigue. Questions during learning strengthen memory and give information to direct learning. Questions after learning reduce forgetting.
  • This study compared two groups of students, one group studied only and the other studied and took a test within the same total time. The group who took a test retained more a week later.
  • The same technology that can accommodate small-screen mobile devices can also help make it easier to embed iFrame delivered assessments in your learning content
  • Measuring the impact of social and informal learning

    1. 1. Measuring the Impact of Social and Informal Learning John Kleeman / Chairman, Questionmark john@questionmark.com Thursday, 26th January 2012Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    2. 2. Learning is Changing  Academic model worked for 2500 yearsSign up  Internet changing things Learn  “Google it”  Changing work roles Test  Deeper understanding of learning Do work Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    3. 3. Only ~10% of learning Learning from your comes from study mentors, conversations (reading, researching, with peers and meetings courses, etc.) (if you pay attention and ask questions) 70+20+10 Formal Learning model of learningLearning From Others 10% 20% Learning By Doing On-The-Job learning by making mistakes, taking 70% time for retrospectives and learning from the experience. Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    4. 4. 70+20+10 model for learning  If you invest in ~70% from 70+20+10, how do you Experience measure the impact? ~20% from  Need to measure Others  Are people learning?  Are people performing better? ~10% from Study Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    5. 5. Assessments? Assessments are a critical part of learning  Quizzes  Surveys  Tests  Exams How can assessments  Measure informal learning?  Measure improved performance? Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    6. 6. Assessing Impact of Social/Informal Learning Initiatives Did they learn it? Did they Are they like it? doing it? Results Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    7. 7. Did they like it?  Use surveys to evaluate attitudes/reactions  Specific content / initiatives  Whole 70+20+10 initiative  Use surveys to identify “success cases” using the Brinkerhoff method  Surveys  Side-by-side with learning  Delivered on mobile devices  More conventional routes Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 7 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    8. 8. Mobile Assessments Questionmark assessments:  Can run via browser  Or on mobile devices via iPhone, iPad or Android Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    9. 9. QR codes can be used to launch surveys easily Books PowerPoint Signage Benefits • Ease of access for increased response rates • Demographic data easily captured • Better data accuracy • Easier analysis of survey results by demographic Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 9 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    10. 10. Survey & Course Evaluation Analytics Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 10 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    11. 11. Did they learn it?  Formative assessments  Help direct learning  Strengthen memory  Engage learners  In 70+20+10 user-generated quizzes provide extra dimension  Measure knowledge and skills  Compare before to after  Compare to benchmark  Compare to others Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 11 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    12. 12. Quiz attached to video in SharePoint Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 12 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    13. 13. Embed Youtube and other hosted video in Questionmark Live items! Embed YouTube or other hosted video directly in question stimulus Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 13 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    14. 14. Questions Through the Learning Process Reduce Forgetting Create Intrigue Strengthen Knowledge/Skills Memory Questions Questions Questions Questions Questions Time Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    15. 15. Quizzes and tests slow forgetting curve  Taking a quiz or test Knowledge Retained gives retrieval 90% 80% practice 70%  Reinforces mental 60% After 5 pathways to make 50% 56% mins 40% future recall more 30% 42% After 1 likely 20% week 10% 0%Data from Test-Enhanced Learning,Roediger and Karpicke 2006. Study only Study, TestPsychological Science Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    16. 16. Embed in a Wiki Embedded in a Wiki Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 16 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    17. 17. Embedding in Blogs, Portals, Wiki’s For Learners For Instructors  Focus the learners’ attention  Easily use assessments  Practice memory recall throughout the learning  Identify what was learned process  Provide seamless user  Provide feedback to assist the experience learner to:  Centralised management of  Correct misconceptions assessment content and  Instill confidence  Point to relevant material results  Impact data on social learning Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 17 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    18. 18. Compare to Benchmarks Benchmark ParticipantTopiclevelOverall Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 18 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    19. 19. Compare to Others Participant Topic level Overall Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 19 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    20. 20. Are they doing it?  In 70+20+10, people are learning  In multiple places  At different times  In different ways  Measuring performance more meaningful than measuring learning  Observational assessments can do this Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 20 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    21. 21. Observational assessment vs. ‘traditional’ quiz, test or exam Traditional assessment • Participant is presented with questions for direct response Observational assessment • Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 21 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    22. 22. What are Observational Assessments?  Observational assessments provide a mechanism for an ‘observer’ to assess a participant  For example, a participant might be observed and rated on…  Performance of a specific task or procedure  Knowledge acquisition and application  Demonstration of skills/abilities that ‘traditional’ assessment may be less reliable in measuring Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 22 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    23. 23. If assessing performance of a task, observer typically rates participant in three areas: Performing thePreparation Wrap Up task Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    24. 24. Typical workflow for “observer” Monitor/observer logs in Selects assessment Selects participant to be observed Answers/submits responses to questions Results/feedback reported to stakeholders Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 24 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    25. 25. Monitor/observer logs in
    26. 26. Selects assessment
    27. 27. Selects participant to be observed
    28. 28. Answers/submits responses to questions
    29. 29. Example applications of Observational AssessmentCertification Observational assessments can be used as critical components of the certification process – in some cases are required by regulatory authorities.ProcessPerformance An organization has delivered a training course to its employees and is now evaluating its effectiveness. By observing and monitoring behavioral changes inor level 3 the workplace before and after the training, the organization will be able to assessassessment: the changes caused by the training and evaluate the return on investment gained.Equipment Observational assessment can be used to record and rate operators skills, abilities, safety practices and adherence to required proceduresoperation:Customer Workplace Assessments can also be used to evaluate retail sales and customer service staff on how well they cope in different situations to assess whereService additional training is required or where personnel excel.evaluation: Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    30. 30. Feedback to Participant on PerformanceResults/feedback reported to stakeholders
    31. 31. Mobile devices Mobile devices such as tablets and smart phones ideal for delivering observational assessments Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    32. 32. The same assessment that is delivered on traditional laptop can be delivered via ahandheld mobile devicesuch as iPad, iPhone or Android phone/tablet
    33. 33. How to measure 70+20+10 impact? Did they learn it? Did they Are they like it? doing it? Results Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    34. 34. Visit Questionmark at Stand 60 Questions? Twitter: @questionmark and @johnkleeman More info: www.questionmark.com/go/observationalCopyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

    ×