Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
Upcoming SlideShare
Loading in …5
×

7 Ways that Online Assessments Help Ensure Compliance

1,315 views

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,315
On SlideShare
0
From Embeds
0
Number of Embeds
401
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

7 Ways that Online Assessments Help Ensure Compliance

  1. 1. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. ​7 Ways That Online Assessments Help Ensure Compliance John Kleeman, Chairman, Questionmark Brian McNamara, Marketing Director, Questionmark
  2. 2. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. What do we mean by “assessment” 7 Ways That Online Assessments Help Ensure Compliance Good practice in using assessments Q&A Agenda Slide 2
  3. 3. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Brian McNamara John Kleeman  Chairman and Founder of Questionmark  Active in the development of industry and technical standards relating to online assessment  Frequent contributor of blog articles focusing on assessment good practice  Marketing Director, Questionmark  15+ years working with organizations in the training/ learning technology space About the Presenters Slide 3
  4. 4. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 4 About Questionmark  Founded in 1988  Assessment Management Systems  Create, deliver and report on quizzes, tests, surveys and exams  Measure knowledge, skills and attitudes  Commonly used for corporate learning, certification and compliance related assessment  More than 1,100 current customers worldwide  Corporate: Many customers in highly regulated industries such as Energy & Utilities; Financial Services; Manufacturing; Healthcare; Medical (Pharmaceuticals, Biotech, Medical devices); Transportation; Retail.  Government and Military  Higher Education
  5. 5. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. What we mean by the term “assessment” Slide 5
  6. 6. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Examples of Suppliers: Hogan, SHL, Insights, Myers Briggs, etc. Examples of Suppliers: Questionmark Survey Monkey and numerous others Examples of Suppliers to author, deliver and analyze: Questionmark Examples of Suppliers of pre- built tests: Total Testing, eSkill, etc Not normally used in companies or education. Assessments Personality assessment Cultural Fit Job Fit Survey Customer Satisfaction Employee Satisfaction Course Evaluation JTA 180/360 On The Job Performance Formative Diagnostic Summative Observational Medical and Psychological Psychiatrists. Courts, etc. Types of Assessment All assessments provide a stimulus and collect data to evaluate Type of Suppliers: 1. Prebuilt Instruments /Assessments 2. Tools to create assessments (i.e. Authoring, Delivery and Analytics)
  7. 7. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 7 Quizzes, Tests, Exams •Questions and scoring algorithms •Scores and feedback at question, topic and assessment level Self-Assessments •Questions related to knowledge, skills, abilities and attitudes •Retains responses for future review and change Surveys •Questions relate to a person’s opinions on a topic •Course evaluations, employee attitude surveys Observational Assessments •Someone assessment a person’s performance, knowledge, skills, and abilities 180 and 360 Assessments •Groups of related individuals are asked questions to assess a person •Leverages on the wisdom of a crowd to help individual development Assess Content and Contributors •People assess value of learning content •Expert locator and acknowledgement Types of Assessments
  8. 8. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Assessments through the Learning Process Evidence Accumulated Windows CDPrepare For Learning Windows CDIn/As Learning Assessments Windows CDOf-Learning Assessments Mentoring Instructor Collaborative & E-learning Work Place
  9. 9. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Assessments through the Learning Process Evidence Accumulated Windows CDPrepare For Learning Windows CDIn/As Learning Assessments Windows CDOf Learning Assessments Windows CD Job Task Analysis Windows CDCourse Evaluations Windows CDNeeds Analysis Mentoring Instructor Collaborative & E-learning Work Place Windows CDSlow Forgetting Curve
  10. 10. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Pre-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: • Create intrigue • Route to course • Inform instructors • Establish benchmarks Windows CDPrepare For Learning Windows CDIn/As Learning Assessments Windows CDOf Learning Assessments
  11. 11. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning In-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: • Encourages focus on learning • Provide instant feedback to instructor • Stimulate and strengthen memory • Correct misconceptions Windows CDPrepare For Learning Windows CDIn/As Learning Assessments Windows CDOf Learning Assessments
  12. 12. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Post-Learning Assessments Evidence Accumulated Mentoring Instructor Collaborative & E-learning Work Place Purpose: • Was learning environment okay? • Knowledge transfer occurred? • Reduce Forgetting Curve • Certifications Windows CDPrepare For Learning Windows CDIn/As Learning Assessments Windows CDOf Learning Assessments
  13. 13. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Criterion referenced vs. Norm referenced assessment Norm Referenced What is the participant’s knowledge …compared to others in a group? •Typically used for standardized college placement tests like the SAT, ‘competition’ tests (e.g. advancement test for limited number of openings) •Pass/Fail can only be determined after the assessment Criterion Referenced What is the participant’s knowledge …compared to a benchmark? •Typically used for certification tests, licensing exams, end of course exams, etc. •Typically Pass/Fail score is determined before the assessment Compliance oriented assessments are generally criterion referenced
  14. 14. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Reliable: dependable, repeatable, consistent  Valid: measures the specific knowledge and skills that were intended to be measured Reliability and validity Slide 14 Figure 1: Reliable but not Valid Figure 2: Not reliable, not Valid Figure 3: Reliable and Valid
  15. 15. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. ​7 Ways That Online Assessments Help Ensure Compliance Slide 15
  16. 16. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Regulators encourage them Demonstrate commitment Find problems early Document understanding after training Increase knowledge / reduce forgetting Reduce cost & time for compliance training Reduce human error 7 reasons why assessments mitigate compliance risk Slide 16
  17. 17. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #1 : Regulators encourage them Slide 17
  18. 18. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  US FDIC:  “Once personnel have been trained on a particular subject, a compliance officer should periodically assess employees on their knowledge and comprehension of the subject matter”  WHO good manufacturing principles  “Continuing training should also be given, and its practical effectiveness periodically assessed.”  US Dept of Health & Human Services  “Contractors should consider using tests or other mechanisms to determine the trainees’ comprehension of the training concepts presented.” Three of many, many, many examples Slide 18
  19. 19. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #2 : Demonstrate commitment Slide 19
  20. 20. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Because employees follow their leaders  Charles Jennings (August 2012) “… common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance” http://charles-jennings.blogspot.co.uk/2012/08/compliance-training-does-it-really-work.html Why is it important to demonstrate top level commitment?
  21. 21. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  If you require all employees and managers to  Take assessments every 6 or 12 months  Remedial training and retake assessment if fail  Assessments seen as fair  Major stride in demonstrating commitment How do assessments demonstrate commitment? Slide 21
  22. 22. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  “Prosecutors also should determine whether the corporation's employees are adequately informed about the compliance program and are convinced of the corporation's commitment to it. This will enable the prosecutor to make an informed decision as to whether the corporation has adopted and implemented a truly effective compliance program” US Department of Justice Slide 22
  23. 23. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #3 : Find problems early Slide 23
  24. 24. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Look at topic and assessment scores  By department  By geography  Over time  Look at wrong answers  Look at test failures  Assessments touch every employee: a unique tool How tests/quizzes/exams can find problems early
  25. 25. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Surveys identify issues in culture and ethics  Most employees want to help. From employee of bank involved in 2012 LIBOR scandal: “Always happy to help, leave it with me, Sir”.  UK Ministry of Justice Bribery Act Guidelines “Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti- bribery policies”. How surveys can find problems early Slide 25
  26. 26. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #4 : Document understanding after training Slide 26
  27. 27. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Because regulators require it  common regulatory requirement, e.g. OECD recommends on ethics/compliance programme: measures designed to ensure periodic communication, and documented training for all levels of the company  Because makes business sense  Trained employees more effective  “If you can not measure it, you can not improve it.“ Why document training for compliance?
  28. 28. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Take attendance Get employee signatures Test to check understanding Common Methods of Documenting Training Slide 28
  29. 29. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How to do  Trainer records attendance or takes photo of attendees  Record online attendees  Pros  Easy to do  Measures who turns up  Cons  Did they stay the whole time?  Did they pay attention?  Did they get it? Take attendance Slide 29
  30. 30. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How to do  Get employees to sign a form confirming that they attended training AND that they understand the rules  Pros  Gets employee confirmation  Employee committed to follow rules  Cons  People may sign but not take seriously  People may sign without attending or understanding Get employee signatures Slide 30
  31. 31. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Not where safety is involved, according to 2005 United States Appeal Court: “merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”  Complete General Const. Co. v. OSHRC, 2005 U.S. App. LEXIS 5197 (6th Cir.), March 29, 2005 Does getting an employee to sign give legal protection? Slide 31
  32. 32. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How to do:  After training, test employees  People who fail test undergo remedial training  Pros  Genuinely check understanding  Allows improvement of weaknesses  Feedback to improve training  Cons  More effort  You have to follow through & take action on failures Test to check understanding Slide 32
  33. 33. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #5 : Increase knowledge / reduce forgetting Slide 33
  34. 34. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Compliance Training Slide 34 Receive information Working memory Long term memory
  35. 35. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Cue • Question • Other stimulus Search • Retrieve from long term memory Act • Act or respond  Research shows retrieval practice helps future retrieval  Successful retrieval modifies memory making it more retrievable in future Training useful if can retrieve Slide 35
  36. 36. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  120 students divided into groups randomly  Given reading comprehension passages used in TOEFL  Some students  Study for 14 mins  Some students  Study for 7 mins  Recall material for 7 mins  Compared results after 5 mins and 1 week Roediger & Karpicke experiment 2006 Slide 36
  37. 37. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% After 5 mins After 1 week Restudy Quiz How much learners retained Slide 37
  38. 38. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning in the Actual World They don’t get it All! Slide 38
  39. 39. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Learning Curve Slide 39 Ouch! The Forgetting Curve Time Knowledge/Skills Learning Experience Learning Experience Learning Experience Questions Questions Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall
  40. 40. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. In the Actual World… we forget some of it! Slide 40
  41. 41. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slow the Forgetting Curve Slide 41 Time Knowledge/Skills Learning Experience Learning Experience Learning Experience Questions Questions Questions Questions Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve
  42. 42. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #6 : Reduce cost & time for compliance training Slide 42
  43. 43. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Take test If pass, skip training Otherwise do training Testing out of compliance training Slide 43
  44. 44. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Management Report Example A) Number of people 1,000 B) Time for training if no testing (2 hours each) 2,000 hours C) Time to take test (20 mins / test) 333 hours D) 25% pass test and skip training E) 75% do compliance training 1,500 hours F) Hours of training avoided (B – C - E) 177 hours G) Average cost / hour $100 H) Money saved (F * G) $17,700 Saving money by “testing out” Slide 44
  45. 45. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Many regulators permit “testing out” to reduce compliance training  ABA Bank Compliance magazine article 2009 interviewed some US regulators  OTS: Testing out acceptable  OCC: Agree, providing testing program well structured. May be some exceptions  FDIC : Agree  Check with your regulator Will the regulator allow?
  46. 46. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reason #7 : Reduce human error Slide 46
  47. 47. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 22% 23% 29% 15% 6% 5% Root causes of human error (data from UK MHRA 2011, graph by Questionmark) Process/procedure incorrect Procedural steps omitted Concentration error Training misunderstood Training missing Poor communication/rushing
  48. 48. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Assessments can fix  Training misunderstood (15%)  Training missing or out of date (6%)  Procedural steps omitted (23%)  Process or procedure incorrect (22%)  Other causes of error  Poor communication / rushing (5%)  Concentration error (29%) Assessments Slide 48 22% 23% 29% 15% 6% 5% Process/procedure incorrect Procedural steps omitted Concentration error Training misunderstood Training missing Poor communication/rushing
  49. 49. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Regulators encourage them Demonstrate commitment Find problems early Document understanding after training Increase knowledge / reduce forgetting Reduce cost & time for compliance training Reduce human error Summary on why to use assessments Slide 49
  50. 50. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Good practice in using assessments Slide 50
  51. 51. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Create scenario questions Assess application not just knowledge
  52. 52. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Evaluation Synthesis Analysis Application Comprehension Knowledge  Questions better if use application / comprehension NOT just knowledge Blooms Taxonomy
  53. 53. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  What does a yellow traffic light mean?  Stop  Go  Caution  If you are driving toward an intersection and the light turns from yellow to red, what should you do?  Speed up and cross the intersection  Stop suddenly  Stop gradually Example Slide 53
  54. 54. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Use feedback at the topic level It is actionable Slide 54
  55. 55. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  What to do  Divide assessment by topic areas  Give individual feedback  Topic scores  Resources or advice to improve weak topics  Analyze at department or organization level  Topic feedback is actionable  Direct participant to specific resources to improve learning  Direct compliance to areas knowledge is weak How topic feedback can help
  56. 56. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Feedback based topics of questions Slide 56 Corrective feedback points participant to appropriate learning resources “Topics” of questions answered in assessment Score on topic
  57. 57. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  “participant can immediately see where they need further facilitation”  “We have done extremely well in terms of increasing our overall pass mark … by using topic feedback” Real life financial services compliance Slide 57
  58. 58. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Consider observational assessments Measure skills and behavior Slide 58
  59. 59. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Traditional assessment • Participant is presented with questions for direct response Observational assessment • Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities Observational assessment vs. ‘traditional’ quiz, test or exam Slide 59
  60. 60. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Preparation Performing the task Wrap Up Observer typically rates participant in 3 areas Smartphones/tablets make more practical
  61. 61. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Real life example of observational assessments for learning & compliance  $10 billion medical equipment manufacturer  Company certifies employees by administering observational assessments  Assessments are key to both learning and compliance initiatives  Observational assessment process:  Sales rep is trained on a particular product  Field trainer  Observes rep demonstrate the product’s use  Using an iPad, completes an observational assessment  Rates performance on a 3-point scale:  “Performed without Prompting”  “Performed with Prompting”  “Unable to Perform.”
  62. 62. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Use item analysis Pick out the bad questions Slide 62
  63. 63. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Difficulty index: P value = % answered correctly  Number from 0.0 to 1.0  Too high means too easy  Too low means too hard or perhaps confusing  Correlation/discrimination : correlation between item score and test score  < 0.2 low discrimination, review item  > 0.3 good Item analysis key numbers
  64. 64. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 64
  65. 65. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Set a pass score Slide 65
  66. 66. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  A test with a pass score of 70%  A test with a pass score of 80%  A test with a pass score of 90%  It depends on how hard the questions are Which is the best test for compliance?
  67. 67. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Think about pass or “cut” scores Good compliance practice Not so good practice  Set pass or cut score to be minimal acceptable competence  Passing test demonstrates competence  Someone who fails does not  Pick a number out of a hat to use that as the pass score Slide 67
  68. 68. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Angoff method: What is the % chance that a borderline candidate will get question right?  How it works  Poll SMEs  Consider marginal candidates and the probability of getting specific questions right (0-100%)  Make a mathematical calculation to determine cut score Angoff Method Slide 68
  69. 69. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Angoff Method Example What is the % chance that a borderline candidate will get question right? SME A SME B SME C Total Q1 75% 75% 75% 75.00% Q2 70% 80% 80% 76.67% Q3 65% 75% 70% 70.00% Q4 60% 85% 90% 78.33% Q5 80% 80% 85% 81.67% Q6 80% 80% 80% 80.00% Q7 75% 80% 75% 76.67% Q8 65% 90% 65% 73.33% Q….. 75% 80% 75% 76.67% Q50 65% 85% 65% 71.67% Totals 71% 81% 76% 76% Slide 69
  70. 70. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  One of our customers in the US military summed it up this way… this method is:  Defensible  Easy to use  Requires only minimal training to implement  Widely accepted Why use the Angoff Method? Slide 70
  71. 71. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Have participants agree a code of conduct Help them be honest Slide 71
  72. 72. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Fraud needs:  Motivation  Opportunity  Rationalization Fraud triangle Slide 72
  73. 73. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Motivation depends on stakes Slide 73 Higher Stakes High Medium Low Medium Stakes Low Stakes What’s at Stake? Life and Limb Promotion & Jobs & Legal Concern Educational Exams Tests Elearning & Surveys
  74. 74. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Impersonation Content Theft Cheating Opportunity Slide 74 Monitoring/proctoring helpful, e.g. having manager supervise employees
  75. 75. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Reduce rationalization to cheat  Make the program fair  Communicate widely that it is fair  A code of conduct (honor code) can help  Idea borrowed from universities  Employees commit at start of assessment to be honest Rationalization Slide 75
  76. 76. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 1. Use scenario questions – test above knowledge 2. Use topic feedback 3. Consider observational assessments 4. Use item analysis 5. Set a pass score 6. Use a code of conduct Summary on good practice Slide 76
  77. 77. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Excellent book on assessment  Criterion-Referenced Test Development  by Shrock and Coscarelli More good practice Slide 77
  78. 78. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Questionmark White Papers www.questionmark.com/go/whitepapers More good practice Slide 78
  79. 79. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. What questions do you have? Slide 79
  80. 80. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Thank you! John Kleeman, Chairman, Questionmark Brian McNamara, Marketing Director, Questionmark Slide 80

×