Assess to comply:                                            How else can you be sure that                                ...
John Kleeman         John Kleeman MA MBCS C.Eng             Worked in assessment software 25 years             Chairman...
Setting the scene             Increasing regulation             Companies expected to perform             Business and ...
Agenda             5 reasons why assessments mitigate compliance risk              1.   Demonstrate commitment           ...
Common assessments in compliance             Medium and higher stakes               Internal exams               Post-c...
Reason #1 : Demonstrate commitment                                                                                        ...
Why is it important to demonstrate top               level commitment?   Because employees follow their leaders       Ch...
How do assessments demonstrate                           commitment?             If you require all employees and manager...
Fine of UK£5.5m for not testing, partly                    due to lack of commitment             UK FSA 2009:            ...
Reason #2 : Find problems early                                                                                           ...
How surveys can find problems early          Surveys identify issues in culture and ethics          Most employees want ...
How tests/quizzes/exams can find                 problems early   Look at topic and assessment scores       By departmen...
Reason #3 : Document understanding           after training                                                               ...
Why document training for compliance?   Because regulators require it       common regulatory requirement, e.g. OECD rec...
Common Methods of Documenting Training                                                                                    ...
Take attendance          How to do              Trainer records attendance or takes photo of attendees              Rec...
Get employee signatures          How to do              Get employees to sign a form confirming               that they ...
Does getting an employee to sign give                     legal protection?          Not where safety is involved accordi...
What do regulators think?          SEC on FCPA compliance:           Titan’s only related ‘policy’ is a statement in Tita...
Test to check understanding          How to do:              After training, test employees              People who fai...
Reason #4 : Increase knowledge /           reduce forgetting                                                              ...
Why is St Louis famous?   Because of the    famous arch?   Should it be for    Washington    University in St.    Louis ...
Compliance Training         Receive        Working                       Long term       information      memory          ...
Training useful if can retrieve                  • Question                                                            Re...
Roediger & Karpicke experiment 2006                       (in St. Louis)          120 students divided into groups random...
How much learners retained      90%      80%      70%      60%      50%                                                   ...
Marsh, Roediger, Bjork & Bjork 2007.           Multiple choice quizzes aid long term retention      60%      50%      40% ...
Learning in the Actual World                   They don’t get it All!                                                     ...
Learning Curve                                Purpose:                                • Content repetition                ...
In the Actual World                                   They Forget It!                                                     ...
Slow the Forgetting Curve                                     Purpose:                                     • Memory retrie...
Reason #5 : Reduce cost & time for           compliance training                                                          ...
Testing out of compliance training                                                 If pass, skip                          ...
Example of how can save money     Management Report Example     A) Assessment Completions                                 ...
Will the regulator allow?   Many regulators permit “testing out” to reduce    compliance training       ABA Bank Complia...
Summary on why to use assessments1.   Demonstrate commitment2.   Find problems early3.   Document understanding after trai...
Good practice in using assessments             5 key practical steps for compliance assessments                           ...
Create scenario questionsAssess application not just knowledge                                                            ...
Blooms Taxonomy                     Evaluation                                                                   Question...
Example          What does a yellow                                                            If you are driving       ...
Use feedback at the topic level           It is actionable                                                                ...
How topic feedback can help   What to do       Divide assessment by topic areas       Give individual feedback        ...
Feedback based topics of questions    “Topics” of                                                                  Correct...
Make assessments accessible           Any large organization will have visually impaired or other           participants w...
Accommodations to consider               Disability                                                              Accommoda...
Be careful about          Screenshots in questions          Use of color          Long text that is hard to read       ...
Have participants agree a code of           conduct           Help them be honest                                         ...
Fraud triangle          Fraud needs:              Motivation              Opportunity              Rationalization    ...
Motivation depends on stakes                                                                                              ...
Opportunity      Impersonation                              Content Theft                                                 ...
Rationalization          Reduce rationalization to cheat              Make the programme fair              Communicate ...
Consider observational assessments           Measure skills and behavior                                                  ...
Observational assessment vs. ‘traditional’                 quiz, test or exam                                             ...
Observer typically rates participant in 3 areas                                            Performing thePreparation      ...
Actionable good practice       1.   Create scenario questions       2.   Use feedback at the topic level       3.   Make a...
More good practice and resources          Excellent book on assessment              Criterion-Referenced Test Developmen...
Your questions                                                                                                            ...
Upcoming SlideShare
Loading in...5
×

Assess to Comply: how else can you be sure that employees understand? ECOA 2012

18,999

Published on

Assess to Comply: how else can you be sure that employees understand? Presentation by John Kleeman, Questionmark (www.questionmark.com)
ECOA 20th Annual Ethics & Compliance Conference
St Louis, October 2012

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
18,999
On Slideshare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
10
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • From criminologist Donald Cressey
  • Assess to Comply: how else can you be sure that employees understand? ECOA 2012

    1. 1. Assess to comply: How else can you be sure that employees understand? John Kleeman, Questionmark (www.questionmark.com) ECOA 20th Annual Ethics & Compliance Conference St Louis, October 2012Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    2. 2. John Kleeman  John Kleeman MA MBCS C.Eng  Worked in assessment software 25 years  Chairman and Founder of Questionmark  Questionmark used by 1000+ organizations for compliance and other assessments  Contact:  Email: john@questionmark.com  Twitter : @johnkleeman @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 2 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    3. 3. Setting the scene  Increasing regulation  Companies expected to perform  Business and regulators international (e.g. FCPA, UK Bribery Act)  Regulators levying heavier penalties  Several companies fined US$300m+ in 2012  Can assessments make compliance easier and reduce risk? @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 3 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    4. 4. Agenda  5 reasons why assessments mitigate compliance risk 1. Demonstrate commitment 2. Find problems early 3. Document understanding after training 4. Increase knowledge / reduce forgetting 5. Reduce cost & time for compliance training  5 key pieces of practical advice on using assessments @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 4 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    5. 5. Common assessments in compliance  Medium and higher stakes  Internal exams  Post-course tests  Diagnostic tests  Observational/workplace assessments  Lower stakes assessments  Course evaluation surveys  Employee attitude surveys  Job task analysis surveys  Practice tests  Quizzes during learning  360 degree performance assessments @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 5 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    6. 6. Reason #1 : Demonstrate commitment @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 6 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    7. 7. Why is it important to demonstrate top level commitment? Because employees follow their leaders  Charles Jennings (August 2012) “… common thread that runs through almost all high-profile compliance catastrophes. It is that the top-tier executives and middle managers in the organisations simply don’t model the behaviours that will lead to a culture of compliance” @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    8. 8. How do assessments demonstrate commitment?  If you require all employees and managers to  Take assessments every 6 or 12 months  Remedial training and retake assessment if fail  Assessments seen as fair  Major stride in demonstrating commitment @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 8 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    9. 9. Fine of UK£5.5m for not testing, partly due to lack of commitment  UK FSA 2009:  “Aon Ltd should have ensured that appropriate members of staff – particularly those in the Aviation and Energy divisions – received focused training in relation to this area and were tested on their understanding of the relevant risks involved. Effective training and testing in this regard would have emphasised to staff the importance of carrying out effective due diligence prior to authorising an Overseas Third Party for payment.” @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 9 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    10. 10. Reason #2 : Find problems early @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 10 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    11. 11. How surveys can find problems early  Surveys identify issues in culture and ethics  Most employees want to help. From employee of bank involved in recent LIBOR scandal: “Always happy to help, leave it with me, Sir”.  UK Ministry of Justice Bribery Act Guidelines “Staff surveys, questionnaires and feedback from training can also provide an important source of information on effectiveness and a means by which employees and other associated persons can inform continuing improvement of anti- bribery policies”. @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 11 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    12. 12. How tests/quizzes/exams can find problems early Look at topic and assessment scores  By department  By geography  Over time Look at wrong answers Look at test failures Assessments touch every employee: a unique tool @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    13. 13. Reason #3 : Document understanding after training @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 13 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    14. 14. Why document training for compliance? Because regulators require it  common regulatory requirement, e.g. OECD recommends on ethics/compliance programme: measures designed to ensure periodic communication, and documented training for all levels of the company Because makes business sense  Trained employees more effective  “If you can not measure it, you can not improve it.“ @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    15. 15. Common Methods of Documenting Training Test to check understanding Get employee signatures Take attendance @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 15 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    16. 16. Take attendance  How to do  Trainer records attendance or takes photo of attendees  Record online attendees  Pros  Easy to do  Measures who turns up  Cons  Did they stay the whole time?  Did they pay attention?  Did they get it? @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 16 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    17. 17. Get employee signatures  How to do  Get employees to sign a form confirming that they attended training AND that they understand the rules  Pros  Gets employee confirmation  Employee committed to follow rules  Cons  People may sign but not take seriously  People may sign without attending or understanding @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 17 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    18. 18. Does getting an employee to sign give legal protection?  Not where safety is involved according to 2005 United States Appeal Court: “merely having an individual sign a form acknowledging his responsibility to read the safety manual is insufficient to insure that the detailed instructions contained therein have actually been communicated.”  Complete General Const. Co. v. OSHRC, 2005 U.S. App. LEXIS 5197 (6th Cir.), March 29, 2005 @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 18 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    19. 19. What do regulators think?  SEC on FCPA compliance: Titan’s only related ‘policy’ is a statement in Titan’s Code of Ethics, which all Titan employees were required to sign annually, stating employees must be familiar with and strictly adhere to such provisions … Titan did not enforce that policy nor did it provide its employees with any information concerning the FCPA or its purposes”.  UK FSA example of poor practice: “… employees were merely required to sign a form to confirm that they were aware of the firm’s AML procedures, but their understanding was never tested.” @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 19 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    20. 20. Test to check understanding  How to do:  After training, test employees  People who fail test undergo remedial training  Pros  Genuinely check understanding  Allows improvement of weaknesses  Feedback to improve training  Cons  More effort  You have to follow through & take action on failures @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 20 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    21. 21. Reason #4 : Increase knowledge / reduce forgetting @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 21 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    22. 22. Why is St Louis famous? Because of the famous arch? Should it be for Washington University in St. Louis psychology research? @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    23. 23. Compliance Training Receive Working Long term information memory memory @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 23 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    24. 24. Training useful if can retrieve • Question  Research at Washington Cue • Other stimulus University in St. Louis (and elsewhere) shows retrieval practice helps • Retrieve from long term memory future retrieval Search  Successful retrieval • Act or respond modifies memory Act making it more retrievable in future @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 24 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    25. 25. Roediger & Karpicke experiment 2006 (in St. Louis)  120 students divided into groups randomly  Given reading comprehension passages used in TOEFL  Some students  Study for 14 mins  Some students  Study for 7 mins  Recall material for 7 mins  Compared results after 5 mins and 1 week @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 25 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    26. 26. How much learners retained 90% 80% 70% 60% 50% Restudy 40% Quiz 30% 20% 10% 0% After 5 mins After 1 week @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 26 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    27. 27. Marsh, Roediger, Bjork & Bjork 2007. Multiple choice quizzes aid long term retention 60% 50% 40% 30% No quiz Quiz 20% 10% 0% SAT II Bloom Definition Bloom Application @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 27 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    28. 28. Learning in the Actual World They don’t get it All! @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 28 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    29. 29. Learning Curve Purpose: • Content repetition • Memory retrieval practice • Strengthens memory recall Ouch! The Forgetting Curve Knowledge/Skills Experience Experience Questions Experience Questions Learning Learning Time Learning @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 29 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    30. 30. In the Actual World They Forget It! @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 30 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    31. 31. Slow the Forgetting Curve Purpose: • Memory retrieval practice • Strengthens memory recall • Reduce Forgetting Curve Knowledge/Skills Experience Experience Questions Experience Questions Questions Questions Learning Learning Time Learning @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 31 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    32. 32. Reason #5 : Reduce cost & time for compliance training @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 32 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    33. 33. Testing out of compliance training If pass, skip Otherwise Take test training do training @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 33 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    34. 34. Example of how can save money Management Report Example A) Assessment Completions 1000 B) Pass the assessment and test out 375 C) Do the compliance training 625 D) Time for compliance training 1.5 hours E) Hours of training avoided (B * D) 562.5 F) Average cost / hour $70 G) Money saved (E * F) $39,375 @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 34 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    35. 35. Will the regulator allow? Many regulators permit “testing out” to reduce compliance training  ABA Bank Compliance magazine article 2009 interviewed some US regulators  OTS: Testing out acceptable  OCC: Agree, providing testing program well structured. May be some exceptions  FDIC : Agree Check with your regulator @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    36. 36. Summary on why to use assessments1. Demonstrate commitment2. Find problems early3. Document understanding after training4. Increase knowledge / reduce forgetting5. Reduce cost & time for compliance training @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    37. 37. Good practice in using assessments 5 key practical steps for compliance assessments @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 37 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    38. 38. Create scenario questionsAssess application not just knowledge @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    39. 39. Blooms Taxonomy Evaluation  Questions better if use application Synthesis / comprehension Analysis Application Comprehension Knowledge NOT just knowledgeCopyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    40. 40. Example  What does a yellow  If you are driving traffic light mean? toward an intersection  Stop and the light turns from  Go yellow to red, what  Caution should you do?  Speed up and cross the intersection  Stop suddenly  Stop gradually @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 40 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    41. 41. Use feedback at the topic level It is actionable @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 41 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    42. 42. How topic feedback can help What to do  Divide assessment by topic areas  Give individual feedback  Topic scores  Resources or advice to improve weak topics  Analyze at department or organization level Topic feedback is actionable  Direct participant to specific resources to improve learning  Direct compliance to areas knowledge is weak @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    43. 43. Feedback based topics of questions “Topics” of Corrective feedback questions Score on points participant to answered in topic appropriate learning assessment resources Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 43 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    44. 44. Make assessments accessible Any large organization will have visually impaired or other participants with accessibility needs @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 44 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    45. 45. Accommodations to consider Disability Accommodation Loss of vision Screen reader support Visual impairment Contrast and text size D/deafness, hard of hearing Text alternatives Motor impairment Keyboard accessible Cognitive impairment Text size, extra time Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 45 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    46. 46. Be careful about  Screenshots in questions  Use of color  Long text that is hard to read  Simulations & graphics using Flash  Time limits (may need to extend for some)  Make sure your assessment software supports accessible assessments @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 46 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    47. 47. Have participants agree a code of conduct Help them be honest @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 47 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    48. 48. Fraud triangle  Fraud needs:  Motivation  Opportunity  Rationalization @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 48 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    49. 49. Motivation depends on stakes What’s at Stake? High Life and Limb Higher Medium Promotion & Jobs & Legal Concern Stakes Low Educational Exams Medium Stakes Tests Low Stakes Elearning & Surveys Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 49 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    50. 50. Opportunity Impersonation Content Theft Cheating Monitoring/proctoring helpful, e.g. having manager supervise employees @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 50 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    51. 51. Rationalization  Reduce rationalization to cheat  Make the programme fair  Communicate widely that it is fair  A code of conduct (honor code) can help  Idea borrowed from universities  Employees commit at start of assessment to be honest @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 51 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    52. 52. Consider observational assessments Measure skills and behavior @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 52 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    53. 53. Observational assessment vs. ‘traditional’ quiz, test or exam Traditional assessment • Participant is presented with questions for direct response Observational assessment • Observer is presented with questions to ‘rate’ participant to measure knowledge, skills and abilities Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 53 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    54. 54. Observer typically rates participant in 3 areas Performing thePreparation Wrap Up task Smartphones/tablets make more practical Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    55. 55. Actionable good practice 1. Create scenario questions 2. Use feedback at the topic level 3. Make assessments accessible 4. Have participants agree a code of conduct 5. Consider observational assessments @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 55 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    56. 56. More good practice and resources  Excellent book on assessment  Criterion-Referenced Test Development  Shrock and Coscarelli  Questionmark white papers on good practice in compliance and learning assessments  www.questionmark.com/whitepapers  Papers by Professor Roediger at Washington University in St Louis  http://psych.wustl.edu/memory/publications/  What’s the best way to document that training has taken place  Article by John Kleeman in Compliance & Ethics Professional, July/August 2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 56 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    57. 57. Your questions @theECOA | #AECC2012 Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 57 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×