Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
Upcoming SlideShare
Loading in...5
×

The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing

1,003

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,003
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing

  1. 1. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 2013 Users Conference Baltimore | March 3rd – 6th The Half-Time Strategy: Saving Your Organization Time and Money with Diagnostic Testing John LoBianco Steven Torkel PwC
  2. 2. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  John LoBianco  Senior Associate, Evaluation & Assessment Team  Steven Torkel  Director, Evaluation & Assessment Team  Special thanks to Sean Farrell, Senior Manager on the E&A Team, who has been instrumental in this work. Your Presenters Slide 2
  3. 3. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  A global network of professional services firms  180,000 staff members and partners across the network  People development is a top priority  PwC was awarded #1 in Training Magazine’s Top 125 for 3 years in a row, and was placed into the “Hall of Fame”  Highly mobile and virtual workforce Slide 3
  4. 4. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How our approach to diagnostic assessment has evolved over the past three years  Best practices for building defensible assessments for the purpose of testing-out of training  Half-time Strategy  Modeling tool Session Objectives Slide 4
  5. 5. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Diagnostic Assessment Slide 5  Measures current level of knowledge and skill in specific topic areas.  Results help drive decisions about future development activities.
  6. 6. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Why do diagnostic assessments? Slide 6
  7. 7. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. How our approach has evolved Slide 7 2010 • Tax 1 self-studies • New employees 2011 • Tax 1 self-studies • New employees • Enhancements 2012 • Mandatory Training • Experienced Partners Future • Classroom Training • Integrating assessments with on-the-job learning
  8. 8. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  New tax hires must attend Tax 1 after joining the firm  College graduates with varying levels of experience  First professional job  First exposure to PwC technical training The Tax 1 Program Slide 8  Program shortened from two weeks to one week Four years ago:
  9. 9. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Tax 1 Re-design Slide 9  Several self-studies were created as advance preparation  New hires were required to complete these self-studies before attending Tax 1  Learners could not always find the time to complete the self-studies
  10. 10. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Test-outs for three of the self-studies were created:  New hires could save time by not having to complete the self-studies  Optional  The three test-outs were comprised of 25, 28, and 21 questions  “One and done” How could we ease the burden on our learners while still maintaining quality? Slide 10
  11. 11. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  On average, there was a slight increase in training time across the learner population after the test-outs were implemented  Why didn’t we achieve the benefit we were hoping for?  Not enough learners passed the test-out  Test-outs were too long How much time did we save our learners? Slide 11
  12. 12. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Combined the three test-outs into one “consolidated” test-out  Identified and removed poorly performing items  Adjusted cut score  Required learners to register for test-out  Implemented “half-time” strategy How did we improve the test-out process? Slide 12
  13. 13. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Half-Time Strategy Slide 13
  14. 14. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Each section of the test-out had 20 questions  Each section of the test-out was split in half  Learners needed to pass the first half of each section to proceed to the second half of that section  Learners answered anywhere from 30 to 60 questions  Saved learners time by bypassing questions they would probably answer incorrectly Tax 1 Test-out Half-time Strategy Slide 14
  15. 15. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Test-out Half-time Strategy Slide 15 Course A (10 questions in 1st half) Course A (10 questions in 2nd half) Course B Course C Course B Course C Learner scores 80% or above Learner scores less than 80% Learner scores less than 80% Learner scores 80% or above Learner scores 80% or above (10 questions in 1st half) (10 questions in 2nd half) (10 questions in 1st half) (10 questions in 2nd half) Learner is then taken to Course B Learner is then taken to Course C Learner reaches end of test-out
  16. 16. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Learners were told their results for each section End of Test-out Feedback Slide 16 Course A Course B Course C
  17. 17. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  58 hours of training time (about 5%) were saved  871 total hours taking the self-studies  164 total hours taking the test-outs  1093 potential total training hours Time savings were modest Slide 17
  18. 18. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Mandatory Training test-out option Slide 18  Mandatory Training for Partners  Yearly training event  Option to take a short online test in lieu of completing a 90 minute self- study learning activity  Testout designed to be more challenging than the end of course assessment  Available only to partners with 2+ years experience
  19. 19. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Benefit: Expected time savings for the practice Slide 19  Difficult to quantify as it depends on the number of partners who take the optional test-out, as well as the percentage who pass  Sample estimates based on:  A learner population of 1500 partners  90 minute self study learning activity  18 minute test-out activity Good Better Best % Taking Test out 50% 70% 90% % Passing 40% 60% 75% Training Hours Saved 225 (10%) 630 (28%) 1114 (50%)
  20. 20. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Parallel test development: Developed over 50 questions mapped to course content for both the test-out and self-study end of course assessment  Pilot-tested all questions with 68 partners and received 400+ comments  Used statistical item analysis to isolate best performing questions with appropriate level of difficulty  Limited testing time while still maintaining quality  17 question test-out  Half-time strategy Rigorous test development procedures improved the quality of the test-out Slide 20
  21. 21. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  17 question test split in two parts:  First half 9 questions and second half 8 questions  Split the questions to even out content and difficulty where possible  Passing score of 80% (14 of 17)  Need to “pass” first half to proceed to second half  First half passing score:  6 of 9 which meant 8 of 8 was needed on the second half  First half passing score set to allow total pass if perfect on second half Test-out Half-time Strategy Slide 21
  22. 22. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Partners were satisfied with the test-out process Slide 22 “I thought this was a great option. Saved a lot of time and it is still not easy to take the test so it also ensures appropriate focus and knowledge.” “I thought it was a great improvement, and permitted those of us who watch these areas closely to get through the requirement (which is a good requirement to have) easily and efficiently.” 86%
  23. 23. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Success Factors Slide 23 • 231 Partners took the test-out; 20% of target audience • Reasons for not participating: - Want to take the self-study - Could not find time to complete Participation Rate • 67% Pass Rate • Of those who did not pass, 31% were stopped at the half-way point, saving them time Pass Rate • 14 minutes average test time; 9 minutes half-time • Saved 236 hours in total; 11% of training time Time Saved • 86% satisfied with test-out • Increased flexibility for partners • Favorable feedback Partner Feedback
  24. 24. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Modeling Tool Interactive Demo Slide 24 Assumptions Data Learner Population 1000 % Taking Test-Out 50% % Passing Test-Out 50% Minutes for Training 90 # Questions in Test-Out 15 Minutes Per Question 1 Additional minutes for test out (e.g. instructions) 3 Total Minutes for Test-Out 18 Total Possible Training Hours 1500 Path # of Learners Test-Out Hours Training Hours Total Hours Learner passed the test out 250 75 75 Learner did not pass the test out 250 75 375 450 Learner did not attempt the test-out 500 750 750 Total 1000 150 1125 1275 Hours Saved 225 15% Change assumptions to estimate hours saved by using a test out strategy.
  25. 25. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Future Deliverables Slide 25  Classroom Training  Integrating assessments with on-the-job learning  Traditional training model  Curriculum-based  Mandatory courses  On-the-job learning {70 / 20 / 10}  Use diagnostic assessment to better target on-the-job learning as well as formal training
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×