Presentation on certification testing within the pharmaceutical industry presented at the 2007 SPBT Conference

1. Designing a Sales Representative Certification Program Through Validated Measurement and Comprehensive Assessment Greg Sapnar, Bristol-Myers Squibb Practitioner’s Perspective Steven Just, Pedagogue Solutions Psychometric & Legal Testing Perspective Tuesday, June 19 th 2:00 -3:30 Hollywood, Florida, Diplomat 4

5. Certification ? Do you Certify?

8. Four Types of Certification

10. Certification should be viewed as a program rather than an activity

11. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer Evaluation Deliver Assessments Deliver

12. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

14. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

17. Assessment Process & Resources Requirements Establish objectives Establish objectives Create test questions Validate Test (½ day meeting) Establish cut-score Report Outcomes Analyze & Interpret Outcomes Create Role-Play Scenarios & checklist Validate Role-Play Scenarios & checklist (full day meeting) Establish Cut Scores for Role-Play Train Raters & Establish Rater Reliability for Role-Play Implement Test Report Outcomes Analyze & Interpret Outcomes Implement Role Play Evaluation Role-Play Assessment Knowledge Assessment Validate objectives Validate objectives The above process illustrates basic requirements for test development. Rigor may be increased if needed to support higher levels of decision making.

18. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

20. Governance and Administration

22. Defining Terms

30. Legal Issues

37. Remediation and Consequences

42. Global Considerations

43. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

44. Job Competency Analysis

50. Creating Fair, Valid and Reliable Assessments

51. Knowledge Assessments

52. What is Validity?

58. Pilot the Test

61. What is Test Reliability?

63. Setting Passing Scores

69. Angoff Method: Example Item 1 2 3 4 5 Total Percent Judge 1 .75 .80 .75 .90 .95 4.15 83% Judge 2 .80 .90 .75 .90 .75 4.10 82% Judge 3 .85 1.00 .90 .80 .85 4.40 88% Averaging the totals for each Judge Cut Score= 84%

70. Performance Testing

73. Scoring Rubric: Example

75. Descriptive Behaviors: Example

78. Recertification

80. Ebbinghaus Curve of Forgetting

81. Ebbinghaus Curve of Forgetting

83. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

84. Some Early Results

86. Certification Development Framework The Driver Business Case Requirements Standards Governance & Administration Re-certification & Maintenance Global Considerations Develop Assessments Groundwork Design Program Policy and Procedures Develop Evaluate Evaluation Deliver Assessments Deliver Reference: Hale, Judith (2000). Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. San Francisco: Jossey-Bass/Pfeiffer

90. Questions? Gregory Sapnar Bristol-Myers Squibb [email_address] 609-897-4307 Steven B. Just Ed.D. Pedagogue Solutions [email_address] 609-921-7585 x12