Fire & Police Selection, Inc. Hiring the Most Qualified Firefighters…. … While Avoiding  (and Defending) Lawsuits
Contact Information Dan A. Biddle, Ph.D. Stacy L. Bell, M.S. CEO, Biddle Consulting Group, Inc./Fire & Police Selection, I...
Biddle Consulting Group, Inc. <ul><li>Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO)...
Fire & Police Selection, Inc. <ul><li>Over 80 Physical Ability Test installations at fire departments across the U.S. </li...
Fire & Police Selection, Inc. <ul><li>Overview of Today’s Presentation: </li></ul><ul><ul><li>The modern litigation landsc...
Approach to Test Validation <ul><li>The Quickly Changing Climate of Personnel Testing… </li></ul><ul><li>EEOC </li></ul><u...
The Quickly Changing Climate of Personnel Testing…
Adverse Impact & Test Validation -Uniform Guidelines & Civil Rights Act of 1991 <ul><li>Amends Section 703 of the 1964 Civ...
How Can Testing Practices be Challenged? Title VII Disparate Impact Discrimination Flowchart “ OR ” Practice, Procedure, o...
Validation Standards <ul><li>Court Precedence </li></ul><ul><li>Title VII: Uniform Guidelines on Employee Selection Proced...
What is Validation? <ul><li>Traditionally, it is making sure a selection procedure (i.e., test) measures what it is design...
Uniform Guidelines Style Validation… <ul><li>Content Validity   </li></ul><ul><ul><li>Section14C </li></ul></ul><ul><li>Cr...
What Types of Validation are REALLY USED in Practice?   <ul><li>Content  (80%) </li></ul><ul><li>Criterion  (15%) </li></u...
<ul><li>Cannot  alone support inferences about mental processes </li></ul><ul><ul><li>Cannot measure traits (such as intel...
Content Validity Essential #1: Job Analysis Research <ul><li>Level 1: Job Description/Job Analysis Review </li></ul><ul><l...
What Ratings are Typically Collected During a Content Validity Job Analysis? (Level 2-3) <ul><li>For  Job Duties  ratings ...
<ul><li>For  KSA s ratings are typically collected for: </li></ul><ul><ul><li>Frequency </li></ul></ul><ul><ul><li>Importa...
Content Validity Essential #2: Targeting Key KSAs for Test Building <ul><li>Only certain KSAs can be tested… </li></ul><ul...
Firefighter Job Analysis  Duties   Sample yes yes 4 4.00 8.25 Takes inventory and records and reports equipment and appara...
Firefighter Job Analysis  KSAPCs   Sample 3 1 HQ 4.13 6.38 A5, A20, C6, F11 Ability to maintain respect for other’s emotio...
Selection Plan <ul><li>Identify appropriate weights for the testing components. </li></ul><ul><ul><li>Completing a selecti...
Selection Plan— Sample  *The score from this selection procedure will be combined with scores from other selection procedu...
Job  Duties Operationally defined KSAOs Other KSAOs Selection  Devices (e.g., application form, tests, interviews) Content...
<ul><li>Pass/Fail Cutoffs: </li></ul><ul><ul><li>“ Normal Expectations of Acceptable Proficiency in the Workplace” (Guidel...
Using Test Scores <ul><li>Ranking  assumes one applicant is  reliably  more qualified than the other </li></ul><ul><li>Ban...
Pass/Fail Cutoffs <ul><li>Some major considerations: </li></ul><ul><ul><li>NOT ADVISED:  </li></ul></ul><ul><ul><ul><li>Us...
Using Test Scores <ul><li>Tests can be used on a pass/fail basis to ensure that candidates possess the  minimum competency...
Comparison Between “Test Use” Options Low Moderate High # Item Requirements Low Moderate High Reliability Requirements Hig...
Weighting/Combining Scores Some thoughts... Job Performance Cognitive  Ability Experience/ Background Interpersonal Person...
Sample Data—Cognitive and Personality Test Scores  Total Scores 156 69 W M 175 70 W F 161 70 B M 158 71 B M M M M Gender W...
Sample Data—Cognitive and Personality Test Scores  Combined Test Scores M M M M F M M Gender W W B B W B W Ethnicity 63 69...
Steps for Weighting Tests Using Content Validity Methods <ul><li>Select a panel of 4-12 Subject-Matter Experts (SMEs) who ...
Standardizing Scores <ul><li>Before individual test scores can be weighted and combined, they should be  standard scored ....
Building a Balanced Hiring Program by Sorting the Key Firefighter Competencies for Success <ul><li>In 2008, FPSI obtained ...
U.S. Fire Department Input & Opinions Personal Characteristics 55% Cognitive/Academic Skills & Abilities 45% National Fire...
Proposed Solutions for Testing Cognitive/Academic Competencies  for Entry-Level Firefighters Notes: CV: Content Validity; ...
Proposed Solutions for Personal Characteristics  for Entry-Level Firefighters Notes: CV: Content Validity; CRV: Criterion-...
Selecting the Ideal Firefighter Critical Skills & Abilities Test “A” Test “B” Reading Ability Mathematical Ability Spatial...
Challenges to Building a Balanced Testing Program and Recommendations for Success <ul><li>The most significant challenge t...
Failure to Use a “Balanced Test”   <ul><li>May have limited validity. </li></ul><ul><li>Less minority representation in th...
Fire & Police Selection, Inc.  Contact us at: <ul><li>Mailing Address </li></ul><ul><ul><li>193 Blue Ravine Rd. Suite 270 ...
Upcoming SlideShare
Loading in …5
×

Hiring the Most Qualified Firefighter While Avoiding Lawsuits

1,769 views

Published on

Keep your department EEO litigation free and maintain a balanced workforce. Reducing litigation and achieving a diverse workforce are related goals because departments that have proactive plans to build a diverse workforce also tend to be involved in fewer lawsuits. While achieving one does not guarantee the other, the strategically-managed fire department will strive for both simultaneously.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,769
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • All From 14C4 of Guidelines.
  • Hiring the Most Qualified Firefighter While Avoiding Lawsuits

    1. 1. Fire & Police Selection, Inc. Hiring the Most Qualified Firefighters…. … While Avoiding (and Defending) Lawsuits
    2. 2. Contact Information Dan A. Biddle, Ph.D. Stacy L. Bell, M.S. CEO, Biddle Consulting Group, Inc./Fire & Police Selection, Inc. 193 Blue Ravine, Ste. 270 Folsom, CA 95630 1-888-990-3473 www.biddle.com | www.FPSI.com Email: dan@biddle.com | s [email_address]
    3. 3. Biddle Consulting Group, Inc. <ul><li>Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) </li></ul><ul><li>EEO Litigation Support / OFCCP (federal contracting) Audit Support </li></ul><ul><li>Compensation Analyses / Test Development and Validation </li></ul><ul><li>Published: Adverse Impact and Test Validation , 2 nd Ed., as a practical guide for HR professionals </li></ul><ul><li>Editor & Publisher: EEO Insight an industry e-Journal </li></ul><ul><li>Creator and publisher of a variety of productivity </li></ul><ul><li>Software/Web Tools: </li></ul><ul><ul><li>OPAC ® (Administrative Skills Testing) </li></ul></ul><ul><ul><li>CritiCall ® (9-1-1 Dispatcher Testing) </li></ul></ul><ul><ul><li>AutoAAP ™ (Affirmative Action Software and Services) </li></ul></ul><ul><ul><li>C 4 ™ (Contact Center Employee Testing) </li></ul></ul><ul><ul><li>Encounter ™ (Video Situational Judgment Test) </li></ul></ul><ul><ul><li>Adverse Impact Toolkit ™ (free online at www.disparateimpact.com) </li></ul></ul><ul><ul><li>AutoGOJA ® (Automated Guidelines Oriented Job Analysis ® ) </li></ul></ul>Industry Leader
    4. 4. Fire & Police Selection, Inc. <ul><li>Over 80 Physical Ability Test installations at fire departments across the U.S. </li></ul><ul><li>Over 800 personnel and fire and police clients across the U.S. and Canada </li></ul><ul><li>Entry-level written, physical, interview assessments, manipulative skills tests, cognitive ability tests, and personality tests available for selection of public safety personnel </li></ul><ul><li>Post-Academy certification tests </li></ul><ul><li>Fire Promotional tests and assessment exercises </li></ul><ul><li>Work Styles Inventory for predicting firefighter and police officer job performance </li></ul><ul><li>Comprehensive Examination Battery for Entry-Level Firefighters </li></ul><ul><li>Situational Judgment/Writing Ability Test for Entry-Level Police Officers </li></ul><ul><li>New National Fire Select Test (NFST ™) for Entry-Level Firefighters </li></ul>
    5. 5. Fire & Police Selection, Inc. <ul><li>Overview of Today’s Presentation: </li></ul><ul><ul><li>The modern litigation landscape </li></ul></ul><ul><ul><li>Litigation triggers </li></ul></ul><ul><ul><li>Validation 101: Building and using tests </li></ul></ul><ul><ul><li>A quick tour through an “ideally weighted” test for firefighters </li></ul></ul>
    6. 6. Approach to Test Validation <ul><li>The Quickly Changing Climate of Personnel Testing… </li></ul><ul><li>EEOC </li></ul><ul><li>DOJ </li></ul><ul><li>DOL (OFCCP—Office of Federal Contract </li></ul><ul><li>Compliance Programs) </li></ul><ul><li>State EEO Offices of Fair Employment </li></ul><ul><li>Private Plaintiff Groups </li></ul>
    7. 7. The Quickly Changing Climate of Personnel Testing…
    8. 8. Adverse Impact & Test Validation -Uniform Guidelines & Civil Rights Act of 1991 <ul><li>Amends Section 703 of the 1964 Civil Rights Act (Title VII) </li></ul><ul><li>(k)(1)(A). An unlawful employment practice based on </li></ul><ul><li>disparate impact is established under this title only if: </li></ul><ul><li>  </li></ul><ul><li>A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR , </li></ul><ul><li>  A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice , and the respondent refuses to adopt such alternative employment practice. </li></ul><ul><li>  </li></ul>
    9. 9. How Can Testing Practices be Challenged? Title VII Disparate Impact Discrimination Flowchart “ OR ” Practice, Procedure, or Test (PPT) Adverse Impact? YES NO Is the PPT Valid? YES NO Alternative Employment Practice? NO Defendant Prevails YES Plaintiff Prevails END Plaintif f Prevails
    10. 10. Validation Standards <ul><li>Court Precedence </li></ul><ul><li>Title VII: Uniform Guidelines on Employee Selection Procedures (1978): EEOC, DOJ, CSB, DOL </li></ul><ul><li>Principles for the Validation and Use of Personnel Selection Procedures (2003) </li></ul><ul><ul><li>Division 14 of the American Psychological Association (the Society for Industrial and Organizational Psychology, or SIOP) </li></ul></ul><ul><li>Standards for Educational and Psychological Testing, published by: </li></ul><ul><ul><li>American Educational Research Association </li></ul></ul><ul><ul><li>American Psychological Association, and </li></ul></ul><ul><ul><li>National Council on Measurement in Education </li></ul></ul>
    11. 11. What is Validation? <ul><li>Traditionally, it is making sure a selection procedure (i.e., test) measures what it is designed to measure </li></ul><ul><li>In a legal realm, a selection procedure is valid if it can be proven by an employer that it is “…job related for the position in question and consistent with business necessity.” </li></ul>
    12. 12. Uniform Guidelines Style Validation… <ul><li>Content Validity </li></ul><ul><ul><li>Section14C </li></ul></ul><ul><li>Criterion Validity </li></ul><ul><ul><li>Section 14B </li></ul></ul><ul><li>Construct Validity </li></ul><ul><ul><li>Section 14D </li></ul></ul>
    13. 13. What Types of Validation are REALLY USED in Practice? <ul><li>Content (80%) </li></ul><ul><li>Criterion (15%) </li></ul><ul><li>Construct (5%) </li></ul><ul><li>Why? </li></ul><ul><ul><li>Simplicity </li></ul></ul><ul><ul><li>Defensibility </li></ul></ul>
    14. 14. <ul><li>Cannot alone support inferences about mental processes </li></ul><ul><ul><li>Cannot measure traits (such as intelligence, aptitude, personality, commonsense, judgment, leadership, spatial ability) </li></ul></ul><ul><li>Not appropriate for those KSAs that will be expected to learn on the job </li></ul>Content Validity
    15. 15. Content Validity Essential #1: Job Analysis Research <ul><li>Level 1: Job Description/Job Analysis Review </li></ul><ul><li>Level 2: Test Customization Survey (TCS) </li></ul><ul><li>Level 3: Full Job Analysis (GOJA) </li></ul>
    16. 16. What Ratings are Typically Collected During a Content Validity Job Analysis? (Level 2-3) <ul><li>For Job Duties ratings are typically collected for: </li></ul><ul><ul><li>Frequency </li></ul></ul><ul><ul><li>Importance * </li></ul></ul><ul><ul><ul><li>Guidelines , 14.C[1,2,4] </li></ul></ul></ul><ul><ul><li>Best Worker * </li></ul></ul><ul><ul><ul><li>Guidelines , 14.C[9] </li></ul></ul></ul><ul><ul><li>Fundamental </li></ul></ul><ul><ul><ul><li>Americans with Disabilities Act </li></ul></ul></ul><ul><ul><li>Assignable </li></ul></ul><ul><ul><ul><li>Americans with Disabilities Act </li></ul></ul></ul>* = required under federal Guidelines for Content Validity
    17. 17. <ul><li>For KSA s ratings are typically collected for: </li></ul><ul><ul><li>Frequency </li></ul></ul><ul><ul><li>Importance * </li></ul></ul><ul><ul><ul><li>Guidelines, 14.C[1,2,4] </li></ul></ul></ul><ul><ul><li>Best Worker </li></ul></ul><ul><ul><ul><li>Guidelines, 14.C[9] </li></ul></ul></ul><ul><ul><li>Minimum v. Desirable Qualifications </li></ul></ul><ul><ul><ul><li>Guidelines, 14.C[4] </li></ul></ul></ul><ul><ul><li>Level needed upon entry </li></ul></ul><ul><ul><ul><li>Guidelines, 5.F, 14.C[1] </li></ul></ul></ul><ul><ul><li>Level needed for success </li></ul></ul><ul><ul><ul><li>knowledge only </li></ul></ul></ul>* = required under federal Guidelines for Content Validity What Ratings are Typically Collected During a Content Validity Job Analysis? (Level 2-3)
    18. 18. Content Validity Essential #2: Targeting Key KSAs for Test Building <ul><li>Only certain KSAs can be tested… </li></ul><ul><ul><li>Only “concrete” KSAs can be tested under content validity (cannot test abstract KSAs) </li></ul></ul><ul><ul><li>KSAs need to be “operationally defined” </li></ul></ul><ul><ul><li>Important/Critical for the job </li></ul></ul><ul><ul><li>Needed “Day One” on the job </li></ul></ul><ul><ul><li>Duty/KSA Linkages </li></ul></ul>
    19. 19. Firefighter Job Analysis Duties Sample yes yes 4 4.00 8.25 Takes inventory and records and reports equipment and apparatus missing or needing repair or replacement. A3 no yes 4 3.13 8.25 Recognizes the need and makes recommendations for entries in daily log. A2 no yes 5 4.88 8.75 Arrives on time, receives and follows orders, instructions, regarding the work to be done. A1 Assignable Fundamental Best Worker Importance Frequency Domain A Station Duties   Supervisor Ratings Firefighter/EMT-I, Firefighter-PM and Fire Engineer Duties  
    20. 20. Firefighter Job Analysis KSAPCs Sample 3 1 HQ 4.13 6.38 A5, A20, C6, F11 Ability to maintain respect for other’s emotional needs following a crisis. O55 4 1 MQ 4.63 7.63 A5, A20, E1, F8, G2 Willing to remain calm under pressure or heavy workloads, not easily given to hostility, anxiousness, or vulnerability. Wiling to make sensible decisions under pressure. O54 4 1 MQ 4.13 7.25 A1, A2, A24, C5 Willing to be resourceful, goal-oriented, and proactive in bringing tasks to completion. O53 Level Needed Upon Entry Best Worker MQ vs. HQ Importance Frequency This KSAPC is linked to the following duties. Domain O Knowledges, Skills, Abilities, and Personal Characteristics (KSAPCs) : The following knowledge skills, abilities, and personal characteristics affect work behavior and are necessary or important for successful job performance.   Supervisor Ratings Firefighter/EMT-I, Firefighter-PM, and Fire Engineer KSAPCs      
    21. 21. Selection Plan <ul><li>Identify appropriate weights for the testing components. </li></ul><ul><ul><li>Completing a selection plan is the step in the validation process where the key, measurable KSAPCs and Physical Requirements are laid out as targets to be assessed by the selection process. </li></ul></ul><ul><ul><li>A selection plan distills the complete list of KSAPCs and Physical Requirements into only those that can and should be tested by one or more selection procedures in the overall selection process. </li></ul></ul><ul><ul><li>The result is a detailed plan that identifies the critical KSAPCs to measure, five necessary ratings, and how the device should be used (e.g., pass/fail, banding, etc.) </li></ul></ul><ul><ul><li>The final step is to identify the weights for the various components. </li></ul></ul>
    22. 22. Selection Plan— Sample *The score from this selection procedure will be combined with scores from other selection procedures in the selection process. P/F MQ 3.1 N/A 3.5 1.4 Basic math skills MQ 3.4 N/A 3.9 1.9 Upper body strength P/F MQ 3.9 3.4 3.5 2.6 Knowledge of Fire Codes R MQ 3.7 N/A 3.5 3.9 Verbal communication skills R MQ 4 N/A 3.2 4.2 Interpersonal and team work skills Background Check Structured Interview Written Test Application Form Minimum/Helpful Qualification (MQ/HQ) Importance (1-5) Level Needed (Knowledges Only) (1-4) Level Needed Upon Entry (1-4) Best Worker (1-5) Knowledge, Skills, Ability, Personal Characteristic, or Physical Requirement Selection Procedure and Use ( P ass/Fail, R ank, or C ombine*) KSAPC/Physical Requirement Rating KSAPC/ Phys Req.
    23. 23. Job Duties Operationally defined KSAOs Other KSAOs Selection Devices (e.g., application form, tests, interviews) Content Validity Essential #3: Connecting the Test to the Job Content Valid!
    24. 24. <ul><li>Pass/Fail Cutoffs: </li></ul><ul><ul><li>“ Normal Expectations of Acceptable Proficiency in the Workplace” (Guidelines, 5H) </li></ul></ul><ul><ul><li>Modified Angoff (U.S. v. South Carolina, USSC) </li></ul></ul><ul><li>Banding: </li></ul><ul><ul><li>Substantially Equally Qualified Applicants </li></ul></ul><ul><ul><li>Statistically Driven (use Std. Error of Difference) </li></ul></ul><ul><li>Ranking: </li></ul><ul><ul><li>Is there adequate score dispersion? </li></ul></ul><ul><ul><li>Does the test have high reliability? (e.g., >.85) </li></ul></ul><ul><ul><li>Is the KSA performance differentiating? </li></ul></ul><ul><li>Weighted/combined with other tests </li></ul><ul><ul><li>How are the weights related to the job </li></ul></ul><ul><ul><li>Do they come from the job analysis or SME ratings? </li></ul></ul>Content Validity Essential #4: (Test Usage)
    25. 25. Using Test Scores <ul><li>Ranking assumes one applicant is reliably more qualified than the other </li></ul><ul><li>Banding considers the unreliability of the test battery and “ties” applicants </li></ul><ul><li>Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified” </li></ul>
    26. 26. Pass/Fail Cutoffs <ul><li>Some major considerations: </li></ul><ul><ul><li>NOT ADVISED: </li></ul></ul><ul><ul><ul><li>Using an arbitrary cutoff (e.g., 70%) </li></ul></ul></ul><ul><ul><ul><li>Using applicant scores to benchmark (e.g., setting cutoff scores at mean-SD of applicant scores) </li></ul></ul></ul><ul><ul><li>ADVISED: </li></ul></ul><ul><ul><ul><li>Consider “Normal expectations of acceptable proficiency in the workplace” (Guidelines, 5H) </li></ul></ul></ul><ul><ul><ul><ul><li>Usually requires SME-level data or ratings </li></ul></ul></ul></ul><ul><ul><ul><li># Applicants that can be feasibly processed </li></ul></ul></ul><ul><ul><li>CONSIDER: </li></ul></ul><ul><ul><ul><li>Is the test supported by content validity information or criterion-related information? </li></ul></ul></ul><ul><ul><ul><li>How critical are the KSAs measured? </li></ul></ul></ul><ul><ul><ul><li>Does the test measure “baseline” or “differentiating” KSAs? </li></ul></ul></ul><ul><ul><ul><li>How would current incumbents perform on this test? </li></ul></ul></ul>
    27. 27. Using Test Scores <ul><li>Tests can be used on a pass/fail basis to ensure that candidates possess the minimum competency levels needed for the job. </li></ul><ul><li>Tests can be used in a ranked fashion whereby criterion-related validity evidence supports that higher test performance statistically predicts better job performance (or, content validity can also be used per UGESP 14C9). </li></ul><ul><li>Tests can be used by weighting/combining multiple test scores whereby a “compensatory” or combined score is used. </li></ul><ul><li>Banding is a recommended “balanced” solution because it uses the test results in ways that are supportable by the tests themselves. </li></ul>
    28. 28. Comparison Between “Test Use” Options Low Moderate High # Item Requirements Low Moderate High Reliability Requirements High Moderate Low Development Time High Moderate/Controllable Restrictive/Controllable Applicant Flow High Moderate Low Cost Low Moderate High Utility Low Moderate High Litigation &quot;Red Flag&quot; High High Low Defensibility Low Moderate High Adverse Impact Low Moderate High Validation Requirements Pass/Fail Cutoffs Banding Ranking Factor
    29. 29. Weighting/Combining Scores Some thoughts... Job Performance Cognitive Ability Experience/ Background Interpersonal Personality Factors
    30. 30. Sample Data—Cognitive and Personality Test Scores Total Scores 156 69 W M 175 70 W F 161 70 B M 158 71 B M M M M Gender W B W Ethnicity 63 68 72 Cognitive Score 156 225 175 Personality Score
    31. 31. Sample Data—Cognitive and Personality Test Scores Combined Test Scores M M M M F M M Gender W W B B W B W Ethnicity 63 69 70 71 70 68 72 Cognitive % Score 156 156 161 158 175 225 175 Personality Score 73% 79% 81% 81% 81% 83% 83% Combined % Score
    32. 32. Steps for Weighting Tests Using Content Validity Methods <ul><li>Select a panel of 4-12 Subject-Matter Experts (SMEs) who are truly experts in the content and “functional areas” of the target position. Ensure diversity on the panel. </li></ul><ul><li>Provide a copy of the Job Analysis for each SME. Be sure that the Job Analysis itemizes the various job duties and KSAPCs that are important or critical to the job. </li></ul><ul><li>Provide each SME with a copy of each test (or a highly detailed description of the content of the test if confidentiality issues prohibit SMEs from viewing the actual test). </li></ul><ul><li>Explain the confidential nature of the workshop, the overall goals and outcomes, and ask the SMEs to sign confidentiality agreements. </li></ul><ul><li>Discuss and review with SMEs the content of each test and the KSAPCs measured by each. Also discuss the extent to which certain tests may be better measures of certain KSAPCs than others. Factors such as the vulnerability of certain tests to fraud, reliability issues, and others should be discussed. </li></ul><ul><li>Provide a survey to SMEs that asks them to distribute 100 points among the tests that will be combined. Be sure that they consider the importance levels of the KSAPCs measured by the tests, and the job duties to which they are linked, when completing this step. </li></ul><ul><li>Detect and remove outlier SMEs from the data set. </li></ul><ul><li>Calculate the average weight for each test. These averages are the weights to use when combining the test into a composite score. </li></ul>
    33. 33. Standardizing Scores <ul><li>Before individual test scores can be weighted and combined, they should be standard scored . This process normalizes the scores and is a necessary step to place different tests on a level playing field. </li></ul><ul><li>Standardizing test scores is simple: </li></ul><ul><ul><li>Convert raw scores into Z scores by subtracting each applicant’s score from the average (mean) score of all applicants and dividing this value by the standard deviation (of all applicant total scores). </li></ul></ul><ul><ul><li>Multiply the Z scores by the respective weights of the test components and a final score for each applicant can then be calculated. </li></ul></ul>
    34. 34. Building a Balanced Hiring Program by Sorting the Key Firefighter Competencies for Success <ul><li>In 2008, FPSI obtained input from over 140 chief-level officers who were asked to weigh the importance of key critical skills and abilities required for successful entry-level firefighter job performance </li></ul><ul><li>Developed cutoff score by working with a panel of 12 Chiefs </li></ul><ul><li>Results </li></ul><ul><ul><li>The process identified the skills and abilities that are most critical for entry-level firefighters. With input from over 140 fire departments, these ratings are being used to build a written test that properly weighs the most important qualifications for the entry-level firefighter job. With this type of input from fire executives across the country, the National Fire Select Test™ is the first test of its type! </li></ul></ul>
    35. 35. U.S. Fire Department Input & Opinions Personal Characteristics 55% Cognitive/Academic Skills & Abilities 45% National Fire Select Test™ Cognitive/Academic Component Strategic Decision- Making 14% Problem Solving 14% Map Reading 10% Writing Ability 15% Reading Ability 18% Mathematical Reasoning 15% Mechanical Ability 14% Personal Characteristics Component Thoroughness/ Attention To Detail 9% Public Relations 9% Sensitivity 8% Emotional Stability 10% Proactive/ Goal-Oriented 8% Truthfulness/ Integrity 13% Allegiance/ Loyalty 10% Working Under Stress 10% Teamwork 12% Following Orders 11%
    36. 36. Proposed Solutions for Testing Cognitive/Academic Competencies for Entry-Level Firefighters Notes: CV: Content Validity; CRV: Criterion-related validity. CV/CRV Using CV, measure mechanical comprehension skills such as leverage, force, and mechanical/ physics contexts regarding weights, shapes, and distances. Also can measure spatial reasoning (when using a CRV validation strategy). 12% Mechanical Ability CV While a Structured Interview is the best tool for measuring this skill (because the applicant can be asked to apply this skill in firefighter-specific scenarios), some level of this skill can be measured using word problems or other contexts supplied in written format where applicants can consider cause/effect of certain actions. 13% Strategic Decision-Making CRV Measure using word problems measuring reasoning skills in job-rich contexts. 15% Problem-Solving CV Measure using maps and related questions asking applicants how they would maneuver to certain locations. Include directional awareness. 8% Map Reading CV Measure using writing passages or word recognition lists, sentence clarity, and/or grammar evaluation items. 12% Writing CV While a Structured Interview is the best tool for measuring this skill (because the skill includes verbal and non-verbal aspects), some level of this skill can be measured using word recognition lists or sentence clarity items. 15% Verbal Communication CV Measure using either (1) “Test Preparation Manual” approach (where applicants are given a manual and asked to study it for a few weeks prior to taking the test based on the Manual); or (2) a short reading passage containing material at a similar difficulty/context to the job that applicants are allowed to study during the testing session and answer related test items. 14% Reading CV Use written or “work sample” format; measure using a limited number of multiple-choice items. Balance various types of math skills (add/subtract/multiple, etc.). 10% Math Typical Validation Method Proposed Testing Solution Weight Cognitive/Academic (45% Overall Importance)
    37. 37. Proposed Solutions for Personal Characteristics for Entry-Level Firefighters Notes: CV: Content Validity; CRV: Criterion-related validity. 11% Following Orders 9% Thoroughness/Attention to Detail CV/CRV These competencies can be effectively measured using either an SJT (using a CV or CRV strategy) or a Conscientiousness (CS) scale (using a CRV strategy). A CS test can be developed using just 20-30 items (using likert- type responses). Such tests are typically successful in predicting job performance in fire settings. 8% Proactive/Goal-Oriented 8% Sensitivity 10% Emotional Stability 9% Public Relations 13% Truthfulness/Integrity 10% Allegiance/Loyalty 10% Working Under Stress CV/CRV Under a CV strategy, a Situational Judgment Test (SJT) can be used for measuring these skills. Alternatively, a custom personality test can be developed using CRV. while these types of assessments can measure if an applicant knows the most appropriate response (using an SJT) or the best attitude or disposition (personality test), they are limited in that they cannot measure whether an applicant would actually respond in such a way. For these reasons, measuring the underlying traits that tend to generate these positive behaviors is typically the most effective strategy. Structured Interviews can also provide useful insight into these types of competencies, as well as background and reference evaluations. However, these tools are time consuming and expensive, so measuring these areas in the testing stage is an effective strategy. 12% Teamwork Typical Validation Method Proposed Testing Solution Weight Personal Characteristics (55% Overall)
    38. 38. Selecting the Ideal Firefighter Critical Skills & Abilities Test “A” Test “B” Reading Ability Mathematical Ability Spatial Abilities Mechanical Aptitudes Writing Ability Interpersonal Skills Decision Making Test “C” Map Reading NFST™
    39. 39. Challenges to Building a Balanced Testing Program and Recommendations for Success <ul><li>The most significant challenge to building an effective testing program for entry-level firefighters lies with testing the “soft skills” (personal characteristics). This is because these skills—such as teamwork and interpersonal skills—are crucial ingredients for success but they are the most difficult to measure in a typical testing format. </li></ul><ul><li>For example, developing a math test is easy; developing a test for measuring teamwork skills is not—but the latter was rated as more important for overall job success! </li></ul><ul><li>The reason for this is that many skills and abilities are “concrete” (opposed to theoretical and abstract). An applicant’s math skills can readily be tapped using questions that measure numerical skills at the same level these skills are required on the job. Abstract or soft skills like teamwork are more difficult to measure during a two-hour testing session. </li></ul>
    40. 40. Failure to Use a “Balanced Test” <ul><li>May have limited validity. </li></ul><ul><li>Less minority representation in the final eligibility list. </li></ul><ul><li>A testing process that leaves out (or underestimates) important cognitive/academic skills will likely result in a significant number of cadets washing out of the academy, and can also lead to poor performance on the job. </li></ul><ul><li>A testing process that over-measures cognitive/academic skills and under-measures personal characteristics could lead to a group of book-smart firefighters who lack the skills necessary to work cooperatively in the close living conditions required by firefighters. </li></ul><ul><li>Greater potential for lawsuits which may result in lowering standards. </li></ul>
    41. 41. Fire & Police Selection, Inc. Contact us at: <ul><li>Mailing Address </li></ul><ul><ul><li>193 Blue Ravine Rd. Suite 270 </li></ul></ul><ul><ul><li> Folsom, CA. 95630 </li></ul></ul><ul><li>Website </li></ul><ul><ul><li>www.FPSI.com </li></ul></ul><ul><ul><li>www.NationalFireSelect.com </li></ul></ul><ul><li>Telephone </li></ul><ul><ul><li>916.294.4242 x. 245 </li></ul></ul><ul><li>E-Mail </li></ul><ul><ul><li>[email_address] </li></ul></ul>

    ×