How Big is the Test Preparation Industryand What are Employers Doing toCounter It?April 16, 2005Cheryl PaullinPersonnel Decisions Research Institutes, Inc.In Paullin, C. (Chair). Maintaining test security in a “cheating” culture. Practitioner Forumconducted at the 20th Annual Conference of the Society for Industrial and OrganizationalPsychology, Los Angeles, CA.
April 16, 2005 Paullin: Test Prep Industry 2Disclaimer• I am not saying that test preparation and cheatingare the same thing.• I am not saying that it is unethical to provide testpreparation or test coaching services. I do believethat there are unethical providers out there.• Test prep is a good thing to the extent it:– Improves relevant skills or knowledge.– Improves general test-taking skills.– Reduces text anxiety.
April 16, 2005 Paullin: Test Prep Industry 3Method• Searched Research literature – keywords:– [employment] test preparation– test coaching• Searched the Internet, libraries, and bookstores forinformation about organizations that provide testpreparation services.• Talked with colleagues and clients about whatemployers are doing to counter the test prepindustry.
April 16, 2005 Paullin: Test Prep Industry 4Research Literature• Sackett, Schmitt, Ellingson, & Kabin (2001)*– Reviewed test coaching programs, specifically toevaluate impact (if any) on subgroup score differenceson standardized tests.• Conclusions -- Test Coaching Programs:– Benefit minority & nonminority examinees slightly, butdo not reduce subgroup score differences.– Are viewed very positively by examinees.– Can have a large practical impact, e.g., when one ortwo points is the difference between passing & failing acertification exam.*American Psychologist, Vol 56 (4), 302-318.
April 16, 2005 Paullin: Test Prep Industry 5Type of Exams Covered• Education Program Admission (SAT, GRE,GMAT, LSAT, MCAT)• Proficiency/Placement (GED, High School ExitExams, Advanced Placement, English & ForeignLanguage Skills)• Employment– Selection (ASVAB, USPS Exam 470, Police Officer,Civil Service, Fire Service).– Certification/Licensure (Health Care Professions,Teacher, Accountant, Engineer, Real Estate, FinancialPlanning, Cosmetology, Information Technology).
April 16, 2005 Paullin: Test Prep Industry 6Scope of Test Prep Industry• Number of orgs offering test prep generallycorresponds to size of examinee population(i.e., the market):– Dozens for large-scale admission exams and fornationwide or regional certification, licensure, oremployment exams.– At least one or two for virtually all location- oremployer-specific exams administered on a regularbasis to more than a few hundred people.
April 16, 2005 Paullin: Test Prep Industry 7Scope of Test Prep Industry (con’t)• Test prep is avail for > 100 different exams.• For anything other than printed test guides,almost impossible to evaluate quality priorto buying the service/materials.• One website compares services (more later).
April 16, 2005 Paullin: Test Prep Industry 8Test Prep Activities• Test Prep Activities– Test guide books (avail for > 100 different exams).– Practice tests (CD-based or online avail for 75-100 exams).– Prep courses (CD-based, online, or live avail for 20-30 exams).– Personal tutoring (for several high-stakes exams, e.g., SAT,GMAT, Medical Licensure, Bar Exam, CPA).• Practice Tests– Vary in extent to which they simulate real testing conditions.• P&P, scoring key & explanation provided.• Real-time administration & scoring with feedback -- access to itembank for X number of tests or X number of hours.
April 16, 2005 Paullin: Test Prep Industry 9Test Prep for a Fee• Providers covering the widest array ofexams:– Kaplan– Learning Express– Thomson Learning– Princeton Review– Cliff’s Test Prep• Supplemented by an uncountable number of “Dr.Paullin’s Best Test Prep Ever” websites.
April 16, 2005 Paullin: Test Prep Industry 10Costs & Claims• Costs– $10-$50 for Test Guide books.– $100-$300 for CD-based course, or limited online access topractice tests.– Several hundred dollars for a lengthy online course, live course, orseveral hours of personal tutoring.– Costs are higher for exams related to high-income professions,e.g., medical licensure, dental licensure, bar exam, CPA.• Claims– Many test prep orgs offer a money-back guarantee unless Xhappens, e.g., earning a passing score.– Most say little or nothing about qualifications of the test prepdeveloper(s); a few say course is taught by or materials weredeveloped by someone who scored high on the exam in question.
April 16, 2005 Paullin: Test Prep Industry 11Comparing Test Prep Providerswww.test-preparation.net• For each of 23 exams:– Which orgs offer test prep services.– Type and cost of prep activities.• Doesn’t evaluate quality of materials orproviders.
April 16, 2005 Paullin: Test Prep Industry 12Test Prep as a Public Service• Public Sector Jobs– Several federal government websiteswww.usajobs.opm.gov.– Public Service Employees Networkwww.pse-net.com.• College entrance – www.number2.com• Labor Unions – Not much on websites at nationallevel; may be more info at local branches.
April 16, 2005 Paullin: Test Prep Industry 13Test Developers/Sponsors• Several offer test prep, some of it free, e.g.,:– College Board – SAT.– ETS (Educational Testing Service) – many tests.– ACT (American College Testing) – ACT.– American Association of Medical Colleges – MCAT.– National Association of Pharmacy Boards – PCAT.– Project Management Institute – PMI Certification.– Pearson VUE – Nursing Certification.– Experior Online -- state-specific certification exams.
April 16, 2005 Paullin: Test Prep Industry 14What are Employers Doing toCounter Test Prep Industry?• Relying on predictors other than multiple-choicetests, e.g., interviews.• Outsourcing the testing process – countering testprep industry someone else’s problem.• Changing test content frequently.– Multiple forms Adaptive testing.– New test for each mass administration.– Easier to do with computer-based & online testing.
April 16, 2005 Paullin: Test Prep Industry 15What are Employers Doing? (con’t)• Providing their own test prep materials.• Warning test-takers to be cautious whenbuying test prep services.• (rare) Legally pursuing test preporgs/individuals that make fraudulentclaims or provide access to controlled testcontent.
April 16, 2005 Paullin: Test Prep Industry 16Bogus Credentials• Issue of increasing concern.• Search term “Diploma Mill” 10 websites.• What’s an employer to do?– http://ope.ed.gov/accreditation -- US Office of Post-Secondary Education.– Hire an organization that does education verification.– If possible, assess KSAs directly.
April 16, 2005 Paullin: Test Prep Industry 17Test Security in Litigation• Attorneys may request, or court may order, thattest items and scoring key be shared with partiesto litigation or read into the court record.• What should the I/O Psychologist do?• Principles of Psychologists and Code of Conduct (AmericanPsychological Association, 1992) -- If psychologists’ ethicalresponsibilities conflict with law, psychologists make known theircommitment to the Ethics Code and take steps to resolve theconflict in a responsible manner. (Standard 1.02)
April 16, 2005 Paullin: Test Prep Industry 18Test Security in Litigation (con’t.)• Standards for Educational and Psychological Testing (1999) state:When sensitive test documents are challenged, it may be appropriate toemploy a third party, using a closely supervised secure procedure toconduct a review of the relevant materials. Such secure procedures aregenerally preferable to placing tests, manuals, and an examinee’s testresponses in the public record.• American Psychologist editorial (1999): The psychologist should askthe court to allow delivery of the secure materials only to psychologistsor professionals who are bound by the an ethical code to maintain theirsecurity. If this isn’t feasible or isn’t allowed, then request that thecourt issue a protective order prohibiting the parties from makingcopies of the materials, requiring that the materials be returned to thepsychologist at the close of litigation, and ordering that the record besealed if test questions or answers are admitted in court.