International Testing: Crossing GlobalBorders While Protecting Your Own Steve Addicott Vice President, Caveon Test Security Tom Gibbs Senior Analyst, SHL Group Aimee Hobby Rhodes, JD Director of Exam Security, CFA Institute
Agenda Intros Security Threats to International Test Programs Steve Addicott, Caveon Employment Testing: The Stakes are High, and They’re Global, Too Tom Gibbs, SHL Group Assessing Country Risks Aimee Hobby Rhodes, CFA Institute Q&A
About Caveon Test Security Starting 9th year Offering the “Three Ps” Proven Practical Protection Certification/Licensure & Education Clientele Pre-eminence Testing globally We’ve seen a few things…
International Test Threats Proxy Test Taking Braindumps & “Rogue Review” Courses Nefarious Test Centers See above Unusual practices
Nefarious Test Centers NOT vendor-owned centers Example of Unusual Test Taking Test Center Hours 6 am to 6pm 130 tests taken at odd hours, ie 2am Candidates unable to schedule tests when Test Center is closed WHAT IS GOING ON? And, Why are these candidates scoring much higher than others?
Braindumps and Rogue Review Courses Test Prep Courses advertising & selling live test items Prevalence of stolen-answer key usage may exceed 20% How are items stolen? Infiltrate a top vendor’s premium security, or Bribe a remote test center? Evidence points to unauthorized decryption of exam after downloading to test site servers
Proxies and “Gunmen”Proxy Test Takers: Make Big Money Analysis uncovered a large proxy test taking service operating in at least six testing sites: USA, Bangladesh, Turkey, UAE, and an unknown country. 500 test instances @ $1000 per instance in 6 months Own and operate test centers Are branching out
Overview of presentation The user model It is 24/7/365 and its online Why the stakes are high The war for talent and the current economic conditions Why the stakes are global The war for talent is everywhere Are we managing the cheats and the pirates? The value of web patrols and a clear security framework Summary and further information We are publishing more about our security processes
24/7/365 The user model is increasingly not proctored and not delivered via group testing Delivered online with a variety of security features Linear On-the Fly Testing (LOFT) and Computer Adaptive Testing (CAT) Systems for managing item exposure and answer key exposure Supported by a security framework including data forensics, web patrols and incident response procedures (see next slide) Initial version has been operating since 2002 Verify has been operating since 2006 Now available in 26 languages
Test security monitoring Security Breach Process •Collect Evidence Web Patrols •Report to Legal •Communications with Suspected suspected offender security breach Data Forensics Reporting •Run analysis on data specific(through clients/business/other) to test, date range and client of suspected breach No Continue Item Bank monitoring compromised? Yes Replace compromised items
Further Information...Burke, E. (2009). Preserving the integrity of onlinetesting. Industrial and Organizational Psychology, 2, 35-38.This is a special edition of I/O Psychology on this topicand can be accessed via www3.interscience.wiley.com/journal/118902544/home
The stakes in the employment market haveincreased
Quotes from a well known graduaterecruitment portal“Being the honest idiot in a sea full off successful liarswill get you nowhere” “I put sweat, blood and tears into improving my CV and am absolutely sickened that lazy, lying cheating graduates are getting jobs they have done nothing to deserve”“Have to say that you shouldnt lie about what you canget caught out on but anything else is fair game. Reallife is about communication, persuasion andnegotiation”“Its important that you are able to describe in detailyour fake job just in case you are asked about it”
The war for talent is local and globalGeography is becoming less of a constraint
An applicant could be applying for a job located anywhere from anywhere
So, how do we know if we are managing the cheats and the pirates?
Web patrols are telling us that there is athreat but we can detect it and manage it
Web patrols are also telling us that theapplicants know there is security and that there are consequences
Internet discussion from April 12th 2010 ttsstt “With all the psychometric testing for a number of jobs i have applied for i am wondering how many people cheat and get help from family/friends while doing the test. From a few people ive spoken too I get the sense that it is MORE than common …Thanks guys” underthepump: “I didnt for two reasons, well two reasons on top of being honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time limit. You dont exactly have time to sit there and come to a consensus and different people work at different paces. Personally having someone else sitting there would annoy me more than it would help” brendan58: “what underthepump said pretty much. they make you sit a supervised test in the AC and it would be quiet a shame to get rejected based on fluctuating test results … just need to have the confidence to complete them yourself” nebasil: “There is no point in cheating, how do you know the person doing the test for you will do better than you? Id rather trust myself than someone else” Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy who went to his AC, he absolutely got owned at the verification test there and he was pointed and told off by the HR. HR was like "how can u perform absolutely so low", SHL is to test ur honesty as well, not just brilliance”
Coming soon …Lievens, F., and Burke, E. (2011). Dealing with the threats inherent inunproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational andOrganizational Psychology (due for publication this year)First scientific journal article describing features designed to promotethe security of online high stakes testingBartram, D., and Burke, E. (In press). Industrial/OrganizationalTesting Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.).Handbook of Test Security. New York, NY: Routledge.This also includes a review of faking good on self-report (typicalperformance) measures such as personality questionnaires
Aimee RhodesDirector of Examination Security CFA Institute
Due Diligence for International Testing CFA InstituteCFA Institute awards the CFA Charter to candidates who successfullycomplete 3 levels of testing and meet other education and work-related criteria.In 2010, approximately 200,000 candidates registered for CFA Examsadministered in 89 countries across the globe. Exams areadministered twice a year.All three levels of the CFA Exam are paper and pencil.CFA Institute also gives the CIPM designation and exam, which is CBT.
Country Risk AssessmentTwo Different Assessments: Focus on Legal, Regulatory & Tax Focus on ability to protect rights and intellectual property (including test items) in a country. CFA Institute does this prior to opening a test center in a new country.
Legal, Regulatory & Tax What governmental agencies, if any, must we interact with in order to administer our tests in the way our program is designed? What Taxes do we owe based on our program?- keep in mind other activities in the country- inquiries on a “no name” basis
Assessment to Open a New Test CenterPractice established in 2009Multi-part assessmentGoal is to examine risks & exposures unique to each country
Demographics and Political Landscape Hi, we’re new here! Understand the country to which you are going Social and Economic history – informs how you may interact with regulators, staff, and candidates Propensity for unrest and how it generally unfolds.
Protection of Intellectual Property Why is our logo on that guy’s test? To which IP conventions is the country a signatory? What protections are in place on a national basis? How do you register your trademark? Is it a first to file system? Automated or manual? Are service marks recognized? Is there institutional capacity to address IP issues? Judicial resources and corruption should be considered.
Corruption & Crime Are you gonna lose your shirt and how will you get it back when you do? Transparency International Corruption Index – also consider if the areas of corruption are related to your business. Anti-corruption legislation and enforcement – is it robust? Violent street crime in area of your test – does it endanger candidates and staff? Do the police have the ability and desire to be helpful? Is this a location in which you would want to involve the police?
Ability to Investigate What can you do when you lose your stuff? Is it safe for your home office staff to travel to the location for investigation? Is a visa required? How long to get it? If home office staff cannot go – what is your plan? Outside security vendor? Do you have one identified?
First-Hand Experience of other Organizations Hey, did you guys lose your shirt? Reach out to other organizations that test in the location you are evaluating. Requires building relationships with other organizations. This can be invaluable. You compare what your research reveals to real experiences.
Customs Hey, our stuff doesn’t seem to be getting to our testing staff! Understand how to work with the customs officials. U.S. Department of Commerce can advise and may intervene if necessary – cultivate this relationship. International Trade Office (part of Dept. of Commerce). Talk to your delivery agent – what is their experience in that country? Know your timelines.
Current Test Centers Do we really need a test center here? Proximity of nearest centers in other countries? Ease of travel for candidates to adjacent countries – understand that not everyone can go everywhere. Does expense of travel keep potential examinees out of your program?
Testing Partners and Locations So you decided to take the plunge – with whom are you swimming and in which city? Is your testing partner in the country? If not, who is? Which city (or locations) are best for our test?