Caveon Webinar - International Testing 11-15-2011

International Testing: Crossing GlobalBorders While Protecting Your Own Steve Addicott Vice President, Caveon Test Security Tom Gibbs Senior Analyst, SHL Group Aimee Hobby Rhodes, JD Director of Exam Security, CFA Institute

Agenda Intros Security Threats to International Test Programs  Steve Addicott, Caveon Employment Testing: The Stakes are High, and They’re Global, Too  Tom Gibbs, SHL Group Assessing Country Risks  Aimee Hobby Rhodes, CFA Institute Q&A

Steve AddicottVice President, Caveon

About Caveon Test Security Starting 9th year Offering the “Three Ps”  Proven  Practical  Protection Certification/Licensure & Education Clientele  Pre-eminence  Testing globally We’ve seen a few things…

International Test Threats Proxy Test Taking Braindumps & “Rogue Review” Courses Nefarious Test Centers  See above  Unusual practices

Nefarious Test Centers NOT vendor-owned centers Example of Unusual Test Taking  Test Center Hours 6 am to 6pm  130 tests taken at odd hours, ie 2am  Candidates unable to schedule tests when Test Center is closed  WHAT IS GOING ON?  And, Why are these candidates scoring much higher than others?

Braindumps and Rogue Review Courses Test Prep Courses advertising & selling live test items Prevalence of stolen-answer key usage may exceed 20% How are items stolen?  Infiltrate a top vendor’s premium security, or  Bribe a remote test center?  Evidence points to unauthorized decryption of exam after downloading to test site servers

Proxies and “Gunmen”Proxy Test Takers: Make Big Money  Analysis uncovered a large proxy test taking service operating in at least six testing sites: USA, Bangladesh, Turkey, UAE, and an unknown country.  500 test instances @ $1000 per instance in 6 months Own and operate test centers Are branching out

A Gunman for Hire September 10, 20xx Dear sir: We can take XTest exam service for you now in china, please fill out the attached file and send to me. Well finish the exam within one week. The price now for XTest exam service is USD 1000, you can send half payment before exam, half payment after exam. Best regards, Mr. Zhao9 © Caveon Test Security

Gunman September 12, 20xx Hello: We only accept WESTERN UNION payment, you can send half before exam, half after exam. If you do not have a passport, you can send me a copy of your drivers ID, you can scan it and send to me. Please fill out the attached file and send to me when you want to start the XTest exam. Cheers. Mr. Zhao10 © Caveon Test Security

Gunman October 16, 20xx My answer for you question: 1. Testing ID,when we register XTest for you, the test system will automatically give you the ID. 2. After exam, XTest will send you the XTest certification directly from USA. 3. When we finish XTest for you, you can check exam result by XTest official site,which can verify your exam result. For the payment, you can send half before exam, half after we finish XTest for you, XTest will send you certification within 4-8 weeks. When you want to start the exam, please let me know. Cheers. Mr.Zhao11 © Caveon Test Security

Gunman October 27, 20xx Hello: Our X test centre is being checked until now, Im afraid we should wait for a little more time to start the XTest. Do you need YTest or ZTest exam service? I can prepare for you next week. Please let me know your decision, thanks. Best regards. Mr. Zhao12 © Caveon Test Security

Gunman November 10, 2007 The payment is OK, thanks. Now I send you XTest info: [Link to Site] Test ID* or XTest Certification ID: XTest11327691 Password: 123456 ***Please keep your personal information updated always. Youd better login the XTest site each week. You can modify the password now. If you or your friend need YTest, ZTest exam in the future, you can contact me, there are XTest certification status in the XTest site, please pay attention to it. Contact me if you have any question. Best regards, Mr.Zhao13 © Caveon Test Security

Tom GibbsSenior Analyst, SHL

Overview of presentation The user model  It is 24/7/365 and its online Why the stakes are high  The war for talent and the current economic conditions Why the stakes are global  The war for talent is everywhere Are we managing the cheats and the pirates?  The value of web patrols and a clear security framework Summary and further information  We are publishing more about our security processes

The User Model

24/7/365 The user model is increasingly not proctored and not delivered via group testing Delivered online with a variety of security features  Linear On-the Fly Testing (LOFT) and Computer Adaptive Testing (CAT)  Systems for managing item exposure and answer key exposure  Supported by a security framework including data forensics, web patrols and incident response procedures (see next slide) Initial version has been operating since 2002 Verify has been operating since 2006 Now available in 26 languages

Test security monitoring Security Breach Process •Collect Evidence Web Patrols •Report to Legal •Communications with Suspected suspected offender security breach Data Forensics Reporting •Run analysis on data specific(through clients/business/other) to test, date range and client of suspected breach No Continue Item Bank monitoring compromised? Yes Replace compromised items

Further Information...

Further Information...Burke, E. (2009). Preserving the integrity of onlinetesting. Industrial and Organizational Psychology, 2, 35-38.This is a special edition of I/O Psychology on this topicand can be accessed via www3.interscience.wiley.com/journal/118902544/home

Further Information...

The stakes in the employment market haveincreased

Quotes from a well known graduaterecruitment portal“Being the honest idiot in a sea full off successful liarswill get you nowhere” “I put sweat, blood and tears into improving my CV and am absolutely sickened that lazy, lying cheating graduates are getting jobs they have done nothing to deserve”“Have to say that you shouldnt lie about what you canget caught out on but anything else is fair game. Reallife is about communication, persuasion andnegotiation”“Its important that you are able to describe in detailyour fake job just in case you are asked about it”

The war for talent is local and globalGeography is becoming less of a constraint

An applicant could be applying for a job located anywhere from anywhere

So, how do we know if we are managing the cheats and the pirates?

Web patrols are telling us that there is athreat but we can detect it and manage it

Web patrols are also telling us that theapplicants know there is security and that there are consequences

Internet discussion from April 12th 2010 ttsstt “With all the psychometric testing for a number of jobs i have applied for i am wondering how many people cheat and get help from family/friends while doing the test. From a few people ive spoken too I get the sense that it is MORE than common …Thanks guys” underthepump: “I didnt for two reasons, well two reasons on top of being honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time limit. You dont exactly have time to sit there and come to a consensus and different people work at different paces. Personally having someone else sitting there would annoy me more than it would help” brendan58: “what underthepump said pretty much. they make you sit a supervised test in the AC and it would be quiet a shame to get rejected based on fluctuating test results … just need to have the confidence to complete them yourself” nebasil: “There is no point in cheating, how do you know the person doing the test for you will do better than you? Id rather trust myself than someone else” Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy who went to his AC, he absolutely got owned at the verification test there and he was pointed and told off by the HR. HR was like "how can u perform absolutely so low", SHL is to test ur honesty as well, not just brilliance”

Cheating and piracy is a fact

It is global!

It requires constant vigilance

It requires innovation and commitment

Coming soon …Lievens, F., and Burke, E. (2011). Dealing with the threats inherent inunproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational andOrganizational Psychology (due for publication this year)First scientific journal article describing features designed to promotethe security of online high stakes testingBartram, D., and Burke, E. (In press). Industrial/OrganizationalTesting Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.).Handbook of Test Security. New York, NY: Routledge.This also includes a review of faking good on self-report (typicalperformance) measures such as personality questionnaires

Thank You

Aimee RhodesDirector of Examination Security CFA Institute

Due Diligence for International Testing CFA InstituteCFA Institute awards the CFA Charter to candidates who successfullycomplete 3 levels of testing and meet other education and work-related criteria.In 2010, approximately 200,000 candidates registered for CFA Examsadministered in 89 countries across the globe. Exams areadministered twice a year.All three levels of the CFA Exam are paper and pencil.CFA Institute also gives the CIPM designation and exam, which is CBT.

Country Risk AssessmentTwo Different Assessments: Focus on Legal, Regulatory & Tax Focus on ability to protect rights and intellectual property (including test items) in a country. CFA Institute does this prior to opening a test center in a new country.

Legal, Regulatory & Tax What governmental agencies, if any, must we interact with in order to administer our tests in the way our program is designed? What Taxes do we owe based on our program?- keep in mind other activities in the country- inquiries on a “no name” basis

Assessment to Open a New Test CenterPractice established in 2009Multi-part assessmentGoal is to examine risks & exposures unique to each country

Demographics and Political Landscape Hi, we’re new here! Understand the country to which you are going Social and Economic history – informs how you may interact with regulators, staff, and candidates Propensity for unrest and how it generally unfolds.

Protection of Intellectual Property Why is our logo on that guy’s test? To which IP conventions is the country a signatory? What protections are in place on a national basis? How do you register your trademark? Is it a first to file system? Automated or manual? Are service marks recognized? Is there institutional capacity to address IP issues? Judicial resources and corruption should be considered.

Corruption & Crime Are you gonna lose your shirt and how will you get it back when you do? Transparency International Corruption Index – also consider if the areas of corruption are related to your business. Anti-corruption legislation and enforcement – is it robust? Violent street crime in area of your test – does it endanger candidates and staff? Do the police have the ability and desire to be helpful? Is this a location in which you would want to involve the police?

Ability to Investigate What can you do when you lose your stuff? Is it safe for your home office staff to travel to the location for investigation? Is a visa required? How long to get it? If home office staff cannot go – what is your plan? Outside security vendor? Do you have one identified?

First-Hand Experience of other Organizations Hey, did you guys lose your shirt? Reach out to other organizations that test in the location you are evaluating. Requires building relationships with other organizations. This can be invaluable. You compare what your research reveals to real experiences.

Customs Hey, our stuff doesn’t seem to be getting to our testing staff! Understand how to work with the customs officials. U.S. Department of Commerce can advise and may intervene if necessary – cultivate this relationship. International Trade Office (part of Dept. of Commerce). Talk to your delivery agent – what is their experience in that country? Know your timelines.

Current Test Centers  Do we really need a test center here?  Proximity of nearest centers in other countries?  Ease of travel for candidates to adjacent countries – understand that not everyone can go everywhere.  Does expense of travel keep potential examinees out of your program?

Testing Partners and Locations So you decided to take the plunge – with whom are you swimming and in which city? Is your testing partner in the country? If not, who is? Which city (or locations) are best for our test?

Questions? Thanks! steve.addicott@caveon.com tom.gibbs@shlgroup.com aimee.rhodes@cfainstitute.org

Caveon Webinar - International Testing 11-15-2011

by caveonweb on Nov 17, 2011

Accessibility

Tags

Upload Details

Usage Rights

2 Embeds 783

Statistics

Caveon Webinar - International Testing 11-15-2011 — Presentation Transcript

http://www.caveon.com	781
http://www.linkedin.com	2