Caveon Webinar - International Testing 11-15-2011


Published on

Join Caveon clients CFA Institute and SHL as they share best practices for protecting your intellectual property internationally.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Caveon Webinar - International Testing 11-15-2011

  1. 1. International Testing: Crossing GlobalBorders While Protecting Your Own Steve Addicott Vice President, Caveon Test Security Tom Gibbs Senior Analyst, SHL Group Aimee Hobby Rhodes, JD Director of Exam Security, CFA Institute
  2. 2. Agenda Intros Security Threats to International Test Programs  Steve Addicott, Caveon Employment Testing: The Stakes are High, and They’re Global, Too  Tom Gibbs, SHL Group Assessing Country Risks  Aimee Hobby Rhodes, CFA Institute Q&A
  3. 3. Steve AddicottVice President, Caveon
  4. 4. About Caveon Test Security Starting 9th year Offering the “Three Ps”  Proven  Practical  Protection Certification/Licensure & Education Clientele  Pre-eminence  Testing globally We’ve seen a few things…
  5. 5. International Test Threats Proxy Test Taking Braindumps & “Rogue Review” Courses Nefarious Test Centers  See above  Unusual practices
  6. 6. Nefarious Test Centers NOT vendor-owned centers Example of Unusual Test Taking  Test Center Hours 6 am to 6pm  130 tests taken at odd hours, ie 2am  Candidates unable to schedule tests when Test Center is closed  WHAT IS GOING ON?  And, Why are these candidates scoring much higher than others?
  7. 7. Braindumps and Rogue Review Courses Test Prep Courses advertising & selling live test items Prevalence of stolen-answer key usage may exceed 20% How are items stolen?  Infiltrate a top vendor’s premium security, or  Bribe a remote test center?  Evidence points to unauthorized decryption of exam after downloading to test site servers
  8. 8. Proxies and “Gunmen”Proxy Test Takers: Make Big Money  Analysis uncovered a large proxy test taking service operating in at least six testing sites: USA, Bangladesh, Turkey, UAE, and an unknown country.  500 test instances @ $1000 per instance in 6 months Own and operate test centers Are branching out
  9. 9. A Gunman for Hire September 10, 20xx Dear sir: We can take XTest exam service for you now in china, please fill out the attached file and send to me. Well finish the exam within one week. The price now for XTest exam service is USD 1000, you can send half payment before exam, half payment after exam. Best regards, Mr. Zhao9 © Caveon Test Security
  10. 10. Gunman September 12, 20xx Hello: We only accept WESTERN UNION payment, you can send half before exam, half after exam. If you do not have a passport, you can send me a copy of your drivers ID, you can scan it and send to me. Please fill out the attached file and send to me when you want to start the XTest exam. Cheers. Mr. Zhao10 © Caveon Test Security
  11. 11. Gunman October 16, 20xx My answer for you question: 1. Testing ID,when we register XTest for you, the test system will automatically give you the ID. 2. After exam, XTest will send you the XTest certification directly from USA. 3. When we finish XTest for you, you can check exam result by XTest official site,which can verify your exam result. For the payment, you can send half before exam, half after we finish XTest for you, XTest will send you certification within 4-8 weeks. When you want to start the exam, please let me know. Cheers. Mr.Zhao11 © Caveon Test Security
  12. 12. Gunman October 27, 20xx Hello: Our X test centre is being checked until now, Im afraid we should wait for a little more time to start the XTest. Do you need YTest or ZTest exam service? I can prepare for you next week. Please let me know your decision, thanks. Best regards. Mr. Zhao12 © Caveon Test Security
  13. 13. Gunman November 10, 2007 The payment is OK, thanks. Now I send you XTest info: [Link to Site] Test ID* or XTest Certification ID: XTest11327691 Password: 123456 ***Please keep your personal information updated always. Youd better login the XTest site each week. You can modify the password now. If you or your friend need YTest, ZTest exam in the future, you can contact me, there are XTest certification status in the XTest site, please pay attention to it. Contact me if you have any question. Best regards, Mr.Zhao13 © Caveon Test Security
  14. 14. Tom GibbsSenior Analyst, SHL
  15. 15. Overview of presentation The user model  It is 24/7/365 and its online Why the stakes are high  The war for talent and the current economic conditions Why the stakes are global  The war for talent is everywhere Are we managing the cheats and the pirates?  The value of web patrols and a clear security framework Summary and further information  We are publishing more about our security processes
  16. 16. The User Model
  17. 17. 24/7/365 The user model is increasingly not proctored and not delivered via group testing Delivered online with a variety of security features  Linear On-the Fly Testing (LOFT) and Computer Adaptive Testing (CAT)  Systems for managing item exposure and answer key exposure  Supported by a security framework including data forensics, web patrols and incident response procedures (see next slide) Initial version has been operating since 2002 Verify has been operating since 2006 Now available in 26 languages
  18. 18. Test security monitoring Security Breach Process •Collect Evidence Web Patrols •Report to Legal •Communications with Suspected suspected offender security breach Data Forensics Reporting •Run analysis on data specific(through clients/business/other) to test, date range and client of suspected breach No Continue Item Bank monitoring compromised? Yes Replace compromised items
  19. 19. Further Information...
  20. 20. Further Information...Burke, E. (2009). Preserving the integrity of onlinetesting. Industrial and Organizational Psychology, 2, 35-38.This is a special edition of I/O Psychology on this topicand can be accessed via
  21. 21. Further Information...
  22. 22. The stakes in the employment market haveincreased
  23. 23. Quotes from a well known graduaterecruitment portal“Being the honest idiot in a sea full off successful liarswill get you nowhere” “I put sweat, blood and tears into improving my CV and am absolutely sickened that lazy, lying cheating graduates are getting jobs they have done nothing to deserve”“Have to say that you shouldnt lie about what you canget caught out on but anything else is fair game. Reallife is about communication, persuasion andnegotiation”“Its important that you are able to describe in detailyour fake job just in case you are asked about it”
  24. 24. The war for talent is local and globalGeography is becoming less of a constraint
  25. 25. An applicant could be applying for a job located anywhere from anywhere
  26. 26. So, how do we know if we are managing the cheats and the pirates?
  27. 27. Web patrols are telling us that there is athreat but we can detect it and manage it
  28. 28. Web patrols are also telling us that theapplicants know there is security and that there are consequences
  29. 29. Internet discussion from April 12th 2010 ttsstt “With all the psychometric testing for a number of jobs i have applied for i am wondering how many people cheat and get help from family/friends while doing the test. From a few people ive spoken too I get the sense that it is MORE than common …Thanks guys” underthepump: “I didnt for two reasons, well two reasons on top of being honest :p … 1) They make you do a supervised one at ACs anyway … 2) Time limit. You dont exactly have time to sit there and come to a consensus and different people work at different paces. Personally having someone else sitting there would annoy me more than it would help” brendan58: “what underthepump said pretty much. they make you sit a supervised test in the AC and it would be quiet a shame to get rejected based on fluctuating test results … just need to have the confidence to complete them yourself” nebasil: “There is no point in cheating, how do you know the person doing the test for you will do better than you? Id rather trust myself than someone else” Huynhzie: “lol cheat now, ur going to get screwed in there. there was a PWC guy who went to his AC, he absolutely got owned at the verification test there and he was pointed and told off by the HR. HR was like "how can u perform absolutely so low", SHL is to test ur honesty as well, not just brilliance”
  30. 30. Cheating and piracy is a fact
  31. 31. It is global!
  32. 32. It requires constant vigilance
  33. 33. It requires innovation and commitment
  34. 34. Coming soon …Lievens, F., and Burke, E. (2011). Dealing with the threats inherent inunproctored Internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational andOrganizational Psychology (due for publication this year)First scientific journal article describing features designed to promotethe security of online high stakes testingBartram, D., and Burke, E. (In press). Industrial/OrganizationalTesting Case Studies. In Wollack, J. A. and Fremer, J. F. (Eds.).Handbook of Test Security. New York, NY: Routledge.This also includes a review of faking good on self-report (typicalperformance) measures such as personality questionnaires
  35. 35. Thank You
  36. 36. Aimee RhodesDirector of Examination Security CFA Institute
  37. 37. Due Diligence for International Testing CFA InstituteCFA Institute awards the CFA Charter to candidates who successfullycomplete 3 levels of testing and meet other education and work-related criteria.In 2010, approximately 200,000 candidates registered for CFA Examsadministered in 89 countries across the globe. Exams areadministered twice a year.All three levels of the CFA Exam are paper and pencil.CFA Institute also gives the CIPM designation and exam, which is CBT.
  38. 38. Country Risk AssessmentTwo Different Assessments: Focus on Legal, Regulatory & Tax Focus on ability to protect rights and intellectual property (including test items) in a country. CFA Institute does this prior to opening a test center in a new country.
  39. 39. Legal, Regulatory & Tax What governmental agencies, if any, must we interact with in order to administer our tests in the way our program is designed? What Taxes do we owe based on our program?- keep in mind other activities in the country- inquiries on a “no name” basis
  40. 40. Assessment to Open a New Test CenterPractice established in 2009Multi-part assessmentGoal is to examine risks & exposures unique to each country
  41. 41. Demographics and Political Landscape Hi, we’re new here! Understand the country to which you are going Social and Economic history – informs how you may interact with regulators, staff, and candidates Propensity for unrest and how it generally unfolds.
  42. 42. Protection of Intellectual Property Why is our logo on that guy’s test? To which IP conventions is the country a signatory? What protections are in place on a national basis? How do you register your trademark? Is it a first to file system? Automated or manual? Are service marks recognized? Is there institutional capacity to address IP issues? Judicial resources and corruption should be considered.
  43. 43. Corruption & Crime Are you gonna lose your shirt and how will you get it back when you do? Transparency International Corruption Index – also consider if the areas of corruption are related to your business. Anti-corruption legislation and enforcement – is it robust? Violent street crime in area of your test – does it endanger candidates and staff? Do the police have the ability and desire to be helpful? Is this a location in which you would want to involve the police?
  44. 44. Ability to Investigate What can you do when you lose your stuff? Is it safe for your home office staff to travel to the location for investigation? Is a visa required? How long to get it? If home office staff cannot go – what is your plan? Outside security vendor? Do you have one identified?
  45. 45. First-Hand Experience of other Organizations Hey, did you guys lose your shirt? Reach out to other organizations that test in the location you are evaluating. Requires building relationships with other organizations. This can be invaluable. You compare what your research reveals to real experiences.
  46. 46. Customs Hey, our stuff doesn’t seem to be getting to our testing staff! Understand how to work with the customs officials. U.S. Department of Commerce can advise and may intervene if necessary – cultivate this relationship. International Trade Office (part of Dept. of Commerce). Talk to your delivery agent – what is their experience in that country? Know your timelines.
  47. 47. Current Test Centers  Do we really need a test center here?  Proximity of nearest centers in other countries?  Ease of travel for candidates to adjacent countries – understand that not everyone can go everywhere.  Does expense of travel keep potential examinees out of your program?
  48. 48. Testing Partners and Locations So you decided to take the plunge – with whom are you swimming and in which city? Is your testing partner in the country? If not, who is? Which city (or locations) are best for our test?
  49. 49. Questions? Thanks!