2. Outline 01 PDF Paperless Testing‣ Computer-based testing‣ YACBTS - Yet another CBT “system”‣ Characteristics of a P2T document‣ Anatomy of a PDF‣ The process of creating a P2T document‣ Usability, security and other amenities‣ The Adobe in-house smack-down (Adobe Acrobat vs Adobe Reader vs LiveCycle RE)‣ Let’s start our business (i.e., let’s talk about money)‣ A “CBT” in your pocket‣ P2T and adaptive testing: preliminary reflections‣ Three final questions
3. Computer-based testing 02 PDF Paperless TestingA Computer-Based Testing (CBT) is a method of administering tests bymeans of electronic devices. Advantages of this method over traditionalpaper&pencil testing (PPT) have been exhaustively demonstrated in theliterature. We will not illustrate them here, as we believe that you know themwell.Over the years, the Italian Air Force Selection Center has acquired (andused) several CBT systems, accumulating a fairly reasonable experience onthe matter. Generally speaking, we are pretty satisfied with them, but havealso found some undesirable drawbacks: ‣ installation (or update) of CBTs isn’t always a breeze; ‣ license plans may be pricey; ‣ while possible, it isn’t easy to incorporate proprietary material (e.g., old paper&pencil tests) into CBTs.
4. YACBTS - Yet another CBT “system” 03 PDF Paperless TestingWith the intention of adding a new tool to the ones already in use, our teamstrived to develop a CBT system that could address these drawbacks.We were not interested in creating a full-fledged system, rather finding atechnique for rapidly authoring and delivering psychological tests/surveys.The basic assumptions were that such CBT system should be: ‣ cross-platform and not dependent from complex infrastructures; ‣ free from cumbersome installation/update procedures; ‣ easy to manage/set up; ‣ cheap.We focused our attention on the PDF format and its capabilities ofinteracting “intelligently” with the user.Following a Aha Erlebnis, we realized that we could make a PDF behave asif it was a tiny little CBT system. This was the beginning of the PDFPaperless Testing project, a.k.a. P2T.
5. YACBTS - Yet another CBT “system” 04 PDF Paperless TestingQ How can a PDF document (P2T) be a tiny little CBT system?A If properly authored, a PDF document can: ‣ behave in specific ways as events happen (e.g., document is opened/closed, page xx is selected or not and so on); ‣ accept user inputs through the use of form elements (e.g., text- fields, check-boxes, list-boxes, push-buttons and so on); ‣ perform a wide array of calculations.Q How can a PDF document (P2T) do these fancy things?A By means of some programming instructions that resides in the PDF itself. In other words, everything is inside the PDF document. There is no need of installing softwares.
6. Characteristics of a P2T document 05 PDF Paperless TestingA super-duper P2T document can: ‣ perform a login procedure by checking test-taker personal information; ‣ track time and show to the test-taker a countdown watch; ‣ code test-taker’s answers; ‣ calculate raw and standardized scores; ‣ present a chart of the results; ‣ store test-taker’s data.We have started our P2T project with fixed length testsand surveys, but we think that it could be possible tocreate adaptive tests (more on this later).
10. Anatomy of a PDF 09 PDF Paperless TestingQ What is XML?A Extensible Markup Language (XML) is a set of rules for encoding chunks of data in machine-readable form. An XML document is just a text file that stores data in a way that information is made inherently meaningful. The information is organized in a tree-like structure and can be easily retrieved by accessing its nodes (or meaningful tags).Q Do I have to change the XML code each time I make a P2T?A Yes, but it’s an easy task. We used XML to deal with test-takers information (e.g., name, surname, gender, etc.) and test specifications (i.e., test duration, number of items, keys, etc.). XML code doesn’t entail advanced programming skills.
12. The process of creating a P2T Test 11 PDF Paperless TestingQ Now that I have my vitaminized PDF what am I supposed to to?A Well, just put it in a computer and run it!Q What kind of software will run my vitaminized PDF?A There are two choices. Test-takers may use Acrobat Standard (full capabilities, but is not free) or Acrobat Reader (some limitations, but it is free). Technical Note: If the test-takers use Acrobat Reader, our vitaminized PDF has to be given “extended capabilities” and can be run from no more than 500 workstations (but without limits on the number of times the document is filled in).
13. Usability, security and other amenities 12 PDF Paperless Testing PDF Paperless Testing Traditional CBTTest creationTest session setupUsability Apple Linux WindowsData securityData integrityData Handling
14. The Adobe in-house smack-down 13 PDF Paperless TestingOur P2T document relies on special functions that are fully operational if thetest taker uses Acrobat Standard and somewhat limited with the freeversion of Acrobat Reader. The Reader’s limitations can be circumvented.Let’s see three different scenarios. 1 2 3 Test-takers use Test-takers use Test-takers use Acrobat Acrobat Acrobat Reader, and Reader, and Standard (there the P2T the P2T is no need of document has document has “Reader been “Reader been “Reader enabling” the Enabled” using Enabled” using P2T document) Acrobat Pro LiveCycle RE (server-based) affordability affordability affordability functionalities functionalities functionalities
15. Let’s start our business 14 PDF Paperless TestingWe can hypothesize two options based on a medium-sized PC Lab (40computers). The first one stems from the necessity of investing theminimum amount of money. Everything will work (with few limitations).The second one is the best compromise (in our opinion) between costsand functionalities. Test Development Test Administration Costs 1 Nr. 1 License Adobe Acrobat Pro + Nr. 40 Acrobat Reader (free) ~500 euro Unlimited number of deployable tests Check prices with your local Adobe reseller on a maximum of 500 workstations 2 Nr. 1 License Adobe Acrobat Pro + Nr. 40 Licenses Acrobat Standard ~10.000 euro Unlimited number of deployable tests Check prices with your local Adobe reseller; inquire also for volume licensing plans
16. A “CBT” in your pocket 15 PDF Paperless Testing Since the P2T documents don’t require to be installed, they can 19 mm be carried (and deployed) anywhere. Should you be worried about security, the P2T documents may be stored on an IronKey drive (encrypted with high-speed military-grade hardware encryption). When an IronKey drive is plugged into a laptop or desktop computer, the user must authenticate with a password before75 mm data and applications are accessible. If a thief tries to break into an IronKey and enters 10 incorrect passwords, the Cryptochip will securely erase all of the stored information. This ensures no trace of your data can be recovered from the device. All encryption and password verification are performed in hardware, and cannot be disabled by worms, viruses or other 4Gb malware. 99$ Check prices with your national Iron Key reseller
19. Three final questions 18 PDF Paperless TestingQ Can you show us a real example?A Yes, sure.Q Are you an Adobe or Iron-Key reseller?A Nope.Q Can you disclose your code to us?A We will be happy to do so (at no costs). If you are interested in P2T, contact us before leaving. Maj Pierpaolo Calanna (firstname.lastname@example.org) Maj Andrea Mastrorilli (email@example.com) Italian Air Force Selection Centre