PDF Testing


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

PDF Testing

  1. 1. P2t PDF® Paperless TestingIAMPS 2011
  2. 2. Outline 01 PDF Paperless Testing‣ Computer-based testing‣ YACBTS - Yet another CBT “system”‣ Characteristics of a P2T document‣ Anatomy of a PDF‣ The process of creating a P2T document‣ Usability, security and other amenities‣ The Adobe in-house smack-down (Adobe Acrobat vs Adobe Reader vs LiveCycle RE)‣ Let’s start our business (i.e., let’s talk about money)‣ A “CBT” in your pocket‣ P2T and adaptive testing: preliminary reflections‣ Three final questions
  3. 3. Computer-based testing 02 PDF Paperless TestingA Computer-Based Testing (CBT) is a method of administering tests bymeans of electronic devices. Advantages of this method over traditionalpaper&pencil testing (PPT) have been exhaustively demonstrated in theliterature. We will not illustrate them here, as we believe that you know themwell.Over the years, the Italian Air Force Selection Center has acquired (andused) several CBT systems, accumulating a fairly reasonable experience onthe matter. Generally speaking, we are pretty satisfied with them, but havealso found some undesirable drawbacks: ‣ installation (or update) of CBTs isn’t always a breeze; ‣ license plans may be pricey; ‣ while possible, it isn’t easy to incorporate proprietary material (e.g., old paper&pencil tests) into CBTs.
  4. 4. YACBTS - Yet another CBT “system” 03 PDF Paperless TestingWith the intention of adding a new tool to the ones already in use, our teamstrived to develop a CBT system that could address these drawbacks.We were not interested in creating a full-fledged system, rather finding atechnique for rapidly authoring and delivering psychological tests/surveys.The basic assumptions were that such CBT system should be: ‣ cross-platform and not dependent from complex infrastructures; ‣ free from cumbersome installation/update procedures; ‣ easy to manage/set up; ‣ cheap.We focused our attention on the PDF format and its capabilities ofinteracting “intelligently” with the user.Following a Aha Erlebnis, we realized that we could make a PDF behave asif it was a tiny little CBT system. This was the beginning of the PDFPaperless Testing project, a.k.a. P2T.
  5. 5. YACBTS - Yet another CBT “system” 04 PDF Paperless TestingQ How can a PDF document (P2T) be a tiny little CBT system?A If properly authored, a PDF document can: ‣ behave in specific ways as events happen (e.g., document is opened/closed, page xx is selected or not and so on); ‣ accept user inputs through the use of form elements (e.g., text- fields, check-boxes, list-boxes, push-buttons and so on); ‣ perform a wide array of calculations.Q How can a PDF document (P2T) do these fancy things?A By means of some programming instructions that resides in the PDF itself. In other words, everything is inside the PDF document. There is no need of installing softwares.
  6. 6. Characteristics of a P2T document 05 PDF Paperless TestingA super-duper P2T document can: ‣ perform a login procedure by checking test-taker personal information; ‣ track time and show to the test-taker a countdown watch; ‣ code test-taker’s answers; ‣ calculate raw and standardized scores; ‣ present a chart of the results; ‣ store test-taker’s data.We have started our P2T project with fixed length testsand surveys, but we think that it could be possible tocreate adaptive tests (more on this later).
  7. 7. Anatomy of a PDF 06 PDF Paperless TestingPDFs may be conceived as a stratification of layers, each of which servesdifferent purposes. In developing our P2T documents, we utilized some ofthese layers in order to implement the basic functionalities of a CBT system. F Form layer; contains form elements (e.g., check-boxes, push-buttons, etc.) for accepting user inputs A Annotations layer; used to display (in combination with T) graphical representations of data T Typographic layer; contains graphics and texts (i.e., items texts) ...and behind the scenes... X XML layer; used to store data (e.g., test specifications, test-takers information) J JavaScript layer; contains the programming code that makes the PDF a P2T document This is where the magic happens!
  8. 8. Anatomy of a PDF 07 PDF Paperless TestingQ What is Javascript?A JavaScript is an object-oriented scripting language used to make documents (i.e., web-pages, pdf files) more interactive. JavaScript is a language that lets you make your pages interact with the user and respond to what he does. Acrobat Standard and Acrobat Reader are capable of understanding Javascript code without the need of installing additional softwares.Q Is Javascript some obscure and exoteric language?A Absolutely not. It is one of the scripting languages most used nowadays (e.g., interactive web pages), relatively easy yet powerful enough to accomplish complex tasks. We are sure that in your unit there are smart IT guys who know it pretty well.
  9. 9. Anatomy of a PDF 08 PDF Paperless TestingQ Do I have to change the Javascript code each time I make a P2T?A The answer is no. But it really depends on the ability of your computer experts to write reusable code. The code we implemented can be adapted to different test scenarios, with a minimum amount of changes (or no changes at all).Q Do P2T intelligent functions require a lot of code writing?A The answer is no. But it really depends on the ability of your computer experts to write optimized code. It took us approximately 1.000 lines of code to implement the basic functionalities of a CBT system (Examinee Registration and Scheduling, Test delivery, Post-Examination Processing). Such a figure, in terms of programming effort, is low. And..hey! we are not good programmers.
  10. 10. Anatomy of a PDF 09 PDF Paperless TestingQ What is XML?A Extensible Markup Language (XML) is a set of rules for encoding chunks of data in machine-readable form. An XML document is just a text file that stores data in a way that information is made inherently meaningful. The information is organized in a tree-like structure and can be easily retrieved by accessing its nodes (or meaningful tags).Q Do I have to change the XML code each time I make a P2T?A Yes, but it’s an easy task. We used XML to deal with test-takers information (e.g., name, surname, gender, etc.) and test specifications (i.e., test duration, number of items, keys, etc.). XML code doesn’t entail advanced programming skills.
  11. 11. The process of creating a P2T document 10 PDF Paperless TestingThe shrinkUsing programs such as Microsoft Word, the psychologistassembles a test (fixed length, multiple choice items in thescenario illustrated today) by putting items, graphics, multimediaand whatever he thinks the test/survey may deserve.He then saves the document as a PDF file.The geekUsing Adobe Acrobat Pro, the computer expert opens the PDFfile, places the interactive elements (e.g., check-boxes), puts theJavascript code, attaches two XML files: one relates to test-takers, the other to test specifications.He then hands back the vitaminized PDF (or P2T) file to thepsychologist who can deploy it.
  12. 12. The process of creating a P2T Test 11 PDF Paperless TestingQ Now that I have my vitaminized PDF what am I supposed to to?A Well, just put it in a computer and run it!Q What kind of software will run my vitaminized PDF?A There are two choices. Test-takers may use Acrobat Standard (full capabilities, but is not free) or Acrobat Reader (some limitations, but it is free). Technical Note: If the test-takers use Acrobat Reader, our vitaminized PDF has to be given “extended capabilities” and can be run from no more than 500 workstations (but without limits on the number of times the document is filled in).
  13. 13. Usability, security and other amenities 12 PDF Paperless Testing PDF Paperless Testing Traditional CBTTest creationTest session setupUsability Apple Linux WindowsData securityData integrityData Handling
  14. 14. The Adobe in-house smack-down 13 PDF Paperless TestingOur P2T document relies on special functions that are fully operational if thetest taker uses Acrobat Standard and somewhat limited with the freeversion of Acrobat Reader. The Reader’s limitations can be circumvented.Let’s see three different scenarios. 1 2 3 Test-takers use Test-takers use Test-takers use Acrobat Acrobat Acrobat Reader, and Reader, and Standard (there the P2T the P2T is no need of document has document has “Reader been “Reader been “Reader enabling” the Enabled” using Enabled” using P2T document) Acrobat Pro LiveCycle RE (server-based) affordability affordability affordability functionalities functionalities functionalities
  15. 15. Let’s start our business 14 PDF Paperless TestingWe can hypothesize two options based on a medium-sized PC Lab (40computers). The first one stems from the necessity of investing theminimum amount of money. Everything will work (with few limitations).The second one is the best compromise (in our opinion) between costsand functionalities. Test Development Test Administration Costs 1 Nr. 1 License Adobe Acrobat Pro + Nr. 40 Acrobat Reader (free) ~500 euro Unlimited number of deployable tests Check prices with your local Adobe reseller on a maximum of 500 workstations 2 Nr. 1 License Adobe Acrobat Pro + Nr. 40 Licenses Acrobat Standard ~10.000 euro Unlimited number of deployable tests Check prices with your local Adobe reseller; inquire also for volume licensing plans
  16. 16. A “CBT” in your pocket 15 PDF Paperless Testing Since the P2T documents don’t require to be installed, they can 19 mm be carried (and deployed) anywhere. Should you be worried about security, the P2T documents may be stored on an IronKey drive (encrypted with high-speed military-grade hardware encryption). When an IronKey drive is plugged into a laptop or desktop computer, the user must authenticate with a password before75 mm data and applications are accessible. If a thief tries to break into an IronKey and enters 10 incorrect passwords, the Cryptochip will securely erase all of the stored information. This ensures no trace of your data can be recovered from the device. All encryption and password verification are performed in hardware, and cannot be disabled by worms, viruses or other 4Gb malware. 99$ Check prices with your national Iron Key reseller
  17. 17. P2T and adaptive testing 16 PDF Paperless TestingSo far, we have talked about fixed length tests. Our P2T documents arecharacterized by a mixture of static and dynamic elements. The static partis represented by the fixed set of items the test-takers are asked to deal with.Entities such as check-boxes, list-boxes and push-buttons are the dynamicside of the experience. Finally, the javascript and the XML code make ourP2T documents self-administering and self-correcting.At this stage of the project, the Italian Air force didn’t explore the possibility ofmaking a step ahead with adaptive testing, but tried to collect somepreliminary information on the issue, finding that it is possible for a PDFdocument to [i] connect to an external datasource (i.e., item bank) and [ii]present items dynamically (i.e., with no static part at all).What we don’t know is whether it is feasible to implement computationally-intensive tasks (e.g., ability estimate) using Adobe’s products.Anyhow, the piece of software that creates completely dynamic PDFdocuments is called LiveCycle Designer (included in the Acrobat Pro Suite)and is based upon XFA technology (not covered here).
  18. 18. P2T and adaptive testing 17 PDF Paperless Testing Fixed length Tests/surveys Adaptive testingAuthoring software Adobe Acrobat Pro LiveCycle DesignerTechnology AcroForm, Javascript XFA, JavascriptSelf-contained testDynamic contentCross platform dep.Coding experience
  19. 19. Three final questions 18 PDF Paperless TestingQ Can you show us a real example?A Yes, sure.Q Are you an Adobe or Iron-Key reseller?A Nope.Q Can you disclose your code to us?A We will be happy to do so (at no costs). If you are interested in P2T, contact us before leaving. Maj Pierpaolo Calanna (p.calanna@gmail.com) Maj Andrea Mastrorilli (andrea.mastrorilli@aeronautica.difesa.it) Italian Air Force Selection Centre