Test Evaluation Computerised Tests Overview June 2010 'Developments in Automated Testing' Garry Cleere Q-Validus Dublin
Overview 1. Background 2. Plan  3. Develop  4. Item Formats (samples) 5. Administer 6. Score & Report 7. Analyze 8. Commun...
Background <ul><li>1. Background </li></ul><ul><li>Significant Evolution in Automated Test Evaluation Systems (ATES) </li>...
Plan <ul><li>2. Plan </li></ul><ul><li>‘ From the ground up’ </li></ul><ul><li>Validity and quality </li></ul><ul><li>Expe...
Develop <ul><li>3. Test Development </li></ul><ul><li>New formality ; process: Item writers / scripting process  </li></ul...
Item Formats <ul><li>4. Test Item Formats </li></ul><ul><li>From the ‘humble MCQ’ … </li></ul><ul><li>Through to the Image...
Sample Item Formats
Test Administration <ul><li>5. Tests Administration </li></ul><ul><li>Candidate management </li></ul><ul><li>Self registra...
Score and Report <ul><li>6. Scoring and Reporting </li></ul><ul><li>Formal Test Blueprint </li></ul><ul><li>Pass / Fail re...
Psychometric Analysis <ul><li>7. Psychometric Analysis </li></ul><ul><li>'P' Value </li></ul><ul><li>Discrimination value ...
Communications <ul><li>8. Communications </li></ul><ul><li>Sensitivity / responsibility / validity? </li></ul><ul><li>Info...
Security <ul><li>9. Security </li></ul><ul><li>Increasing threats </li></ul><ul><li>Tests Stakes involved  </li></ul><ul><...
Accessibility <ul><li>10. Accessibility </li></ul><ul><li>The  BCS ( B ritish Computer Society)  consider the following  c...
Final Comments <ul><li>1. Final Comments </li></ul><ul><li>Bandwidth and delivery </li></ul><ul><li>In-applications </li><...
Garry Cleere Managing Director Q-Validus Dublin www.q-validus.com [email_address] Tel. 00 353 1 716 3740
Upcoming SlideShare
Loading in …5
×

QV Automated Testing Overview

300 views

Published on

Developments in Automated Testing

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
300
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

QV Automated Testing Overview

  1. 1. Test Evaluation Computerised Tests Overview June 2010 'Developments in Automated Testing' Garry Cleere Q-Validus Dublin
  2. 2. Overview 1. Background 2. Plan 3. Develop 4. Item Formats (samples) 5. Administer 6. Score & Report 7. Analyze 8. Communicate 9. Security 10. Accessibility 11. Final Comments
  3. 3. Background <ul><li>1. Background </li></ul><ul><li>Significant Evolution in Automated Test Evaluation Systems (ATES) </li></ul><ul><li>Changes overtime: platforms / construct / content / process enhancement </li></ul><ul><li>Impact of a range of new technologies on ATES </li></ul><ul><li>Implications of greater bandwidth for delivery and format </li></ul><ul><li>Configuration options </li></ul><ul><li>Impact of greater processing power </li></ul><ul><li>Impacts of Legislation, Laws and Guidelines </li></ul><ul><li>Increasing security concerns </li></ul>
  4. 4. Plan <ul><li>2. Plan </li></ul><ul><li>‘ From the ground up’ </li></ul><ul><li>Validity and quality </li></ul><ul><li>Expert Group (SME's) </li></ul><ul><li>Formal Syllabus Definition </li></ul><ul><li>Test Blueprinting </li></ul><ul><li>‘ Soft’ Characterisation? </li></ul><ul><li>CTT (Characterisation Test Template) </li></ul><ul><li>Linear / sequential </li></ul><ul><li>Item banking </li></ul><ul><li>Accessibility considerations </li></ul><ul><li>‘ Universal Design’ </li></ul>
  5. 5. Develop <ul><li>3. Test Development </li></ul><ul><li>New formality ; process: Item writers / scripting process </li></ul><ul><li>Building to a formal test blueprint </li></ul><ul><li>Item writing workshops </li></ul><ul><li>Item Development using studio tools </li></ul><ul><li>Graphics and designers </li></ul><ul><li>Readability considerations </li></ul><ul><li>Accessibility considerations </li></ul><ul><li>Test Banking </li></ul><ul><li>Posting to the ATS engine </li></ul><ul><li>Pilot testing, results analysis </li></ul><ul><li>Test form release </li></ul><ul><li>Ongoing validation / maintenance </li></ul>
  6. 6. Item Formats <ul><li>4. Test Item Formats </li></ul><ul><li>From the ‘humble MCQ’ … </li></ul><ul><li>Through to the Image supported MCQ </li></ul><ul><li>Further range of engaging theoretical formats: </li></ul><ul><li>- Drag n’ Drop; FITB; Matching; Re-ordering </li></ul><ul><li>Hotspot item formats </li></ul><ul><li>Simulations type items </li></ul><ul><li>Light simulation item formats </li></ul><ul><li>Rich simulation item formats </li></ul><ul><li>In-applications testing (providing a ‘real’ or authentic test experience) </li></ul><ul><li>High Face Validity </li></ul><ul><li>“ Form and Function” and item format congruency with Syllabus </li></ul>
  7. 7. Sample Item Formats
  8. 8. Test Administration <ul><li>5. Tests Administration </li></ul><ul><li>Candidate management </li></ul><ul><li>Self registration </li></ul><ul><li>Candidate group import facilities </li></ul><ul><li>Global programmes administration features </li></ul><ul><li>Different territories, countries, languages </li></ul><ul><li>Clocks, alarms, reminders … </li></ul><ul><li>Much enhanced breadth of reporting </li></ul><ul><li>New registrations management features </li></ul><ul><li>Automatic LMS imports </li></ul><ul><li>Integration with LMS </li></ul>
  9. 9. Score and Report <ul><li>6. Scoring and Reporting </li></ul><ul><li>Formal Test Blueprint </li></ul><ul><li>Pass / Fail reporting </li></ul><ul><li>Percentage reporting </li></ul><ul><li>Summative v. formative assessment </li></ul><ul><li>Diagnostic results </li></ul><ul><li>Final state comparison </li></ul><ul><li>Spelling accommodations </li></ul><ul><li>Primer tests / mock tests </li></ul><ul><li>Range of reports </li></ul><ul><li>Depth / detail in tests reports </li></ul><ul><li>Psychometric analysis </li></ul>
  10. 10. Psychometric Analysis <ul><li>7. Psychometric Analysis </li></ul><ul><li>'P' Value </li></ul><ul><li>Discrimination value </li></ul><ul><li>Point Bi-serial indicators </li></ul><ul><li>Measuring the item’s performance </li></ul><ul><li>Granular item analysis looking to the: ‘anticipated outcomes for borderline or just capable students’ </li></ul><ul><li>Cheating and analysis outputs </li></ul><ul><li>Test validity </li></ul><ul><li>Comparable / parallel test forms </li></ul><ul><li>Weakly versus strongly parallel test forms </li></ul>
  11. 11. Communications <ul><li>8. Communications </li></ul><ul><li>Sensitivity / responsibility / validity? </li></ul><ul><li>Information about the test </li></ul><ul><li>Impact of collateral materials </li></ul><ul><li>Branding and programme message </li></ul><ul><li>Value of Support materials </li></ul><ul><li>Preparation for the tests </li></ul><ul><li>Value of Tests primers, mock tests </li></ul><ul><li>Purpose of test instruments </li></ul><ul><li>Centre communications </li></ul><ul><li>Holistic experience </li></ul><ul><li>Centre Support </li></ul><ul><li>Valuable resources </li></ul>
  12. 12. Security <ul><li>9. Security </li></ul><ul><li>Increasing threats </li></ul><ul><li>Tests Stakes involved </li></ul><ul><li>High / moderate / low stakes </li></ul><ul><li>Appropriateness (Risk / Reputation) </li></ul><ul><li>‘ 50 ways to cheat’ - firm in the US who specialize </li></ul><ul><li>Tests for sale on the Internet </li></ul><ul><li>Learning from test analysis </li></ul><ul><li>Invigilation procedures </li></ul><ul><li>Login validation </li></ul><ul><li>Biometrics </li></ul><ul><li>Cameras </li></ul><ul><li>Range of security measures </li></ul>
  13. 13. Accessibility <ul><li>10. Accessibility </li></ul><ul><li>The BCS ( B ritish Computer Society) consider the following categories for AT ES accessibilty: </li></ul><ul><li>  </li></ul><ul><li>Sensory (hearing and visual ) ; </li></ul><ul><li>Cognitive (dyslexia and other specific learning difficulties); </li></ul><ul><li>Physical (mobility); </li></ul><ul><li>Behavioral (emotional or social). </li></ul><ul><li>New Drivers: Assistive technologies; Plain language ; Visual ; Hearing impaired. </li></ul>
  14. 14. Final Comments <ul><li>1. Final Comments </li></ul><ul><li>Bandwidth and delivery </li></ul><ul><li>In-applications </li></ul><ul><li>Imaginative item formats </li></ul><ul><li>Multi-lingual </li></ul><ul><li>Global Administration </li></ul><ul><li>Feature rich systems </li></ul><ul><li>Exciting Item formats </li></ul><ul><li>Process bound development </li></ul><ul><li>Continuing evolution </li></ul>
  15. 15. Garry Cleere Managing Director Q-Validus Dublin www.q-validus.com [email_address] Tel. 00 353 1 716 3740

×