Innovations in IT and how it can (and will…) impact assessment
1. Gaining advantage through assessment
Growing Talent
In Europe
What’s Next?
Innovations in IT and how it can (and will…)
impact assessment
2. Gaining advantage through assessment
Growing Talent
In Europe
Who Are We?
▪ Mark Molenaar
Chief Information Officer
Cito/Questify
▪ Jason Fagan
Software Developer
Trifork
9/25/2014 2
3. Gaining advantage through assessment
Growing Talent
In Europe
What Do We Do?
▪ Projects together
▪ CvTE – Facet
▪ IBE – Switzerland
▪ CBR – Dutch Driver’s License (Theory)
▪ Using (open) standards
▪ QTI in particular
9/25/2014 3
4. Growing Talent
In Europe
Gaining advantage through assessment
Facet (CvTE)
5. Growing Talent
In Europe
Gaining advantage through assessment
IBE (Zwitserland)
7. Gaining advantage through assessment
Growing Talent
In Europe
Recent Developments
▪ Moving away from paper based assessments
▪ First to computer, now to tablets
▪ 5 years ago the iPad did not exist!
▪ Launched in 2010
▪ Lot of apps developed, also in the area of
assessment
▪ Large target audience
9/25/2014 7
8. ▪ A natural evolution to BYOD and interactive testing
Gaining advantage through assessment
Growing Talent
In Europe
(R)Evolution?
Paper. Computer
Touch-
Screen
Tablet/Ph
ablet
Virtual-
Reality?
9/25/2014 8
9. Growing Talent
In Europe
What’s the Impact?
Gaining advantage through assessment
9/25/2014 9
10. Gaining advantage through assessment
Growing Talent
In Europe
Where’s the research?
▪ Pushing products?
▪ Research required
▪ What’s the impact?
▪ And is it intentional…
▪ Case sample: shift towards tablets
▪ Math Garden Research
▪ Short Comparison Study Tablet/PC delivery
9/25/2014 10
11. Gaining advantage through assessment
Growing Talent
In Europe
Math Garden Research on Tablet Use
▪ “a fun and stimulating
online environment in
which children can train
and develop their
mathematical skills
playfully at their own level”
▪ Research by
Matthieu Brinkhuis,
University of Amsterdam
9/25/2014 11
12. Growing Talent
In Europe
Gaining advantage through assessment
Results & Trends
9/25/2014 12
13. Gaining advantage through assessment
Growing Talent
In Europe
Comparison PC/Tablet – First results
▪ Short comparison study
▪ 1 school in primary education (PE)
▪ 25 candidates
▪ Average age: 11
▪ 2 schools in secondary education (SE)
▪ 62 candidates
▪ Average age: 13/14
▪ Each candidate 1 PC, 1 tablet test, 1 survey
▪ Consisting of 5 items and types
▪ Research by Saskia Wools, Cito
9/25/2014 13
15. Gaining advantage through assessment
Growing Talent
In Europe
Results: Device Usage
60
50
40
30
20
10
0
Totaal Computer Laptop Tablet
60
50
40
30
20
10
25
20
15
10
5
25
20
15
10
5
9/25/2014 15
Number of students
DEVICES @ HOME
(PRIMARY EDUCATION)
Owned Shared Total
0
Totaal Computer Laptop Tablet
Number of students
DEVICES @ HOME
(SECONDARY EDUCATION)
Owned Shared Total
0
Computer Laptop Tablet
Aantal leerlingen
DEVICE USAGE
(PRIMARY EDUCATION)
< once a week once a week few times a week every day
0
Computer Laptop Tablet
Aantal leerlingen
DEVICE USAGE
(SECONDARY EDUCATION)
< once a week once a week few times a week every day unknown
17. Gaining advantage through assessment
Growing Talent
In Europe
Results: Average Response Time
17
▪ Short answer item (multiple gaps):
9/25/2014
30000
25000
20000
15000
10000
5000
0
PE_PC PE_Tablet SE_PC SE_Tablet
Average response time in ms
18. Gaining advantage through assessment
Growing Talent
In Europe
First conclusions/confirmations
▪ Tablet-use is on the rise
▪ Children with tablets at home prefer tablets
▪ Could lead to Differential Item Functioning (DIF)
▪ Construction for tablet requires thought
▪ Keyboard popup, auto-complete, etc.
▪ Not all types are equal
▪ Hotspot/drag&drop preferred on tablet
▪ Short/long answer preferred on PC
▪ Response times can differ greatly
9/25/2014 18
19. Growing Talent
In Europe
So What’s Next?
Gaining advantage through assessment
9/25/2014 19
20. Gaining advantage through assessment
Growing Talent
In Europe
Current & Coming Developments
9/25/2014 20
21. Growing Talent
In Europe
Gaining advantage through assessment
New technology?
9/25/2014 21
22. Growing Talent
In Europe
Gaining advantage through assessment
Dangers?
9/25/2014 22
23. Growing Talent
In Europe
Gaining advantage through assessment
Possibilities!
9/25/2014 23
24. Growing Talent
In Europe
Gaining advantage through assessment
More possibilities…
25. Gaining advantage through assessment
Growing Talent
In Europe
Takeaways
▪ The future is coming fast…
▪ But there is still time…
▪ Use it to add value
▪ Not because it is cool…
▪ Not doing the same tricks differently…
▪ Further research is needed!
▪ Especially on comparison & equivalence
9/25/2014 25
26. Growing Talent
In Europe
Gaining advantage through assessment
9/25/2014 26
27. Gaining advantage through assessment
Questions & Discussion
Growing Talent
In Europe
What possibilities do you see?
28. Gaining advantage through assessment
Growing Talent
In Europe
Thank you very much for your attendance
▪ Come see us at our booths
▪ Contact:
▪ jason.fagan@trifork.nl
▪ mark.molenaar@cito.nl
▪ saskia.wools@cito.nl (PC/Tablet research)
▪ m.j.s.brinkhuis@uva.nl (MathGarden research)
▪ Twitter #EATP2014
Please leave your name or business card to receive the
presentation and stay informed
9/25/2014 28
Editor's Notes
DescriptionIT is driving innovation in many fields of industry, including assessment. But how do these innovations impact the processes we know? What is the real added value and when is it only distracting from original purpose? When does cutting edge become bleeding edge? In this session we will discuss the possibilities and pitfalls of innovations which could leverage mobility of talent.
We will discuss one example more extensively: tablets have become a common platform for many test drivers and can benefit (employee) mobility to great extent. But very little research has been done in comparing the results of “classical” Computer (PC/Mac) based Testing (CBT) and tablet-based assessment. Many comparison-studies have been performed on similar content using CBT and PPT (Paper & Pencil Testing) deliveries, but empirical data on tablets is lacking. During this session, the first results of a comparison-study will be presented and discussed.
Other innovations in mobility are already underway; augmented reality (Google Glass), next generation Virtual Reality (Oculus Rift, Sony Project Morpheus) and other wearable devices (smart watches like Pebble and more health-oriented devices) are on the horizon. What are the possibilities for these innovations in IT for employee mobility and what would their impact be? The session will conclude with an outlook on these developments and suggestions for further research.
Hand over to Jason
Switch to Jason
CvTE = College voor Toetsen en Examens, Exam board for nationwide exams
Dutch examination body.
Move away from paper-based exams to purely digital content with a view to adaptive testing.
Our first implementation using the new QTI Assessment Delivery Engine.
High and low stake exams.
Distributed examination system
Schools host the system.
Variety of computer hardware and operating systems.
Users take exams using a secure browser.
Institute for Educational Evaluation of the University of Zurich.
Eliminate paper-based exams
Provides teachers with the ability to easily create practice tests.
Runs on a wide range of devices, BYOD
High and low stake exams and practice exams.
Adaptive testing algorithms.
Dutch Driving License Agency
Multiple centres across the country.
Windows touch screen computers (no keyboard or mouse).
Our first 100% touch based implementation.
Switch to Mark
Mark: polish thi slide
Most have computer, some own tablet; everyone has computer
Laptop and tablet are used more frequently a week