Using Standards to Create Best-of-Breed Assessment Solutions
•
1 like•497 views
Powerpoint presented by Mark Molenaar and Thomas Garrard at the E-ATP 2014 Budapest
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
Similar to Using Standards to Create Best-of-Breed Assessment Solutions
Similar to Using Standards to Create Best-of-Breed Assessment Solutions (20)
More from Cito
More from Cito (20)
Recently uploaded
Recently uploaded (20)
Using Standards to Create Best-of-Breed Assessment Solutions
- 1. Innovations In Testing.org Using Standards to Create Best-of-Breed Assessment Solutions Mark Molenaar, Cito (Questify) Thomas Garrard, Open Assessment Technologies (TAO)
- 2. Innovations In Testing.org About Us…
- 3. Innovations In Testing.org About Standards… Source: www.xkcd.com
- 4. Innovations In Testing.org Standards we love… ▪ Storage: SCSI, S-ATA ▪ Peripherals: PCI, USB ▪ Networking: Ethernet, TCP/IP, 802.11x ▪ Web browsing: HTML5 ▪ Music: MP3, WAV ▪ Video: MPEG-4, H.264 ▪ Documents: PDF, ODF 10/2/2014 4
- 5. Innovations In Testing.org Powerful Impact ▪ 12th August 1981 ▪ IBM released the IBM PC 10/2/2014 5
- 6. ▪ Disaggregation of Hardware and Software resulted Innovations In Testing.org Consequences in phenomenal growth ▪ Trend continues today ▪ $267B in 2011 (Gartner) ▪ Will the assessment industry follow a similar path towards disaggregation? 10/2/2014 6
- 7. Innovations In Testing.org Consequences? ▪ In the assessment industry ▪ Separating the assessment content from the authoring and delivering platforms ▪ Enabling cooperation between different platforms’ functionalities ▪ The ball is beginning to roll… ▪ Todays industry remains highly vertical ▪ Shift towards horizontals currently slow and often ad hoc ▪ Game changing technology is within our grasp 10/2/2014 7
- 8. Innovations In Testing.org Standards we chose… ▪ Learning Tools Interoperability (LTI) ▪ Tailor your system by seamless tool integration ▪ Question and Test Interoperability (QTI) ▪ Keep you tests and items as you move ▪ Others worth mentioning… ▪ DEP (Dutch Exam Profile) ▪ QTI profile ▪ LIS (Learning Information Services) ▪ Including Outcomes ▪ APIP (QTI profile) ▪ Including Portable Custom Interaction ▪ TinCan (Experience API) ▪ By ADL (SCORM) ▪ And many, many more… Published by IMS Global Learning Consortium http://www.imsglobal.org
- 9. Innovations In Testing.org Where do they fit in? QTI
- 10. Innovations In Testing.org And in our case… LTI: Take an exam from a training course LTI: Take an exam from a training course LTI: Launch a TAO test from Questify QTI: Move quizzes to professional assessment tools QTI: use your items in various assessment contexts
- 11. Innovations In Testing.org Learning Tools Interoperability ▪ For integration of learning modules ▪ Instead of exchanging data ▪ Quick development & adoption by industry ▪ LMS like Moodle, Sakai, etc. ▪ Tools like Khan Academy, ▪ Popular versions ▪ 1.0 (May 2010): a.k.a. Basic LTI ▪ 1.1 (March 2012): pass back outcome ▪ 1.1.1 (August 2012): mentor role ▪ 2.0 (february 2014): more extensible, advanced outcomes
- 12. Innovations In Testing.org LTI Basic Flow
- 13. Innovations In Testing.org So far for Theory… LTI: Take an exam from a training course LTI: Take an exam from a training course LTI: Launch a TAO test from Questify QTI: Move quizzes to professional assessment tools QTI: use your items in various assessment contexts
- 14. Innovations In Testing.org LTI Experience ▪ Designed to exchange services ▪ Does not regulate content offered by such services ▪ What if a resource such as instructional video is removed? ▪ LTI 1.1 limited result return [0..1] ▪ Does not provide context information ▪ LTI 2.0 includes new services to get more information about test-taker 10/2/2014 14
- 15. Innovations In Testing.org Question & Test Interoperability ▪ Exchanging item & test data ▪ Preventing vendor lock-in? ▪ Or integrating systems? ▪ Interoperability is key ▪ Therefore basic ▪ But extensions possible… ▪ Impact on interoperability? ▪ Popular versions ▪ 1.2 (2002): abstract ▪ 2.0 (2005): only item-level ▪ 2.1 (2006-2012): final(ly) ▪ 2.2: development, planned 2015 ▪ 3.0 (aQTI): integration with APIP, use HTML5
- 16. Innovations In Testing.org So far for Theory… LTI: Take an exam from a training course LTI: Take an exam from a training course LTI: Launch a TAO test from Questify QTI: Move quizzes to professional assessment tools QTI: use your items in various assessment contexts
- 17. Innovations In Testing.org QTI Experience ▪ Compliance - bar set very low ▪ Only 3 interactions required (out of 21) ▪ Choice / Extended text / Text entry ▪ Discretion given to vendors ▪ Rendering interpretation of interactions is left to vendors ▪ Lack of psychometric validation 10/2/2014 17
- 18. Innovations In Testing.org Conclusion ▪ QTI and LTI ▪ A tool for the job ▪ But not a magic bullet ▪ Use them ▪ Improve them ▪ Build upon them 10/2/2014 18
- 19. Questions & Discussion Innovations In Testing.org
Editor's Notes
- Division: Mark intro Tom Standards (general) Mark standard IMS & demo LTI Tom QTI demo & wrap-up
- Cito: based in Arnhem, 600 employees, subsidiaries in Germany & Turkey; key partners in USA, Japan O.A.T: Luxembourg based Cito investor, supporter
- Switch to Tom Standards tend to be open nowadys, but they can be proprietary... Such as blu-ray. And, there are often competing standards... Video had to fend off betamax and blu-ray beat HD-DVD. So, standards can be open or proprietary, they can compete with other standards... So what really makes a good standard? Well, usage of course. The more people using the standard, the more useful it is.
- In Computing, the purpose of standards is to create interoperability between components, systems and applications – be they from different vendors, organisations or teams. Interoperability has clear technological advantages, but also major economic benefits as it can reduce software development cost, acquisition cost, operating cost, as well as support and maintenance cost.
- The 12th of August 1981 was the day IBM released the first IBM PC… They opened the standard allowing anyone to make similar PCs This catapulted, what became known simply as the PC, to market domination – and IBM didn’t do badly out of it too… they’re the worlds largest consulting and services company in the world; or the third biggest software company… in fact they’re outright one of the largest companies in the world. They dominate because of a market they opened using standards… software… The companies trying to compete by staying proprietary fell by the wayside… Commodore, Atari, Acorn… Well, all except Apple who staunchly bucked the trend from their creation in 1984 until 2006 when even they switched to a IBM PC derived intel architecture…
- The disaggregation of hardware and software resulted in phenomenal growth… The software and related services sector’s real contribution to U.S. GDP exceeded $267 billion in 2011. Which was an annual growth rate of 14% compared to 2% for all industries combined The chart shows the Gross Value add of software & services within the UK which hit over 30 billion in 2012 But, I’m sure we already know the IT industry as a whole is very successful, but, will the assessment IT industry follow in the footsteps of the wider industry to achieve similar growth? Will we head towards disaggregation of platform and content?
- Separating the content and authoring platforms is a step that parallels the wider IT industry’s step in the early 80s. This step allows companies to focus on areas where they add value and really make a difference without having to incur the costs of developing every part of an end to end solution. And if you happen to add value at every stage then there are still cases where someone else offers a ‘critical feature’… it’s a very sad day when your whole platform is disregarded simply because of one missing feature… especially when you could improve the service elsewhere. Compliance to standards is just beginning to become popular, the industry remains fairly vertical, but there is a slow, often project by project adoption beginning to happen. But there is a reluctance to opening up and particularly to changing the revenue model – For example, we had a recent project where Open Source and Standards were must have requirements. The tech leaders in the organisation clearly saw the value of interoperability, but… once the legal & business department got involved the first question was “how do we lock this down and protect the IP” This slow adoption will ramp up very quickly when we move to the cloud… the elastic nature and lower costs of an infrastructure like Amazon AWS lend themselves exceedingly well to large scale assessment. Purely based on compute costs, we have been able to simulate 100,000 test takers for less than $600… that’s less than one cent per user… and it included the cost of simulating the user as well!
- Switch to Mark
- Switch to Tom – import a QTI package...
- Compliance bar is set rather low... QTI requires self certication, and does not require much of the standard to be implemented, in fact, you can get away with only 3 of the 21 specified interactions. This means that consumers of the standard need to be aware that platform they are utilising may or may not support a business critical feature. Even the very same feature may be different between vendors... For example, TAO’s own associate interaction is a ‘click and drop’. ie click once to select and then again to drop... Rather than the traditional drag and drop used by some other vendors. In fact all of our ‘drag and drops’ are actually ‘click and drops’ because, importantly, they are more accessible – requiring less motor skills. But, also because touch screen platorms use swipe gestures for other features. But, of course, this interpretation begs the question of whether an item can still be considered the same from a psychometric perspective.
- So… QTI and LTI are a tool… not a magic bullet - but as far as interoperability is concerned, they are the best tools we have. Use them – they will reduce costs of interoperability and ensure you are not ‘locked in’ Improve them by contacting ims members such as ourselves and making feature requests. Or… of course… contact ims themselves! Build upon them… prepare guidelines of what features must be implemented and how your psychometric requirements must be met. Then share that and try to get consensus.