The document discusses the history and evolution of e-assessment over the past 10 years. It notes that e-assessment began being widely used in the late 1990s and involves online testing using computer technology, mainly multiple choice questions. The types of e-assessment include summative high-stakes tests and formative assessments. Recent benefits of e-assessment include immediate feedback, assessment validity, and flexible administration. However, benefits are not automatic and poor design may achieve no benefits. The document also describes a case study of developing an e-assessment system for logic courses that provides automated scoring and feedback.

1. 10 Years of E-assessment & Math
M. Antonia Huertas
Computer Science Department
UOC
04/04/13 e-math 2013, UNED, Madrid 1

2. Introduction: E-assessment today
• The term ‘e-assessment’ began to be used widely from
the late 1990s
• Ofqual (Office of the Qualifications and Examinations
Regulator UK) defines e-assessment (2009) as:
“the use of electronic systems for the development,
operation and delivery of accredited qualification assessment
or the collection of performance evidence, which contributes
to the awarding of a unit or an accredited qualification”
http://www.ofqual.gov.uk/64.aspx
• E-assessments are tests which learners take onscreen
or online using computer technology. They are mainly
formed by multiple choice questions (MCQ)
04/04/13 e-math 2013, UNED, Madrid 2

3. Types of e-assessment in formal
education
• Summative, ‘high stakes’ assessments – examinations
where the results matter to the candidate and the
institution, and hence are both carefully regulated and
of public interest.
• Formative (or assessments for learning) – those
embedded within learning activities, are designed for
learners and teachers to monitor and plan their work.
• Technologies employed in these two settings may be
similar, approaches to risk and innovation are very
different.
04/04/13 e-math 2013, UNED, Madrid 3

4. Today use and benefits
• Benefits:
– Immediate feedback
– Assessment validity
– Space and time flexibility
– More efficient administration
• Not all benefits apply to all e-assessment use
• Poorly designed or implemented e-
assessment may achieve no benefits at all
• Benefits are not ‘automatic’
04/04/13 e-math 2013, UNED, Madrid 4

5. E-Learning technologies today
• Education is influenced by a lot of emerging
technologies
• http://c4lpt.co.uk/top-100-tools-2012/
• http://www.edtechmagazine.com/k12/article/
2012/11/6-hot-trends-educational-
technology-infographic
04/04/13 e-math 2013, UNED, Madrid 5

6. Features in e- assessment formats
• Question types: Short answers, Essay, Multiple Choice Questions (MCQ),
Multiple Response Questions (MRQ, Fill in the Blanks (FIB), Matchin,
Crossword puzzles.
04/04/13 e-math 2013, UNED, Madrid 6

7. Beyond multiple-choice questions
Automated scoring & feedback
• Automated scoring
– Mathematics marking
 Moodle + Wiris use in UOC experimental “Introduction of Maths for
Engineering”
– Programming languages & formal notations
– Short free text answer
• Automated feedback
– Intelligent Tutoring Systems (ITSs)
– Example for mathematics:
 ‘Carnegie Learning’s Cognitive Tutor’ at secondary and HE in USA
 ‘eFit’ for mathematics at lower secondary schools in Germany
04/04/13 e-math 2013, UNED, Madrid 7

8. Beyond multiple-choice questions
Adaptive assessment
• The system chooses which questions to ask next, based
on the candidate’s responses to previous questions.
(+) Better measure of student’s capabilities in a given time
(- ) Difficulty to ensure that the adaptivity functions correctly
• Technical methods of implementation commonly used
include dynamic item selection from a pool of items
based on statistical methods
04/04/13 e-math 2013, UNED, Madrid 8

9. Beyond multiple-choice questions
Simulations and serious gaming
• Used when students interaction with the real thing
is impractical (danger, too time consuming, too
expensive)
• Simulation technology is quite varied.
– Generic simulation engines (such as Expert Systems),
tailored for educational purpose
– But also computationally relatively simple software,
relying rather on rich and varied media
• Serious gaming refers to games with an educational
purpose. Rigorous assessment is a key area of
development in serious gaming
04/04/13 e-math 2013, UNED, Madrid 9

10. E-assessement generations
• We are in the Computer-based Testing Era:
– To improve the efficiency of testing procedures.
– E-assessment is centred on testing
• Generation 1 (1994-2002):
– Computerised conventional testing
– Automated scoring
• Generation 2 (2002-2012):
– Computerised adaptive testing
– MCQ & Moodle kind of systems
– Automated feedbak
04/04/13 e-math 2013, UNED, Madrid 10

11. Current and future e-Assessment strategies
Source : IPTS on the basis of Bennett, 2010; Bunderson, et al., 1989; Martin, 2008).
04/04/13 e-math 2013, UNED, Madrid 11

12. E-assessement generations
(the future)
• We are going to the Embedded Assessment Era
– Embedding assessment within the learning process
– Based on ‘Learning Analytics’ (Data analysis + beheavoural
tracking)
– Intelligent tutoring systems
– Automated feedback
• Generation 3:
– Continuous measurement to monitoring student’s achievement
trajectory
• Generation 4:
– Personalised feedback and tutoring
– Intelligent monitoring by means of knowledge engeeniering
• Requires technological advances
04/04/13 e-math 2013, UNED, Madrid 12

13. A Generation 2-3 case study
• Logic is an introductory course offered under the
Computer Science Degree programme at UOC.
• Looking for an e-assessment system to implement
formative e-assessment
• Design and implementation of an e-assessment
system for both knowledge and skill
• Design and implementation of a methodology for a
pilot in a real course in UOC
04/04/13 e-math 2013, UNED, Madrid 13

14. Architecture of the system
04/04/13 e-math 2013, UNED, Madrid 14

15. Formalization of the pilot
04/04/13 e-math 2013, UNED, Madrid 15

16. Formalization of the pilot
04/04/13 e-math 2013, UNED, Madrid 16

17. Results
• Formative continuous assessment is more correlated with
summative final assessment in the pilot group than in the
control group
• Helping to implement formative e-assessment, when skill
and knowledge acquisition have to be assessed
• From the statistical study it is not possible to state that the
student skills were improved (the ones evaluated in the
final examination), but they are not worsened although in
the pilot group students had a higher workload
• Teachers have the realistic chance to switch the previous
assessment model for a complete formative assessment
• Needing improvements: technologically and
methodologically.
04/04/13 e-math 2013, UNED, Madrid 17

18. My 10 years of e-assessment & Math
2002
• Logic
• MCQ tests
(automated scoring)
• Common test for all
students
2006
• Algebra
• Automated scoring
(Wiris)
• Individual test (n
kinds of exercises
with parameters)
2009
• Logic
• ITS for Learning Logic
• Automated feedback
step by step
excercises
2012
• Logic
• E-assessment system
(Moodle + ITS)
• Automated scoring
and feedback for
knowledge and skills
04/04/13 e-math 2013, UNED, Madrid 18

19. Obrigada
Thank you
Gracias
04/04/13 e-math 2013, UNED, Madrid 19

20. 04/04/13 e-math 2013, UNED, Madrid 20

21. 04/04/13 e-math 2013, UNED, Madrid 21

22. 04/04/13 e-math 2013, UNED, Madrid 22

23. 04/04/13 e-math 2013, UNED, Madrid 23

24. 04/04/13 e-math 2013, UNED, Madrid 24

25. 04/04/13 e-math 2013, UNED, Madrid 25

26. 04/04/13 e-math 2013, UNED, Madrid 26

27. 04/04/13 e-math 2013, UNED, Madrid 27