Presentation of EvalCourse in ISELEAR 2013, Salamanca, Spain.

1. A generative computer
language to customize
online learning
assessment
Antonio Balderas, Iván Ruíz-Rube, Juan M. Dodero, Manuel Palomo-Duarte
A generative computer language to customize online learning assessment

2. Table of contents
» Introduction
» EvalCourse
» Case study
» Conclusions and future research
A generative computer language to customize online learning assessment

3. A generative computer language to customize online learning assessment
Introduction
– Importance of competences in Education:
• Subject specific (knowledge)
• Generic (abilities, capacities and knowledge that any student
should develop regardless of his/her subject area)
– Assessing generic skills could not be a scalable task:
• We hardly have enough time to meet planning
• More difficult as the number of activities to evaluate increases

4. A generative computer language to customize online learning assessment
Introduction
– The scope of our work is related to LMS (Learning
Management System)
– Each file, access or assignment done by a student is
registered in the system
– This collected information would be used as objective
indicators of students' proficiency
– Indicators are automatically-obtained statements that help
determine if the competence level is met

5. A generative computer language to customize online learning assessment
Introduction
»We have developed a Domain-Specific
Language to customize online learning
assessments in Moodle: EvalCourse
» Using EvalCourse lecturers can easily
retrieve indicators from information stored
in the Moodle activity logs without any
technical knowledge on databases or
computer programming

6. A generative computer language to customize online learning assessment
EvalCourse
» EvalCourse is a plugin for the Eclipse IDE
that processes queries in a domain-specific
language that we have developed
to get indicators

7. A generative computer language to customize online learning assessment
EvalCourse

8. A generative computer language to customize online learning assessment
EvalCourse
» Reserved words and formats
Evidence name_of_the_indicator:
get students
show milestones | participation| access | interaction
in assignment | forum | campus | workshop [list of ids]
[between YYYY-MM-DD and YYYY-MM-DD].

9. EvalCourse
» Example of use:
– Students have been submitting several tasks during the course
– We are going to asses their performance in the competence of
planning and time managment from the number of tasks that
have been uploaded in time
Evidence Milestones_list:
get students
show milestones
in assignment.
A generative computer language to customize online learning assessment

10. A generative computer language to customize online learning assessment
EvalCourse
» List of quantitative information
Id Username Total In-time Overdue In-Progress In-time rate
1 John 5 3 2 0 60
2 Mary 5 1 0 4 20
3 Jane 5 5 0 0 100
4 Michael 5 3 2 0 60
5 Walter 5 4 1 0 80

11. A generative computer language to customize online learning assessment
EvalCourse
» List of quantitative information
Id Username Total In-time Overdue In-Progress In-time rate
1 John 5 3 2 0 60
2 Mary 5 1 0 4 20
3 Jane 5 5 0 0 100
4 Michael 5 3 2 0 60
5 Walter 5 4 1 0 80

12. A generative computer language to customize online learning assessment
EvalCourse
» A group of graphs that gives an overview
of the results that can support the
interpretation of the data

13. A generative computer language to customize online learning assessment
EvalCourse
» A Pentaho transformation file

14. A generative computer language to customize online learning assessment
Case study
» The case study we analyze was developed
in University of Cadiz, Spain
» In a compulsory course on Language
Processors II of its degree on Computer
Science
» 36 students enrolled the course in 2012/13
academic year
» The course evaluation was done manually
and we later applied EvalCourse

15. A generative computer language to customize online learning assessment
Case study
The competences we wanted to assess
were:
» Planning and time management
» Teamwork
» Interpersonal skills
» Leadership

16. A generative computer language to customize online learning assessment
Case study
Extraction of indicators from milestones results
– Students had to work in small teams
– Every team had to carry out five milestones during the course
– Every milestone had a deadline
– For each team, only one member had to upload each activity

17. A generative computer language to customize online learning assessment
Case study
Extraction of indicators from milestones results
– Planning and time management
• All their milestones submitted in time
– Teamwork
• Five activities submitted exactly

18. Case study
Extraction of indicators from milestones results
Evidence PlanTeam:
get students
show milestones
in assignment.
A generative computer language to customize online learning assessment

19. A generative computer language to customize online learning assessment
Case study
Extraction of indicators from forum results
– Students had to work in small teams
– Each team in the course had a forum for internal
communication
– As the indicator of Interpersonal skills of each student we
calculate the total forum participation
– As the indicator of Leadership we took into account the amount
of debates that each student started

20. Case study
Extraction of indicators from forum results
Evidence forum_participation:
get students
show participation
in forum.
A generative computer language to customize online learning assessment

21. A generative computer language to customize online learning assessment
Case study
– The grades that the supervisor manually calculated for every
student were closed to the indicators:
• The best works were also those who performed better on planning
and time management and teamwork skills
• Few students who participated in the forum, but students with
good marks in both indicators really stood out on the competences

22. Conclusions and future research
– EvalCourse: a generative computer language to customize
online learning assessments
– Lecturers were allowed to retrieve desired indicators from
information in the LMS using a very simple syntax
– In the case study, we assessed students' performance in the
competences of Leadership, interpersonal skills, Planning and
Teamwork
A generative computer language to customize online learning assessment

23. Conclusions and future research
– Results are promising
– The interpretation that we did about indicators was closely
related to the results that the lecturer noticed during the course
– Further study are needed to get a strong conclusion on the
validity of the specific usage of the information retrieved by
EvalCourse
A generative computer language to customize online learning assessment

24. Conclusions and future research
We have two future lines of work ahead:
– Identify others indicators stored in LMS that can be used to
assess competences
– Expand our generative computer language to be used in others
LMS
A generative computer language to customize online learning assessment

25. A generative computer
language to customize
online learning
assessment
http://assembla.com/spaces/evalcourse
Thank you very much for your attention.
Questions? Comments?
A generative computer language to customize online learning assessment
Acknowledgments: