Martin Schön Martin Ebner Life Long Learning Social Learning Graz University of Technology Graz University of Technology Mandellstraße 13, A-8010 Graz Münzgrabenstraße 35A/I, A-8010 Graz +433168734931 +433168738540 firstname.lastname@example.org email@example.comWhy micro testing is a necessity for teachingand learning.Learning analytics as teacher’s assistant.Microlearning 6.0 6th International Conference 2012, July 9th & 10th
ContentOr: this is a story,b) how we began to think about an educational problem (multiplication tables)c) how we tried to solve it with a small applicationd) and how this leaded us to Learning Analytics & Educational Data Mining
IdeaAt the Graz University of Technology some students of Martin Ebner hadto develop applications during their studies of information technology(Master of Science).Task: Simulate a diagnostic situation: A student is introduced to a teacher.He/she tries to gain an overview of the tasks which the students can doreliable well. This should be done in an optimal way.
Task• The application should test the competence in practicing the multiplication tables• The system should provide appropriate tasks according to the competence grade of the learner.• The system should ensure that already well-done exercises are repeated and practiced. After succeeding a problem the probability for a repeated display should decrease (similar to a Leitner System )
Checklist cont. 2• Nevertheless the tasks should tend to be challenging.• In general the system should be motivating and show that learning can be fun.• The system should record and safe fine-grained data of all done exercises, test results and the current competence grade of the learner in order to prepare the next sessions in an adequate way.• It should relieve the teachers from the unsolvable task of remembering all data from every child – it should be user as an information processing tool• That means the teachers should get help and no additional time consuming tasks
Checklist – cont. 3• during this spring we defined another goal: We wanted to give teachers the opportunity to set up and watch a whole class - Today we are able to present a database for organizing schools, classes and individuals – with an interface for the conventional desktop browsers and also for Android or iOS operating mobile devices.• the web-application is written in php/mysql –• the apps in objectiveC (iPhone/iPad) & JAVA (Android).
Measurement• the goal is to generate a complete table to inform learners as well as teachers in a deterministic way about the competences in every single task, in every single multiplication factThis is opposite to our experience in interviews:• Teachers talk about quantity: “I have 4 pupils who …. And 6 who … ….
Algorithm The presentation of the tasks is not only randomly generated.. At the very first contact a moderate problem is presented. According to the results more or less difficult tasks are following. The competence level of the test subject is afterwards estimated and after every task recalculated. This determines the difficulty of the next tasks.
Competence level1. estimate the competence level:2. After every new solved or unsolved problem we choose a smooth way for adopting this first estimation of competence to the experience during the sessions
Difficulty• Without any empirical data we decided to use the ranking for the tables as follows (You´ll find this hierarchy in most didactical concepts): easy …. 1, 2, 4, 3, 5, 8, 6, 7, 9 … difficult• we transformed these ranking in difficulty levels between 1 and 0.• We discussed to integrate a statistical founded ranking, but at this point we fear, the teachers would be somewhere confused about the results. I’ll speak about this problem later.
Stored Data• The answers of the learners are marked with 0,1 or 2: • 1 shows that the user knew the correct answer once • 2 indicates that the student had two consecutive correct answers (we say a question is “well known”) • 0 indicates, the last answer was incorrect (or this item was presented never before)Additional stored: a time line with needed time, results, task no., ever presented,
Selecting the next item We use a random number between 0 and 1 to decide, which category is activated to generate the next multiplication problem: Therefore three cases are defined:• Case 1: If the random number is 0<x<0.05 than a well-known question (marked with 2) is chosen.• Case 2: The random number is 0.05 > x >= 0.15 than a known question marked with 1 is chosen.• Case 3: The random number is x > 0.15 than an unknown question out of the extended and actual learning area is chosen.These parameters can be changed for the total system, not for Indivuduals
Extended Area• Extended area means the idea, that we choose items not only in the learning area, that means under the level of competence.• To produce some dynamic and the chance to get a higher level, we add now 0.15 to the actual estimation of the degree of competence (0 …1) of the student
Study• first research study was carried out at a primary school in Austria.• Begin: summer semester 2011• 43 pupils of the primary school Laubegg (age: 9-10).• at least 4 weeks. Some of the learners ignored this time restriction and played the game again and again over months.• Learners learned on computers at the school as well as on their personal computers at home.• 12.926 answers where given which means that on average each learner answered 308 questions- they did 3.4 times the whole multiplication table.• Bearing in mind that there was no real pressure from teacher’s side using the program it is a considerable pleasant high number.• Furthermore it can be stated that pupils seemed to enjoy using the application or at least get not bored.
Demotivated learnerOne learner with a weak performanceattracted our attention because of avery high number of trials (513). Adetailed inspection showed that he didnot work very intensively. In the firsttwo tasks he/she failed, then eleventasks were ok, his performance roseabruptly. However, afterwards, hecontinued approximately 400 times towait the whole answering timewithout doing anything but asking fora new assignment.
Motivated captainThe most diligent learner.In the beginning, theassignments were solvedcorrectly, then some mistakesoccurred, afterwards alearning process can berecognized and finally withsome occasional mistakes thelearner works on a highperformance level.Obviously, the learner washighly motivated to deal withthe assignments given by theprogram.
Medium learnerIn the beginning, the learner mademistakes in every secondassignment (0.5), followed by 7mistakes consecutively. This is thereason for the big decrease (0.15).Afterwards, the learner gave anumber of right answers and therate of correct answers increasedback to 0.5. In the following phasean up and down can be seen till anumber of right consecutiveanswers helps to reach a level of0.7. But then the number ofmistakes rose again and the ratewent down to about 0.5.
We don‘t know (now):More than half of the learners did not reach the 100 percent level (= 90items are well-known). Therefore we have to think about this group oflearners: Perhaps they• didn´t get used to / have problems with the interface• do not know the necessary operations; are not able to solve the learning problem correctly• misinterpret an assignment•are distracted by the environment, are badly concentrated for severalreasons.-> Could we gather more information with additional devices to knowmore?
Educational Data Mining• At this point, after all the preparations and programming, with the first incoming data we began to realize that we had taken a step into a whole new age.• We had begun to program and improve a simple cybernetic loop. At the beginning we saw just the necessity to store the performance data. Now we find us in the situation, that we want to collect “everything” – We want to store all tracks left by a learner. And we want even to produce more data, even in that we use additional instruments and sensors.• We realized that this idea of collecting more and more data implies completely new perspectives to perceive and to reconstruct some learning processes (not any!)
Learning AnalyticsLearning analytics is an educational approachaimed to improve education and facilitate learningby the systematic analysis and interpretation of digital tracks.Through the use of more and more intelligent ubiquitousinformation technology for communication and organization,more and more events are created for leaving digital traces.With the increasing data floods also the importance of thisparadigm increases.
Interim conclusions• It is as easy as never before in history to collect data• We perceive much more details about the learners – the learning process – as never before!!• Teachers get precise information• This application was designed for diagnostic purposes, to get an overview on problems with multiplication facts. Now we see, it can be effectively used for training and facilitate learning
Informationprocessing• The matrix at the left shows the history: The beige colour indicates “well known” results, the brown is “one right answer”. The rabbit comes from left to right to the carrot and catches one - if the pupil produces consecutive right answers.
DetailsIf you point over with the mouse on the matrix, you see the multiplication fact behind the symbol.
Register (it`s simple!)If you want to test this application you can use “noschool” for an individual registration.If you want to administer a whole school and acouple of class lists … firstname.lastname@example.org to send you anadministration-account.
AnalysisI´ll show here some graphs of the collected statstical data.We made the experience, that the teachers in our interviewsare interested in very short compact information, overviews.Therefore the following graphs are only for scientifc purposesWe know, that if we would produce too much information manyteachers will ignore the entire system ….
New findingsThis new applications are public available since March 2012.
Well - knownPerhaps someone is interested to get more information on thedifference of signing the tasks with „known“ and well known:
Results for teachersIf the analysis shows more than 12 correct answers during the last 20tasks the traffic light will show green, with less than 5 correct answers itis indicated red, otherwise the student and of course also teachers will seeyellow. In retrospection to the current situation this would lead only to5% red and yellow alerts.5% red and yellow alerts.
Future Work• closer look at the learners: perhaps more intelligent analyses of the data? – more data – more concepts? Hints for Learners?• Mathe-Multi-Trainer (multi digit multiplication)• Cooperative Learning• Test and Training in Reading
Acknowledgements We express our gratitude to the teachers of the primary school in Laubegg (Styria, Austria) as well as all participating school children. We are equally indebted to our funding agency “Internet Foundation Austria (IPA)” for supporting our ideas and helping us with netidee to work on the future of education.