MoToLi Score: An innovative approach to reading assessment
•Download as PPTX, PDF•
0 likes•415 views
In recent years numerous large-scale reading assessments have been performed in over 50 countries, mainly using EGRA (Early Grade Reading Assessment). The importance of such studies is significant and necessary to define policy, however their cost remains extremely high, mainly because they required well-trained enumerators. To address this issue, Et4d (Education Technology For Development) has developed a digital application - MoToLi Score-, which can assess reading acquisition and estimate reading fluency -number of words read per minute- using a multiple choice test. This method enables testing to be performed very efficiently and could be a valuable alternative to the currently used oral tests such as EGRA or ASER. MoToLi Score presents various very valuable strengths: - The test is very easy to administer, because it does not require a trained enumerator. The child can do the test on his own; therefore s/he is also less impressed or disturbed by the testing process. Even very low-level skills can be assessed, which is important to identify very early on whether the students are acquiring the reading skills according to the curriculum. - Testing can become part of a classroom routine. By using continuous assessment, the teacher can adapt his or her instruction to needs of all learners so all of them will have the chance to learn and succeed. - Results are immediate and easy to understand, a simple grading scale can be established. A statistical module is being developed to compare student and schools as well as student’s progression. If schools have access to internet, an online management system could be put in place, allowing data collection at a regional or national level.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
MoToLi Score: An innovative approach to reading assessment
- 1. An innovative approach to reading fluency assessment
- 2. What is MoToLi Score • A mobile application for assessment of reading fluency • Multiple choice questions based on audio as target word/letter
- 3. What is MoToLi Score
- 4. Results Measured • Answers: 1. Number/Percentage of correct answer 2. Number/Percentage of incorrect or skipped answer • Number of quizzes answered correctly in a minute
- 5. Correlation with Egra • To access the correlation of MoToli Score with EGRA a study was conducted in Nicaragua in November 2015. • The results were very good, and the application was adjusted to further improve the correlation.
- 6. Methodology • Team of assessors • Implementation context • Test design • Test procedure • Results measured • Population tested
- 7. Team of assessors Supervisor: Micaela Gomez trained by RTI had 6 years of EGRA experience. Assessors: Nidia Reyes and Mercedes Aguilar had 4 of EGRA experience. => Very qualified team.
- 8. Assessors Training Nidia and Mercedes received a 40 minute training to MoToLi Score.
- 9. Implementation Context The test was performed very poor community, in a school with low reading level
- 10. Implementation Context The school was very welcoming, the children behaved very well, and despite a very large number of children in a very small space, there were no visible discipline issues.
- 11. Test Design EGRA: The original version designed by RTI for Nicaragua in 2008, except that the upper case letters were removed. MoToLi Score: The letter test had 28 quizzes, and the word test 30 quizzes. The 30 target words were taken from the EGRA test, and most of the distractors.
- 12. Test Procedure 1. The assessor introduction 2. EGRA practice 3. EGRA is performed for letters and then for words 4. MoToLi Score practice 5. MoToLi Score for letters and then words
- 13. Test Procedure
- 14. Test Procedure Some students were tested without any assistance
- 15. EGRA Results Measured • Number of letters read correctly per minute • Number of words read correctly per minute • Percentage of letters read correctly • Percentage of words read correctly
- 16. MoToLi Results Measured • Number of letter quizzes answered correctly per minute • Number of word quizzes answered correctly per minute • Percentage of letter quizzes answered correctly • Percentage of word quizzes answered correctly
- 17. Population tested • Kindergarten: 12 students • 1st grade : 98 students • 2nd grade: 17 students
- 18. Results Letters Fluency Correlation ratio: 0.77 95% confidence interval [0.65 ,0.89 ] -20 0 20 40 60 80 100 0 5 10 15 20 25 30 35 Egralpm MoToLi qpm
- 19. Results Words Fluency Correlation ratio: 0.86 95% confidence interval [0.76 ,0.95] -20 0 20 40 60 80 100 120 0 5 10 15 20 25 30 35 Egrawpm MoToLi qpm
- 20. Results Letters Accuracy Correlation ratio: 0.84 95% confidence interval [0.74 ,0.95] 0.00 0.20 0.40 0.60 0.80 1.00 1.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20 Egra Motoli
- 21. Results Words Accuracy Correlation ratio: 0.86 95% confidence interval [0.76 ,1] -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20 Egra MoToLi
- 22. Issues identified during trial • Teacher taught letter’s sound and not the letter’s name • The quality of the audio was not perfect • There was a lot of noise outside
- 24. Issues identified during trial • Kids guess rather than skip • Non readers could manage with some luck to get a few correct answers using MoToLi • Very fast readers did not always try to answer fast on MoToLi Score’s quizzes
- 26. MoToLi Score - Improvements • The application stops in the following cases: – 5 consecutive incorrect answers – 4 times tapping on the same – 40 sec with no correct answer • Enable the use of upper case letters
- 27. MoToLi Score - New Features • Comprehension test • Data collection with synchronization of data with a server • A statistic module
- 28. Procedure Recommendations • Test the app with about 10 good readers • Sound’s volume as high as possible • Practice with 5 quizzes • Make students are aware that time is measured
- 29. Strength of MoToLi Score • Result not reliant on enumerator • Training of an enumerator less than 1 hour • Assessment can be performed by anybody Reduces cost of assessment • Easy to administer even in classroom setting • Curriculum based assessment Can be tailored for continuous assessment
- 30. Strength of MoToLi Score • Results can be synchronized immediately for analysis • Results include geo-localization More information available • Children favor the digital test versus the paper oral test • Digital test limits the stress factor Child friendly
- 31. Possible Use of MoToLi Score • In the classroom for continuous assessment, screening and diagnostic • Program evaluations, especially for monitoring of new methods • District or national reading assessment campaign
- 32. THANK YOU
Editor's Notes
- Motoli Score – An innovative approach to readin
- Motoli score is a mobile application used for the assessment of reading fluency. It uses multiple choice, the target is an audio file, and the student has to choose between 4 possible answer
- Let’s what how it works
- The result measured concern accuracy and speed. For the accuracy we measure the percentage of correct answer and for speed we measure the number of quizzed answered correctly in a minute, which is highly correlated with eth number of words read per minute.
- To access the correlation of MoToli Score with EGRA a study was conducted in Nicaragua in November 2015. The results were very good. The trial also enabled us to identify a few way to adjust the application in order to improve the correlation
- Methodology: I will present to you the methodology used for the test, the following aspects will be described in details: - Who were the assessors, what was their competency level? - In which context did we implement the test? - How was the test designed? - What was the test procedure? - Which results did we measure? - What population of children did we test?
- Team of assessors: Micaela Gomez was supervising and coordinating the assessment. Micaela was trained by RTI during the first EGRA implementation in Nicaragua in 2008. She has been training assessors and running EGRA evaluations for the past 6 years. Nidia Reyes and Mercedes Aguilar were performing the tests. They have both been EGRA enumerators for over 4 years. The team was very qualified to perform the test and to follow the EGRA procedure. We encountered no logistic issues.
- Assessors Training: Nidia and Mercedes received a 40 minute training to MoToLi Score. They were first taught how to use the tablet and the test as well as a few technical troubleshooting issues. After this short preparation, they simulated the test situation with each other. Let’s watch Mercedes and Nidia simulate a testing situation.
- Implementation Context: The test was performed in la Escuala Vera Cruz, a school located in Ciudad Sandino about 30 kilometers north of Managua. This very poor community was selected because of its low results for reading, which were diagnosed during previous EGRA evaluation.
- Test Design: EGRA: The EGRA version used was the original version designed by RTI for Nicaragua in 2008, except that the upper case letters were removed. MoToLi Score: The letter test had 28 quizzes, and the word test 30 quizzes. Each quiz entailed 1 target word (or letter) and 3 distractors. The 30 target words were taken from the EGRA test, and most of the distractor words were also part of the EGRA evaluation, but some distractors were chosen based on their similarities with the target words.
- Test Procedure: Each assessor performed EGRA and MoToLi for each student. The test procedure was: The assessor introduces herself and collects information related to the student (Name, sex, age, grade, class). The assessor explains what the test is about and why we are doing the test in order to limit the stress factor. The student does a short EGRA practice. EGRA is performed for letters and then for words. The student does a short MoToLi Score practice. The student performs MoToLi Score for letters and then words.
- Let’s watch a child performing MoToLi Score assessment
- Some students had to retake MoToLi Score because of technical issues or because of a high level of noise. They then took the test on their own without any assistance but with headphones. Let’s watch a child performing the test by him self
- For EGRA, we measured: Number of letters read correctly per minute using the letter name and not the sound as the correct answer. Number of words read correctly per minute. Percentage of letters read correctly. Percentage of words read correctly.
- For MoToLi, we measured: Number of letter quizzes answered correctly per minute Number of word quizzes answered correctly per minute. Percentage of letter quizzes answered correctly. Percentage of word quizzes answered correctly.
- Population tested: We mainly wanted to focus on 1st graders, who had just finished their first year of school, and who were therefore expected to be able to read. However we also included some students from kindergarten just with the letter evaluation, and some 2nd graders just with the word evaluation. Kindergarten: 12 students 1st grade : 98 students 2nd grade: 17 students
- Result Letters fluency: Correlation ratio: 0.77 95% confidence interval [0.65 ,0.89 ]
- Result: words fluency: Correlation ratio: 0.86 95% confidence interval [0.76 ,0.95]
- Result Letter accuracy: Correlation ratio: 0.84 95% confidence interval [0.74 ,0.95]
- Result words accuracy: Correlation ratio: 0.86 95% confidence interval [0.76 ,1]
- Issues identified during trial The teachers of the Escuela Vera Cruz did not teach the letter’s name but the letter’s sound, however the letter’s sound is considered wrong for the standard EGRA procedure. In one extreme case a girl read only 10 letters correctly per minute but 40 words per minute. However, since the student knew the letters, she had a good result with MoToLi Score, which seems more accurate than EGRA. The quality of the audio was not perfect, partially because the audio files were not recorded locally, and the accent must have been slightly different, only 1 student had a perfect score for letters. There was a lot of noise outside and, because we wanted to be able to observe the child answering, we did not always use headphones. Noise can induce inaccuracy of the evaluation.
- Let’s listen to the level of noise, which can make test conditions very difficult.
- Culturally it seems normal for the kids to guess rather than to skip or say that they don’t know. For instance one student while reading the EGRA words recited the name of all the animals she knew. Some students with very low reading skills (close to “0”) could manage with some luck to get a few correct answers using MoToLi. A little girl for instance always tapped on the same box. However, we have identified a few ways to limit this occurrence, which will be shared at the end of this report. At the other end of the spectrum, very fast readers did not attempt to answer fast on MoToLi Score’s quizzes, unless they were specifically asked to do so. Kids could therefore benefit from a practice test a little longer than the 3 word test which was used in this trial.
- Let’s watch how this little girl who does not know how to read a word manages to get a luck related score by tapping always on the same box.
- MoToLi Score – Improvements The application should stop if the students is a non-reader, just like EGRA stops if no letters/words are read correctly in the first line. We will make the application stop in the following cases: 5 consecutive incorrect answers 4 times tapping on the same box 40 sec with no correct answer A new version of MoToLi score will enable the use of upper case letters
- MoToLi Score - New Features We will add a comprehension test based on multiple choice. The data collection component will be improved to enable synchronization of data with a server. A statistic module will be added to enable a quick overview of the results for a given population.
- Procedure Recommendations We recommend testing the application with about 10 good readers, to make sure that they can hear the target words correctly. Volume of sound files should be as high as possible. The practice test should include 5 quizzes, in order to acquire the validation gesture. Make sure that the students are aware that they should try to answer as fast as they can.
- Strength of MoToLi Score Result not reliant on enumerator Training of an enumerator less than 1 hour Assessment can be performed by anybody => Reduces cost of assessment Easy to administer even in classroom setting Curriculum based assessment => Can be tailored for continuous assessment
- Strength of MoToLi Score Results can be synchronized immediately for analysis Results include geo-localization = >More information available Children favor the digital test versus the paper oral test Digital test limits the stress factor Child friendly
- Possible Use of MoToLi Score MoToLi Score could be used in the classroom for continuous assessment, screening and diagnostic. MoToLi Score can enable program evaluations, especially when new methods are being tested, and results are needed before a wide scale deployment. District or national reading assessment campaign could benefit from MoToLi Score’s easy data collection features.