Reflections from various Evaluations of ICT projects - Benita Williams
Reflectionsfrom variousEvaluations ofICT projectsBenita Williams, EvaluationConsultantFeedback Research &Analyticsbwilliams@feedbackra.co.za
Benefits of ICTs for education ICT skills important in the job market Options for conveying broad range of interactive content is enhanced It makes administering learning easier (under some circumstances) Assessment feedback can be used immediately to enhance learning. Allow for self-paced learning (under certain circumstances) Long term it can drive cost down Eliminates the total dependence on the teachers as only facilitator of learning
Context Lessons learnt from 7 project evaluations Learner Project: Maths content rolled out in a government funded school lab via the internet (after school, and two periods per week) Learner Project : Maths content rolled out in a government funded school lab via the internet blended with face to face tuition, and use of non-lab computers (After school) Learner Project : Maths content installed locally on laptop computers, and used by maths teachers in their class (During class) Learner Project : Maths and science content delivered via video (DVD sets / touchpad computers) after school Teacher Project: Laptops with remote maths software accessed via internet to enhance teacher knowledge Teacher Project: ECD practitioners trained to use computers to stimulate learners Leadership Course material rolled out via universities’ e- learning platforms
Possible Implementationmodels One to many e.g. the teacher or a learner uses the technology to convey content to class vs. One to one e.g. the teacher takes the learners to the computer lab to engage with some content Primary site of learning (used to deliver content) vs. Second site of learning (enrich or expand) Used to remediate vs. used to teach new content Group paced (All learners engage with similar content at the same time) vs. self paced (Each learner on their own trajectory)
Possible implementationoptions Typical infrastructure roll-out models Use what is in the school, e.g. use the GOL labs Supplement the ICT infrastructure - Use the lab as is, but add your own server / connectivity Provide ICT infrastructure in the school e.g. laptops, tablets Cost of hardware + software + training+ maintenance + cost of participation+ security
Reflection 1: Don’t assumeeveryone will respond the same Diffusion of innovations models
Reflection 2: Not only the“What”, but the “Who” Implementation team requires the following competence: Project design (Phases, link with other education activities) Project management (Meeting deadlines, reporting, accountability) Relationship management (Schools, District, Province, Donor) Logistics coordination (arranging transport, catering) Technical expertise (Set up and trouble shoot, product specific and more general ICT skills) Generic education expertise (How schools work, how schools in townships typically work) Content specific education expertise (E.g. Maths or science etc.) Contextual understanding (E.g. Township schools) Monitoring and evaluation (Using the information as it emerges)
Reflection 3: Don’t assume therewill be support for your project Arrange access: open the class and set up the computers so that valuable teaching time does not get lost Supervision of the learners: helping the learners with technical issues, helping the learners with content issues, watching them to make sure they cope and are not busy with other activities Learner follow-up: Communicating with the learners about when the sessions will run, follow up if learners are absent Staff follow-up: Making sure that who-ever was responsible for supervising a session/ arranging access is available, feedback of results, flagging of issues to follow up Communicating with the school stakeholders, project implementers, project managers, district officials (content specialists and e-learning specialists) Collecting attendance data / other monitoring data
Reflection 4: Expect interferenceand plan for contingencies Other projects (Donor funded, government sponsored) Exam preparations Labour action Theft of hardware Betterchance in the first two terms Strong buy-in necessary
Reflection 5: If you involveteachers… Will they be required to work extra hours? Manage expectations Will they be more comfortable with using the ICTs than their learners? What is in it for them? How can their practice / knowledge be enhanced? When you do training - Are you training them on what they need to know to use the ICTs (software knowledge, ICT troubleshooting skills, use of ICT as pedagogic tool, subject content knowledge)
Reflection 6: Down time - Thedownfall of any ICT programme Testsystem stability up front Plan for scheduled maintenance Have a back-up plan Have on the spot trouble shooting skills (is one person sufficient?) Have extra equipment
Reflection 7: Consider yourminimum criteria for participants Language skills ICT use levels Content mastery School functionality Geographic spread Commitment of individual participants Is it wise to exclusively rely on districts to select schools, and teachers to select learners for participation?
Reflection 8: A lot more than ICTonly You must provide the hardware and software and the maintenance Training of users on multiple competencies Relationship building and constant communication Motivate kids to participate Provide incentives Think of contextual factors like transport, and food
Reflection 9: Be clear about yourexit strategy - Up front Willthey keep the software or hardware? Who will pay for internet / maintenance What use do you expect to occur after the project – are participants equipped to do this?
Reflection 10: Ask the rightevaluation questionsDid it work? Other important questions Is the content relevant? (Content review) Is the content user friendly for the intended users (Heuristics Evaluation) Was it implemented at the requisite “dosage” level for it to possibly work? (Fidelity monitoring) Can it effect change? (Experimental design) At what cost (to participants and donors) (Cost analysis) Then only, can you start to answer: Did it work (Quasi- experimental design) Does it work better than “something else” (comparative analysis), or how does it work with “something else”