This presentation was given at a meeting of the National Partnership for Education on 30 October 2012

1. Reflections
from various
Evaluations of
ICT projects
Benita Williams, Evaluation
Consultant
Feedback Research &
Analytics
bwilliams@feedbackra.co.za

2. Benefits of ICTs for education
 ICT skills important in the job market
 Options for conveying broad range of
interactive content is enhanced
 It makes administering learning easier (under
some circumstances)
 Assessment feedback can be used immediately
to enhance learning.
 Allow for self-paced learning (under certain
circumstances)
 Long term it can drive cost down
 Eliminates the total dependence on the
teachers as only facilitator of learning

3. Context
 Lessons learnt from 7 project evaluations
 Learner Project: Maths content rolled out in a government
funded school lab via the internet (after school, and two
periods per week)
 Learner Project : Maths content rolled out in a government
funded school lab via the internet blended with face to
face tuition, and use of non-lab computers (After school)
 Learner Project : Maths content installed locally on laptop
computers, and used by maths teachers in their class
(During class)
 Learner Project : Maths and science content delivered via
video (DVD sets / touchpad computers) after school
 Teacher Project: Laptops with remote maths software
accessed via internet to enhance teacher knowledge
 Teacher Project: ECD practitioners trained to use computers
to stimulate learners
 Leadership Course material rolled out via universities’ e-
learning platforms

4. Possible Implementation
models
 One to many e.g. the teacher or a learner uses
the technology to convey content to class vs.
One to one e.g. the teacher takes the learners
to the computer lab to engage with some
content
 Primary site of learning (used to deliver content)
vs. Second site of learning (enrich or expand)
 Used to remediate vs. used to teach new
content
 Group paced (All learners engage with similar
content at the same time) vs. self paced (Each
learner on their own trajectory)

5. Possible implementation
options
Typical infrastructure roll-out models
 Use what is in the school, e.g. use the GOL labs
 Supplement the ICT infrastructure - Use the lab
as is, but add your own server / connectivity
 Provide ICT infrastructure in the school e.g.
laptops, tablets
 Cost of hardware + software + training+
maintenance + cost of participation+ security

6. Reflection 1: Don’t assume
everyone will respond the same
 Diffusion of innovations models

7. Reflection 2: Not only the
“What”, but the “Who”
 Implementation team requires the following
competence:
 Project design (Phases, link with other education activities)
 Project management (Meeting deadlines, reporting,
accountability)
 Relationship management (Schools, District, Province, Donor)
 Logistics coordination (arranging transport, catering)
 Technical expertise (Set up and trouble shoot, product specific
and more general ICT skills)
 Generic education expertise (How schools work, how schools in
townships typically work)
 Content specific education expertise (E.g. Maths or science
etc.)
 Contextual understanding (E.g. Township schools)
 Monitoring and evaluation (Using the information as it emerges)

8. Reflection 3: Don’t assume there
will be support for your project
 Arrange access: open the class and set up the computers
so that valuable teaching time does not get lost
 Supervision of the learners: helping the learners with
technical issues, helping the learners with content issues,
watching them to make sure they cope and are not busy
with other activities
 Learner follow-up: Communicating with the learners about
when the sessions will run, follow up if learners are absent
 Staff follow-up: Making sure that who-ever was responsible
for supervising a session/ arranging access is available,
feedback of results, flagging of issues to follow up
 Communicating with the school stakeholders, project
implementers, project managers, district officials (content
specialists and e-learning specialists)
 Collecting attendance data / other monitoring data

9. Reflection 4: Expect interference
and plan for contingencies
 Other projects (Donor funded,
government sponsored)
 Exam preparations
 Labour action
 Theft of hardware
 Betterchance in the first two terms
 Strong buy-in necessary

10. Reflection 5: If you involve
teachers…
 Will they be required to work extra hours?
Manage expectations
 Will they be more comfortable with using the
ICTs than their learners?
 What is in it for them? How can their practice
/ knowledge be enhanced?
 When you do training - Are you training them
on what they need to know to use the ICTs
(software knowledge, ICT troubleshooting
skills, use of ICT as pedagogic tool, subject
content knowledge)

11. Reflection 6: Down time - The
downfall of any ICT programme
 Testsystem stability up front
 Plan for scheduled maintenance
 Have a back-up plan
 Have on the spot trouble shooting skills (is
one person sufficient?)
 Have extra equipment

12. Reflection 7: Consider your
minimum criteria for participants
 Language skills
 ICT use levels
 Content mastery
 School functionality
 Geographic spread
 Commitment of individual participants
 Is it wise to exclusively rely on districts to select
schools, and teachers to select learners for
participation?

13. Reflection 8: A lot more than ICT
only
 You must provide the hardware and software
and the maintenance
 Training of users on multiple competencies
 Relationship building and constant
communication
 Motivate kids to participate
 Provide incentives
 Think of contextual factors like transport, and
food

14. Reflection 9: Be clear about your
exit strategy - Up front
 Willthey keep the software or hardware?
 Who will pay for internet / maintenance
 What use do you expect to occur after
the project – are participants equipped to
do this?

15. Reflection 10: Ask the right
evaluation questions
Did it work?
 Other important questions
 Is the content relevant? (Content review)
 Is the content user friendly for the intended users
(Heuristics Evaluation)
 Was it implemented at the requisite “dosage” level for it
to possibly work? (Fidelity monitoring)
 Can it effect change? (Experimental design)
 At what cost (to participants and donors) (Cost analysis)
 Then only, can you start to answer: Did it work (Quasi-
experimental design)
 Does it work better than “something else” (comparative
analysis), or how does it work with “something else”