SlideShare a Scribd company logo
1 of 76
Download to read offline
1
By
One Global Economy
Moustafa Mourad
Alyssa Perez
Colin Richardson
and
Martine Holston
Final Report | October 2014
Digital Inclusion
Social Impact Evaluation
Acknowledgements
One Global Economy would like to express its gratitude to the Symantec Corporation
for its continued support for OGE and the grant that has allowed us to undertake this
evaluation in particular. For a number of years Symantec has supported OGE’s mission
of serving low income communities in many countries including India and Mexico,
and most recently, featured One Global Economy in its online brand campaign.
Table of Contents
Acknowledgements............................................................... i
Executive Summary...............................................................1
1. Background..................................................................... 3
1.1	 Goal and Objectives of the Impact Evaluation............................ 3
1.2	 The Information-Based Ecosystem Strategy.............................. 4
1.3	 Internet Access Points................................................... 5
1.4	Training................................................................. 6
Digital Literacy................................................................. 6
Entrepreneurship and Technology....................................... 6
Community Connectors..................................................7
Internet Safety...........................................................7
1.5	 Local Content.................................................................. 8
1.6	 Country Programs............................................................. 9
South Africa................................................................... 9
Egypt.........................................................................10
India.......................................................................... 11
Mexico........................................................................ 12
2. Evaluation Design and Methodology.......................................... 13
2.1	 Goal and Objectives of the Impact Evaluation........................... 13
2.2	 Developing the Evaluation Design and Framework......................... 14
Theory of Change............................................................ 15
Results Chain Model......................................................... 17
Success Formula.............................................................. 18
Content for Impact: Skill Categories......................................... 19
Indicators.....................................................................20
2.3	 Evaluation Design and Framework.......................................... 21
Program Monitoring......................................................... 21
Outcome Evaluation......................................................... 22
Data Collection Tools........................................................ 22
2.4	 The Evaluation Toolkit........................................................23
Digital Inclusion Social Impact Evaluation
3. Implementing the Evaluation....................................................25
3.1 	 Results Chain Model and Success Formula..................................26
3.2 	 End-User Survey.............................................................. 27
Sampling Strategy and Location............................................. 27
Data Collection............................................................... 27
Data Entry, Management and Analysis.....................................28
3.3 	 Routine Reporting............................................................29
4. Initial Evaluation Results.......................................................... 31
4.1 	 Google Analytics............................................................. 31
Total and Unique Views...................................................... 31
Average Number of Page Views
and Average Time Spent on The Platform.................................. 31
Top Content..................................................................33
4.2	 Survey Participants...........................................................35
4.3 	 Background Characteristics of Respondents................................35
4.4 	 Training and Access..........................................................38
4.5 	 Access Local Content........................................................ 41
4.6 	 Take Informed Action........................................................43
4.7 	 Increased Opportunities In Income.........................................46
Use Identity to Participate in the Local and Global Economy..............46
Start a Business...............................................................46
Career Advancement and Employment..................................... 47
4.8 	 Increased Opportunities in Health, Education, and Citizenship............50
Increased Opportunities in Health.......................................... 51
Increased Opportunities in Education....................................... 52
Increased Opportunities in Social Development............................53
4.9 	 Transform Communities.....................................................55
5. Conclusions and Lessons Learned............................................... 57
6. The Road Forward................................................................... 61
Appendix A: Data Collection Methods Along the Results Chain...................63
Appendix B: Impact Evaluation Toolkit................................................64
Appendix C: Top Five Articles by Country by Year...................................65
Evaluation Dictionary..................................................................... 67
Mission Definitions.................................................................68
References................................................................................. 71
1
One Global Economy began its Impact Evaluation program a year ago
after receiving a grant from the Symantec Corporation to evaluate existing
programs in Egypt, India, Mexico, and South Africa, and to build its internal
capacity to continue to monitor and evaluate future programs. Over the
course of the program, One Global Economy (OGE) has learned a great deal
about the processes of monitoring and evaluation, and about the people and
the communities that OGE serves.
The program began with OGE developing its theory of change, or how its
programs improve the lives of the people it aims to serve. This helped clarify
and determine the specific social impact metrics to be measured through
the evaluation.
The next step in the process was the design of a community survey to be
administered in all four countries in Arabic, Hindi, Spanish, and English.
OGE received 4,364 completed surveys, which provided a trove of data about
people who had either attended training delivered by OGE or had accessed its
online content on one of its digital platforms. Consequently, OGE was able to
learn more about the people it serves and how they use and view the content
and training.
OGE focuses on employment as a solid indicator of joining the economic
mainstream. Therefore, all of OGE’s trainings start with teaching basic
digital literacy so that participants will be able to apply those skills to both
the job search as well as increasing their employability. OGE’s findings about
employment were very encouraging. 86% of survey respondents did not have
a job before they participated in a training, but of those, 35% were employed
by the time the evaluation process had begun. Half of the respondents, who
reported they were employed, had found their job online.
Google Analytics has allowed OGE to see which pages of its content are the
most trafficked, how many pages people visit, how long users stay on a page,
and other hard data. Through combining the Google analytics data with the
data gathered from the community survey, OGE gained a much more detailed
and nuanced understanding of the impact of its online content and how
users adopt global best practices to improve their health, education, and
social development.
35%
of previously unemployed
survey respondents were
employed by the time the
evaluation process had begun
EXCUTIVE SUMMARY
2 Digital Inclusion Social Impact Evaluation
The survey data showed that 99% of respondents, who accessed content on
OGE’s digital platforms, took some kind of action. The most popular content
dovetailed with the feedback from the stakeholders’ engagement process
that guided the content creation to develop the platform in each country.
Employment and job-seeking skills were most popular in Egypt, health and
sanitary practices in India, citizenship in South Africa, and job skills in Mexico.
Another significant finding related to how people use OGE’s content
and training revealed that 86% of respondents from Egypt and 60% of
respondents from South Africa reported teaching other people in their
communities about what they learned from the digital platform or in an
OGE training. These results indicate that OGE’s content and training have
a multiplicative effect on the communities it serves.
The Impact Evaluation Program has bolstered OGE’s capacity to continually
monitor its programs. OGE has worked with its partner organizations to
help them develop performance indicators for programs delivered together.
Additionally, OGE has created a monthly reporting tool and a training
reporting tool that its partner organizations and local managers will use to
provide regular feedback to OGE about the progress of its programs, express
what needs improvement, and to share lessons they have learned.
In developing an evaluation methodology and a host of tools, OGE has
gained both a deeper understanding about the social impact its programs
generate as well as developed a greater capacity in monitoring them. OGE
has already begun applying lessons learned from this program. Firstly, OGE is
implementing the regular reporting. Secondly, OGE will continue to deliver
programs that proved successful. Lastly, OGE has identified content areas that
it would like to develop and improve. OGE looks forward to implementing the
evaluation methodology to measure the impact of future programs as well as
refining programs based on the results of the evaluation.
99%
of survey respondents who
accessed content on OGE’s
digital platforms, took some
kind of action
86%
of survey respondents from
Egypt reported teaching other
people in their communities
about what they learned from
the digital platform or in an
OGE training
3
1.1 Goal and Objectives of the Impact Evaluation
One Global Economy (OGE) works with under-served communities in over 15
countries across five continents to create and implement information-based
development strategies to help households and individuals raise their standard
of living, join the economic mainstream, and innovate local solutions.
OGE has a diverse portfolio of programs, each suited to the unique community
it works in through a process of local stakeholder meetings and collaboration
with the community and local non-governmental organizations (NGOs). These
programs focus on access to broadband Internet connections, creating relevant
digital content in local languages, and training people in areas of digital
literacy, entrepreneurship, and techniques to gain the skills necessary to gain
employment and upward mobility.
Since its founding in 2005, OGE has launched 17 international community
development websites. These digital platforms (Note: Words in bold
orange font are defined in the Evaluation Dictionary) contain vital
community development content on topics such as education, employment,
entrepreneurship, and health written in local languages for readers at a 4th-
grade reading level. The content housed on the digital platforms is composed of
both global best practices and locally generated articles that focus on the needs
of the community for which it is written. OGE’s digital platforms now reach
more than 2.5 million unique visitors per year and since its founding in 2005,
8 million people have benefited from One Global Economy’s empowering
community development information. In addition, OGE has trained thousands
of people on digital literacy and assisted in the opening of Internet access
points to enable people to have the ability to access the local content on the
digital platforms.
The purpose of this Impact Evaluation Program was to assess the long-term
impact of OGE’s existing programs on mission-driven outcomes and to build
OGE’s capacity to continually monitor and evaluate its programs going forward.
This is particularly important because OGE’s programs serve very different
communities, address a wide variety of issues, and do so across diverse levels of
development and urbanization.
BACKGROUND1
DEFINITION
Digital Platform
International community
development websites that
contain vital community
development content on topics
such as education, employment,
entrepreneurship, and health
written in local languages for
readers at a 4th-grade reading
level. The content housed on the
digital platforms is composed of
both global best practices and
locally-generated articles that
focus on the needs of the
community for which it is written.
DEFINITION
Digital Literacy
The ability to find, evaluate, utilize,
share, and create content using
information technologies and
the Internet (Cornell University);
the ability to perform tasks
effectively in a digital environment.
Literacy includes the ability to
read and interpret media (text,
sound, images), to reproduce
data and images through digital
manipulation, and to evaluate and
apply new knowledge gained from
digital environments (Jones-Kavalier
and Flannigan).
4 Digital Inclusion Social Impact Evaluation
This impact evaluation focused on OGE’s programs in South Africa, Mexico,
Egypt, and India to design and implement its methodology. There are common
aspects of OGE’s work in all of those countries, but each country had different
programs, content, and training curricula. The framework and tools created
as part of this evaluation are meant to provide other organizations working in
digital literacy and inclusion with tools to adapt to measure their impact and
improve programming.
1.2 The Information-based Ecosystem Strategy
OGE’s digital inclusion model addresses three barriers to participation in the
information economy: lack of access to internet services, lack of access to
actionable local content in local languages, and lack of digital literacy skills
through what it calls the Information-based Ecosystem Strategy (see Figure 1).
LACK OF ACCESS TO THE INTERNET
In the information age, access to the Internet is vital to participate in the global
economy. Without access to the Internet, the information available online
will not reach underserved communities. OGE partners with community
technology centers around the world that serve rural and low-income
communities, and offer access to the Internet for free, or at a low cost that is
considered affordable by the local residents. Through its partnerships with
these local organizations, OGE creates a network of access points, where the
digital platform is hosted in each country.
LACK OF ACTIONABLE CONTENT IN LOCAL LANGUAGES
There is a disparity between global language populations and content
available online in those language. For example, in 2012, “Arabic content
on the web represented just 3 percent of the total digital content online,
yet Arabic speakers make up more than 5 percent of the global population”
(Franceschi Bicchierai 2012). Online content in local languages that are not
as widely spoken such as Kinyarwanda and Swahili account for even less, at
below 1 percent. Consequently, digital content focused on developmental
themes would account for even less, making it challenging for individuals in
developing countries to access information in their language that is aimed at
helping them to improve their standard of living. With more people accessing
the Internet in developing countries through their mobile devices and local
telecenters, there is increased potential for online information to contribute to
the improvement of development outcomes in the areas of income, health, and
social development.
5
Beehive content on the digital platform is designed to address community
development priorities identified by local stakeholders through a highly
participatory engagement and consensus building approach. The content
combines global best practices with highly localized information, written in
the local language for readers with low literacy levels.
LACK OF DIGITAL LITERACY SKILLS
Without the skills necessary to access information on the Internet, people will
not utilize access points and cannot find the actionable content available in
their language. OGE provides training to partner computer center managers
on ways they can support digital literacy in their community, launch demand-
driven programs and services that address local needs, and teach users how
to navigate the Beehive as well as access other existing online resources that
provide global best practice information.
OGE’s three-pronged strategy has successfully helped to create an ecosystem
that has enabled millions in underserved communities to access high quality,
actionable content on developmental themes.
1.3 Internet Access Points
OGE works with in-country partner organizations to utilize their existing
Internet access points as places to deliver trainings, hold stakeholder
engagement meetings, and teach the community about its digital platforms.
OGE helps to build the capacity of the local access point managers by drawing
on its experience and best practices to help the centers to be more organized
and to run more smoothly.
Figure 1. Information-based
Ecosystem for Digital Inclusion
6 Digital Inclusion Social Impact Evaluation
1.4 Training
DIGITAL LITERACY
Digital literacy is critical to each person’s success in the 21st
century and to
maximizing the benefit of OGE’s programs, which is why it is the bedrock of OGE’s
training programs. “Digital literacy is the ability to find, evaluate, utilize, share,
and create content using information technologies and the Internet” (Cornell
University). It is the ability to perform tasks effectively in a digital environment.
“Literacy includes the ability to read and interpret media (text, sound, images), to
reproduce data and images through digital manipulation, and to evaluate and apply
new knowledge gained from digital environments” (Jones-Kavalier and Flannigan).
Email, social networking, word processing, and using the Internet are
necessary skills for anyone seeking gainful employment, a better job, or to
expand his or her business. OGE teaches these and more advanced skills in
many workshops. Using email and being able to navigate the internet open up
a huge amount of opportunities to save money, take advantage of government
programs, educate oneself, and make the kind of informed decisions that allow
people to improve their own lives.
OGE firmly believes that access, content and training are complementary
components of digital literacy, in that they work together to improve and
expand livelihood opportunities. Without training on how to use technology
properly, and training on how it can be incorporated into daily life, and applied
to finding information and solutions for development challenges, access
would not play such a transformative role. Similarly, access does not benefit
underserved populations if there does not exist content that meets them where
they are in terms of their language, education level, and local context. This is
why all of OGE’s training programs include digital literacy.
ENTREPRENEURSHIP AND TECHNOLOGY
Entrepreneurship is fundamental to growing today’s developing economies
and technology and globalization allow for unprecedented entrepreneurial
opportunities. OGE focuses on equipping emerging entrepreneurs with the
digital and mobile skills necessary to enter the formal business world and
confront everyday business challenges. Empowering entrepreneurs with core
soft and hard skills builds their potential to start and grow their enterprises,
and contributes to small business creation at the community-level.
OGE’s Information-based Ecosystem Strategy supports and strengthens its
entrepreneurship-focused programming by leveraging access, content, and
training to build the capacity of aspiring entrepreneurs. The entrepreneurial
training teaches basic technology skills to encourage technology skills application
as a valuable tool for small business growth. OGE guides the development
of essential business planning materials through providing templates and
instructions for creating a business plan, budget, and marketing plan amongst
others. OGE also provides online tools, localized information on entrepreneurship,
7
and mentoring support though mobile technology to entrepreneurs to help them
grow their existing businesses. Further, OGE helps small businesses access and
use banking services by teaching basic financial literacy to help small business
owners take out loans and to use government programs and financial institutions
that will help keep their businesses stable and profitable.
One such training program is OGE’s Mobile Entrepreneur’s Program
(MEP), which emphasizes the use of digital and mobile tools to grow and
improve small businesses. Workshops teach and monitor the achievement of
a series of online skills such as establishing an online presence for a business,
incorporating the use of social media into marketing plans, creating a LinkedIn
profile for online professional networking, monitoring business bank accounts
online and through SMS alerts, and conducting online research to increase
business knowledge and to access government or private funding, or additional
support through local organizations and services. Consists of modules
focused on similar career development themes and skills, but with a focus on
marketing individuals’ entrepreneurial skill sets to obtain jobs in a related field.
COMMUNITY CONNECTORS
The Community Connectors curriculum in digital literacy and leadership is
focused on teaching young people how to apply digital and mobile tools to
become actively involved in community development initiatives, leaders in
their communities and technology ambassadors for their peers and families. The
program teaches a combination of information and communications technology
(ICT) skills and soft skills (planning and management, time management, project
management, public speaking, self-confidence, etc). The program varies in each
country and consists of additional modules addressing entrepreneurship, career
development, Internet safety etc., and can focus on helping youth improve
academic performance or find jobs through technology skills application
depending on the target population (See Table A, OGE Training Programs).
Community Connectors graduates have applied skills learned to lead workshops
at their community centers to teach community members technology skills,
conduct research on community and economic development topics to map out
projects their community may implement, develop local businesses, and identify
income-generating opportunities. The program fosters youth empowerment and
skills-building for personal, professional, and social development.
INTERNET SAFETY
Internet safety has been incorporated as a focus in training in Mexico and India,
where OGE has a partnership with Symantec, and has carried out projects
promoting the involvement of locally-based Symantec employee volunteers.
Local employees and engineers have worked with OGE and its partners to
develop curricula for workshops and Internet safety campaigns focused on
teaching first-time Internet users how to protect their identities online, how
to protect their computers from viruses, and how to identify common threats
such as information theft and fraud. Symantec has also contributed content on
online safety, which is published on each digital platform.
DEFINITION
Mobile Entrepreneur’s
Program (MEP)
The Mobile Entrepreneurs Program
(MEP) emphasizes the use of
digital and mobile tools to grow
and improve small businesses.
Workshops teach and monitor
the achievement of a series of
online skills such as establishing
an online presence for a business,
incorporating the use of social
media into marketing plans,
creating a LinkedIn profile for
online professional networking,
monitoring business bank accounts
online and through SMS alerts,
and conducting online research to
increase business knowledge and
to access government or private
funding, or additional support
through local organizations and
services. Consists of modules
focused on similar career
development themes and skills,
but with a focus on marketing
individuals’ entrepreneurial skill
sets to obtain jobs in a related field.
8 Digital Inclusion Social Impact Evaluation
Community Connectors
Community Technology
Center Management
Mobile Entrepreneurs
Program
Internet
Safety
South Africa Digital Literacy, Leadership,
Community Development
Digital Literacy &
Business Management
for Community
Computer Centers
Digital & Mobile Literacy
for Entrepreneurship,
Business Planning,
Marketing
N/A
Mexico Digital Literacy, Leadership,
Entrepreneurship, Internet
Safety, Community
Development
Digital Literacy &
Business Management
for Community
Computer Centers
N/A Internet &
Personal ID
Safety
India Digital Literacy, Leadership,
Internet Safety, Job-Searching
& Career Development/
Professional Skills
Digital Literacy &
Business Management
for Community
Computer Centers
N/A Internet &
Personal ID
Safety
Egypt Digital Literacy, Leadership,
Entrepreneurship, Business,
Job-Searching & Career
Development/Professional Skills
N/A N/A N/A
1.5 Local Content
One Global Economy develops digital platforms written in local languages
that are accessible to people with a 4th grade reading level. OGE’s community
development-focused content is a combination of global best practices and
highly localized information. Best practice information covers topics that
are relevant all over the world, such as education, employment, health, and
money, and focus on teaching people specific best practices such as how to
write a business plan, how to select a school for their child, and basic prenatal
care. Highly localized content complements these global best practices by
providing information on how and where to access local resources, services,
and government programs that benefit underserved populations and help them
to take action to improve their lives in these areas.
Content is produced by local editors, NGOs , and student journalists. The
process begins with a stakeholders engagement workshop, where a roadmap is
created to guide the development of the digital platform, and then the content
is created over a period of 4-6 months. Each digital platform is launched as a
public website once there is content about all community development topics
identified in the workshop. Digital platforms are launched publicly with at least
200 highly localized articles. New content is added on specific topics as OGE
develops more partnerships with local NGOs. OGE expands upon content on
specific focus areas to complement its training programs. The digital platform
is hosted, maintained and checked for quality of content by OGE.
Table A. OGE Training Programs
9
1.6 Country Programs
SOUTH AFRICA
OGE has worked in South Africa since 2007 when it established the Durban
Centre, a community technology center hosted at the Austerville Public Library.
The Durban Centre serves residents of South Durban and the Austerville
community, and offers free access to the Internet and the South Africa digital
platform, known as a “Beehive”, as well as hosts workshops in digital literacy
for youth and adults.
In (2008, OGE expanded its work in South Africa through a 5-year commitment
to the Clinton Global Initiative (CGI) with partners, Cisco, Siyafunda, and
Appleseeds Academy. Through the CGI, the partners established a network of
Community Knowledge Centers (CKCs) across Sub-Saharan Africa (in Kenya,
South Africa, Rwanda, Ethiopia, Nigeria, and Uganda). In South Africa, the
partners trained Siyafunda community telecentre managers in digital literacy
and business management skills essential to running sustainable computer
centers that effectively serve their surrounding communities by building upon
their basic offerings to offer demand-driven programs.
The partners also developed and launched Community Connectors, a youth
technology, leadership, and entrepreneurship training program and trained
partner centers in the implementation and delivery of the program to local youth.
In addition to the South Africa digital platform, OGE launched portals
localized to Durban, Johannesburg, and Cape Town. Over the last five years,
OGE has continued to work in partnership with Siyafunda to deliver its Mobile
Entrepreneurs Program, which has supported over 1,500 South African street
vendors, aspiring entrepreneurs, and small business owners through training
and skills building in technology and entrepreneurship skills. The training
is complemented with text messages to reinforce learning and disseminate
helpful information, tips, and advice for growing and improving enterprises.
PROGRAM SNAPSHOT
South Africa
Year Started:
2007
Key Activities:
Establishing a network of
Community Knowledge Centers
(CKCs), Community Connectors
Program, Mobile Entrepreneurs
Program, and Beehives for
South Africa, Johannesburg,
Cape Town, and Durban
People Trained:
Over 2,500
Access Points Opened:
Over 80
Focus Areas:
Entrepreneurship and
youth leadership
DEFINITION
Beehive
The name of OGE’s digital platform
in South Africa, Egypt, and India.
10 Digital Inclusion Social Impact Evaluation
EGYPT
OGE’s work in Egypt has a particular emphasis on employment and
entrepreneurship, where it has been working with local partner, CEOSS
(Coptic Evangelical Organization for Social Services) and HANDS (Hands
Along the Nile), based in Alexandria, Virginia, since 2011 to connect young
people to livelihood opportunities through Asset Mapping, access to global best
practices and highly localized content on the Egypt digital platform (known
as the Beehive), and training in digital literacy, entrepreneurship, and career
development skills.
Through its Employment Through Information Technology program, OGE has
mapped the skills of approximately 23,000 young men and women along
with the outsourcing and employment opportunities of over 5,000 employers
to support opportunity identification and job matching through online and
mobile platforms. OGE and CEOSS have trained over 1,200 young people in
marginalized communities throughout Cairo, Minya, and Beni-Suef at 10 local
computer centers established through the program.
OGE launched the Egypt Beehive in addition to two localized portals created
for Cairo and Minya, which feature global best practice information and
community development-focused content in Arabic. The Beehives have served
over a million individuals since their launch in early 2013.
PROGRAM SNAPSHOT
Egypt
Year Started:
2011
Key Activities:
Asset Mapping, Online &
Mobile Job-Matching System,
Egypt, Cairo & Minya Beehives,
Community Connectors
People Trained:
Over 1,200
Access Points Opened:
10 (in Cairo, Minya & Beni-Suef)
Focus Areas:
Employment and
Entrepreneurship
11
PROGRAM SNAPSHOT
India
Year Started:
2010
Key Activities:
Beehives for India, Mumbai,
and Pune in English, Hindi, and
Marathi; working with Pratham
Infotech Foundation; training
programs; education programs
People Trained:
765
Access Points Opened:
10
Focus Areas:
Jobs, health, digital literacy
INDIA
OGE’s India Digital Inclusion Program focused on integrating the core
components of local internet access, localized online content, and digital
literacy and professional skills training into new programming and services at
10 computer centers run by OGE’s local partner, Pratham InfoTech Foundation
(PIF) in Maharashtra. OGE and (PIF) trained over 4,000 people in digital
literacy, technology, leadership, entrepreneurship, and community service, as
well as working with local health professionals to educate 488 people—most of
whom are young women—about health and sanitation.
The India Digital Inclusion Program has a particular emphasis on employment.
The program offers coaching services to help participants begin their
employment search, practice interviewing, job placement, and inclusion in an
online employment database. Many participants have learned how to create
and/or access email accounts and to use them for job-seeking and professional
purposes.
The third aspect of this program is OGE’s digital platform known as the Beehive
(www.india.thebeehive.org). Viewed by over 200,000 people in the last year,
the Indian Beehive is composed of articles about financial literacy, employment,
and health written in English, Hindi, and Marathi for an audience with a 4th-
grade reading level.
12 Digital Inclusion Social Impact Evaluation
MEXICO
OGE launched the Mexico digital platform, InfoFácil.mx, in 2010 and the
Toluca digital platform (Toluca.InfoFácil.mx) in 2013. OGE has been working
with University partner, Universidad Autónoma del Estado de México (UAEM)
in Toluca and its journalism department to engage students in the development
of highly localized content for the two platforms by providing them with small
stipends, community service and internship credit in addition to incorporating
content creation into curricula, and offering the Mexico and Toluca digital
platforms as an online publishing outlet. Through engaging student journalists,
over 500 articles were produced for the Mexico platforms over a period of
three years/ to date. Since their launch, more than 3.3 million unique visitors
are accessing the digital platforms in Mexico, with over 300,000 accessing it
via their mobile phones.
OGE also launched its Community Connectors program in three rural villages
in Guanajuato (El Garbanzo, San Agustin, and Victoria), where it works with
local partner, Choice Humanitarian, and University partner, Universidad
Technológica del Norte del Estado de Guanajuato, to support the development
of sustainable community technology centers. With the past year, OGE also
implemented the youth program at three semi-urban high schools in Toluca, in
partnership with UAEM student trainers, training a total of approximately 150
youth in Mexico. In 2013, OGE delivered educational workshops in Internet
safety for first-time Internet users to over 500 Toluca residents together with
student trainers and locally based volunteer Symantec engineers.
PROGRAM SNAPSHOT
Mexico
Year Started:
2010
Key Activities:
Beehives for Mexico &
Toluca, Student Journalist
Beehive Editors Program,
Community Technology
Center Management Training,
Community Connectors,
Internet Security Campaign
People Trained:
Over 200
Access Points Opened:
4
Focus Areas:
Health, Employment,
Education, Internet Safety,
Youth Leadership
13
There is a heightened awareness among the science and technology community
regarding the social impact potential of connectivity and economic inclusion
through digital and mobile tools. Academics and think tanks report on the
positive effects of access to the Internet, yet there is a need for real-life analysis
of the effects digital inclusion has on improving the lives of those in poverty.
OGE developed this Impact Evaluation to address this need and to quantify
its social impact potential and results, gather data that indicates lasting social
value of its digital inclusion programs, and track progress towards its defined
social impact targets.
In this Impact Evaluation, OGE sought to provide a guide for a rigorous analysis
of the “lift,” or margin of improvement, of its digital inclusion services upon
low-income populations in developing countries who are new to the Internet
and lack skills necessary to utilize the Internet to improve their lives. This
evaluation examines the lift in terms of the economic and human development
effects on individuals who participate in the digital online platform as users, in
the Community Connectors digital literacy training programs, and the patrons
of OGE’s partner Community Knowledge Centers.
2.1 Goal and Objectives of the Impact Evaluation
Through this Impact Evaluation, OGE sought to track the impact of the
digital inclusion program on participants’ economic opportunities and quality
of life, as expressed in terms of human development indicators: access to
opportunities that increase their income, education, and improve their health.
In developing a methodology to assess the impact of OGE’s training and
programs, this Impact Evaluation aimed to create a standardized way of
evaluating impact in digital literacy programs for the benefit of other
organizations doing similar work. Digital inclusion programs pose unique
challenges in designing and implementing an impact evaluation. By nature,
these programs are not place-based interventions; while the topics are
localized geographically the content can be accessed globally. Unlike traditional
development work, OGE’s programs are not defined in geographic terms
and the organization is not focused on working in specific communities for
extended periods of time. Therefore, this evaluation required the development
EVALUATION DESIGN
& METHODOLOGY2
OGE’s Mission
One Global Economy works
with under-served communities
to create and implement
information-based development
strategies to help households
and individuals raise their
standard of living, join the
economic mainstream, and
innovate local solutions
14 Digital Inclusion Social Impact Evaluation
of new methods and tools to assess the impact of OGE’s programs and of
digital inclusion programs in general. Thus, one of the main goals of this
evaluation was to create a toolkit of adaptable tools and frameworks for other
organizations to implement evaluations in this expanding niche.
This organizational capacity-building project also served to strengthen focus
and codify an outcomes-based approach for implementation across all OGE
country programs in the future.
THE OBJECTIVES OF THE IMPACT EVALUATION WERE TO:
•	 Articulate a theory of change of how OGE’s training and programs lead to
mission-driven outcomes
•	 Develop key qualitative and quantitative performance and impact
indicators
•	 Create a comprehensive strategy to gather data for the key indicators,
including routine programmatic reporting and follow-up with participants
–	 Implement a follow-up survey in partner computer centers and online
in four actively-funded countries (Egypt, India, Mexico and South Africa)
–	 Develop internal tools for more efficient communications regarding
progress toward impact metrics
•	 Create a toolkit with adaptable templates that provides a comprehensive
guide to implementing an impact evaluation for digital inclusion
•	 Analyze the process and outcome of this initial impact evaluation with
attention to improving programming, outcomes, and refining the
evaluation methodology
•	 Build internal capacity to monitor and evaluate the impact of Digital
Inclusion programs for future projects
•	 Share the findings, results, and conclusions publicly
2.2 Developing the Evaluation Design & Framework
The process of developing the evaluation design and framework included:
•	 Creating a theory of change to articulate how OGE achieves its impact.
This causal logic of how OGE’s programs deliver results is the underpinning
of the impact evaluation and was used to create its framework, indicators,
and data collection methods.
15
•	 Adapting two program planning tools, the Results Chain Model and the
Success Formula, to further map out the inputs, strategies, and outputs that
lead to OGE’s hypothesized impact. These two exercises identified the process
that leads to impact as well as what the impact looks like.
•	 Creating a comprehensive list of indicators using these two tools to ensure
all aspects of programs were measured as part of the evaluation. Each step
of the Results Chain was included, as well as a clear definition of all the skill
and content categories identified through the Success Formula process. The
comprehensive list of indicators was distilled into a list of key indicators, tested
in this evaluation and to be used as a menu of programmatic indicators for
future planning.
•	 Using the list of indicators to develop data collection tools and systems to
measure each of the key indicators identified in this process.
THEORY OF CHANGE
OGE began the impact evaluation with the question: What effect does digital
literacy and access to the Internet have on social and economic development
of individuals and communities? To answer this question, the first step of the
evaluation was to hypothesize how OGE achieves its impact – the organization’s
theory of change.
The evaluation team deconstructed the organization’s mission by creating
internal definitions of each piece of the mission (see Evaluation Dictionary:
Mission Definitions) to articulate the organization’s theory of change:
Strengthening access to digital and mobile tools by partnering with local
computer centers, training computer center managers and patrons in digital
literacy and business skills and providing access to locally-written, actionable
content on developmental themes in their own languages will enable households
and individuals to take informed action that will transform their lives and
their communities.
Figure 2. Evaluation
Methodology Development
16 Digital Inclusion Social Impact Evaluation
More specifically, the OGE training curricula combined with both locally
generated content and global best practices in key issue areas provides users
with the skills and information they need to take significant steps towards
participating in both local and global economies. Through digital inclusion,
users and their communities can raise their standard of living and join the
economic mainstream by increasing their economic, health, and educational
opportunities, such as starting a small business, improving their livelihood
prospects, improving their ability to make informed health, education, and
business decisions; and increasing their participation in the local community.
A key element of OGE’s Theory of Change is that the individuals in the
communities OGE serves must take action to complete the cycle of change.
While OGE fosters the information ecosystem with local partners and creates
and shares opportunities for people to take action to improve their lives, OGE
believes firmly that the sustainable solution is one in which local participants
are the key actors, leading local progress.
Figure 3. One Global Economy’s Theory of Change
17
RESULTS CHAIN MODEL
The evaluation team then applied the theory of change to two program
planning tools to further articulate how the organization achieves its impact.
Through clearly defining and articulating the mechanisms and processes that
lead to mission-driven outcomes, the team sought to create a comprehensive
evaluation model that encompasses all elements of OGE’s programming to
best assess and estimate the “lift” and a set of program planning tools adapted
to OGE’s theory of change that could be used for future programs. Thus, these
tools were used both to help guide the evaluation by looking retrospectively
at OGE’s program models and as an exercise to develop planning tools that
would be used in all future programs and adapted for other organizations and
included in the Evaluation Toolkit.
First the team used the World Bank’s results chain model found in the
Impact Evaluation in Practice. A results chain shows cause and effect: how the
sequence of program inputs (e.g. staff and other resources) and activities lead
to specific program outputs, and how these outputs are directly related to
accomplishment of program goals (i.e. impact). This model shows how specific
program activities directly lead to the desired program goal and what must be
done in order to achieve programmatic targets and results. It also documents
what goes into a program (the inputs), which is integral for replicability and
scalability of programs.
The evaluation team created an overall results chain model for the organization
as well as adaptations for country-level programs. We used this structure
to succinctly visualize and describe each of OGE’s programmatic activities,
including their anticipated proximate and distal impact. The results chain
demonstrates the cause and effect relationship of OGE’s programming.
Moving forward, OGE will require program staff at the country level to create
a Results Chain Model for their program so that they can understand how
programmatic activity results in the desired program goal and what must be
done in order to achieve programmatic targets and results. By creating a model
that showcases how the program’s theory of change is directly tied to the
program’s specific focus and goals, stakeholders create a common vision for
the program, which serves as a roadmap for both implementing and evaluating
the program.
DEFINITION
Results Chain Model
Template form of the results chain
for country teams to fill in at the
beginning of a program, a practice
for them to see how programmatic
activities are directly tied to longer-
term, mission-driven outcomes.
18 Digital Inclusion Social Impact Evaluation
Note: The full version of the Digital Inclusion Results Chain includes a section for
listing indicators along each step of the chain. Country teams list the inputs, activities,
outputs, outcomes, and impact for their programs and then create a list of indicators
to measure each.
SUCCESS FORMULA
OGE also adapted the Success Formula framework from the firm Mission
Measurement and its CEO Jason Saul’s book Benchmarking for Nonprofits as
a second program planning tool for this evaluation.
The Digital Inclusion Success Formula is a universal tool that enables One
Global Economy and other digital inclusion nonprofits to consider its social
impact as a simple equation. At the end of the formula is the ultimate impact
the organization aims to achieve through programming, created through the
sum of the intermediate steps at a strategic, organizational level.
The Success Formula makes OGE’s Theory of Change actionable and
measurable. At the top of the Success Formula is the underlying principle
that acquiring digital literacy and key skills combined with access to both
locally generated content and global best practices in key issue areas provides
users with the skills and information they need to take significant steps
towards participating in both local and global economies. The Success
Formula is used to outline the strategy and activities that will be used in
the key issue areas (income, health, education, and social development) of
the program, and the performance indicators that will be used to measure
progress and success.
Figure 4. OGE Digital Inclusion Results Chain
Adapted from Gertler, Martinez et al.
Impact Evaluation in Practice.
The World Bank 2011.
19
CONTENT FOR IMPACT: SKILL CATEGORIES
To further refine the model, we brainstormed and defined the main content
areas of OGE’s programming, which together we believe are critical to
achieving OGE’s desired impact.
We called these groupings of content “Skill Categories” because the goal
of the content in digital inclusion programming is building capacity and
informed action. Part of OGE’s theory of change is that access to and taking
informed action on these specific content areas are what create the desired
impact as described in OGE’s mission statement.
The four primary Skill Categories we identified are: digital literacy, business
and entrepreneurship, livelihood and employment, and personal/social/
community development. Of note, Digital Literacy is a fundamental skill,
because it creates the ability to access content in the other Skill Categories.
The Skill Categories were further broken down into sub-skills, again using
the criteria that these skills are imperative to achieving the desired results
and are a fundamental part of OGE’s training curricula and programmatic
activities. The skills and sub-skills identified in this exercise were
fundamental to creating indicators and creating training assessments as
part of the routine reporting component of the Impact Evaluation.
Figure 5. OGE Digital Inclusion Success Formula
Note: The full version of the Digital Inclusion Success Formula includes sections for listing program strategies and indicators
for income, health, education, and social development. Country teams list the strategies they will use in their programs to
create change in these categories, and then create a list of indicators to measure each.
DEFINITION
Digital Literacy
The ability to find, evaluate, utilize,
share, and create content using
information technologies and
the Internet (Cornell University);
the ability to perform tasks
effectively in a digital environment.
Literacy includes the ability to
read and interpret media (text,
sound, images), to reproduce
data and images through digital
manipulation, and to evaluate and
apply new knowledge gained from
digital environments (Jones-Kavalier
and Flannigan).
20 Digital Inclusion Social Impact Evaluation
INDICATORS
The indicator table serves as the backbone of the evaluation. The evaluation
team developed indicators, or means to measure the comprehensive model
of the theory of change. Indicators were created for each step in the Results
Chain, and within each step for the skill categories. Creating the indicators
in this way ensured that we measured all aspects of the process of delivering
programs and the components that are imperative to creating impact.
The indicators for inputs, activities, and outputs are process indicators, which
measure what is put into a program and the deliverables of programmatic
activities. The indicators for outcomes and impact are outcome indicators,
which measure the lasting impact of the programs on the participants.
To create outcome indicators tied to OGE’s mission of raising the standard of
living, join the economic mainstream, and innovate local solutions, we used the
Human Development Index, published by the United Nations Development
Program. This Index uses composite indices to measure health (life
expectancy), education (mean years of education), and income (gross
national income, GNI, per capita) to rank countries in terms of development.
While the indicators themselves cannot be used at the individual and
household level, we used these broad categories to define standard of living
for the purposes of this impact evaluation.
Figure 6. Skill categories and sub-skills
DEFINITION
Process Indicator
Indicators to measure whether
planned activities took place, add
more details in relation to the
product (“output”) of the activity,
and/or monitor the quality of the
activities conducted, based on
a number of established quality
criteria or standards. (WHO)
DEFINITION
Outcome indicator
Indicators that measure the
objectives of an intervention, that
is its ‘results’, its outcome. They are
the result of both the “quantity”
(“how many”) and quality (“how
well”) of the activities implemented.
See full definition in the
Evaluation Dictionary
21
OGE therefore defines standard of living as the income, education, and
health opportunities accessed by a given individual or household. It includes
the following:
•	 INCOME: Changes in annual income and savings, employment and career
growth, entrepreneurship.
•	 EDUCATION: Pursue or attain GED, applied to or enrolled in university.
•	 HEALTH: Accessing quality health care systems, disease management,
access preventative medicine (prenatal care, sanitation and water access,
family planning, immunizations, infant care, nutrition)
By using OGE’s mission statement to identify and define the final outcomes
in the results chain, we aimed to ensure that this impact evaluation measures
OGE’s results against its mission, reinforcing the mission at all levels and
integrating the mission in its evaluation of itself. All indicators were assessed
to be specific, measurable, attributable, realistic, and targeted.
2.3 Evaluation Design and Framework
The impact evaluation applied a conventional social science approach using
a non-experimental design. We collected information on the impact of
OGE’s programs and activities on the desired outcomes through standardized
routine program monitoring and a follow-up survey with training
participants and digital platform users.
The nature of OGE’s programs renders an experimental (randomized) or
quasi-experimental (comparison group) approach infeasible. Therefore, the
evaluation compared participants’ utilization of information technology and
key socio-economic indicators before and after training.
The evaluation used several methods to measure changes in the skill
acquisition, utilization, and impact of information technology and resources
from OGE’s programs. The evaluation methods bolstered routine reporting
to monitor programs and assess input, process, and immediate outcomes of
programs and initiated a survey to evaluate outcomes.
PROGRAM MONITORING
By measuring the resources, activities, and program achievements,
organizations can track the causal logic of program outcomes, determine if
interventions were carried out as planned, if the interventions reached the
targeted beneficiaries, and if the interventions were performed on schedule,
and what the results from the interventions were for the beneficiaries.
22 Digital Inclusion Social Impact Evaluation
As part of the evaluation, OGE developed three components to program
monitoring:
•	 A STANDARDIZED MONTHLY REPORTING FORM and system to collect
program data regularly, including data on the resources used (inputs),
activities completed, and direct program outputs. Field staff and partners
complete this form on a monthly basis to record progress towards key
program objectives, which will directly feed into the impact evaluation.
•	 A STANDARDIZED TRAINING REPORT FORM, which includes the content
of the training, what curricula were used and how they were adapted, and
the number of people trained. In addition, post-training assessments were
created to ascertain the level of proficiency in the key skills taught during
the training.
•	 A TRAFFIC MONITORING TOOL to track OGE’s international digital
platforms using Google Analytics. This monitoring tool allows OGE to
gather significant data about the people who are using the digital platforms.
OUTCOME EVALUATION
To measure the impact of digital inclusion programs, it is necessary to survey
participants and measure behavior and socio-economic changes of those who
participated in trainings and/or accessed information via the digital platform.
•	 IMPACT SURVEY: A standardized survey given both online and through
active follow-up with program participants after they have been trained
in one or more of OGE’s programs (e.g. Digital Inclusion, Community
Connectors, Entrepreneurs, etc). The purpose of this survey is to find out
the longer-term impact of the training on participants’ lives and socio-
economic condition. The survey was conducted in four countries where
OGE has conducted trainings—specifically, in India, Mexico, Egypt, and
South Africa—for this initial evaluation.
DATA COLLECTION TOOLS
As part of creating the indicator table, we identified when and how the data
would be collected and from whom. This process identified the different data
collection tools that were necessary and when they should be implemented.
To develop the data collection tools, we used the indicators to create
questions. We systematically went through the indicator table to ensure
that all data required to measure the indicators will be collected.
A summary of data collection along the results chain is presented in Figure
7. By measuring indicators along the results chain, the impact evaluation
methodology prevents producing a ‘black box’ that only identifies whether or
not the predicted results occurred without explaining why or how the results
were achieved. See Appendix A for a detailed description of data collection
methods across the results chain.
23
As we refined the data collection tools, we refined and updated the indicator
table. The survey questionnaire went through several rounds of reviews with
the team to streamline, simplify and ensure that high-quality data would be
collected that would measure the desired outcomes of the evaluation and
then was sent to local partners for review and feedback.
2.4 The Evaluation Toolkit
One of the goals of this Impact Evaluation was to create a methodology to
enable the organization and its partners to better measure the social and
economic impact of its Digital Inclusion Programs so that the lessons can
be used to refine and scale future programs. Therefore, as part of the Impact
Evaluation, OGE developed an Evaluation Toolkit to codify the process, tools,
and implementation of a digital inclusion impact evaluation to be used
in the future for all of OGE’s digital inclusion programming and for other
organizations doing similar programming to adapt and use it to measure
their own impact.
This toolkit is a resource to help the management team, programmatic staff,
and partners succeed at enabling digital inclusion toward developing their own
communities. The purpose of this toolkit is to enable better digital inclusion
practitioners to understand, define, measure and improve their impact.
In the Impact Toolkit, OGE offers several guides for localizing the methodology,
including a Success Indicators Planner that breaks down OGE’s theory of
change and definitions of digital inclusion success. This will help to cultivate
local critical thinking, ownership, and leadership of digital inclusion progress
and measurement on the ground. Throughout the process of implementing
this initial impact evaluation, we incorporated lessons learned into the toolkit
and refined the framework, data collection tools, and systems.
Figure 7. Data collection summary along the results chain
24 Digital Inclusion Social Impact Evaluation
This Digital Inclusion Impact Evaluation Toolkit features capacity building
guides, templates, and exercises for One Global Economy (OGE), its
international partners, and other Digital Inclusion organizations to better
define, plan and measure against its proposed outcomes (see Figure 8). OGE
is sharing this universal toolkit with templates designed to be customized for
each country/program with the collaboration of local partners, and enables
partners to adapt and help execute the evaluation in their country.
For a detailed list of the contents of the Impact Evaluation Toolkit, see
Appendix B.
Figure 8. The Digital Inclusion
Impact Evaluation Toolkit
25
The evaluation team included OGE programmatic and leadership staff in
collaboration with an evaluation consultant who provided technical guidance
and assistance. The team began planning for the evaluation in the fall of 2013.
The Impact Evaluation methodology, including the Theory of Change, Results
Chain Model, Success Framework, indicators, and data collection methods
were designed and created over the course of five months, from October 2013
to February 2014.
Once the indicators were created and the methodology outlined, the team
began designing an end-user survey and internal monitoring tools in early 2014
and implementing them in the summer. The analysis of all data and writing of
the final report took place between August and September 2014 (see Figure 9).
The purpose of this impact evaluation was to be formative for continued
evaluation of impact. As such, this evaluation was carried out in only four
countries to test and improve the methodology. Moving forward, OGE will
continue to implement the routine reporting created as part of this evaluation
(monthly, training, and traffic reports) in all of its current and future country
programs and will implement subsequent surveys to continue to assess the
impact of its programs.
OGE chose India, South Africa, Mexico, and Egypt as the countries in which
to conduct this formative Impact Evaluation for a number of reasons. Firstly,
OGE had ongoing programs in each of those countries, which meant it had on-
the-ground partners who could help contact people who had taken trainings
and accessed content. Secondly, these countries represent a diverse array of
communities OGE serves, ranging from the densely urban areas of Mumbai
to very rural villages in the Mexican countryside with many gradations in
between. Thirdly, these four countries covered a wide array of different
programs that OGE employs.
IMPLEMENTING
THE EVALUATION3
26 Digital Inclusion Social Impact Evaluation
3.1 Results Chain Model and Success Formula
The Results Chain Model and Success Formula are program planning tools,
and served two functions in this evaluation. First, we used these tools to
retrospectively outline the theory of change and strategies for achieving the
desired outcomes of OGE’s programming overall. This was an essential step
to help develop a comprehensive set of indicators that would drive the data
collection methodology of the evaluation. Second, we field tested these tools
with country teams to gain consensus and improve the tools so that they could
be included in the Evaluation Toolkit as program planning tools – used in
all future OGE programs and adapted by other organizations when planning
similar programs.
The feedback from country teams in South Africa and Mexico was that
these were useful tools in looking at the program as a whole and better
understanding the relationship between program outputs and impact.
Managers in these countries were able to participate in a step-by-step process
of diagramming how these programs are designed to help their beneficiaries.
This clarified the strategies through which real change is affected and
highlighted the importance of their programmatic activities.
Of note the team in South Africa used an innovative approach to completing
the Success Formula: they completed the formula from the perspective of
one person who couldn’t find a job to figure out the best way to work with
them. This approach was incorporated into the directions for completing
Figure 9. Impact Evaluation Timeline
2013 2014
27
the Success Formula and results chain, but from a more holistic perspective:
completing the exercises several times over with different client avatars, each
with different problems and needs, as a needs assessment, program planning,
and evaluation tool. For example in Mexico, the team of student journalists
reflected on how articles on various locally relevant topics contributed to
community, personal, and social development, and how different formats
for disseminating global best practice information provided access to more
services and resources locally.
Overall, a challenge with these tools was getting program teams to be as specific
as possible about programmatic activity. Without guidance, local teams often
wrote blanket statements that were not particularly meaningful or measurable.
To help guide teams in the future, the evaluation team created a list of program
activities and indicators for them to include, adapt, or expand upon.
3.2 End-User Survey
SAMPLING STRATEGY AND LOCATION
The end-user survey is the most significant surveying OGE has ever undertaken
of the communities it serves. OGE posted links to the survey on the local
online portals in each country to try to get online users to participate and
actively followed-up with past program participants in communities to conduct
the survey in-person.
The survey was implemented in Mexico, Egypt, South Africa, and India. The
target population was anyone who had either visited a local online portal
or had attended a training course with an OGE curriculum. This is a very
demographically unspecific set of people, but that was necessary given the
diverse population that OGE serves.
DATA COLLECTION
A standardized survey questionnaire was uploaded to SurveyGizmo, a web-
based survey program that allows for skip logic, multiple choice, and write-in
responses. A link to the survey was posted to the digital platform in each of
the four countries, with translations in Arabic, Hindi, Spanish, and English
available. In addition, staff of OGE’s partners administered the questionnaire
in-person with hard copies to members of their community.
Data collection began on March 15 and continued through July 22, 2014.
It is impossible to overstate how critical OGE’s in-country partners were to
the success of this survey. OGE worked with Pratham InfoTech Foundation in
India; the Coptic Evangelical Organization of Social Services (CEOSS) in Egypt;
28 Digital Inclusion Social Impact Evaluation
Siyafunda in South Africa; and Universidad Autónoma del Estado de México
in Mexico. OGE works with these partners to implement its programs in each
country. Not only do these NGOs have a deep reach into the communities
where OGE works, but in many cases they also run computer or community
centers where OGE could get people from the community who had participated
in its programs to take the survey. The partners met with people in their
communities, conducted outreach, and encouraged people to take the survey.
DATA ENTRY, MANAGEMENT AND ANALYSIS
Most individual respondents completed the survey on SurveyGizmo. The hard
copies of the survey were directly entered into SurveyGizmo by members of
OGE’s partner NGOs to ensure standardized data between online and in-
person respondents.
The analysis plan used the Success Formula and Results Chain as a guide
to analyze the key performance indicators identified during the process of
creating those tools. For demographic and direct program impact variables,
data was compared between countries to see how the different programs
affected outcomes. For longer-term impact variables, data was compared
between those who did and did not participate in a training to see the long-
term effects of the training on key impact variables, particularly the impact of
training on taking informed action in key development areas. Furthermore, we
analyzed the data by time since training to uncover the longer-term impact of
the program and training.
Overall, this analysis was conducted with an eye towards understanding the
effect of content and training and how we can improve our programs to
achieve greater impact.
A criterion of p<0.05 was used to assess the statistical significance of
differences between groups. Differences with p<0.01 were noted as
highly significant.
29
3.3 Routine Reporting
A major goal of the Impact Evaluation was to build OGE’s capacity to
monitor and evaluate its programs. Therefore, as part of the evaluation, a
comprehensive routine reporting system was developed and implemented
(see Section 3.2 above).
OGE collaborated with its partners in each of the countries to create
the routine reporting templates, including monthly reporting forms, a
standardized training report form, and a traffic monitoring tool. The Monthly
Routine Reporting Form and the Training Report Form together create
a comprehensive internal monitoring system for OGE. The purpose of this
system is to:
•	 STANDARDIZE INFORMATION ACROSS COUNTRY PROGRAMS. The same
information will be collected from all of OGE’s programs using this form.
With standardized information, we can track programmatic activities and
progress organization-wide. Regular updates on program activities ensure
the history of a program is documented thoroughly.
•	 ENABLE “REAL-TIME” REPORTING. Activities are updated monthly as
they happen, rather than waiting until a report is due. If updates program
milestones are given on an on-going basis, the burden of giving information
for reports at the end of the period is reduced. Rather than going back and
asking country teams what happened during the reporting period, staff can
generate a report using the data that has been entered already.
•	 CONSOLIDATED SOURCE OF PROGRAM INFORMATION.
The Monthly Routine Reporting Form collects information on major activities
(e.g. trainings or workshops, computer centers opened, outreach activities, and
community activities), content developed for the digital platform, computer
center activity (e.g. users and center development activities), follow-up with
training participants, and stories of people innovating local solutions, quotes
and testimonials, and other notes. The Training Report Form is part of a larger
set of tools for designing and evaluating trainings, which include choosing
curricula and teaching objectives and post-training evaluations of skills
acquired. The Training Report Form reports on the content and objectives of
the training, the attendees, and the results of post-training assessments.
The reporting system utilizes a web-based survey design, Survey Gizmo, with
skip logic: answers to initial questions direct the respondent to subsequent
questions to complete information specific to that type of activity and allows
them to skip questions that are irrelevant.
DEFINITION
Routine Programmatic
Reporting
Standardized monthly report form
submitted by the country team to
headquarters, including directions
and a description of how this
reporting fits into the larger impact
evaluation and OGE’s mission.
DEFINITION
Training Report Forms
Standardized training report form
submitted by the country team
within one month of training to
headquarters, including directions
and a description of how this
reporting fits into the larger impact
evaluation and OGE’s mission.
30 Digital Inclusion Social Impact Evaluation
By including the local partners in the development of the reporting system,
OGE was able to integrate their feedback to make the system more accurate,
relevant, and feasible. For example, the South Africa Manager provided
helpful feedback for localizing the Monthly Reporting Form, to the Mobile
Entrepreneurs Program specific to South Africa, by suggesting that the form
integrate more skills in the checklists and that the form include a space for
local managers to provide testimonials from participants, which are gathered
regularly. Walking in-country managers and partners through their own
success formula and results chain often helps them gain a better perspective
on the work that they are doing.
Google Analytics allows OGE to track the information of digital platform users.
By tracking information using the Traffic Monitoring Tool, OGE can learn
which pages are viewed the most and for the longest amount of time in different
areas, where users come to the digital platforms from, and which topics resonate
the most. This allows OGE to give partner organizations accurate feedback
about articles written by community members, as well as pointing OGE towards
the issues that are most pertinent to different communities.
DEFINITION
Traffic Monitoring Tool
A form to collect and consolidate
information from Google Analytics.
31
4.1 Google Analytics
TOTAL AND UNIQUE VIEWS
At the time of this report, OGE has had 10,756,356 sessions and 9,386,745
unique views on its digital platforms since 2005. As seen in Figure 10, unique
views have been steadily increasing yearly for OGE digital platforms overall.
Unique views on the Mexico digital platform have remained consistent since
July 2011 at an average of 450,000 unique views per year. The Egypt digital
platform, which started in 2013, had over 570,000 unique views that year.
AVERAGE NUMBER OF PAGE VIEWS
AND AVERAGE TIME SPENT ON THE PLATFORM
Two ways to measure online engagement are the number of pages viewed in
a session and the amount of time spent on the website. OGE monitors these
two metrics on all of its web properties using Google Analytics (Figure 11).
INITIAL EVALUATION
RESULTS
Figure 10. Unique views per
year by country and OGE overall
4
32 Digital Inclusion Social Impact Evaluation
In South Africa the number of pages visited per session has dropped a small
amount from just over two pages per session to about 1.8 pages per session
while the time spent on the website has increased from about 1:05 to 1:18.
There has been a similar trend in Mexico where page views decreased from 1.8
per session to 1.4 while the time on the sites has risen dramatically from 1:25 to
about 2 minutes. This signals a deeper engagement with the material.
India has trended oppositely, where page views have gone up from an average
of 2.85 to 3 and time spent on the web property has fallen from about 3:30 to
about 2:55. Though the time on the site has decreased, almost three minutes
spent on the site remains higher than the industry average of 10 to 20 seconds
(Nielsen 2011) and the “average of a little over one minute on a newspaper
website” (Grabowicz 2014).
Figure 11. Average number of pages viewed and time spent per session on the digital platform
33
TOP CONTENT
As seen in Figure 12, health, education, and employment are consistently
among most accessed content. Because each portal is localized to its respective
region, programmatic efforts focus on relevant topics, and their significance to
the target population is often highlighted by trends in users’ activity as seen in
the diversity between countries in these results.
For example, South Africa shows the most views for citizenship-related content
compared to Mexico. In South Africa, the stakeholders’ engagement workshop
revealedthatmanySouthAfricanresidentswerenotawareoftheirrightsascitizens
under the then new post-Apartheid democracy. Through developing content with
localNGOs,OGEdefinedcitizenshipasakeycommunitydevelopmentpriority, and
featureditasthefifthmaintopic.ForEgyptandIndiathisfifthtopicisagriculture.
SouthAfrica’scitizenship-focusedcontentoutlinescitizens’ basicrightsandexplains
SouthAfrica’ssystemofgovernmentinalanguageuserscanunderstandin orderto
access local services, initiate processes for obtaining licenses and certificates, and
become informed and active citizens through participating in elections.
In other cases, top content indicates hot issues and can inform article creation for
the development of more specific content addressing those topics. For example,
soon after the launch of the Mexico digital platform, a review of Google Analytics
and comments from users on the platform showed that there was a demand for
more employment-focused articles. OGE, its local team, and student editors at
UAEM (partner University in Mexico) focused on developing articles aimed at
young people and job seekers, increasing the number of online resources that
help people understand how to apply for jobs online, learn interview skills, and
link them directly to Mexico-based online job-search engines. The demand for
livelihood-focused content addressing employment and education issues is also
illustrated by the high percentage of content viewed on women and teenagers,
which remains a top topic due to additional content OGE developed to guide
high school dropouts, single mothers, and adults through choosing the best
way to complete their education and earn degrees and certificates.
The structure of the program and its components in each individual country can
also explain why certain topics are more frequently trafficked over others. For
example, in Egypt, the goal of the project is to link young people to employment
opportunities. Content on the Egypt digital platform therefore largely focuses on
connecting individuals to resources addressing issues of financial literacy, income
generation, entrepreneurship, agribusiness, and best practices for microenterprise
development. In India, a recent program focused on implementing health camps
to teach basic sanitation practices, preventative healthcare, and adolescent health
at the partner centers. Many of the health camps have focused on sexual health
and serve young women, who lack access to such information. Top content
viewed on health in general, and articles on safe sex, and date rape drug awareness
are reflective of a need for adolescent health information.
See Appendix C for the top five articles by year for each country and OGE overall.
34 Digital Inclusion Social Impact Evaluation
MOBILE VS. DESKTOP USERS
The percentage of people accessing OGE’s digital platforms from mobile
devices steadily increased since 2011. In South Africa between 2013 and
2014, 53% of people accessing the digital platform did so via mobile devices.
Though the percent of mobile users are still low in Egypt (9%), India (22%),
and Mexico (24%), they are not insubstantial. All four countries are showing
substantial upward trends, with the percent of mobile users doubling in all
four countries in the last year. This trend has serious consideration for future
programs, which will be discussed later in this report.
Figure 12. Top content topics by country by year
Figure 13. Percent of all users accessing the digital platform with a mobile device by country by year
35
4.2 Survey Participants
OGE collected a total of 4,676 surveys between March 15 and July 22, 2014.
Incomplete surveys (n=308) and surveys from respondents living in countries
not included in the evaluation (n=8) were removed from the analysis. A total
of 4,364 surveys were included in the analysis: 2,407 from Egypt; 1,292 from
India; 491 from South Africa; and 174 from Mexico (see Figure 14).
The number of respondents in total and in each country is sufficient for OGE’s
purposes because the total is large enough to test for statistical significance and
each country’s total reflects the number of participants in OGE programs. For
instance, OGE works with CEOSS, one of the largest social service NGOs in
Egypt. The programs reach a lot of people and therefore the most surveys were
collected in that country. In contrast, OGE’s work in Mexico is in a few very
rural villages so the 174 completed surveys represent a very large number of the
participants in OGE’s programs there.
4.3 Background Characteristics of Respondents
Table B presents the demographics of survey respondents. This is a very
detailed and representative snapshot of OGE’s constituency, and it reflects
OGE’s focus on populations underrepresented in the economic mainstream,
particularly women, young people, and the unemployed.
Overall, 59% of the respondents were women; while the gender of respondents
varied significantly between countries the range was not large (56% in Mexico
to 65% in South Africa). See Spotlight: Gender for a detailed analysis of the
impact of OGE’s programs on women.
Figure 14. Data for analysis
by country
36 Digital Inclusion Social Impact Evaluation
Of note, most respondents were between the ages of 18 and 34 at the time of
the survey (90%). This is partially because of the large amount of respondents
in Egypt (55% of the total survey) who were of that age (2,400 of the 2,407).
However, that trend held across the other countries as well, with 78% of Indian,
91% of Mexican, and 85% of South African respondents below the age of 30.
This reflects the emphasis OGE’s programs have on training young people
because youth around the world often lack job prospects and job skills, and
employment as a young adult is one of the best routes out of poverty. Young
people, who are more likely to have free time and less likely to have families,
are particularly able to take advantage of the job skills and entrepreneurial
programs that OGE offers.
Almost half of all Egyptians surveyed (48%) are unemployed even though 69%
have some college experience or have graduated from college. This speaks
to the difficult economic situation that many Egyptians face, but also to the
benefits of becoming as job-ready and as good at the soft skills of interviewing
and networking as possible. In India, despite a low unemployment rate, many
of the respondents were students (27%) or homemakers (25%). These are two
populations that can greatly benefit from digital literacy and job skills as well
as entrepreneurial training because of the opportunities they provide for small
business creation and joining the job force.
37
Egypt
(N=2,407)
India
(N=1,292)
Mexico
(N=174)
South Africa
(N=491)
Total
(N=4,364)
Gender*
Male 1,010 (42.2%) 526 (40.8%) 76 (43.7%) 168 (35.3%) 1,780 (41.1%)
Female 1,382 (57.8%) 763 (59.2%) 98 (56.3%) 308 (64.7%) 2,551 (58.9%)
Age**
12-18 3 (0.1%) 2 (0.2%) 69 (39.7%) 2 (0.4%) 76 (1.7%)
18-24 1,376 (57.2%) 598 (46.4%) 66 (37.9%) 255 (52.3%) 2,295 (52.7%)
25-34 1,022 (42.5%) 406 (31.5%) 25 (14.4%) 157 (32.2%) 1,610 (37.0%)
35-44 4 (0.2%) 197 (15.3%) 9 (5.2%) 53 (10.9%) 263 (6.0%)
45-54 0 74 (5.7%) 4 (2.3%) 15 (3.1%) 93 (2.1%)
55-64 0 9 (0.7%) 0 6 (1.2%) 15 (0.3%)
65+ 0 3 (0.2%) 1 (0.6%) 0 4 (0.1%)
Marital status**
Married 594 (24.9%) 642 (50.3%) 25 (14.5%) 49 (10.2%) 1,310 (30.3%)
Separated/Divorced/
Widowed
23 (1.0%) 23 (1.8%) 7 (4.0%) 19 (4.0%) 72 (1.7%)
Single 1,771 (74.1%) 612 (47.9%) 141 (81.5%) 413 (85.8%) 2,937 (68.0%)
Highest education**
Elementary 16 (0.7%) 246 (19.0%) 15 (8.6%) 2 (0.4%) 279 (6.4%)
Middle school 10 (0.4%) 323 (25.0%) 53 (30.5%) 9 (1.9%) 395 (9.1%)
Some high school 43 (1.8%) 214 (16.6%) 20 (11.5%) 54 (11.0%) 331 (7.6%)
High school grad 416 (17.3%) 232 (18.0%) 32 (18.4%) 305 (62.1%) 985 (22.6%)
Some college or
technical school
258 (10.7%) 173 (13.4%) 16 (9.2%) 60 (12.2%) 507 (11.6%)
College or technical
school graduate
1,553 (64.5%) 97 (7.5%) 32 (18.4%) 61 (12.4%) 1,743 (39.9%)
Other 111 (4.6%) 7 (.5%) 6 (3.4%) 0 124 (2.8%)
Employment**
Homemaker 118 (5.0%) 320 (24.8%) 14 (8.2%) 9 (1.9%) 461 (10.7%)
Retired 1 (0.1%) 5 (0.4%) 2 (1.2%) 0 8 (0.1%)
Self-employed 246 (10.4%) 88 (6.8%) 8 (4.7%) 44 (9.2%) 386 (9.0%)
Student 46 (2.0%) 349 (27.1%) 95 (55.6%) 145 (30.2%) 635 (14.8%)
Unemployed 1,139 (48.4%) 62 (4.8%) 9 (5.2%) 197 (41.0%) 1,407 (32.8%)
Working for
an employer
801 (34.1%) 465 (36.1%) 43 (25.1%) 85 (17.7%) 1,394 (32.5%)
* p<0.05 ** p<0.01
Table B. Background characteristics of respondents
38 Digital Inclusion Social Impact Evaluation
4.4 Training and Access
As seen in Figure 15, 3,562 people who took the survey (82%) participated in
one or more trainings by OGE or one of its partners between 2009 and 2014.
Most respondents reported participating in a training between 2012 and
2014, largely from respondents in Egypt and India.
The large numbers of people participating in trainings in Egypt and India is
mainly due to three factors: 1) Egypt and India were the largest number of
respondents to the survey, 2) the partners’ large reach in these countries,
and 3) the nature and scale of OGE’s programs in these countries.
In both India and Egypt, OGE works with larger, local NGOs, Pratham and
CEOSS, respectively, both of which have a greater reach than its partners
in Mexico. OGE implemented a large-scale, multi-year project in Egypt,
designed to train at least 1,200 youth. In India, OGE launched its training
curriculum at Pratham’s community technology centers in Pune, training
over 2,900 individuals in 2012, and built the capacity of 10 centers reaching
15 communities in Maharashtra State in 2013-2014 to implement a number
of training programs that served over 4,000 people. Siyafunda, OGE’s partner
in South Africa, also has a large network of community technology centers
located in mostly rural townships in Johannesburg. However, OGE only
works with about 5-8 of their centers per year, as the project is smaller in
scale. Finally, in Mexico, OGE works with three centers in very small, remote
villages, as well as small classes at local high schools in Toluca, in partnership
with UAEM.
Missing training date from 204 respondents.
Figure 15. Training by
country by year
39
In Egypt and India, most respondents (92% and 93% respectively) had
participated in any training by OGE or one of its partners compared to half
the respondents in South Africa (62%) and a quarter in Mexico (29%). The
most common trainings were Career guidance and job hunting (73% of those
who participated in a training) and Community Connectors (66%).
It is important to note that the training programs in each country all focus
on digital literacy, technology adoption, and leadership skills. The leadership
focus varies in each country depending on the target population. In Mexico,
leadership skills taught are core soft and hard skills that can be exercised in
an academic or professional setting, or used to become actively involved in
community development initiatives. In Egypt and South Africa, there is an
emphasis on entrepreneurship and management, and in South Africa, India
and Egypt, on employment. Therefore, there exists great overlap in training
content from country to country, largely due to the fact that digital literacy
is very much connected to and has the impact to improve a number of other,
more specific skill sets pertaining to peoples’ livelihoods.
It can be inferred from these results and based on each country’s unique
program offerings that not every respondent indicated the training they
necessarily participated in. For example, the Mobile Entrepreneurs
Program, only offered in South Africa, consists of modules focused on
entrepreneurship, but also contains modules that teach career development
skills so that entrepreneurs also learn how to get jobs, if the choose, that
allow them to utilize their entrepreneurial skillset. Therefore, some of the
respondents may have said they had received training in career guidance
and other job-related skills if they participated in this training. In Mexico,
respondents who indicated they received career guidance and job training
could represent adults who participated in Community Connector-led
technology workshops at their local centers, which are aimed at parents,
many of which are first-time Internet users, who are interested in learning
how to look and apply for jobs online.
40 Digital Inclusion Social Impact Evaluation
Table C. Background characteristics of respondents
Egypt
(N=2,407)
India
(N=1,292)
Mexico
(N=174)
South Africa
(N=491)
Total
(N=4,364)
Any training**
2,204 (91.6%) 1,205 (93.3%) 51 (29.3%) 306 (62.3%) 3,766 (86.3%)
Career guidance and
job hunting** 2,142 (97.2%) 483 (40.1%) 12 (23.5%) 104 (34.0%) 2,741 (72.8%)
Community Connectors**
1,822 (82.7%) 559 (46.4%) 32 (62.8%) 72 (25.5%) 2,485 (66.0%)
Business**
97 (4.4%) 29 (2.4%) 7 (13.7%) 112 (36.6%) 245 (6.5%)
Health camp**
0 224 (18.6%) 0 0 224 (6.0%)
Digital literacy**
0 166 (13.8%) 1 (2.0%) 7 (2.3%) 174 (4.6%)
Internet security and
IT adoption** 1 (0.1%) 86 (7.1%) 9 (17.7%) 7 (2.3%) 103 (2.7%)
Mobile Entrepreneurship**
10 (0.5%) 0 0 84 (27.5%) 94 (2.5%)
Professional and soft skills**
0 8 (0.6%) 4 (7.8%) 15 (4.9%) 27 (0.7%)
Managers**
4 (0.2%) 0 6 (11.8%) 6 (2.0%) 16 (0.4%)
Other (not specified) 5 (0.2%) 116 (9.6%) 1 (2.0%) 5 (1.7%) 127 (3.4%)
None 220 (9.1%) 95 (7.4%) 123 (70.7%) 191 (38.9%) 629 (14.4%)
* p<0.05	 ** p<0.01
The majority (71%) of respondents in Egypt reported having Internet
access at home; in the other countries Internet access in the home varied
from 14% in India to 41% in Mexico (Table D). In India, respondents most
commonly reported that they accessed the Internet at a CKC (84%) while
in Mexico almost half of the respondents reported accessing the Internet at
home (40%), school (43%), and at a CKC (37%) in addition to many using
telecenters (29%) and work (19%). In South Africa, the majority said they
accessed the Internet at a CKC (67%); while some reported accessing the
Internet at home (20%), school (15%), or at a telecenter (14%).
41
Egypt
(N=2,407)
India
(N=1,292)
Mexico
(N=174)
South Africa
(N=491)
Total
(N=4,364)
Where do you access the Internet?
At home**
1,692 (70.3%) 173 (13.4%) 70 (40.2%) 101 (20.6%) 2,036 (46.7%)
At school**
14 (0.6%) 14 (1.1%) 75 (43.1%) 77 (15.7%) 180 (4.1%)
At work**
171 (7.1%) 112 (8.7%) 33 (19.0%) 36 (7.3%) 352 (8.1%)
At a telecenter**
120 (5.0%) 8 (0.6%) 50 (28.7%) 67 (13.7%) 245 (5.6%)
At a CKC**
457 (19.0%) 1,086 (84.1%) 65 (37.4%) 329 (67.0%) 1,937 (44.4%)
On a mobile phone**
58 (2.4%) 155 (12.0%) 3 (1.7%) 12 (2.4%) 228 (5.2%)
Using a mobile
broadband USB stick** 20 (0.8%) 0 0 0 20 (0.5%)
At a library 0 0 0 1 (0.2%) 1 (0.1%)
Have internet
access at home** 1,714 (71.2%) 183 (14.2%) 72 (41.4%) 138 (28.1%) 2,107 (48.3%)
* p<0.05	 ** p<0.01
Table D. Background characteristics of respondents
4.5 Access Local Content
The high percentage of respondents who have accessed the digital platform
in Egypt and India is reflective of the reach of OGE’s partners in those
countries, as well as the high number of access points. Only 25% of
respondents in South Africa reported ever accessing the digital platform.
This number is somewhat surprising because OGE’s partner in South
Africa also has a significant reach and number of access points, however
this could be due to the small sample size taken from South Africa. If OGE
had surveyed a larger number of respondents from a greater variety of
townships where its access points are located, results might have shown
a higher percentage of people who have accessed the platform. The small
percentage of people who have ever accessed the digital platform in Mexico
is representative of the scale of OGE’s work and the number of access points
(three), which are located in small and rural towns.
42 Digital Inclusion Social Impact Evaluation
Table E. Read about topics on the digital platform
Egypt
(N=2,407)
India
(N=1,292)
Mexico
(N=174)
South Africa
(N=491)
Total
(N=4,364)
Ever accessed the
digital platform** 1,758 (73.0%) 1,114 (86.2%) 44 (25.3%) 124 (25.3%) 3,040 (69.7%)
Topics accessed:
Agriculture**
317 (13.2%) 400 (31.0%) 14 (8.1%) 23 (4.7%) 754 (17.3%)
Citizenship**
109 (4.5%) 165 (12.7%) 10 (5.8%) 26 (5.3%) 310 (7.1%)
Health**
1,046 (43.5%) 998 (77.2%) 26 (14.9%) 47 (9.6%) 2,117 (48.5%)
Education**
1,004 (41.7%) 906 (70.1%) 21 (12.1%) 58 (11.8%) 1,989 (45.6%)
Recommended articles
to other people** 1,495 (85.0%) 325 (29.2%) 9 (17.7%) 74 (59.7%) 1,917 (63.1%)
* p<0.05	 ** p<0.01
Figure 16, which shows the top three content areas by country, demonstrates
that content on the digital platform is tied to programmatic themes, but
also how content that the popularity of content is not always directly related
to training topics and may serve as key resources in supporting peoples’
livelihood development.
For example, while the programmatic focus in OGE’s programs in Egypt
was employment and entrepreneurship, the program served some rural
communities where agribusinesses account for many of the small businesses.
Therefore it is unsurprising that agriculture was a popular topic among
people accessing the Egypt digital platform content. All three of India’s top
three content areas are reflective of training topics and services offered at
partner centers. In this case, the India results correlate to program objectives
focused on supporting people’s improvement in health, education, and
employment. In Mexico, there is no training program focused solely on
health, so the large percentage of people viewing health content could be
indicative of both the demand for and the value of localized information
on the topic. A key section of the Mexico digital platform is the Homework
Help section, which was developed to include links to online educational
resources such as tutorials and activities for over 100 concepts in math,
science, reading and other core subjects for primary, secondary, and high
schools students, aimed at guiding parents in helping their children improve
their academic achievement. According to Google Analytics, more than
a million users have accessed the Homework Help section, which could
contribute to education being among a top content area for Mexico. South
Africa’s program has a strong emphasis on employment and entrepreneurship,
which is also captured by the top content results.
43
All programs emphasize using and recommending others to the digital
platform for the purpose of accessing global best practice information and
community development-focused content on locally relevant issues and
topics. The digital inclusion ecosystem OGE has created with its partners—
access, content, and training—is designed to support word of mouth and
actions taken to increase technology adoption and usage of the digital
platform in the communities it serves.
Respondents were asked if they recommended articles on the digital platform
to others, and if so what topics they recommended. There was wide variation
between countries in both the frequency of recommending articles and
they type of content. Respondents from Egypt were most likely to report
recommending articles on the digital platform (85%), followed by South
Africa (60%) and Mexico (52%). Respondents from India were least likely to
report recommending articles on the digital platform to others. The topics
that were recommended mirrored the top content in each country.
4.6 Take Informed Action
The core of all of OGE’s programs is to help people to be better able to help
themselves by acquiring skills and gaining access to better information
via information technology to make informed decisions about their lives.
We measured this by asking not only what actions people took after they
attended a training or after they accessed content on the digital platform,
but if those actions improved their lives by gaining employment, starting a
business, earning more money, increasing their savings, and improving their
health and/or education.
Figure 16. Top three content
areas by country
Figure 17. Ever recommended
articles and top three topics
recommended on the digital
platform
44 Digital Inclusion Social Impact Evaluation
Figure 18 shows the how often respondents in different countries report they
go online. It is very encouraging that more than half of respondents reported
that they go online at least once each day (orange bars in Figure 19). However,
respondents in India reported going online with much less frequency: only
29% reported going online daily and 43% reported going online only once
every few months (vs. less than 2% of respondents in the other countries). This
speaks to the need for computer centers and other points of access in India and
the benefit of online training and content in the other countries.
Figure 19 shows that only 29% of all respondents use email for personal
communication and 10% use it for business communication. This is a
critically important skill, particularly in the 21st
century economy. It is
possible that so few people use email for business because many of the
respondents are either unemployed or have a job that does not require
sending email. That said, the low numbers of people using email for personal
use are indicative of low levels of digital literacy given that so many of the
respondents reported going online regularly.
It is very important to have an online identity for finding and getting a job
as well as making the types of connections that can lead to professional
advancement. Email is one way to create an online identity and social
Figure 18. How often do you go online?
45
networking is another, which is why it is encouraging that almost two thirds
of respondents reported using online social networks (61%). This represents
a great opportunity to teach people how to use social networking to form ties
to their communities and to advance their professional lives, particularly in
Egypt where unemployment is high, Internet use is very regular, and many
respondents (43%) reported using the Internet to search and apply for jobs.
One reason for this high number is the OGE’s job-matching program where
Egyptians post their skills on an online database and employers post job
openings. Lastly, the data makes it clear that many people use the internet
to try to improve their lives by studying, researching, applying for jobs, and
social networking rather than just for frivolous purposes.
Less than 1% of respondents said they used the internet for downloading
music and watching movies online [data not shown].
Figure 19. How often do you go online?
** p <0.01
46 Digital Inclusion Social Impact Evaluation
4.7 Increased Opportunities in Income
USE IDENTITY TO PARTICIPATE IN THE LOCAL
AND GLOBAL ECONOMY
Over a quarter of the respondents reported having bank accounts and using
online banking, as seen in Table E (27%). However, there was wide variation
across countries: only 2% of respondents in Egypt reported having a bank
account compared to 70% in South Africa. However, of those who had bank
accounts, those from Egypt were much more likely to access their bank
account online (69% of those with a bank account, compared to 16% overall).
Very few respondents reported paying their bills online (5%).
Egypt
(N=2,407)
India
(N=1,292)
Mexico
(N=174)
South Africa
(N=491)
Total
(N=4,364)
Have a bank account**
55 (2.3%) 751 (58.1%) 39 (22.4%) 344 (70.1%) 1,189 (27.3%)
Access bank
account online** 38 (69.1%) 56 (7.5%) 21 (53.9%) 71 (20.6%) 186 (15.6%)
Pay bills online**
74 (3.1%) 68 (5.3%) 25 (14.4%) 31 (6.3%) 198 (4.5%)
* p<0.05	 ** p<0.01
START A BUSINESS
OGE administers business trainings that aim to help people start businesses
and employ best practices to maximize the chances that these businesses will
be successful. In these trainings people learn how to create an operational
budget, apply for loans and grants, and create marketing and business plans
amongst other activities.
Of the 245 respondents who said they had participated in one of OGE’s
business trainings, 46 reported that they started a business after training (19%).
Those who started a business were much more likely to have implemented
business best practices than those that did not, such as creating operational
budgets (57% vs. 19%), a business plan (70% vs. 44%), and a marketing plan
(52% vs. 19%). However, many respondents implemented these best practices
without starting a business. For example, 44% of respondents who didn’t start
a business after the business training stated that they created a business plan.
Since the majority of respondents attended a training within the last year, the
survey may have captured people in the process of starting a business. With
increased time since the training, future results may show a higher proportion
of trainees starting a business. [data not shown]
Table E. Read about topics on the digital platform
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report
Social Impact Evaluation Report

More Related Content

Viewers also liked

Impression Management Strategy final ppt
Impression Management Strategy final pptImpression Management Strategy final ppt
Impression Management Strategy final pptPrajakta Patil
 
AGS Members' Day 2015 - Business Practice Presentation
AGS Members' Day 2015 - Business Practice PresentationAGS Members' Day 2015 - Business Practice Presentation
AGS Members' Day 2015 - Business Practice PresentationForumCourt
 
υτθικ,
υτθικ,υτθικ,
υτθικ,ouran
 
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.com
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.comSeslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.com
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.comEcem Yar
 
How did your horror trailer fit the horror genre?
How did your horror trailer fit the horror genre?How did your horror trailer fit the horror genre?
How did your horror trailer fit the horror genre?Lauzalaar96
 
Demand, Supply and Equilibrium
Demand, Supply and EquilibriumDemand, Supply and Equilibrium
Demand, Supply and EquilibriumKonok Mondal
 
chuyên dịch vụ giúp việc quận 3 ở sài gòn
chuyên dịch vụ giúp việc quận 3 ở sài gònchuyên dịch vụ giúp việc quận 3 ở sài gòn
chuyên dịch vụ giúp việc quận 3 ở sài gònmoira229
 
Bassett/HGTV Decor Signage and Displays
Bassett/HGTV Decor Signage and DisplaysBassett/HGTV Decor Signage and Displays
Bassett/HGTV Decor Signage and DisplaysSteve Brown
 

Viewers also liked (14)

Monsters
MonstersMonsters
Monsters
 
Impression Management Strategy final ppt
Impression Management Strategy final pptImpression Management Strategy final ppt
Impression Management Strategy final ppt
 
AGS Members' Day 2015 - Business Practice Presentation
AGS Members' Day 2015 - Business Practice PresentationAGS Members' Day 2015 - Business Practice Presentation
AGS Members' Day 2015 - Business Practice Presentation
 
Mondo fantastico ste
Mondo fantastico steMondo fantastico ste
Mondo fantastico ste
 
Trademarks in turkey
Trademarks in turkeyTrademarks in turkey
Trademarks in turkey
 
υτθικ,
υτθικ,υτθικ,
υτθικ,
 
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.com
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.comSeslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.com
Seslide filmizle - Sesli Sohbet, Sesli Chat | Seslivaybe.com
 
How did your horror trailer fit the horror genre?
How did your horror trailer fit the horror genre?How did your horror trailer fit the horror genre?
How did your horror trailer fit the horror genre?
 
Demand, Supply and Equilibrium
Demand, Supply and EquilibriumDemand, Supply and Equilibrium
Demand, Supply and Equilibrium
 
chuyên dịch vụ giúp việc quận 3 ở sài gòn
chuyên dịch vụ giúp việc quận 3 ở sài gònchuyên dịch vụ giúp việc quận 3 ở sài gòn
chuyên dịch vụ giúp việc quận 3 ở sài gòn
 
Bassett/HGTV Decor Signage and Displays
Bassett/HGTV Decor Signage and DisplaysBassett/HGTV Decor Signage and Displays
Bassett/HGTV Decor Signage and Displays
 
LA TDT EN COLOMBIA
LA TDT EN COLOMBIALA TDT EN COLOMBIA
LA TDT EN COLOMBIA
 
Curriculum Vitae
Curriculum VitaeCurriculum Vitae
Curriculum Vitae
 
Talent undone
Talent undoneTalent undone
Talent undone
 

Similar to Social Impact Evaluation Report

Team Development Process
Team Development ProcessTeam Development Process
Team Development ProcessCecilia Lucero
 
Creation And Development Of The Program
Creation And Development Of The ProgramCreation And Development Of The Program
Creation And Development Of The ProgramAmber Rodriguez
 
A permit approval system for environmental impact assessment by epa
A permit approval system for environmental impact assessment by epaA permit approval system for environmental impact assessment by epa
A permit approval system for environmental impact assessment by epaKingsley Mensah
 
Social Media Analysis for the EAP Program at SUNY Broome
Social Media Analysis for the EAP Program at SUNY BroomeSocial Media Analysis for the EAP Program at SUNY Broome
Social Media Analysis for the EAP Program at SUNY BroomeMatthew Blum
 
Linking learning to business
Linking learning to businessLinking learning to business
Linking learning to businessLaura Overton
 
Smart Chicago Eliminate the Digital Divide UIC Capstone Report
Smart Chicago Eliminate the Digital Divide UIC Capstone Report Smart Chicago Eliminate the Digital Divide UIC Capstone Report
Smart Chicago Eliminate the Digital Divide UIC Capstone Report Smart Chicago Collaborative
 
Focus Group Discussion on UC Browser Bangladesh TVC
Focus Group Discussion on UC Browser Bangladesh TVCFocus Group Discussion on UC Browser Bangladesh TVC
Focus Group Discussion on UC Browser Bangladesh TVCShagufta Rahman
 
WSB Best Practices Report
WSB Best Practices ReportWSB Best Practices Report
WSB Best Practices ReportTim Pehlke
 
The Top 10 Critical Human Capital Issues of 2014 - i4cp
The Top 10 Critical Human Capital Issues of 2014 - i4cpThe Top 10 Critical Human Capital Issues of 2014 - i4cp
The Top 10 Critical Human Capital Issues of 2014 - i4cpJan van Rooyen
 
2014 Student Information Systems Market Trends Report Sample
2014 Student Information Systems Market Trends Report Sample2014 Student Information Systems Market Trends Report Sample
2014 Student Information Systems Market Trends Report SampleThe Tambellini Group
 
Tagoras Association Learning + Technology Report 2014
Tagoras Association Learning + Technology Report 2014Tagoras Association Learning + Technology Report 2014
Tagoras Association Learning + Technology Report 2014Tagoras
 
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...Sanjan Haque
 
Creating a baseline for Corporate CSR Spend on Global Education Initiatives
Creating a baseline for Corporate CSR Spend on Global Education InitiativesCreating a baseline for Corporate CSR Spend on Global Education Initiatives
Creating a baseline for Corporate CSR Spend on Global Education InitiativesSustainable Brands
 
c PJM6610 Foundations of Project Business Analysis.docx
c        PJM6610 Foundations of Project Business Analysis.docxc        PJM6610 Foundations of Project Business Analysis.docx
c PJM6610 Foundations of Project Business Analysis.docxbartholomeocoombs
 
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS Final
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS FinalSOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS Final
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS FinalBrendon Spiteri
 
Social business-roadmap2011
Social business-roadmap2011Social business-roadmap2011
Social business-roadmap2011Claude Super
 
2015PGP083_Kumar Gaurav
2015PGP083_Kumar Gaurav2015PGP083_Kumar Gaurav
2015PGP083_Kumar GauravKumar Gaurav
 

Similar to Social Impact Evaluation Report (20)

Team Development Process
Team Development ProcessTeam Development Process
Team Development Process
 
Creation And Development Of The Program
Creation And Development Of The ProgramCreation And Development Of The Program
Creation And Development Of The Program
 
A permit approval system for environmental impact assessment by epa
A permit approval system for environmental impact assessment by epaA permit approval system for environmental impact assessment by epa
A permit approval system for environmental impact assessment by epa
 
Social Media Analysis for the EAP Program at SUNY Broome
Social Media Analysis for the EAP Program at SUNY BroomeSocial Media Analysis for the EAP Program at SUNY Broome
Social Media Analysis for the EAP Program at SUNY Broome
 
Linking learning to business
Linking learning to businessLinking learning to business
Linking learning to business
 
Joint Learning Update
Joint Learning UpdateJoint Learning Update
Joint Learning Update
 
PAR Guideline 2015
PAR Guideline 2015PAR Guideline 2015
PAR Guideline 2015
 
Smart Chicago Eliminate the Digital Divide UIC Capstone Report
Smart Chicago Eliminate the Digital Divide UIC Capstone Report Smart Chicago Eliminate the Digital Divide UIC Capstone Report
Smart Chicago Eliminate the Digital Divide UIC Capstone Report
 
Focus Group Discussion on UC Browser Bangladesh TVC
Focus Group Discussion on UC Browser Bangladesh TVCFocus Group Discussion on UC Browser Bangladesh TVC
Focus Group Discussion on UC Browser Bangladesh TVC
 
WSB Best Practices Report
WSB Best Practices ReportWSB Best Practices Report
WSB Best Practices Report
 
The Top 10 Critical Human Capital Issues of 2014 - i4cp
The Top 10 Critical Human Capital Issues of 2014 - i4cpThe Top 10 Critical Human Capital Issues of 2014 - i4cp
The Top 10 Critical Human Capital Issues of 2014 - i4cp
 
2014 Student Information Systems Market Trends Report Sample
2014 Student Information Systems Market Trends Report Sample2014 Student Information Systems Market Trends Report Sample
2014 Student Information Systems Market Trends Report Sample
 
Tagoras Association Learning + Technology Report 2014
Tagoras Association Learning + Technology Report 2014Tagoras Association Learning + Technology Report 2014
Tagoras Association Learning + Technology Report 2014
 
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...
Analysing Oxfam Viet Nam’s Participatory Poverty Mapping Analysis project usi...
 
Creating a baseline for Corporate CSR Spend on Global Education Initiatives
Creating a baseline for Corporate CSR Spend on Global Education InitiativesCreating a baseline for Corporate CSR Spend on Global Education Initiatives
Creating a baseline for Corporate CSR Spend on Global Education Initiatives
 
c PJM6610 Foundations of Project Business Analysis.docx
c        PJM6610 Foundations of Project Business Analysis.docxc        PJM6610 Foundations of Project Business Analysis.docx
c PJM6610 Foundations of Project Business Analysis.docx
 
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS Final
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS FinalSOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS Final
SOCIAL MEDIA AND HOW IT IS AFFECTING HUMAN RELATIONS Final
 
Social business-roadmap2011
Social business-roadmap2011Social business-roadmap2011
Social business-roadmap2011
 
AIIM Social Business Roadmap
AIIM Social Business RoadmapAIIM Social Business Roadmap
AIIM Social Business Roadmap
 
2015PGP083_Kumar Gaurav
2015PGP083_Kumar Gaurav2015PGP083_Kumar Gaurav
2015PGP083_Kumar Gaurav
 

Social Impact Evaluation Report

  • 1. 1 By One Global Economy Moustafa Mourad Alyssa Perez Colin Richardson and Martine Holston Final Report | October 2014 Digital Inclusion Social Impact Evaluation
  • 2. Acknowledgements One Global Economy would like to express its gratitude to the Symantec Corporation for its continued support for OGE and the grant that has allowed us to undertake this evaluation in particular. For a number of years Symantec has supported OGE’s mission of serving low income communities in many countries including India and Mexico, and most recently, featured One Global Economy in its online brand campaign.
  • 3. Table of Contents Acknowledgements............................................................... i Executive Summary...............................................................1 1. Background..................................................................... 3 1.1 Goal and Objectives of the Impact Evaluation............................ 3 1.2 The Information-Based Ecosystem Strategy.............................. 4 1.3 Internet Access Points................................................... 5 1.4 Training................................................................. 6 Digital Literacy................................................................. 6 Entrepreneurship and Technology....................................... 6 Community Connectors..................................................7 Internet Safety...........................................................7 1.5 Local Content.................................................................. 8 1.6 Country Programs............................................................. 9 South Africa................................................................... 9 Egypt.........................................................................10 India.......................................................................... 11 Mexico........................................................................ 12 2. Evaluation Design and Methodology.......................................... 13 2.1 Goal and Objectives of the Impact Evaluation........................... 13 2.2 Developing the Evaluation Design and Framework......................... 14 Theory of Change............................................................ 15 Results Chain Model......................................................... 17 Success Formula.............................................................. 18 Content for Impact: Skill Categories......................................... 19 Indicators.....................................................................20 2.3 Evaluation Design and Framework.......................................... 21 Program Monitoring......................................................... 21 Outcome Evaluation......................................................... 22 Data Collection Tools........................................................ 22 2.4 The Evaluation Toolkit........................................................23 Digital Inclusion Social Impact Evaluation
  • 4. 3. Implementing the Evaluation....................................................25 3.1 Results Chain Model and Success Formula..................................26 3.2 End-User Survey.............................................................. 27 Sampling Strategy and Location............................................. 27 Data Collection............................................................... 27 Data Entry, Management and Analysis.....................................28 3.3 Routine Reporting............................................................29 4. Initial Evaluation Results.......................................................... 31 4.1 Google Analytics............................................................. 31 Total and Unique Views...................................................... 31 Average Number of Page Views and Average Time Spent on The Platform.................................. 31 Top Content..................................................................33 4.2 Survey Participants...........................................................35 4.3 Background Characteristics of Respondents................................35 4.4 Training and Access..........................................................38 4.5 Access Local Content........................................................ 41 4.6 Take Informed Action........................................................43 4.7 Increased Opportunities In Income.........................................46 Use Identity to Participate in the Local and Global Economy..............46 Start a Business...............................................................46 Career Advancement and Employment..................................... 47 4.8 Increased Opportunities in Health, Education, and Citizenship............50 Increased Opportunities in Health.......................................... 51 Increased Opportunities in Education....................................... 52 Increased Opportunities in Social Development............................53 4.9 Transform Communities.....................................................55 5. Conclusions and Lessons Learned............................................... 57 6. The Road Forward................................................................... 61 Appendix A: Data Collection Methods Along the Results Chain...................63 Appendix B: Impact Evaluation Toolkit................................................64 Appendix C: Top Five Articles by Country by Year...................................65 Evaluation Dictionary..................................................................... 67 Mission Definitions.................................................................68 References................................................................................. 71
  • 5. 1 One Global Economy began its Impact Evaluation program a year ago after receiving a grant from the Symantec Corporation to evaluate existing programs in Egypt, India, Mexico, and South Africa, and to build its internal capacity to continue to monitor and evaluate future programs. Over the course of the program, One Global Economy (OGE) has learned a great deal about the processes of monitoring and evaluation, and about the people and the communities that OGE serves. The program began with OGE developing its theory of change, or how its programs improve the lives of the people it aims to serve. This helped clarify and determine the specific social impact metrics to be measured through the evaluation. The next step in the process was the design of a community survey to be administered in all four countries in Arabic, Hindi, Spanish, and English. OGE received 4,364 completed surveys, which provided a trove of data about people who had either attended training delivered by OGE or had accessed its online content on one of its digital platforms. Consequently, OGE was able to learn more about the people it serves and how they use and view the content and training. OGE focuses on employment as a solid indicator of joining the economic mainstream. Therefore, all of OGE’s trainings start with teaching basic digital literacy so that participants will be able to apply those skills to both the job search as well as increasing their employability. OGE’s findings about employment were very encouraging. 86% of survey respondents did not have a job before they participated in a training, but of those, 35% were employed by the time the evaluation process had begun. Half of the respondents, who reported they were employed, had found their job online. Google Analytics has allowed OGE to see which pages of its content are the most trafficked, how many pages people visit, how long users stay on a page, and other hard data. Through combining the Google analytics data with the data gathered from the community survey, OGE gained a much more detailed and nuanced understanding of the impact of its online content and how users adopt global best practices to improve their health, education, and social development. 35% of previously unemployed survey respondents were employed by the time the evaluation process had begun EXCUTIVE SUMMARY
  • 6. 2 Digital Inclusion Social Impact Evaluation The survey data showed that 99% of respondents, who accessed content on OGE’s digital platforms, took some kind of action. The most popular content dovetailed with the feedback from the stakeholders’ engagement process that guided the content creation to develop the platform in each country. Employment and job-seeking skills were most popular in Egypt, health and sanitary practices in India, citizenship in South Africa, and job skills in Mexico. Another significant finding related to how people use OGE’s content and training revealed that 86% of respondents from Egypt and 60% of respondents from South Africa reported teaching other people in their communities about what they learned from the digital platform or in an OGE training. These results indicate that OGE’s content and training have a multiplicative effect on the communities it serves. The Impact Evaluation Program has bolstered OGE’s capacity to continually monitor its programs. OGE has worked with its partner organizations to help them develop performance indicators for programs delivered together. Additionally, OGE has created a monthly reporting tool and a training reporting tool that its partner organizations and local managers will use to provide regular feedback to OGE about the progress of its programs, express what needs improvement, and to share lessons they have learned. In developing an evaluation methodology and a host of tools, OGE has gained both a deeper understanding about the social impact its programs generate as well as developed a greater capacity in monitoring them. OGE has already begun applying lessons learned from this program. Firstly, OGE is implementing the regular reporting. Secondly, OGE will continue to deliver programs that proved successful. Lastly, OGE has identified content areas that it would like to develop and improve. OGE looks forward to implementing the evaluation methodology to measure the impact of future programs as well as refining programs based on the results of the evaluation. 99% of survey respondents who accessed content on OGE’s digital platforms, took some kind of action 86% of survey respondents from Egypt reported teaching other people in their communities about what they learned from the digital platform or in an OGE training
  • 7. 3 1.1 Goal and Objectives of the Impact Evaluation One Global Economy (OGE) works with under-served communities in over 15 countries across five continents to create and implement information-based development strategies to help households and individuals raise their standard of living, join the economic mainstream, and innovate local solutions. OGE has a diverse portfolio of programs, each suited to the unique community it works in through a process of local stakeholder meetings and collaboration with the community and local non-governmental organizations (NGOs). These programs focus on access to broadband Internet connections, creating relevant digital content in local languages, and training people in areas of digital literacy, entrepreneurship, and techniques to gain the skills necessary to gain employment and upward mobility. Since its founding in 2005, OGE has launched 17 international community development websites. These digital platforms (Note: Words in bold orange font are defined in the Evaluation Dictionary) contain vital community development content on topics such as education, employment, entrepreneurship, and health written in local languages for readers at a 4th- grade reading level. The content housed on the digital platforms is composed of both global best practices and locally generated articles that focus on the needs of the community for which it is written. OGE’s digital platforms now reach more than 2.5 million unique visitors per year and since its founding in 2005, 8 million people have benefited from One Global Economy’s empowering community development information. In addition, OGE has trained thousands of people on digital literacy and assisted in the opening of Internet access points to enable people to have the ability to access the local content on the digital platforms. The purpose of this Impact Evaluation Program was to assess the long-term impact of OGE’s existing programs on mission-driven outcomes and to build OGE’s capacity to continually monitor and evaluate its programs going forward. This is particularly important because OGE’s programs serve very different communities, address a wide variety of issues, and do so across diverse levels of development and urbanization. BACKGROUND1 DEFINITION Digital Platform International community development websites that contain vital community development content on topics such as education, employment, entrepreneurship, and health written in local languages for readers at a 4th-grade reading level. The content housed on the digital platforms is composed of both global best practices and locally-generated articles that focus on the needs of the community for which it is written. DEFINITION Digital Literacy The ability to find, evaluate, utilize, share, and create content using information technologies and the Internet (Cornell University); the ability to perform tasks effectively in a digital environment. Literacy includes the ability to read and interpret media (text, sound, images), to reproduce data and images through digital manipulation, and to evaluate and apply new knowledge gained from digital environments (Jones-Kavalier and Flannigan).
  • 8. 4 Digital Inclusion Social Impact Evaluation This impact evaluation focused on OGE’s programs in South Africa, Mexico, Egypt, and India to design and implement its methodology. There are common aspects of OGE’s work in all of those countries, but each country had different programs, content, and training curricula. The framework and tools created as part of this evaluation are meant to provide other organizations working in digital literacy and inclusion with tools to adapt to measure their impact and improve programming. 1.2 The Information-based Ecosystem Strategy OGE’s digital inclusion model addresses three barriers to participation in the information economy: lack of access to internet services, lack of access to actionable local content in local languages, and lack of digital literacy skills through what it calls the Information-based Ecosystem Strategy (see Figure 1). LACK OF ACCESS TO THE INTERNET In the information age, access to the Internet is vital to participate in the global economy. Without access to the Internet, the information available online will not reach underserved communities. OGE partners with community technology centers around the world that serve rural and low-income communities, and offer access to the Internet for free, or at a low cost that is considered affordable by the local residents. Through its partnerships with these local organizations, OGE creates a network of access points, where the digital platform is hosted in each country. LACK OF ACTIONABLE CONTENT IN LOCAL LANGUAGES There is a disparity between global language populations and content available online in those language. For example, in 2012, “Arabic content on the web represented just 3 percent of the total digital content online, yet Arabic speakers make up more than 5 percent of the global population” (Franceschi Bicchierai 2012). Online content in local languages that are not as widely spoken such as Kinyarwanda and Swahili account for even less, at below 1 percent. Consequently, digital content focused on developmental themes would account for even less, making it challenging for individuals in developing countries to access information in their language that is aimed at helping them to improve their standard of living. With more people accessing the Internet in developing countries through their mobile devices and local telecenters, there is increased potential for online information to contribute to the improvement of development outcomes in the areas of income, health, and social development.
  • 9. 5 Beehive content on the digital platform is designed to address community development priorities identified by local stakeholders through a highly participatory engagement and consensus building approach. The content combines global best practices with highly localized information, written in the local language for readers with low literacy levels. LACK OF DIGITAL LITERACY SKILLS Without the skills necessary to access information on the Internet, people will not utilize access points and cannot find the actionable content available in their language. OGE provides training to partner computer center managers on ways they can support digital literacy in their community, launch demand- driven programs and services that address local needs, and teach users how to navigate the Beehive as well as access other existing online resources that provide global best practice information. OGE’s three-pronged strategy has successfully helped to create an ecosystem that has enabled millions in underserved communities to access high quality, actionable content on developmental themes. 1.3 Internet Access Points OGE works with in-country partner organizations to utilize their existing Internet access points as places to deliver trainings, hold stakeholder engagement meetings, and teach the community about its digital platforms. OGE helps to build the capacity of the local access point managers by drawing on its experience and best practices to help the centers to be more organized and to run more smoothly. Figure 1. Information-based Ecosystem for Digital Inclusion
  • 10. 6 Digital Inclusion Social Impact Evaluation 1.4 Training DIGITAL LITERACY Digital literacy is critical to each person’s success in the 21st century and to maximizing the benefit of OGE’s programs, which is why it is the bedrock of OGE’s training programs. “Digital literacy is the ability to find, evaluate, utilize, share, and create content using information technologies and the Internet” (Cornell University). It is the ability to perform tasks effectively in a digital environment. “Literacy includes the ability to read and interpret media (text, sound, images), to reproduce data and images through digital manipulation, and to evaluate and apply new knowledge gained from digital environments” (Jones-Kavalier and Flannigan). Email, social networking, word processing, and using the Internet are necessary skills for anyone seeking gainful employment, a better job, or to expand his or her business. OGE teaches these and more advanced skills in many workshops. Using email and being able to navigate the internet open up a huge amount of opportunities to save money, take advantage of government programs, educate oneself, and make the kind of informed decisions that allow people to improve their own lives. OGE firmly believes that access, content and training are complementary components of digital literacy, in that they work together to improve and expand livelihood opportunities. Without training on how to use technology properly, and training on how it can be incorporated into daily life, and applied to finding information and solutions for development challenges, access would not play such a transformative role. Similarly, access does not benefit underserved populations if there does not exist content that meets them where they are in terms of their language, education level, and local context. This is why all of OGE’s training programs include digital literacy. ENTREPRENEURSHIP AND TECHNOLOGY Entrepreneurship is fundamental to growing today’s developing economies and technology and globalization allow for unprecedented entrepreneurial opportunities. OGE focuses on equipping emerging entrepreneurs with the digital and mobile skills necessary to enter the formal business world and confront everyday business challenges. Empowering entrepreneurs with core soft and hard skills builds their potential to start and grow their enterprises, and contributes to small business creation at the community-level. OGE’s Information-based Ecosystem Strategy supports and strengthens its entrepreneurship-focused programming by leveraging access, content, and training to build the capacity of aspiring entrepreneurs. The entrepreneurial training teaches basic technology skills to encourage technology skills application as a valuable tool for small business growth. OGE guides the development of essential business planning materials through providing templates and instructions for creating a business plan, budget, and marketing plan amongst others. OGE also provides online tools, localized information on entrepreneurship,
  • 11. 7 and mentoring support though mobile technology to entrepreneurs to help them grow their existing businesses. Further, OGE helps small businesses access and use banking services by teaching basic financial literacy to help small business owners take out loans and to use government programs and financial institutions that will help keep their businesses stable and profitable. One such training program is OGE’s Mobile Entrepreneur’s Program (MEP), which emphasizes the use of digital and mobile tools to grow and improve small businesses. Workshops teach and monitor the achievement of a series of online skills such as establishing an online presence for a business, incorporating the use of social media into marketing plans, creating a LinkedIn profile for online professional networking, monitoring business bank accounts online and through SMS alerts, and conducting online research to increase business knowledge and to access government or private funding, or additional support through local organizations and services. Consists of modules focused on similar career development themes and skills, but with a focus on marketing individuals’ entrepreneurial skill sets to obtain jobs in a related field. COMMUNITY CONNECTORS The Community Connectors curriculum in digital literacy and leadership is focused on teaching young people how to apply digital and mobile tools to become actively involved in community development initiatives, leaders in their communities and technology ambassadors for their peers and families. The program teaches a combination of information and communications technology (ICT) skills and soft skills (planning and management, time management, project management, public speaking, self-confidence, etc). The program varies in each country and consists of additional modules addressing entrepreneurship, career development, Internet safety etc., and can focus on helping youth improve academic performance or find jobs through technology skills application depending on the target population (See Table A, OGE Training Programs). Community Connectors graduates have applied skills learned to lead workshops at their community centers to teach community members technology skills, conduct research on community and economic development topics to map out projects their community may implement, develop local businesses, and identify income-generating opportunities. The program fosters youth empowerment and skills-building for personal, professional, and social development. INTERNET SAFETY Internet safety has been incorporated as a focus in training in Mexico and India, where OGE has a partnership with Symantec, and has carried out projects promoting the involvement of locally-based Symantec employee volunteers. Local employees and engineers have worked with OGE and its partners to develop curricula for workshops and Internet safety campaigns focused on teaching first-time Internet users how to protect their identities online, how to protect their computers from viruses, and how to identify common threats such as information theft and fraud. Symantec has also contributed content on online safety, which is published on each digital platform. DEFINITION Mobile Entrepreneur’s Program (MEP) The Mobile Entrepreneurs Program (MEP) emphasizes the use of digital and mobile tools to grow and improve small businesses. Workshops teach and monitor the achievement of a series of online skills such as establishing an online presence for a business, incorporating the use of social media into marketing plans, creating a LinkedIn profile for online professional networking, monitoring business bank accounts online and through SMS alerts, and conducting online research to increase business knowledge and to access government or private funding, or additional support through local organizations and services. Consists of modules focused on similar career development themes and skills, but with a focus on marketing individuals’ entrepreneurial skill sets to obtain jobs in a related field.
  • 12. 8 Digital Inclusion Social Impact Evaluation Community Connectors Community Technology Center Management Mobile Entrepreneurs Program Internet Safety South Africa Digital Literacy, Leadership, Community Development Digital Literacy & Business Management for Community Computer Centers Digital & Mobile Literacy for Entrepreneurship, Business Planning, Marketing N/A Mexico Digital Literacy, Leadership, Entrepreneurship, Internet Safety, Community Development Digital Literacy & Business Management for Community Computer Centers N/A Internet & Personal ID Safety India Digital Literacy, Leadership, Internet Safety, Job-Searching & Career Development/ Professional Skills Digital Literacy & Business Management for Community Computer Centers N/A Internet & Personal ID Safety Egypt Digital Literacy, Leadership, Entrepreneurship, Business, Job-Searching & Career Development/Professional Skills N/A N/A N/A 1.5 Local Content One Global Economy develops digital platforms written in local languages that are accessible to people with a 4th grade reading level. OGE’s community development-focused content is a combination of global best practices and highly localized information. Best practice information covers topics that are relevant all over the world, such as education, employment, health, and money, and focus on teaching people specific best practices such as how to write a business plan, how to select a school for their child, and basic prenatal care. Highly localized content complements these global best practices by providing information on how and where to access local resources, services, and government programs that benefit underserved populations and help them to take action to improve their lives in these areas. Content is produced by local editors, NGOs , and student journalists. The process begins with a stakeholders engagement workshop, where a roadmap is created to guide the development of the digital platform, and then the content is created over a period of 4-6 months. Each digital platform is launched as a public website once there is content about all community development topics identified in the workshop. Digital platforms are launched publicly with at least 200 highly localized articles. New content is added on specific topics as OGE develops more partnerships with local NGOs. OGE expands upon content on specific focus areas to complement its training programs. The digital platform is hosted, maintained and checked for quality of content by OGE. Table A. OGE Training Programs
  • 13. 9 1.6 Country Programs SOUTH AFRICA OGE has worked in South Africa since 2007 when it established the Durban Centre, a community technology center hosted at the Austerville Public Library. The Durban Centre serves residents of South Durban and the Austerville community, and offers free access to the Internet and the South Africa digital platform, known as a “Beehive”, as well as hosts workshops in digital literacy for youth and adults. In (2008, OGE expanded its work in South Africa through a 5-year commitment to the Clinton Global Initiative (CGI) with partners, Cisco, Siyafunda, and Appleseeds Academy. Through the CGI, the partners established a network of Community Knowledge Centers (CKCs) across Sub-Saharan Africa (in Kenya, South Africa, Rwanda, Ethiopia, Nigeria, and Uganda). In South Africa, the partners trained Siyafunda community telecentre managers in digital literacy and business management skills essential to running sustainable computer centers that effectively serve their surrounding communities by building upon their basic offerings to offer demand-driven programs. The partners also developed and launched Community Connectors, a youth technology, leadership, and entrepreneurship training program and trained partner centers in the implementation and delivery of the program to local youth. In addition to the South Africa digital platform, OGE launched portals localized to Durban, Johannesburg, and Cape Town. Over the last five years, OGE has continued to work in partnership with Siyafunda to deliver its Mobile Entrepreneurs Program, which has supported over 1,500 South African street vendors, aspiring entrepreneurs, and small business owners through training and skills building in technology and entrepreneurship skills. The training is complemented with text messages to reinforce learning and disseminate helpful information, tips, and advice for growing and improving enterprises. PROGRAM SNAPSHOT South Africa Year Started: 2007 Key Activities: Establishing a network of Community Knowledge Centers (CKCs), Community Connectors Program, Mobile Entrepreneurs Program, and Beehives for South Africa, Johannesburg, Cape Town, and Durban People Trained: Over 2,500 Access Points Opened: Over 80 Focus Areas: Entrepreneurship and youth leadership DEFINITION Beehive The name of OGE’s digital platform in South Africa, Egypt, and India.
  • 14. 10 Digital Inclusion Social Impact Evaluation EGYPT OGE’s work in Egypt has a particular emphasis on employment and entrepreneurship, where it has been working with local partner, CEOSS (Coptic Evangelical Organization for Social Services) and HANDS (Hands Along the Nile), based in Alexandria, Virginia, since 2011 to connect young people to livelihood opportunities through Asset Mapping, access to global best practices and highly localized content on the Egypt digital platform (known as the Beehive), and training in digital literacy, entrepreneurship, and career development skills. Through its Employment Through Information Technology program, OGE has mapped the skills of approximately 23,000 young men and women along with the outsourcing and employment opportunities of over 5,000 employers to support opportunity identification and job matching through online and mobile platforms. OGE and CEOSS have trained over 1,200 young people in marginalized communities throughout Cairo, Minya, and Beni-Suef at 10 local computer centers established through the program. OGE launched the Egypt Beehive in addition to two localized portals created for Cairo and Minya, which feature global best practice information and community development-focused content in Arabic. The Beehives have served over a million individuals since their launch in early 2013. PROGRAM SNAPSHOT Egypt Year Started: 2011 Key Activities: Asset Mapping, Online & Mobile Job-Matching System, Egypt, Cairo & Minya Beehives, Community Connectors People Trained: Over 1,200 Access Points Opened: 10 (in Cairo, Minya & Beni-Suef) Focus Areas: Employment and Entrepreneurship
  • 15. 11 PROGRAM SNAPSHOT India Year Started: 2010 Key Activities: Beehives for India, Mumbai, and Pune in English, Hindi, and Marathi; working with Pratham Infotech Foundation; training programs; education programs People Trained: 765 Access Points Opened: 10 Focus Areas: Jobs, health, digital literacy INDIA OGE’s India Digital Inclusion Program focused on integrating the core components of local internet access, localized online content, and digital literacy and professional skills training into new programming and services at 10 computer centers run by OGE’s local partner, Pratham InfoTech Foundation (PIF) in Maharashtra. OGE and (PIF) trained over 4,000 people in digital literacy, technology, leadership, entrepreneurship, and community service, as well as working with local health professionals to educate 488 people—most of whom are young women—about health and sanitation. The India Digital Inclusion Program has a particular emphasis on employment. The program offers coaching services to help participants begin their employment search, practice interviewing, job placement, and inclusion in an online employment database. Many participants have learned how to create and/or access email accounts and to use them for job-seeking and professional purposes. The third aspect of this program is OGE’s digital platform known as the Beehive (www.india.thebeehive.org). Viewed by over 200,000 people in the last year, the Indian Beehive is composed of articles about financial literacy, employment, and health written in English, Hindi, and Marathi for an audience with a 4th- grade reading level.
  • 16. 12 Digital Inclusion Social Impact Evaluation MEXICO OGE launched the Mexico digital platform, InfoFácil.mx, in 2010 and the Toluca digital platform (Toluca.InfoFácil.mx) in 2013. OGE has been working with University partner, Universidad Autónoma del Estado de México (UAEM) in Toluca and its journalism department to engage students in the development of highly localized content for the two platforms by providing them with small stipends, community service and internship credit in addition to incorporating content creation into curricula, and offering the Mexico and Toluca digital platforms as an online publishing outlet. Through engaging student journalists, over 500 articles were produced for the Mexico platforms over a period of three years/ to date. Since their launch, more than 3.3 million unique visitors are accessing the digital platforms in Mexico, with over 300,000 accessing it via their mobile phones. OGE also launched its Community Connectors program in three rural villages in Guanajuato (El Garbanzo, San Agustin, and Victoria), where it works with local partner, Choice Humanitarian, and University partner, Universidad Technológica del Norte del Estado de Guanajuato, to support the development of sustainable community technology centers. With the past year, OGE also implemented the youth program at three semi-urban high schools in Toluca, in partnership with UAEM student trainers, training a total of approximately 150 youth in Mexico. In 2013, OGE delivered educational workshops in Internet safety for first-time Internet users to over 500 Toluca residents together with student trainers and locally based volunteer Symantec engineers. PROGRAM SNAPSHOT Mexico Year Started: 2010 Key Activities: Beehives for Mexico & Toluca, Student Journalist Beehive Editors Program, Community Technology Center Management Training, Community Connectors, Internet Security Campaign People Trained: Over 200 Access Points Opened: 4 Focus Areas: Health, Employment, Education, Internet Safety, Youth Leadership
  • 17. 13 There is a heightened awareness among the science and technology community regarding the social impact potential of connectivity and economic inclusion through digital and mobile tools. Academics and think tanks report on the positive effects of access to the Internet, yet there is a need for real-life analysis of the effects digital inclusion has on improving the lives of those in poverty. OGE developed this Impact Evaluation to address this need and to quantify its social impact potential and results, gather data that indicates lasting social value of its digital inclusion programs, and track progress towards its defined social impact targets. In this Impact Evaluation, OGE sought to provide a guide for a rigorous analysis of the “lift,” or margin of improvement, of its digital inclusion services upon low-income populations in developing countries who are new to the Internet and lack skills necessary to utilize the Internet to improve their lives. This evaluation examines the lift in terms of the economic and human development effects on individuals who participate in the digital online platform as users, in the Community Connectors digital literacy training programs, and the patrons of OGE’s partner Community Knowledge Centers. 2.1 Goal and Objectives of the Impact Evaluation Through this Impact Evaluation, OGE sought to track the impact of the digital inclusion program on participants’ economic opportunities and quality of life, as expressed in terms of human development indicators: access to opportunities that increase their income, education, and improve their health. In developing a methodology to assess the impact of OGE’s training and programs, this Impact Evaluation aimed to create a standardized way of evaluating impact in digital literacy programs for the benefit of other organizations doing similar work. Digital inclusion programs pose unique challenges in designing and implementing an impact evaluation. By nature, these programs are not place-based interventions; while the topics are localized geographically the content can be accessed globally. Unlike traditional development work, OGE’s programs are not defined in geographic terms and the organization is not focused on working in specific communities for extended periods of time. Therefore, this evaluation required the development EVALUATION DESIGN & METHODOLOGY2 OGE’s Mission One Global Economy works with under-served communities to create and implement information-based development strategies to help households and individuals raise their standard of living, join the economic mainstream, and innovate local solutions
  • 18. 14 Digital Inclusion Social Impact Evaluation of new methods and tools to assess the impact of OGE’s programs and of digital inclusion programs in general. Thus, one of the main goals of this evaluation was to create a toolkit of adaptable tools and frameworks for other organizations to implement evaluations in this expanding niche. This organizational capacity-building project also served to strengthen focus and codify an outcomes-based approach for implementation across all OGE country programs in the future. THE OBJECTIVES OF THE IMPACT EVALUATION WERE TO: • Articulate a theory of change of how OGE’s training and programs lead to mission-driven outcomes • Develop key qualitative and quantitative performance and impact indicators • Create a comprehensive strategy to gather data for the key indicators, including routine programmatic reporting and follow-up with participants – Implement a follow-up survey in partner computer centers and online in four actively-funded countries (Egypt, India, Mexico and South Africa) – Develop internal tools for more efficient communications regarding progress toward impact metrics • Create a toolkit with adaptable templates that provides a comprehensive guide to implementing an impact evaluation for digital inclusion • Analyze the process and outcome of this initial impact evaluation with attention to improving programming, outcomes, and refining the evaluation methodology • Build internal capacity to monitor and evaluate the impact of Digital Inclusion programs for future projects • Share the findings, results, and conclusions publicly 2.2 Developing the Evaluation Design & Framework The process of developing the evaluation design and framework included: • Creating a theory of change to articulate how OGE achieves its impact. This causal logic of how OGE’s programs deliver results is the underpinning of the impact evaluation and was used to create its framework, indicators, and data collection methods.
  • 19. 15 • Adapting two program planning tools, the Results Chain Model and the Success Formula, to further map out the inputs, strategies, and outputs that lead to OGE’s hypothesized impact. These two exercises identified the process that leads to impact as well as what the impact looks like. • Creating a comprehensive list of indicators using these two tools to ensure all aspects of programs were measured as part of the evaluation. Each step of the Results Chain was included, as well as a clear definition of all the skill and content categories identified through the Success Formula process. The comprehensive list of indicators was distilled into a list of key indicators, tested in this evaluation and to be used as a menu of programmatic indicators for future planning. • Using the list of indicators to develop data collection tools and systems to measure each of the key indicators identified in this process. THEORY OF CHANGE OGE began the impact evaluation with the question: What effect does digital literacy and access to the Internet have on social and economic development of individuals and communities? To answer this question, the first step of the evaluation was to hypothesize how OGE achieves its impact – the organization’s theory of change. The evaluation team deconstructed the organization’s mission by creating internal definitions of each piece of the mission (see Evaluation Dictionary: Mission Definitions) to articulate the organization’s theory of change: Strengthening access to digital and mobile tools by partnering with local computer centers, training computer center managers and patrons in digital literacy and business skills and providing access to locally-written, actionable content on developmental themes in their own languages will enable households and individuals to take informed action that will transform their lives and their communities. Figure 2. Evaluation Methodology Development
  • 20. 16 Digital Inclusion Social Impact Evaluation More specifically, the OGE training curricula combined with both locally generated content and global best practices in key issue areas provides users with the skills and information they need to take significant steps towards participating in both local and global economies. Through digital inclusion, users and their communities can raise their standard of living and join the economic mainstream by increasing their economic, health, and educational opportunities, such as starting a small business, improving their livelihood prospects, improving their ability to make informed health, education, and business decisions; and increasing their participation in the local community. A key element of OGE’s Theory of Change is that the individuals in the communities OGE serves must take action to complete the cycle of change. While OGE fosters the information ecosystem with local partners and creates and shares opportunities for people to take action to improve their lives, OGE believes firmly that the sustainable solution is one in which local participants are the key actors, leading local progress. Figure 3. One Global Economy’s Theory of Change
  • 21. 17 RESULTS CHAIN MODEL The evaluation team then applied the theory of change to two program planning tools to further articulate how the organization achieves its impact. Through clearly defining and articulating the mechanisms and processes that lead to mission-driven outcomes, the team sought to create a comprehensive evaluation model that encompasses all elements of OGE’s programming to best assess and estimate the “lift” and a set of program planning tools adapted to OGE’s theory of change that could be used for future programs. Thus, these tools were used both to help guide the evaluation by looking retrospectively at OGE’s program models and as an exercise to develop planning tools that would be used in all future programs and adapted for other organizations and included in the Evaluation Toolkit. First the team used the World Bank’s results chain model found in the Impact Evaluation in Practice. A results chain shows cause and effect: how the sequence of program inputs (e.g. staff and other resources) and activities lead to specific program outputs, and how these outputs are directly related to accomplishment of program goals (i.e. impact). This model shows how specific program activities directly lead to the desired program goal and what must be done in order to achieve programmatic targets and results. It also documents what goes into a program (the inputs), which is integral for replicability and scalability of programs. The evaluation team created an overall results chain model for the organization as well as adaptations for country-level programs. We used this structure to succinctly visualize and describe each of OGE’s programmatic activities, including their anticipated proximate and distal impact. The results chain demonstrates the cause and effect relationship of OGE’s programming. Moving forward, OGE will require program staff at the country level to create a Results Chain Model for their program so that they can understand how programmatic activity results in the desired program goal and what must be done in order to achieve programmatic targets and results. By creating a model that showcases how the program’s theory of change is directly tied to the program’s specific focus and goals, stakeholders create a common vision for the program, which serves as a roadmap for both implementing and evaluating the program. DEFINITION Results Chain Model Template form of the results chain for country teams to fill in at the beginning of a program, a practice for them to see how programmatic activities are directly tied to longer- term, mission-driven outcomes.
  • 22. 18 Digital Inclusion Social Impact Evaluation Note: The full version of the Digital Inclusion Results Chain includes a section for listing indicators along each step of the chain. Country teams list the inputs, activities, outputs, outcomes, and impact for their programs and then create a list of indicators to measure each. SUCCESS FORMULA OGE also adapted the Success Formula framework from the firm Mission Measurement and its CEO Jason Saul’s book Benchmarking for Nonprofits as a second program planning tool for this evaluation. The Digital Inclusion Success Formula is a universal tool that enables One Global Economy and other digital inclusion nonprofits to consider its social impact as a simple equation. At the end of the formula is the ultimate impact the organization aims to achieve through programming, created through the sum of the intermediate steps at a strategic, organizational level. The Success Formula makes OGE’s Theory of Change actionable and measurable. At the top of the Success Formula is the underlying principle that acquiring digital literacy and key skills combined with access to both locally generated content and global best practices in key issue areas provides users with the skills and information they need to take significant steps towards participating in both local and global economies. The Success Formula is used to outline the strategy and activities that will be used in the key issue areas (income, health, education, and social development) of the program, and the performance indicators that will be used to measure progress and success. Figure 4. OGE Digital Inclusion Results Chain Adapted from Gertler, Martinez et al. Impact Evaluation in Practice. The World Bank 2011.
  • 23. 19 CONTENT FOR IMPACT: SKILL CATEGORIES To further refine the model, we brainstormed and defined the main content areas of OGE’s programming, which together we believe are critical to achieving OGE’s desired impact. We called these groupings of content “Skill Categories” because the goal of the content in digital inclusion programming is building capacity and informed action. Part of OGE’s theory of change is that access to and taking informed action on these specific content areas are what create the desired impact as described in OGE’s mission statement. The four primary Skill Categories we identified are: digital literacy, business and entrepreneurship, livelihood and employment, and personal/social/ community development. Of note, Digital Literacy is a fundamental skill, because it creates the ability to access content in the other Skill Categories. The Skill Categories were further broken down into sub-skills, again using the criteria that these skills are imperative to achieving the desired results and are a fundamental part of OGE’s training curricula and programmatic activities. The skills and sub-skills identified in this exercise were fundamental to creating indicators and creating training assessments as part of the routine reporting component of the Impact Evaluation. Figure 5. OGE Digital Inclusion Success Formula Note: The full version of the Digital Inclusion Success Formula includes sections for listing program strategies and indicators for income, health, education, and social development. Country teams list the strategies they will use in their programs to create change in these categories, and then create a list of indicators to measure each. DEFINITION Digital Literacy The ability to find, evaluate, utilize, share, and create content using information technologies and the Internet (Cornell University); the ability to perform tasks effectively in a digital environment. Literacy includes the ability to read and interpret media (text, sound, images), to reproduce data and images through digital manipulation, and to evaluate and apply new knowledge gained from digital environments (Jones-Kavalier and Flannigan).
  • 24. 20 Digital Inclusion Social Impact Evaluation INDICATORS The indicator table serves as the backbone of the evaluation. The evaluation team developed indicators, or means to measure the comprehensive model of the theory of change. Indicators were created for each step in the Results Chain, and within each step for the skill categories. Creating the indicators in this way ensured that we measured all aspects of the process of delivering programs and the components that are imperative to creating impact. The indicators for inputs, activities, and outputs are process indicators, which measure what is put into a program and the deliverables of programmatic activities. The indicators for outcomes and impact are outcome indicators, which measure the lasting impact of the programs on the participants. To create outcome indicators tied to OGE’s mission of raising the standard of living, join the economic mainstream, and innovate local solutions, we used the Human Development Index, published by the United Nations Development Program. This Index uses composite indices to measure health (life expectancy), education (mean years of education), and income (gross national income, GNI, per capita) to rank countries in terms of development. While the indicators themselves cannot be used at the individual and household level, we used these broad categories to define standard of living for the purposes of this impact evaluation. Figure 6. Skill categories and sub-skills DEFINITION Process Indicator Indicators to measure whether planned activities took place, add more details in relation to the product (“output”) of the activity, and/or monitor the quality of the activities conducted, based on a number of established quality criteria or standards. (WHO) DEFINITION Outcome indicator Indicators that measure the objectives of an intervention, that is its ‘results’, its outcome. They are the result of both the “quantity” (“how many”) and quality (“how well”) of the activities implemented. See full definition in the Evaluation Dictionary
  • 25. 21 OGE therefore defines standard of living as the income, education, and health opportunities accessed by a given individual or household. It includes the following: • INCOME: Changes in annual income and savings, employment and career growth, entrepreneurship. • EDUCATION: Pursue or attain GED, applied to or enrolled in university. • HEALTH: Accessing quality health care systems, disease management, access preventative medicine (prenatal care, sanitation and water access, family planning, immunizations, infant care, nutrition) By using OGE’s mission statement to identify and define the final outcomes in the results chain, we aimed to ensure that this impact evaluation measures OGE’s results against its mission, reinforcing the mission at all levels and integrating the mission in its evaluation of itself. All indicators were assessed to be specific, measurable, attributable, realistic, and targeted. 2.3 Evaluation Design and Framework The impact evaluation applied a conventional social science approach using a non-experimental design. We collected information on the impact of OGE’s programs and activities on the desired outcomes through standardized routine program monitoring and a follow-up survey with training participants and digital platform users. The nature of OGE’s programs renders an experimental (randomized) or quasi-experimental (comparison group) approach infeasible. Therefore, the evaluation compared participants’ utilization of information technology and key socio-economic indicators before and after training. The evaluation used several methods to measure changes in the skill acquisition, utilization, and impact of information technology and resources from OGE’s programs. The evaluation methods bolstered routine reporting to monitor programs and assess input, process, and immediate outcomes of programs and initiated a survey to evaluate outcomes. PROGRAM MONITORING By measuring the resources, activities, and program achievements, organizations can track the causal logic of program outcomes, determine if interventions were carried out as planned, if the interventions reached the targeted beneficiaries, and if the interventions were performed on schedule, and what the results from the interventions were for the beneficiaries.
  • 26. 22 Digital Inclusion Social Impact Evaluation As part of the evaluation, OGE developed three components to program monitoring: • A STANDARDIZED MONTHLY REPORTING FORM and system to collect program data regularly, including data on the resources used (inputs), activities completed, and direct program outputs. Field staff and partners complete this form on a monthly basis to record progress towards key program objectives, which will directly feed into the impact evaluation. • A STANDARDIZED TRAINING REPORT FORM, which includes the content of the training, what curricula were used and how they were adapted, and the number of people trained. In addition, post-training assessments were created to ascertain the level of proficiency in the key skills taught during the training. • A TRAFFIC MONITORING TOOL to track OGE’s international digital platforms using Google Analytics. This monitoring tool allows OGE to gather significant data about the people who are using the digital platforms. OUTCOME EVALUATION To measure the impact of digital inclusion programs, it is necessary to survey participants and measure behavior and socio-economic changes of those who participated in trainings and/or accessed information via the digital platform. • IMPACT SURVEY: A standardized survey given both online and through active follow-up with program participants after they have been trained in one or more of OGE’s programs (e.g. Digital Inclusion, Community Connectors, Entrepreneurs, etc). The purpose of this survey is to find out the longer-term impact of the training on participants’ lives and socio- economic condition. The survey was conducted in four countries where OGE has conducted trainings—specifically, in India, Mexico, Egypt, and South Africa—for this initial evaluation. DATA COLLECTION TOOLS As part of creating the indicator table, we identified when and how the data would be collected and from whom. This process identified the different data collection tools that were necessary and when they should be implemented. To develop the data collection tools, we used the indicators to create questions. We systematically went through the indicator table to ensure that all data required to measure the indicators will be collected. A summary of data collection along the results chain is presented in Figure 7. By measuring indicators along the results chain, the impact evaluation methodology prevents producing a ‘black box’ that only identifies whether or not the predicted results occurred without explaining why or how the results were achieved. See Appendix A for a detailed description of data collection methods across the results chain.
  • 27. 23 As we refined the data collection tools, we refined and updated the indicator table. The survey questionnaire went through several rounds of reviews with the team to streamline, simplify and ensure that high-quality data would be collected that would measure the desired outcomes of the evaluation and then was sent to local partners for review and feedback. 2.4 The Evaluation Toolkit One of the goals of this Impact Evaluation was to create a methodology to enable the organization and its partners to better measure the social and economic impact of its Digital Inclusion Programs so that the lessons can be used to refine and scale future programs. Therefore, as part of the Impact Evaluation, OGE developed an Evaluation Toolkit to codify the process, tools, and implementation of a digital inclusion impact evaluation to be used in the future for all of OGE’s digital inclusion programming and for other organizations doing similar programming to adapt and use it to measure their own impact. This toolkit is a resource to help the management team, programmatic staff, and partners succeed at enabling digital inclusion toward developing their own communities. The purpose of this toolkit is to enable better digital inclusion practitioners to understand, define, measure and improve their impact. In the Impact Toolkit, OGE offers several guides for localizing the methodology, including a Success Indicators Planner that breaks down OGE’s theory of change and definitions of digital inclusion success. This will help to cultivate local critical thinking, ownership, and leadership of digital inclusion progress and measurement on the ground. Throughout the process of implementing this initial impact evaluation, we incorporated lessons learned into the toolkit and refined the framework, data collection tools, and systems. Figure 7. Data collection summary along the results chain
  • 28. 24 Digital Inclusion Social Impact Evaluation This Digital Inclusion Impact Evaluation Toolkit features capacity building guides, templates, and exercises for One Global Economy (OGE), its international partners, and other Digital Inclusion organizations to better define, plan and measure against its proposed outcomes (see Figure 8). OGE is sharing this universal toolkit with templates designed to be customized for each country/program with the collaboration of local partners, and enables partners to adapt and help execute the evaluation in their country. For a detailed list of the contents of the Impact Evaluation Toolkit, see Appendix B. Figure 8. The Digital Inclusion Impact Evaluation Toolkit
  • 29. 25 The evaluation team included OGE programmatic and leadership staff in collaboration with an evaluation consultant who provided technical guidance and assistance. The team began planning for the evaluation in the fall of 2013. The Impact Evaluation methodology, including the Theory of Change, Results Chain Model, Success Framework, indicators, and data collection methods were designed and created over the course of five months, from October 2013 to February 2014. Once the indicators were created and the methodology outlined, the team began designing an end-user survey and internal monitoring tools in early 2014 and implementing them in the summer. The analysis of all data and writing of the final report took place between August and September 2014 (see Figure 9). The purpose of this impact evaluation was to be formative for continued evaluation of impact. As such, this evaluation was carried out in only four countries to test and improve the methodology. Moving forward, OGE will continue to implement the routine reporting created as part of this evaluation (monthly, training, and traffic reports) in all of its current and future country programs and will implement subsequent surveys to continue to assess the impact of its programs. OGE chose India, South Africa, Mexico, and Egypt as the countries in which to conduct this formative Impact Evaluation for a number of reasons. Firstly, OGE had ongoing programs in each of those countries, which meant it had on- the-ground partners who could help contact people who had taken trainings and accessed content. Secondly, these countries represent a diverse array of communities OGE serves, ranging from the densely urban areas of Mumbai to very rural villages in the Mexican countryside with many gradations in between. Thirdly, these four countries covered a wide array of different programs that OGE employs. IMPLEMENTING THE EVALUATION3
  • 30. 26 Digital Inclusion Social Impact Evaluation 3.1 Results Chain Model and Success Formula The Results Chain Model and Success Formula are program planning tools, and served two functions in this evaluation. First, we used these tools to retrospectively outline the theory of change and strategies for achieving the desired outcomes of OGE’s programming overall. This was an essential step to help develop a comprehensive set of indicators that would drive the data collection methodology of the evaluation. Second, we field tested these tools with country teams to gain consensus and improve the tools so that they could be included in the Evaluation Toolkit as program planning tools – used in all future OGE programs and adapted by other organizations when planning similar programs. The feedback from country teams in South Africa and Mexico was that these were useful tools in looking at the program as a whole and better understanding the relationship between program outputs and impact. Managers in these countries were able to participate in a step-by-step process of diagramming how these programs are designed to help their beneficiaries. This clarified the strategies through which real change is affected and highlighted the importance of their programmatic activities. Of note the team in South Africa used an innovative approach to completing the Success Formula: they completed the formula from the perspective of one person who couldn’t find a job to figure out the best way to work with them. This approach was incorporated into the directions for completing Figure 9. Impact Evaluation Timeline 2013 2014
  • 31. 27 the Success Formula and results chain, but from a more holistic perspective: completing the exercises several times over with different client avatars, each with different problems and needs, as a needs assessment, program planning, and evaluation tool. For example in Mexico, the team of student journalists reflected on how articles on various locally relevant topics contributed to community, personal, and social development, and how different formats for disseminating global best practice information provided access to more services and resources locally. Overall, a challenge with these tools was getting program teams to be as specific as possible about programmatic activity. Without guidance, local teams often wrote blanket statements that were not particularly meaningful or measurable. To help guide teams in the future, the evaluation team created a list of program activities and indicators for them to include, adapt, or expand upon. 3.2 End-User Survey SAMPLING STRATEGY AND LOCATION The end-user survey is the most significant surveying OGE has ever undertaken of the communities it serves. OGE posted links to the survey on the local online portals in each country to try to get online users to participate and actively followed-up with past program participants in communities to conduct the survey in-person. The survey was implemented in Mexico, Egypt, South Africa, and India. The target population was anyone who had either visited a local online portal or had attended a training course with an OGE curriculum. This is a very demographically unspecific set of people, but that was necessary given the diverse population that OGE serves. DATA COLLECTION A standardized survey questionnaire was uploaded to SurveyGizmo, a web- based survey program that allows for skip logic, multiple choice, and write-in responses. A link to the survey was posted to the digital platform in each of the four countries, with translations in Arabic, Hindi, Spanish, and English available. In addition, staff of OGE’s partners administered the questionnaire in-person with hard copies to members of their community. Data collection began on March 15 and continued through July 22, 2014. It is impossible to overstate how critical OGE’s in-country partners were to the success of this survey. OGE worked with Pratham InfoTech Foundation in India; the Coptic Evangelical Organization of Social Services (CEOSS) in Egypt;
  • 32. 28 Digital Inclusion Social Impact Evaluation Siyafunda in South Africa; and Universidad Autónoma del Estado de México in Mexico. OGE works with these partners to implement its programs in each country. Not only do these NGOs have a deep reach into the communities where OGE works, but in many cases they also run computer or community centers where OGE could get people from the community who had participated in its programs to take the survey. The partners met with people in their communities, conducted outreach, and encouraged people to take the survey. DATA ENTRY, MANAGEMENT AND ANALYSIS Most individual respondents completed the survey on SurveyGizmo. The hard copies of the survey were directly entered into SurveyGizmo by members of OGE’s partner NGOs to ensure standardized data between online and in- person respondents. The analysis plan used the Success Formula and Results Chain as a guide to analyze the key performance indicators identified during the process of creating those tools. For demographic and direct program impact variables, data was compared between countries to see how the different programs affected outcomes. For longer-term impact variables, data was compared between those who did and did not participate in a training to see the long- term effects of the training on key impact variables, particularly the impact of training on taking informed action in key development areas. Furthermore, we analyzed the data by time since training to uncover the longer-term impact of the program and training. Overall, this analysis was conducted with an eye towards understanding the effect of content and training and how we can improve our programs to achieve greater impact. A criterion of p<0.05 was used to assess the statistical significance of differences between groups. Differences with p<0.01 were noted as highly significant.
  • 33. 29 3.3 Routine Reporting A major goal of the Impact Evaluation was to build OGE’s capacity to monitor and evaluate its programs. Therefore, as part of the evaluation, a comprehensive routine reporting system was developed and implemented (see Section 3.2 above). OGE collaborated with its partners in each of the countries to create the routine reporting templates, including monthly reporting forms, a standardized training report form, and a traffic monitoring tool. The Monthly Routine Reporting Form and the Training Report Form together create a comprehensive internal monitoring system for OGE. The purpose of this system is to: • STANDARDIZE INFORMATION ACROSS COUNTRY PROGRAMS. The same information will be collected from all of OGE’s programs using this form. With standardized information, we can track programmatic activities and progress organization-wide. Regular updates on program activities ensure the history of a program is documented thoroughly. • ENABLE “REAL-TIME” REPORTING. Activities are updated monthly as they happen, rather than waiting until a report is due. If updates program milestones are given on an on-going basis, the burden of giving information for reports at the end of the period is reduced. Rather than going back and asking country teams what happened during the reporting period, staff can generate a report using the data that has been entered already. • CONSOLIDATED SOURCE OF PROGRAM INFORMATION. The Monthly Routine Reporting Form collects information on major activities (e.g. trainings or workshops, computer centers opened, outreach activities, and community activities), content developed for the digital platform, computer center activity (e.g. users and center development activities), follow-up with training participants, and stories of people innovating local solutions, quotes and testimonials, and other notes. The Training Report Form is part of a larger set of tools for designing and evaluating trainings, which include choosing curricula and teaching objectives and post-training evaluations of skills acquired. The Training Report Form reports on the content and objectives of the training, the attendees, and the results of post-training assessments. The reporting system utilizes a web-based survey design, Survey Gizmo, with skip logic: answers to initial questions direct the respondent to subsequent questions to complete information specific to that type of activity and allows them to skip questions that are irrelevant. DEFINITION Routine Programmatic Reporting Standardized monthly report form submitted by the country team to headquarters, including directions and a description of how this reporting fits into the larger impact evaluation and OGE’s mission. DEFINITION Training Report Forms Standardized training report form submitted by the country team within one month of training to headquarters, including directions and a description of how this reporting fits into the larger impact evaluation and OGE’s mission.
  • 34. 30 Digital Inclusion Social Impact Evaluation By including the local partners in the development of the reporting system, OGE was able to integrate their feedback to make the system more accurate, relevant, and feasible. For example, the South Africa Manager provided helpful feedback for localizing the Monthly Reporting Form, to the Mobile Entrepreneurs Program specific to South Africa, by suggesting that the form integrate more skills in the checklists and that the form include a space for local managers to provide testimonials from participants, which are gathered regularly. Walking in-country managers and partners through their own success formula and results chain often helps them gain a better perspective on the work that they are doing. Google Analytics allows OGE to track the information of digital platform users. By tracking information using the Traffic Monitoring Tool, OGE can learn which pages are viewed the most and for the longest amount of time in different areas, where users come to the digital platforms from, and which topics resonate the most. This allows OGE to give partner organizations accurate feedback about articles written by community members, as well as pointing OGE towards the issues that are most pertinent to different communities. DEFINITION Traffic Monitoring Tool A form to collect and consolidate information from Google Analytics.
  • 35. 31 4.1 Google Analytics TOTAL AND UNIQUE VIEWS At the time of this report, OGE has had 10,756,356 sessions and 9,386,745 unique views on its digital platforms since 2005. As seen in Figure 10, unique views have been steadily increasing yearly for OGE digital platforms overall. Unique views on the Mexico digital platform have remained consistent since July 2011 at an average of 450,000 unique views per year. The Egypt digital platform, which started in 2013, had over 570,000 unique views that year. AVERAGE NUMBER OF PAGE VIEWS AND AVERAGE TIME SPENT ON THE PLATFORM Two ways to measure online engagement are the number of pages viewed in a session and the amount of time spent on the website. OGE monitors these two metrics on all of its web properties using Google Analytics (Figure 11). INITIAL EVALUATION RESULTS Figure 10. Unique views per year by country and OGE overall 4
  • 36. 32 Digital Inclusion Social Impact Evaluation In South Africa the number of pages visited per session has dropped a small amount from just over two pages per session to about 1.8 pages per session while the time spent on the website has increased from about 1:05 to 1:18. There has been a similar trend in Mexico where page views decreased from 1.8 per session to 1.4 while the time on the sites has risen dramatically from 1:25 to about 2 minutes. This signals a deeper engagement with the material. India has trended oppositely, where page views have gone up from an average of 2.85 to 3 and time spent on the web property has fallen from about 3:30 to about 2:55. Though the time on the site has decreased, almost three minutes spent on the site remains higher than the industry average of 10 to 20 seconds (Nielsen 2011) and the “average of a little over one minute on a newspaper website” (Grabowicz 2014). Figure 11. Average number of pages viewed and time spent per session on the digital platform
  • 37. 33 TOP CONTENT As seen in Figure 12, health, education, and employment are consistently among most accessed content. Because each portal is localized to its respective region, programmatic efforts focus on relevant topics, and their significance to the target population is often highlighted by trends in users’ activity as seen in the diversity between countries in these results. For example, South Africa shows the most views for citizenship-related content compared to Mexico. In South Africa, the stakeholders’ engagement workshop revealedthatmanySouthAfricanresidentswerenotawareoftheirrightsascitizens under the then new post-Apartheid democracy. Through developing content with localNGOs,OGEdefinedcitizenshipasakeycommunitydevelopmentpriority, and featureditasthefifthmaintopic.ForEgyptandIndiathisfifthtopicisagriculture. SouthAfrica’scitizenship-focusedcontentoutlinescitizens’ basicrightsandexplains SouthAfrica’ssystemofgovernmentinalanguageuserscanunderstandin orderto access local services, initiate processes for obtaining licenses and certificates, and become informed and active citizens through participating in elections. In other cases, top content indicates hot issues and can inform article creation for the development of more specific content addressing those topics. For example, soon after the launch of the Mexico digital platform, a review of Google Analytics and comments from users on the platform showed that there was a demand for more employment-focused articles. OGE, its local team, and student editors at UAEM (partner University in Mexico) focused on developing articles aimed at young people and job seekers, increasing the number of online resources that help people understand how to apply for jobs online, learn interview skills, and link them directly to Mexico-based online job-search engines. The demand for livelihood-focused content addressing employment and education issues is also illustrated by the high percentage of content viewed on women and teenagers, which remains a top topic due to additional content OGE developed to guide high school dropouts, single mothers, and adults through choosing the best way to complete their education and earn degrees and certificates. The structure of the program and its components in each individual country can also explain why certain topics are more frequently trafficked over others. For example, in Egypt, the goal of the project is to link young people to employment opportunities. Content on the Egypt digital platform therefore largely focuses on connecting individuals to resources addressing issues of financial literacy, income generation, entrepreneurship, agribusiness, and best practices for microenterprise development. In India, a recent program focused on implementing health camps to teach basic sanitation practices, preventative healthcare, and adolescent health at the partner centers. Many of the health camps have focused on sexual health and serve young women, who lack access to such information. Top content viewed on health in general, and articles on safe sex, and date rape drug awareness are reflective of a need for adolescent health information. See Appendix C for the top five articles by year for each country and OGE overall.
  • 38. 34 Digital Inclusion Social Impact Evaluation MOBILE VS. DESKTOP USERS The percentage of people accessing OGE’s digital platforms from mobile devices steadily increased since 2011. In South Africa between 2013 and 2014, 53% of people accessing the digital platform did so via mobile devices. Though the percent of mobile users are still low in Egypt (9%), India (22%), and Mexico (24%), they are not insubstantial. All four countries are showing substantial upward trends, with the percent of mobile users doubling in all four countries in the last year. This trend has serious consideration for future programs, which will be discussed later in this report. Figure 12. Top content topics by country by year Figure 13. Percent of all users accessing the digital platform with a mobile device by country by year
  • 39. 35 4.2 Survey Participants OGE collected a total of 4,676 surveys between March 15 and July 22, 2014. Incomplete surveys (n=308) and surveys from respondents living in countries not included in the evaluation (n=8) were removed from the analysis. A total of 4,364 surveys were included in the analysis: 2,407 from Egypt; 1,292 from India; 491 from South Africa; and 174 from Mexico (see Figure 14). The number of respondents in total and in each country is sufficient for OGE’s purposes because the total is large enough to test for statistical significance and each country’s total reflects the number of participants in OGE programs. For instance, OGE works with CEOSS, one of the largest social service NGOs in Egypt. The programs reach a lot of people and therefore the most surveys were collected in that country. In contrast, OGE’s work in Mexico is in a few very rural villages so the 174 completed surveys represent a very large number of the participants in OGE’s programs there. 4.3 Background Characteristics of Respondents Table B presents the demographics of survey respondents. This is a very detailed and representative snapshot of OGE’s constituency, and it reflects OGE’s focus on populations underrepresented in the economic mainstream, particularly women, young people, and the unemployed. Overall, 59% of the respondents were women; while the gender of respondents varied significantly between countries the range was not large (56% in Mexico to 65% in South Africa). See Spotlight: Gender for a detailed analysis of the impact of OGE’s programs on women. Figure 14. Data for analysis by country
  • 40. 36 Digital Inclusion Social Impact Evaluation Of note, most respondents were between the ages of 18 and 34 at the time of the survey (90%). This is partially because of the large amount of respondents in Egypt (55% of the total survey) who were of that age (2,400 of the 2,407). However, that trend held across the other countries as well, with 78% of Indian, 91% of Mexican, and 85% of South African respondents below the age of 30. This reflects the emphasis OGE’s programs have on training young people because youth around the world often lack job prospects and job skills, and employment as a young adult is one of the best routes out of poverty. Young people, who are more likely to have free time and less likely to have families, are particularly able to take advantage of the job skills and entrepreneurial programs that OGE offers. Almost half of all Egyptians surveyed (48%) are unemployed even though 69% have some college experience or have graduated from college. This speaks to the difficult economic situation that many Egyptians face, but also to the benefits of becoming as job-ready and as good at the soft skills of interviewing and networking as possible. In India, despite a low unemployment rate, many of the respondents were students (27%) or homemakers (25%). These are two populations that can greatly benefit from digital literacy and job skills as well as entrepreneurial training because of the opportunities they provide for small business creation and joining the job force.
  • 41. 37 Egypt (N=2,407) India (N=1,292) Mexico (N=174) South Africa (N=491) Total (N=4,364) Gender* Male 1,010 (42.2%) 526 (40.8%) 76 (43.7%) 168 (35.3%) 1,780 (41.1%) Female 1,382 (57.8%) 763 (59.2%) 98 (56.3%) 308 (64.7%) 2,551 (58.9%) Age** 12-18 3 (0.1%) 2 (0.2%) 69 (39.7%) 2 (0.4%) 76 (1.7%) 18-24 1,376 (57.2%) 598 (46.4%) 66 (37.9%) 255 (52.3%) 2,295 (52.7%) 25-34 1,022 (42.5%) 406 (31.5%) 25 (14.4%) 157 (32.2%) 1,610 (37.0%) 35-44 4 (0.2%) 197 (15.3%) 9 (5.2%) 53 (10.9%) 263 (6.0%) 45-54 0 74 (5.7%) 4 (2.3%) 15 (3.1%) 93 (2.1%) 55-64 0 9 (0.7%) 0 6 (1.2%) 15 (0.3%) 65+ 0 3 (0.2%) 1 (0.6%) 0 4 (0.1%) Marital status** Married 594 (24.9%) 642 (50.3%) 25 (14.5%) 49 (10.2%) 1,310 (30.3%) Separated/Divorced/ Widowed 23 (1.0%) 23 (1.8%) 7 (4.0%) 19 (4.0%) 72 (1.7%) Single 1,771 (74.1%) 612 (47.9%) 141 (81.5%) 413 (85.8%) 2,937 (68.0%) Highest education** Elementary 16 (0.7%) 246 (19.0%) 15 (8.6%) 2 (0.4%) 279 (6.4%) Middle school 10 (0.4%) 323 (25.0%) 53 (30.5%) 9 (1.9%) 395 (9.1%) Some high school 43 (1.8%) 214 (16.6%) 20 (11.5%) 54 (11.0%) 331 (7.6%) High school grad 416 (17.3%) 232 (18.0%) 32 (18.4%) 305 (62.1%) 985 (22.6%) Some college or technical school 258 (10.7%) 173 (13.4%) 16 (9.2%) 60 (12.2%) 507 (11.6%) College or technical school graduate 1,553 (64.5%) 97 (7.5%) 32 (18.4%) 61 (12.4%) 1,743 (39.9%) Other 111 (4.6%) 7 (.5%) 6 (3.4%) 0 124 (2.8%) Employment** Homemaker 118 (5.0%) 320 (24.8%) 14 (8.2%) 9 (1.9%) 461 (10.7%) Retired 1 (0.1%) 5 (0.4%) 2 (1.2%) 0 8 (0.1%) Self-employed 246 (10.4%) 88 (6.8%) 8 (4.7%) 44 (9.2%) 386 (9.0%) Student 46 (2.0%) 349 (27.1%) 95 (55.6%) 145 (30.2%) 635 (14.8%) Unemployed 1,139 (48.4%) 62 (4.8%) 9 (5.2%) 197 (41.0%) 1,407 (32.8%) Working for an employer 801 (34.1%) 465 (36.1%) 43 (25.1%) 85 (17.7%) 1,394 (32.5%) * p<0.05 ** p<0.01 Table B. Background characteristics of respondents
  • 42. 38 Digital Inclusion Social Impact Evaluation 4.4 Training and Access As seen in Figure 15, 3,562 people who took the survey (82%) participated in one or more trainings by OGE or one of its partners between 2009 and 2014. Most respondents reported participating in a training between 2012 and 2014, largely from respondents in Egypt and India. The large numbers of people participating in trainings in Egypt and India is mainly due to three factors: 1) Egypt and India were the largest number of respondents to the survey, 2) the partners’ large reach in these countries, and 3) the nature and scale of OGE’s programs in these countries. In both India and Egypt, OGE works with larger, local NGOs, Pratham and CEOSS, respectively, both of which have a greater reach than its partners in Mexico. OGE implemented a large-scale, multi-year project in Egypt, designed to train at least 1,200 youth. In India, OGE launched its training curriculum at Pratham’s community technology centers in Pune, training over 2,900 individuals in 2012, and built the capacity of 10 centers reaching 15 communities in Maharashtra State in 2013-2014 to implement a number of training programs that served over 4,000 people. Siyafunda, OGE’s partner in South Africa, also has a large network of community technology centers located in mostly rural townships in Johannesburg. However, OGE only works with about 5-8 of their centers per year, as the project is smaller in scale. Finally, in Mexico, OGE works with three centers in very small, remote villages, as well as small classes at local high schools in Toluca, in partnership with UAEM. Missing training date from 204 respondents. Figure 15. Training by country by year
  • 43. 39 In Egypt and India, most respondents (92% and 93% respectively) had participated in any training by OGE or one of its partners compared to half the respondents in South Africa (62%) and a quarter in Mexico (29%). The most common trainings were Career guidance and job hunting (73% of those who participated in a training) and Community Connectors (66%). It is important to note that the training programs in each country all focus on digital literacy, technology adoption, and leadership skills. The leadership focus varies in each country depending on the target population. In Mexico, leadership skills taught are core soft and hard skills that can be exercised in an academic or professional setting, or used to become actively involved in community development initiatives. In Egypt and South Africa, there is an emphasis on entrepreneurship and management, and in South Africa, India and Egypt, on employment. Therefore, there exists great overlap in training content from country to country, largely due to the fact that digital literacy is very much connected to and has the impact to improve a number of other, more specific skill sets pertaining to peoples’ livelihoods. It can be inferred from these results and based on each country’s unique program offerings that not every respondent indicated the training they necessarily participated in. For example, the Mobile Entrepreneurs Program, only offered in South Africa, consists of modules focused on entrepreneurship, but also contains modules that teach career development skills so that entrepreneurs also learn how to get jobs, if the choose, that allow them to utilize their entrepreneurial skillset. Therefore, some of the respondents may have said they had received training in career guidance and other job-related skills if they participated in this training. In Mexico, respondents who indicated they received career guidance and job training could represent adults who participated in Community Connector-led technology workshops at their local centers, which are aimed at parents, many of which are first-time Internet users, who are interested in learning how to look and apply for jobs online.
  • 44. 40 Digital Inclusion Social Impact Evaluation Table C. Background characteristics of respondents Egypt (N=2,407) India (N=1,292) Mexico (N=174) South Africa (N=491) Total (N=4,364) Any training** 2,204 (91.6%) 1,205 (93.3%) 51 (29.3%) 306 (62.3%) 3,766 (86.3%) Career guidance and job hunting** 2,142 (97.2%) 483 (40.1%) 12 (23.5%) 104 (34.0%) 2,741 (72.8%) Community Connectors** 1,822 (82.7%) 559 (46.4%) 32 (62.8%) 72 (25.5%) 2,485 (66.0%) Business** 97 (4.4%) 29 (2.4%) 7 (13.7%) 112 (36.6%) 245 (6.5%) Health camp** 0 224 (18.6%) 0 0 224 (6.0%) Digital literacy** 0 166 (13.8%) 1 (2.0%) 7 (2.3%) 174 (4.6%) Internet security and IT adoption** 1 (0.1%) 86 (7.1%) 9 (17.7%) 7 (2.3%) 103 (2.7%) Mobile Entrepreneurship** 10 (0.5%) 0 0 84 (27.5%) 94 (2.5%) Professional and soft skills** 0 8 (0.6%) 4 (7.8%) 15 (4.9%) 27 (0.7%) Managers** 4 (0.2%) 0 6 (11.8%) 6 (2.0%) 16 (0.4%) Other (not specified) 5 (0.2%) 116 (9.6%) 1 (2.0%) 5 (1.7%) 127 (3.4%) None 220 (9.1%) 95 (7.4%) 123 (70.7%) 191 (38.9%) 629 (14.4%) * p<0.05 ** p<0.01 The majority (71%) of respondents in Egypt reported having Internet access at home; in the other countries Internet access in the home varied from 14% in India to 41% in Mexico (Table D). In India, respondents most commonly reported that they accessed the Internet at a CKC (84%) while in Mexico almost half of the respondents reported accessing the Internet at home (40%), school (43%), and at a CKC (37%) in addition to many using telecenters (29%) and work (19%). In South Africa, the majority said they accessed the Internet at a CKC (67%); while some reported accessing the Internet at home (20%), school (15%), or at a telecenter (14%).
  • 45. 41 Egypt (N=2,407) India (N=1,292) Mexico (N=174) South Africa (N=491) Total (N=4,364) Where do you access the Internet? At home** 1,692 (70.3%) 173 (13.4%) 70 (40.2%) 101 (20.6%) 2,036 (46.7%) At school** 14 (0.6%) 14 (1.1%) 75 (43.1%) 77 (15.7%) 180 (4.1%) At work** 171 (7.1%) 112 (8.7%) 33 (19.0%) 36 (7.3%) 352 (8.1%) At a telecenter** 120 (5.0%) 8 (0.6%) 50 (28.7%) 67 (13.7%) 245 (5.6%) At a CKC** 457 (19.0%) 1,086 (84.1%) 65 (37.4%) 329 (67.0%) 1,937 (44.4%) On a mobile phone** 58 (2.4%) 155 (12.0%) 3 (1.7%) 12 (2.4%) 228 (5.2%) Using a mobile broadband USB stick** 20 (0.8%) 0 0 0 20 (0.5%) At a library 0 0 0 1 (0.2%) 1 (0.1%) Have internet access at home** 1,714 (71.2%) 183 (14.2%) 72 (41.4%) 138 (28.1%) 2,107 (48.3%) * p<0.05 ** p<0.01 Table D. Background characteristics of respondents 4.5 Access Local Content The high percentage of respondents who have accessed the digital platform in Egypt and India is reflective of the reach of OGE’s partners in those countries, as well as the high number of access points. Only 25% of respondents in South Africa reported ever accessing the digital platform. This number is somewhat surprising because OGE’s partner in South Africa also has a significant reach and number of access points, however this could be due to the small sample size taken from South Africa. If OGE had surveyed a larger number of respondents from a greater variety of townships where its access points are located, results might have shown a higher percentage of people who have accessed the platform. The small percentage of people who have ever accessed the digital platform in Mexico is representative of the scale of OGE’s work and the number of access points (three), which are located in small and rural towns.
  • 46. 42 Digital Inclusion Social Impact Evaluation Table E. Read about topics on the digital platform Egypt (N=2,407) India (N=1,292) Mexico (N=174) South Africa (N=491) Total (N=4,364) Ever accessed the digital platform** 1,758 (73.0%) 1,114 (86.2%) 44 (25.3%) 124 (25.3%) 3,040 (69.7%) Topics accessed: Agriculture** 317 (13.2%) 400 (31.0%) 14 (8.1%) 23 (4.7%) 754 (17.3%) Citizenship** 109 (4.5%) 165 (12.7%) 10 (5.8%) 26 (5.3%) 310 (7.1%) Health** 1,046 (43.5%) 998 (77.2%) 26 (14.9%) 47 (9.6%) 2,117 (48.5%) Education** 1,004 (41.7%) 906 (70.1%) 21 (12.1%) 58 (11.8%) 1,989 (45.6%) Recommended articles to other people** 1,495 (85.0%) 325 (29.2%) 9 (17.7%) 74 (59.7%) 1,917 (63.1%) * p<0.05 ** p<0.01 Figure 16, which shows the top three content areas by country, demonstrates that content on the digital platform is tied to programmatic themes, but also how content that the popularity of content is not always directly related to training topics and may serve as key resources in supporting peoples’ livelihood development. For example, while the programmatic focus in OGE’s programs in Egypt was employment and entrepreneurship, the program served some rural communities where agribusinesses account for many of the small businesses. Therefore it is unsurprising that agriculture was a popular topic among people accessing the Egypt digital platform content. All three of India’s top three content areas are reflective of training topics and services offered at partner centers. In this case, the India results correlate to program objectives focused on supporting people’s improvement in health, education, and employment. In Mexico, there is no training program focused solely on health, so the large percentage of people viewing health content could be indicative of both the demand for and the value of localized information on the topic. A key section of the Mexico digital platform is the Homework Help section, which was developed to include links to online educational resources such as tutorials and activities for over 100 concepts in math, science, reading and other core subjects for primary, secondary, and high schools students, aimed at guiding parents in helping their children improve their academic achievement. According to Google Analytics, more than a million users have accessed the Homework Help section, which could contribute to education being among a top content area for Mexico. South Africa’s program has a strong emphasis on employment and entrepreneurship, which is also captured by the top content results.
  • 47. 43 All programs emphasize using and recommending others to the digital platform for the purpose of accessing global best practice information and community development-focused content on locally relevant issues and topics. The digital inclusion ecosystem OGE has created with its partners— access, content, and training—is designed to support word of mouth and actions taken to increase technology adoption and usage of the digital platform in the communities it serves. Respondents were asked if they recommended articles on the digital platform to others, and if so what topics they recommended. There was wide variation between countries in both the frequency of recommending articles and they type of content. Respondents from Egypt were most likely to report recommending articles on the digital platform (85%), followed by South Africa (60%) and Mexico (52%). Respondents from India were least likely to report recommending articles on the digital platform to others. The topics that were recommended mirrored the top content in each country. 4.6 Take Informed Action The core of all of OGE’s programs is to help people to be better able to help themselves by acquiring skills and gaining access to better information via information technology to make informed decisions about their lives. We measured this by asking not only what actions people took after they attended a training or after they accessed content on the digital platform, but if those actions improved their lives by gaining employment, starting a business, earning more money, increasing their savings, and improving their health and/or education. Figure 16. Top three content areas by country Figure 17. Ever recommended articles and top three topics recommended on the digital platform
  • 48. 44 Digital Inclusion Social Impact Evaluation Figure 18 shows the how often respondents in different countries report they go online. It is very encouraging that more than half of respondents reported that they go online at least once each day (orange bars in Figure 19). However, respondents in India reported going online with much less frequency: only 29% reported going online daily and 43% reported going online only once every few months (vs. less than 2% of respondents in the other countries). This speaks to the need for computer centers and other points of access in India and the benefit of online training and content in the other countries. Figure 19 shows that only 29% of all respondents use email for personal communication and 10% use it for business communication. This is a critically important skill, particularly in the 21st century economy. It is possible that so few people use email for business because many of the respondents are either unemployed or have a job that does not require sending email. That said, the low numbers of people using email for personal use are indicative of low levels of digital literacy given that so many of the respondents reported going online regularly. It is very important to have an online identity for finding and getting a job as well as making the types of connections that can lead to professional advancement. Email is one way to create an online identity and social Figure 18. How often do you go online?
  • 49. 45 networking is another, which is why it is encouraging that almost two thirds of respondents reported using online social networks (61%). This represents a great opportunity to teach people how to use social networking to form ties to their communities and to advance their professional lives, particularly in Egypt where unemployment is high, Internet use is very regular, and many respondents (43%) reported using the Internet to search and apply for jobs. One reason for this high number is the OGE’s job-matching program where Egyptians post their skills on an online database and employers post job openings. Lastly, the data makes it clear that many people use the internet to try to improve their lives by studying, researching, applying for jobs, and social networking rather than just for frivolous purposes. Less than 1% of respondents said they used the internet for downloading music and watching movies online [data not shown]. Figure 19. How often do you go online? ** p <0.01
  • 50. 46 Digital Inclusion Social Impact Evaluation 4.7 Increased Opportunities in Income USE IDENTITY TO PARTICIPATE IN THE LOCAL AND GLOBAL ECONOMY Over a quarter of the respondents reported having bank accounts and using online banking, as seen in Table E (27%). However, there was wide variation across countries: only 2% of respondents in Egypt reported having a bank account compared to 70% in South Africa. However, of those who had bank accounts, those from Egypt were much more likely to access their bank account online (69% of those with a bank account, compared to 16% overall). Very few respondents reported paying their bills online (5%). Egypt (N=2,407) India (N=1,292) Mexico (N=174) South Africa (N=491) Total (N=4,364) Have a bank account** 55 (2.3%) 751 (58.1%) 39 (22.4%) 344 (70.1%) 1,189 (27.3%) Access bank account online** 38 (69.1%) 56 (7.5%) 21 (53.9%) 71 (20.6%) 186 (15.6%) Pay bills online** 74 (3.1%) 68 (5.3%) 25 (14.4%) 31 (6.3%) 198 (4.5%) * p<0.05 ** p<0.01 START A BUSINESS OGE administers business trainings that aim to help people start businesses and employ best practices to maximize the chances that these businesses will be successful. In these trainings people learn how to create an operational budget, apply for loans and grants, and create marketing and business plans amongst other activities. Of the 245 respondents who said they had participated in one of OGE’s business trainings, 46 reported that they started a business after training (19%). Those who started a business were much more likely to have implemented business best practices than those that did not, such as creating operational budgets (57% vs. 19%), a business plan (70% vs. 44%), and a marketing plan (52% vs. 19%). However, many respondents implemented these best practices without starting a business. For example, 44% of respondents who didn’t start a business after the business training stated that they created a business plan. Since the majority of respondents attended a training within the last year, the survey may have captured people in the process of starting a business. With increased time since the training, future results may show a higher proportion of trainees starting a business. [data not shown] Table E. Read about topics on the digital platform