Successfully reported this slideshow.
One Global Economy
Final Report | October 2014
Social Impact Evaluation
One Global Economy would like to express its gratitude to the Symantec Corporation
for its continued support for OGE and the grant that has allowed us to undertake this
evaluation in particular. For a number of years Symantec has supported OGE’s mission
of serving low income communities in many countries including India and Mexico,
and most recently, featured One Global Economy in its online brand campaign.
Table of Contents
1. Background..................................................................... 3
1.1 Goal and Objectives of the Impact Evaluation............................ 3
1.2 The Information-Based Ecosystem Strategy.............................. 4
1.3 Internet Access Points................................................... 5
1.4 Training................................................................. 6
Digital Literacy................................................................. 6
Entrepreneurship and Technology....................................... 6
1.5 Local Content.................................................................. 8
1.6 Country Programs............................................................. 9
South Africa................................................................... 9
2. Evaluation Design and Methodology.......................................... 13
2.1 Goal and Objectives of the Impact Evaluation........................... 13
2.2 Developing the Evaluation Design and Framework......................... 14
Theory of Change............................................................ 15
Results Chain Model......................................................... 17
Success Formula.............................................................. 18
Content for Impact: Skill Categories......................................... 19
2.3 Evaluation Design and Framework.......................................... 21
Program Monitoring......................................................... 21
Outcome Evaluation......................................................... 22
Data Collection Tools........................................................ 22
2.4 The Evaluation Toolkit........................................................23
Digital Inclusion Social Impact Evaluation
3. Implementing the Evaluation....................................................25
3.1 Results Chain Model and Success Formula..................................26
3.2 End-User Survey.............................................................. 27
Sampling Strategy and Location............................................. 27
Data Collection............................................................... 27
Data Entry, Management and Analysis.....................................28
3.3 Routine Reporting............................................................29
4. Initial Evaluation Results.......................................................... 31
4.1 Google Analytics............................................................. 31
Total and Unique Views...................................................... 31
Average Number of Page Views
and Average Time Spent on The Platform.................................. 31
4.2 Survey Participants...........................................................35
4.3 Background Characteristics of Respondents................................35
4.4 Training and Access..........................................................38
4.5 Access Local Content........................................................ 41
4.6 Take Informed Action........................................................43
4.7 Increased Opportunities In Income.........................................46
Use Identity to Participate in the Local and Global Economy..............46
Start a Business...............................................................46
Career Advancement and Employment..................................... 47
4.8 Increased Opportunities in Health, Education, and Citizenship............50
Increased Opportunities in Health.......................................... 51
Increased Opportunities in Education....................................... 52
Increased Opportunities in Social Development............................53
4.9 Transform Communities.....................................................55
5. Conclusions and Lessons Learned............................................... 57
6. The Road Forward................................................................... 61
Appendix A: Data Collection Methods Along the Results Chain...................63
Appendix B: Impact Evaluation Toolkit................................................64
Appendix C: Top Five Articles by Country by Year...................................65
Evaluation Dictionary..................................................................... 67
One Global Economy began its Impact Evaluation program a year ago
after receiving a grant from the Symantec Corporation to evaluate existing
programs in Egypt, India, Mexico, and South Africa, and to build its internal
capacity to continue to monitor and evaluate future programs. Over the
course of the program, One Global Economy (OGE) has learned a great deal
about the processes of monitoring and evaluation, and about the people and
the communities that OGE serves.
The program began with OGE developing its theory of change, or how its
programs improve the lives of the people it aims to serve. This helped clarify
and determine the specific social impact metrics to be measured through
The next step in the process was the design of a community survey to be
administered in all four countries in Arabic, Hindi, Spanish, and English.
OGE received 4,364 completed surveys, which provided a trove of data about
people who had either attended training delivered by OGE or had accessed its
online content on one of its digital platforms. Consequently, OGE was able to
learn more about the people it serves and how they use and view the content
OGE focuses on employment as a solid indicator of joining the economic
mainstream. Therefore, all of OGE’s trainings start with teaching basic
digital literacy so that participants will be able to apply those skills to both
the job search as well as increasing their employability. OGE’s findings about
employment were very encouraging. 86% of survey respondents did not have
a job before they participated in a training, but of those, 35% were employed
by the time the evaluation process had begun. Half of the respondents, who
reported they were employed, had found their job online.
Google Analytics has allowed OGE to see which pages of its content are the
most trafficked, how many pages people visit, how long users stay on a page,
and other hard data. Through combining the Google analytics data with the
data gathered from the community survey, OGE gained a much more detailed
and nuanced understanding of the impact of its online content and how
users adopt global best practices to improve their health, education, and
of previously unemployed
survey respondents were
employed by the time the
evaluation process had begun
2 Digital Inclusion Social Impact Evaluation
The survey data showed that 99% of respondents, who accessed content on
OGE’s digital platforms, took some kind of action. The most popular content
dovetailed with the feedback from the stakeholders’ engagement process
that guided the content creation to develop the platform in each country.
Employment and job-seeking skills were most popular in Egypt, health and
sanitary practices in India, citizenship in South Africa, and job skills in Mexico.
Another significant finding related to how people use OGE’s content
and training revealed that 86% of respondents from Egypt and 60% of
respondents from South Africa reported teaching other people in their
communities about what they learned from the digital platform or in an
OGE training. These results indicate that OGE’s content and training have
a multiplicative effect on the communities it serves.
The Impact Evaluation Program has bolstered OGE’s capacity to continually
monitor its programs. OGE has worked with its partner organizations to
help them develop performance indicators for programs delivered together.
Additionally, OGE has created a monthly reporting tool and a training
reporting tool that its partner organizations and local managers will use to
provide regular feedback to OGE about the progress of its programs, express
what needs improvement, and to share lessons they have learned.
In developing an evaluation methodology and a host of tools, OGE has
gained both a deeper understanding about the social impact its programs
generate as well as developed a greater capacity in monitoring them. OGE
has already begun applying lessons learned from this program. Firstly, OGE is
implementing the regular reporting. Secondly, OGE will continue to deliver
programs that proved successful. Lastly, OGE has identified content areas that
it would like to develop and improve. OGE looks forward to implementing the
evaluation methodology to measure the impact of future programs as well as
refining programs based on the results of the evaluation.
of survey respondents who
accessed content on OGE’s
digital platforms, took some
kind of action
of survey respondents from
Egypt reported teaching other
people in their communities
about what they learned from
the digital platform or in an
1.1 Goal and Objectives of the Impact Evaluation
One Global Economy (OGE) works with under-served communities in over 15
countries across five continents to create and implement information-based
development strategies to help households and individuals raise their standard
of living, join the economic mainstream, and innovate local solutions.
OGE has a diverse portfolio of programs, each suited to the unique community
it works in through a process of local stakeholder meetings and collaboration
with the community and local non-governmental organizations (NGOs). These
programs focus on access to broadband Internet connections, creating relevant
digital content in local languages, and training people in areas of digital
literacy, entrepreneurship, and techniques to gain the skills necessary to gain
employment and upward mobility.
Since its founding in 2005, OGE has launched 17 international community
development websites. These digital platforms (Note: Words in bold
orange font are defined in the Evaluation Dictionary) contain vital
community development content on topics such as education, employment,
entrepreneurship, and health written in local languages for readers at a 4th-
grade reading level. The content housed on the digital platforms is composed of
both global best practices and locally generated articles that focus on the needs
of the community for which it is written. OGE’s digital platforms now reach
more than 2.5 million unique visitors per year and since its founding in 2005,
8 million people have benefited from One Global Economy’s empowering
community development information. In addition, OGE has trained thousands
of people on digital literacy and assisted in the opening of Internet access
points to enable people to have the ability to access the local content on the
The purpose of this Impact Evaluation Program was to assess the long-term
impact of OGE’s existing programs on mission-driven outcomes and to build
OGE’s capacity to continually monitor and evaluate its programs going forward.
This is particularly important because OGE’s programs serve very different
communities, address a wide variety of issues, and do so across diverse levels of
development and urbanization.
development websites that
contain vital community
development content on topics
such as education, employment,
entrepreneurship, and health
written in local languages for
readers at a 4th-grade reading
level. The content housed on the
digital platforms is composed of
both global best practices and
locally-generated articles that
focus on the needs of the
community for which it is written.
The ability to find, evaluate, utilize,
share, and create content using
information technologies and
the Internet (Cornell University);
the ability to perform tasks
effectively in a digital environment.
Literacy includes the ability to
read and interpret media (text,
sound, images), to reproduce
data and images through digital
manipulation, and to evaluate and
apply new knowledge gained from
digital environments (Jones-Kavalier
4 Digital Inclusion Social Impact Evaluation
This impact evaluation focused on OGE’s programs in South Africa, Mexico,
Egypt, and India to design and implement its methodology. There are common
aspects of OGE’s work in all of those countries, but each country had different
programs, content, and training curricula. The framework and tools created
as part of this evaluation are meant to provide other organizations working in
digital literacy and inclusion with tools to adapt to measure their impact and
1.2 The Information-based Ecosystem Strategy
OGE’s digital inclusion model addresses three barriers to participation in the
information economy: lack of access to internet services, lack of access to
actionable local content in local languages, and lack of digital literacy skills
through what it calls the Information-based Ecosystem Strategy (see Figure 1).
LACK OF ACCESS TO THE INTERNET
In the information age, access to the Internet is vital to participate in the global
economy. Without access to the Internet, the information available online
will not reach underserved communities. OGE partners with community
technology centers around the world that serve rural and low-income
communities, and offer access to the Internet for free, or at a low cost that is
considered affordable by the local residents. Through its partnerships with
these local organizations, OGE creates a network of access points, where the
digital platform is hosted in each country.
LACK OF ACTIONABLE CONTENT IN LOCAL LANGUAGES
There is a disparity between global language populations and content
available online in those language. For example, in 2012, “Arabic content
on the web represented just 3 percent of the total digital content online,
yet Arabic speakers make up more than 5 percent of the global population”
(Franceschi Bicchierai 2012). Online content in local languages that are not
as widely spoken such as Kinyarwanda and Swahili account for even less, at
below 1 percent. Consequently, digital content focused on developmental
themes would account for even less, making it challenging for individuals in
developing countries to access information in their language that is aimed at
helping them to improve their standard of living. With more people accessing
the Internet in developing countries through their mobile devices and local
telecenters, there is increased potential for online information to contribute to
the improvement of development outcomes in the areas of income, health, and
Beehive content on the digital platform is designed to address community
development priorities identified by local stakeholders through a highly
participatory engagement and consensus building approach. The content
combines global best practices with highly localized information, written in
the local language for readers with low literacy levels.
LACK OF DIGITAL LITERACY SKILLS
Without the skills necessary to access information on the Internet, people will
not utilize access points and cannot find the actionable content available in
their language. OGE provides training to partner computer center managers
on ways they can support digital literacy in their community, launch demand-
driven programs and services that address local needs, and teach users how
to navigate the Beehive as well as access other existing online resources that
provide global best practice information.
OGE’s three-pronged strategy has successfully helped to create an ecosystem
that has enabled millions in underserved communities to access high quality,
actionable content on developmental themes.
1.3 Internet Access Points
OGE works with in-country partner organizations to utilize their existing
Internet access points as places to deliver trainings, hold stakeholder
engagement meetings, and teach the community about its digital platforms.
OGE helps to build the capacity of the local access point managers by drawing
on its experience and best practices to help the centers to be more organized
and to run more smoothly.
Figure 1. Information-based
Ecosystem for Digital Inclusion
6 Digital Inclusion Social Impact Evaluation
Digital literacy is critical to each person’s success in the 21st
century and to
maximizing the benefit of OGE’s programs, which is why it is the bedrock of OGE’s
training programs. “Digital literacy is the ability to find, evaluate, utilize, share,
and create content using information technologies and the Internet” (Cornell
University). It is the ability to perform tasks effectively in a digital environment.
“Literacy includes the ability to read and interpret media (text, sound, images), to
reproduce data and images through digital manipulation, and to evaluate and apply
new knowledge gained from digital environments” (Jones-Kavalier and Flannigan).
Email, social networking, word processing, and using the Internet are
necessary skills for anyone seeking gainful employment, a better job, or to
expand his or her business. OGE teaches these and more advanced skills in
many workshops. Using email and being able to navigate the internet open up
a huge amount of opportunities to save money, take advantage of government
programs, educate oneself, and make the kind of informed decisions that allow
people to improve their own lives.
OGE firmly believes that access, content and training are complementary
components of digital literacy, in that they work together to improve and
expand livelihood opportunities. Without training on how to use technology
properly, and training on how it can be incorporated into daily life, and applied
to finding information and solutions for development challenges, access
would not play such a transformative role. Similarly, access does not benefit
underserved populations if there does not exist content that meets them where
they are in terms of their language, education level, and local context. This is
why all of OGE’s training programs include digital literacy.
ENTREPRENEURSHIP AND TECHNOLOGY
Entrepreneurship is fundamental to growing today’s developing economies
and technology and globalization allow for unprecedented entrepreneurial
opportunities. OGE focuses on equipping emerging entrepreneurs with the
digital and mobile skills necessary to enter the formal business world and
confront everyday business challenges. Empowering entrepreneurs with core
soft and hard skills builds their potential to start and grow their enterprises,
and contributes to small business creation at the community-level.
OGE’s Information-based Ecosystem Strategy supports and strengthens its
entrepreneurship-focused programming by leveraging access, content, and
training to build the capacity of aspiring entrepreneurs. The entrepreneurial
training teaches basic technology skills to encourage technology skills application
as a valuable tool for small business growth. OGE guides the development
of essential business planning materials through providing templates and
instructions for creating a business plan, budget, and marketing plan amongst
others. OGE also provides online tools, localized information on entrepreneurship,
and mentoring support though mobile technology to entrepreneurs to help them
grow their existing businesses. Further, OGE helps small businesses access and
use banking services by teaching basic financial literacy to help small business
owners take out loans and to use government programs and financial institutions
that will help keep their businesses stable and profitable.
One such training program is OGE’s Mobile Entrepreneur’s Program
(MEP), which emphasizes the use of digital and mobile tools to grow and
improve small businesses. Workshops teach and monitor the achievement of
a series of online skills such as establishing an online presence for a business,
incorporating the use of social media into marketing plans, creating a LinkedIn
profile for online professional networking, monitoring business bank accounts
online and through SMS alerts, and conducting online research to increase
business knowledge and to access government or private funding, or additional
support through local organizations and services. Consists of modules
focused on similar career development themes and skills, but with a focus on
marketing individuals’ entrepreneurial skill sets to obtain jobs in a related field.
The Community Connectors curriculum in digital literacy and leadership is
focused on teaching young people how to apply digital and mobile tools to
become actively involved in community development initiatives, leaders in
their communities and technology ambassadors for their peers and families. The
program teaches a combination of information and communications technology
(ICT) skills and soft skills (planning and management, time management, project
management, public speaking, self-confidence, etc). The program varies in each
country and consists of additional modules addressing entrepreneurship, career
development, Internet safety etc., and can focus on helping youth improve
academic performance or find jobs through technology skills application
depending on the target population (See Table A, OGE Training Programs).
Community Connectors graduates have applied skills learned to lead workshops
at their community centers to teach community members technology skills,
conduct research on community and economic development topics to map out
projects their community may implement, develop local businesses, and identify
income-generating opportunities. The program fosters youth empowerment and
skills-building for personal, professional, and social development.
Internet safety has been incorporated as a focus in training in Mexico and India,
where OGE has a partnership with Symantec, and has carried out projects
promoting the involvement of locally-based Symantec employee volunteers.
Local employees and engineers have worked with OGE and its partners to
develop curricula for workshops and Internet safety campaigns focused on
teaching first-time Internet users how to protect their identities online, how
to protect their computers from viruses, and how to identify common threats
such as information theft and fraud. Symantec has also contributed content on
online safety, which is published on each digital platform.
The Mobile Entrepreneurs Program
(MEP) emphasizes the use of
digital and mobile tools to grow
and improve small businesses.
Workshops teach and monitor
the achievement of a series of
online skills such as establishing
an online presence for a business,
incorporating the use of social
media into marketing plans,
creating a LinkedIn profile for
online professional networking,
monitoring business bank accounts
online and through SMS alerts,
and conducting online research to
increase business knowledge and
to access government or private
funding, or additional support
through local organizations and
services. Consists of modules
focused on similar career
development themes and skills,
but with a focus on marketing
individuals’ entrepreneurial skill
sets to obtain jobs in a related field.
8 Digital Inclusion Social Impact Evaluation
South Africa Digital Literacy, Leadership,
Digital Literacy &
Digital & Mobile Literacy
Mexico Digital Literacy, Leadership,
Digital Literacy &
N/A Internet &
India Digital Literacy, Leadership,
Internet Safety, Job-Searching
& Career Development/
Digital Literacy &
N/A Internet &
Egypt Digital Literacy, Leadership,
Job-Searching & Career
N/A N/A N/A
1.5 Local Content
One Global Economy develops digital platforms written in local languages
that are accessible to people with a 4th grade reading level. OGE’s community
development-focused content is a combination of global best practices and
highly localized information. Best practice information covers topics that
are relevant all over the world, such as education, employment, health, and
money, and focus on teaching people specific best practices such as how to
write a business plan, how to select a school for their child, and basic prenatal
care. Highly localized content complements these global best practices by
providing information on how and where to access local resources, services,
and government programs that benefit underserved populations and help them
to take action to improve their lives in these areas.
Content is produced by local editors, NGOs , and student journalists. The
process begins with a stakeholders engagement workshop, where a roadmap is
created to guide the development of the digital platform, and then the content
is created over a period of 4-6 months. Each digital platform is launched as a
public website once there is content about all community development topics
identified in the workshop. Digital platforms are launched publicly with at least
200 highly localized articles. New content is added on specific topics as OGE
develops more partnerships with local NGOs. OGE expands upon content on
specific focus areas to complement its training programs. The digital platform
is hosted, maintained and checked for quality of content by OGE.
Table A. OGE Training Programs
1.6 Country Programs
OGE has worked in South Africa since 2007 when it established the Durban
Centre, a community technology center hosted at the Austerville Public Library.
The Durban Centre serves residents of South Durban and the Austerville
community, and offers free access to the Internet and the South Africa digital
platform, known as a “Beehive”, as well as hosts workshops in digital literacy
for youth and adults.
In (2008, OGE expanded its work in South Africa through a 5-year commitment
to the Clinton Global Initiative (CGI) with partners, Cisco, Siyafunda, and
Appleseeds Academy. Through the CGI, the partners established a network of
Community Knowledge Centers (CKCs) across Sub-Saharan Africa (in Kenya,
South Africa, Rwanda, Ethiopia, Nigeria, and Uganda). In South Africa, the
partners trained Siyafunda community telecentre managers in digital literacy
and business management skills essential to running sustainable computer
centers that effectively serve their surrounding communities by building upon
their basic offerings to offer demand-driven programs.
The partners also developed and launched Community Connectors, a youth
technology, leadership, and entrepreneurship training program and trained
partner centers in the implementation and delivery of the program to local youth.
In addition to the South Africa digital platform, OGE launched portals
localized to Durban, Johannesburg, and Cape Town. Over the last five years,
OGE has continued to work in partnership with Siyafunda to deliver its Mobile
Entrepreneurs Program, which has supported over 1,500 South African street
vendors, aspiring entrepreneurs, and small business owners through training
and skills building in technology and entrepreneurship skills. The training
is complemented with text messages to reinforce learning and disseminate
helpful information, tips, and advice for growing and improving enterprises.
Establishing a network of
Community Knowledge Centers
(CKCs), Community Connectors
Program, Mobile Entrepreneurs
Program, and Beehives for
South Africa, Johannesburg,
Cape Town, and Durban
Access Points Opened:
The name of OGE’s digital platform
in South Africa, Egypt, and India.
10 Digital Inclusion Social Impact Evaluation
OGE’s work in Egypt has a particular emphasis on employment and
entrepreneurship, where it has been working with local partner, CEOSS
(Coptic Evangelical Organization for Social Services) and HANDS (Hands
Along the Nile), based in Alexandria, Virginia, since 2011 to connect young
people to livelihood opportunities through Asset Mapping, access to global best
practices and highly localized content on the Egypt digital platform (known
as the Beehive), and training in digital literacy, entrepreneurship, and career
Through its Employment Through Information Technology program, OGE has
mapped the skills of approximately 23,000 young men and women along
with the outsourcing and employment opportunities of over 5,000 employers
to support opportunity identification and job matching through online and
mobile platforms. OGE and CEOSS have trained over 1,200 young people in
marginalized communities throughout Cairo, Minya, and Beni-Suef at 10 local
computer centers established through the program.
OGE launched the Egypt Beehive in addition to two localized portals created
for Cairo and Minya, which feature global best practice information and
community development-focused content in Arabic. The Beehives have served
over a million individuals since their launch in early 2013.
Asset Mapping, Online &
Mobile Job-Matching System,
Egypt, Cairo & Minya Beehives,
Access Points Opened:
10 (in Cairo, Minya & Beni-Suef)
Beehives for India, Mumbai,
and Pune in English, Hindi, and
Marathi; working with Pratham
Infotech Foundation; training
programs; education programs
Access Points Opened:
Jobs, health, digital literacy
OGE’s India Digital Inclusion Program focused on integrating the core
components of local internet access, localized online content, and digital
literacy and professional skills training into new programming and services at
10 computer centers run by OGE’s local partner, Pratham InfoTech Foundation
(PIF) in Maharashtra. OGE and (PIF) trained over 4,000 people in digital
literacy, technology, leadership, entrepreneurship, and community service, as
well as working with local health professionals to educate 488 people—most of
whom are young women—about health and sanitation.
The India Digital Inclusion Program has a particular emphasis on employment.
The program offers coaching services to help participants begin their
employment search, practice interviewing, job placement, and inclusion in an
online employment database. Many participants have learned how to create
and/or access email accounts and to use them for job-seeking and professional
The third aspect of this program is OGE’s digital platform known as the Beehive
(www.india.thebeehive.org). Viewed by over 200,000 people in the last year,
the Indian Beehive is composed of articles about financial literacy, employment,
and health written in English, Hindi, and Marathi for an audience with a 4th-
grade reading level.
12 Digital Inclusion Social Impact Evaluation
OGE launched the Mexico digital platform, InfoFácil.mx, in 2010 and the
Toluca digital platform (Toluca.InfoFácil.mx) in 2013. OGE has been working
with University partner, Universidad Autónoma del Estado de México (UAEM)
in Toluca and its journalism department to engage students in the development
of highly localized content for the two platforms by providing them with small
stipends, community service and internship credit in addition to incorporating
content creation into curricula, and offering the Mexico and Toluca digital
platforms as an online publishing outlet. Through engaging student journalists,
over 500 articles were produced for the Mexico platforms over a period of
three years/ to date. Since their launch, more than 3.3 million unique visitors
are accessing the digital platforms in Mexico, with over 300,000 accessing it
via their mobile phones.
OGE also launched its Community Connectors program in three rural villages
in Guanajuato (El Garbanzo, San Agustin, and Victoria), where it works with
local partner, Choice Humanitarian, and University partner, Universidad
Technológica del Norte del Estado de Guanajuato, to support the development
of sustainable community technology centers. With the past year, OGE also
implemented the youth program at three semi-urban high schools in Toluca, in
partnership with UAEM student trainers, training a total of approximately 150
youth in Mexico. In 2013, OGE delivered educational workshops in Internet
safety for first-time Internet users to over 500 Toluca residents together with
student trainers and locally based volunteer Symantec engineers.
Beehives for Mexico &
Toluca, Student Journalist
Beehive Editors Program,
Center Management Training,
Internet Security Campaign
Access Points Opened:
Education, Internet Safety,
There is a heightened awareness among the science and technology community
regarding the social impact potential of connectivity and economic inclusion
through digital and mobile tools. Academics and think tanks report on the
positive effects of access to the Internet, yet there is a need for real-life analysis
of the effects digital inclusion has on improving the lives of those in poverty.
OGE developed this Impact Evaluation to address this need and to quantify
its social impact potential and results, gather data that indicates lasting social
value of its digital inclusion programs, and track progress towards its defined
social impact targets.
In this Impact Evaluation, OGE sought to provide a guide for a rigorous analysis
of the “lift,” or margin of improvement, of its digital inclusion services upon
low-income populations in developing countries who are new to the Internet
and lack skills necessary to utilize the Internet to improve their lives. This
evaluation examines the lift in terms of the economic and human development
effects on individuals who participate in the digital online platform as users, in
the Community Connectors digital literacy training programs, and the patrons
of OGE’s partner Community Knowledge Centers.
2.1 Goal and Objectives of the Impact Evaluation
Through this Impact Evaluation, OGE sought to track the impact of the
digital inclusion program on participants’ economic opportunities and quality
of life, as expressed in terms of human development indicators: access to
opportunities that increase their income, education, and improve their health.
In developing a methodology to assess the impact of OGE’s training and
programs, this Impact Evaluation aimed to create a standardized way of
evaluating impact in digital literacy programs for the benefit of other
organizations doing similar work. Digital inclusion programs pose unique
challenges in designing and implementing an impact evaluation. By nature,
these programs are not place-based interventions; while the topics are
localized geographically the content can be accessed globally. Unlike traditional
development work, OGE’s programs are not defined in geographic terms
and the organization is not focused on working in specific communities for
extended periods of time. Therefore, this evaluation required the development
One Global Economy works
with under-served communities
to create and implement
strategies to help households
and individuals raise their
standard of living, join the
economic mainstream, and
innovate local solutions
14 Digital Inclusion Social Impact Evaluation
of new methods and tools to assess the impact of OGE’s programs and of
digital inclusion programs in general. Thus, one of the main goals of this
evaluation was to create a toolkit of adaptable tools and frameworks for other
organizations to implement evaluations in this expanding niche.
This organizational capacity-building project also served to strengthen focus
and codify an outcomes-based approach for implementation across all OGE
country programs in the future.
THE OBJECTIVES OF THE IMPACT EVALUATION WERE TO:
• Articulate a theory of change of how OGE’s training and programs lead to
• Develop key qualitative and quantitative performance and impact
• Create a comprehensive strategy to gather data for the key indicators,
including routine programmatic reporting and follow-up with participants
– Implement a follow-up survey in partner computer centers and online
in four actively-funded countries (Egypt, India, Mexico and South Africa)
– Develop internal tools for more efficient communications regarding
progress toward impact metrics
• Create a toolkit with adaptable templates that provides a comprehensive
guide to implementing an impact evaluation for digital inclusion
• Analyze the process and outcome of this initial impact evaluation with
attention to improving programming, outcomes, and refining the
• Build internal capacity to monitor and evaluate the impact of Digital
Inclusion programs for future projects
• Share the findings, results, and conclusions publicly
2.2 Developing the Evaluation Design & Framework
The process of developing the evaluation design and framework included:
• Creating a theory of change to articulate how OGE achieves its impact.
This causal logic of how OGE’s programs deliver results is the underpinning
of the impact evaluation and was used to create its framework, indicators,
and data collection methods.
• Adapting two program planning tools, the Results Chain Model and the
Success Formula, to further map out the inputs, strategies, and outputs that
lead to OGE’s hypothesized impact. These two exercises identified the process
that leads to impact as well as what the impact looks like.
• Creating a comprehensive list of indicators using these two tools to ensure
all aspects of programs were measured as part of the evaluation. Each step
of the Results Chain was included, as well as a clear definition of all the skill
and content categories identified through the Success Formula process. The
comprehensive list of indicators was distilled into a list of key indicators, tested
in this evaluation and to be used as a menu of programmatic indicators for
• Using the list of indicators to develop data collection tools and systems to
measure each of the key indicators identified in this process.
THEORY OF CHANGE
OGE began the impact evaluation with the question: What effect does digital
literacy and access to the Internet have on social and economic development
of individuals and communities? To answer this question, the first step of the
evaluation was to hypothesize how OGE achieves its impact – the organization’s
theory of change.
The evaluation team deconstructed the organization’s mission by creating
internal definitions of each piece of the mission (see Evaluation Dictionary:
Mission Definitions) to articulate the organization’s theory of change:
Strengthening access to digital and mobile tools by partnering with local
computer centers, training computer center managers and patrons in digital
literacy and business skills and providing access to locally-written, actionable
content on developmental themes in their own languages will enable households
and individuals to take informed action that will transform their lives and
Figure 2. Evaluation
16 Digital Inclusion Social Impact Evaluation
More specifically, the OGE training curricula combined with both locally
generated content and global best practices in key issue areas provides users
with the skills and information they need to take significant steps towards
participating in both local and global economies. Through digital inclusion,
users and their communities can raise their standard of living and join the
economic mainstream by increasing their economic, health, and educational
opportunities, such as starting a small business, improving their livelihood
prospects, improving their ability to make informed health, education, and
business decisions; and increasing their participation in the local community.
A key element of OGE’s Theory of Change is that the individuals in the
communities OGE serves must take action to complete the cycle of change.
While OGE fosters the information ecosystem with local partners and creates
and shares opportunities for people to take action to improve their lives, OGE
believes firmly that the sustainable solution is one in which local participants
are the key actors, leading local progress.
Figure 3. One Global Economy’s Theory of Change
RESULTS CHAIN MODEL
The evaluation team then applied the theory of change to two program
planning tools to further articulate how the organization achieves its impact.
Through clearly defining and articulating the mechanisms and processes that
lead to mission-driven outcomes, the team sought to create a comprehensive
evaluation model that encompasses all elements of OGE’s programming to
best assess and estimate the “lift” and a set of program planning tools adapted
to OGE’s theory of change that could be used for future programs. Thus, these
tools were used both to help guide the evaluation by looking retrospectively
at OGE’s program models and as an exercise to develop planning tools that
would be used in all future programs and adapted for other organizations and
included in the Evaluation Toolkit.
First the team used the World Bank’s results chain model found in the
Impact Evaluation in Practice. A results chain shows cause and effect: how the
sequence of program inputs (e.g. staff and other resources) and activities lead
to specific program outputs, and how these outputs are directly related to
accomplishment of program goals (i.e. impact). This model shows how specific
program activities directly lead to the desired program goal and what must be
done in order to achieve programmatic targets and results. It also documents
what goes into a program (the inputs), which is integral for replicability and
scalability of programs.
The evaluation team created an overall results chain model for the organization
as well as adaptations for country-level programs. We used this structure
to succinctly visualize and describe each of OGE’s programmatic activities,
including their anticipated proximate and distal impact. The results chain
demonstrates the cause and effect relationship of OGE’s programming.
Moving forward, OGE will require program staff at the country level to create
a Results Chain Model for their program so that they can understand how
programmatic activity results in the desired program goal and what must be
done in order to achieve programmatic targets and results. By creating a model
that showcases how the program’s theory of change is directly tied to the
program’s specific focus and goals, stakeholders create a common vision for
the program, which serves as a roadmap for both implementing and evaluating
Results Chain Model
Template form of the results chain
for country teams to fill in at the
beginning of a program, a practice
for them to see how programmatic
activities are directly tied to longer-
term, mission-driven outcomes.
18 Digital Inclusion Social Impact Evaluation
Note: The full version of the Digital Inclusion Results Chain includes a section for
listing indicators along each step of the chain. Country teams list the inputs, activities,
outputs, outcomes, and impact for their programs and then create a list of indicators
to measure each.
OGE also adapted the Success Formula framework from the firm Mission
Measurement and its CEO Jason Saul’s book Benchmarking for Nonprofits as
a second program planning tool for this evaluation.
The Digital Inclusion Success Formula is a universal tool that enables One
Global Economy and other digital inclusion nonprofits to consider its social
impact as a simple equation. At the end of the formula is the ultimate impact
the organization aims to achieve through programming, created through the
sum of the intermediate steps at a strategic, organizational level.
The Success Formula makes OGE’s Theory of Change actionable and
measurable. At the top of the Success Formula is the underlying principle
that acquiring digital literacy and key skills combined with access to both
locally generated content and global best practices in key issue areas provides
users with the skills and information they need to take significant steps
towards participating in both local and global economies. The Success
Formula is used to outline the strategy and activities that will be used in
the key issue areas (income, health, education, and social development) of
the program, and the performance indicators that will be used to measure
progress and success.
Figure 4. OGE Digital Inclusion Results Chain
Adapted from Gertler, Martinez et al.
Impact Evaluation in Practice.
The World Bank 2011.
CONTENT FOR IMPACT: SKILL CATEGORIES
To further refine the model, we brainstormed and defined the main content
areas of OGE’s programming, which together we believe are critical to
achieving OGE’s desired impact.
We called these groupings of content “Skill Categories” because the goal
of the content in digital inclusion programming is building capacity and
informed action. Part of OGE’s theory of change is that access to and taking
informed action on these specific content areas are what create the desired
impact as described in OGE’s mission statement.
The four primary Skill Categories we identified are: digital literacy, business
and entrepreneurship, livelihood and employment, and personal/social/
community development. Of note, Digital Literacy is a fundamental skill,
because it creates the ability to access content in the other Skill Categories.
The Skill Categories were further broken down into sub-skills, again using
the criteria that these skills are imperative to achieving the desired results
and are a fundamental part of OGE’s training curricula and programmatic
activities. The skills and sub-skills identified in this exercise were
fundamental to creating indicators and creating training assessments as
part of the routine reporting component of the Impact Evaluation.
Figure 5. OGE Digital Inclusion Success Formula
Note: The full version of the Digital Inclusion Success Formula includes sections for listing program strategies and indicators
for income, health, education, and social development. Country teams list the strategies they will use in their programs to
create change in these categories, and then create a list of indicators to measure each.
The ability to find, evaluate, utilize,
share, and create content using
information technologies and
the Internet (Cornell University);
the ability to perform tasks
effectively in a digital environment.
Literacy includes the ability to
read and interpret media (text,
sound, images), to reproduce
data and images through digital
manipulation, and to evaluate and
apply new knowledge gained from
digital environments (Jones-Kavalier
20 Digital Inclusion Social Impact Evaluation
The indicator table serves as the backbone of the evaluation. The evaluation
team developed indicators, or means to measure the comprehensive model
of the theory of change. Indicators were created for each step in the Results
Chain, and within each step for the skill categories. Creating the indicators
in this way ensured that we measured all aspects of the process of delivering
programs and the components that are imperative to creating impact.
The indicators for inputs, activities, and outputs are process indicators, which
measure what is put into a program and the deliverables of programmatic
activities. The indicators for outcomes and impact are outcome indicators,
which measure the lasting impact of the programs on the participants.
To create outcome indicators tied to OGE’s mission of raising the standard of
living, join the economic mainstream, and innovate local solutions, we used the
Human Development Index, published by the United Nations Development
Program. This Index uses composite indices to measure health (life
expectancy), education (mean years of education), and income (gross
national income, GNI, per capita) to rank countries in terms of development.
While the indicators themselves cannot be used at the individual and
household level, we used these broad categories to define standard of living
for the purposes of this impact evaluation.
Figure 6. Skill categories and sub-skills
Indicators to measure whether
planned activities took place, add
more details in relation to the
product (“output”) of the activity,
and/or monitor the quality of the
activities conducted, based on
a number of established quality
criteria or standards. (WHO)
Indicators that measure the
objectives of an intervention, that
is its ‘results’, its outcome. They are
the result of both the “quantity”
(“how many”) and quality (“how
well”) of the activities implemented.
See full definition in the
OGE therefore defines standard of living as the income, education, and
health opportunities accessed by a given individual or household. It includes
• INCOME: Changes in annual income and savings, employment and career
• EDUCATION: Pursue or attain GED, applied to or enrolled in university.
• HEALTH: Accessing quality health care systems, disease management,
access preventative medicine (prenatal care, sanitation and water access,
family planning, immunizations, infant care, nutrition)
By using OGE’s mission statement to identify and define the final outcomes
in the results chain, we aimed to ensure that this impact evaluation measures
OGE’s results against its mission, reinforcing the mission at all levels and
integrating the mission in its evaluation of itself. All indicators were assessed
to be specific, measurable, attributable, realistic, and targeted.
2.3 Evaluation Design and Framework
The impact evaluation applied a conventional social science approach using
a non-experimental design. We collected information on the impact of
OGE’s programs and activities on the desired outcomes through standardized
routine program monitoring and a follow-up survey with training
participants and digital platform users.
The nature of OGE’s programs renders an experimental (randomized) or
quasi-experimental (comparison group) approach infeasible. Therefore, the
evaluation compared participants’ utilization of information technology and
key socio-economic indicators before and after training.
The evaluation used several methods to measure changes in the skill
acquisition, utilization, and impact of information technology and resources
from OGE’s programs. The evaluation methods bolstered routine reporting
to monitor programs and assess input, process, and immediate outcomes of
programs and initiated a survey to evaluate outcomes.
By measuring the resources, activities, and program achievements,
organizations can track the causal logic of program outcomes, determine if
interventions were carried out as planned, if the interventions reached the
targeted beneficiaries, and if the interventions were performed on schedule,
and what the results from the interventions were for the beneficiaries.
22 Digital Inclusion Social Impact Evaluation
As part of the evaluation, OGE developed three components to program
• A STANDARDIZED MONTHLY REPORTING FORM and system to collect
program data regularly, including data on the resources used (inputs),
activities completed, and direct program outputs. Field staff and partners
complete this form on a monthly basis to record progress towards key
program objectives, which will directly feed into the impact evaluation.
• A STANDARDIZED TRAINING REPORT FORM, which includes the content
of the training, what curricula were used and how they were adapted, and
the number of people trained. In addition, post-training assessments were
created to ascertain the level of proficiency in the key skills taught during
• A TRAFFIC MONITORING TOOL to track OGE’s international digital
platforms using Google Analytics. This monitoring tool allows OGE to
gather significant data about the people who are using the digital platforms.
To measure the impact of digital inclusion programs, it is necessary to survey
participants and measure behavior and socio-economic changes of those who
participated in trainings and/or accessed information via the digital platform.
• IMPACT SURVEY: A standardized survey given both online and through
active follow-up with program participants after they have been trained
in one or more of OGE’s programs (e.g. Digital Inclusion, Community
Connectors, Entrepreneurs, etc). The purpose of this survey is to find out
the longer-term impact of the training on participants’ lives and socio-
economic condition. The survey was conducted in four countries where
OGE has conducted trainings—specifically, in India, Mexico, Egypt, and
South Africa—for this initial evaluation.
DATA COLLECTION TOOLS
As part of creating the indicator table, we identified when and how the data
would be collected and from whom. This process identified the different data
collection tools that were necessary and when they should be implemented.
To develop the data collection tools, we used the indicators to create
questions. We systematically went through the indicator table to ensure
that all data required to measure the indicators will be collected.
A summary of data collection along the results chain is presented in Figure
7. By measuring indicators along the results chain, the impact evaluation
methodology prevents producing a ‘black box’ that only identifies whether or
not the predicted results occurred without explaining why or how the results
were achieved. See Appendix A for a detailed description of data collection
methods across the results chain.
As we refined the data collection tools, we refined and updated the indicator
table. The survey questionnaire went through several rounds of reviews with
the team to streamline, simplify and ensure that high-quality data would be
collected that would measure the desired outcomes of the evaluation and
then was sent to local partners for review and feedback.
2.4 The Evaluation Toolkit
One of the goals of this Impact Evaluation was to create a methodology to
enable the organization and its partners to better measure the social and
economic impact of its Digital Inclusion Programs so that the lessons can
be used to refine and scale future programs. Therefore, as part of the Impact
Evaluation, OGE developed an Evaluation Toolkit to codify the process, tools,
and implementation of a digital inclusion impact evaluation to be used
in the future for all of OGE’s digital inclusion programming and for other
organizations doing similar programming to adapt and use it to measure
their own impact.
This toolkit is a resource to help the management team, programmatic staff,
and partners succeed at enabling digital inclusion toward developing their own
communities. The purpose of this toolkit is to enable better digital inclusion
practitioners to understand, define, measure and improve their impact.
In the Impact Toolkit, OGE offers several guides for localizing the methodology,
including a Success Indicators Planner that breaks down OGE’s theory of
change and definitions of digital inclusion success. This will help to cultivate
local critical thinking, ownership, and leadership of digital inclusion progress
and measurement on the ground. Throughout the process of implementing
this initial impact evaluation, we incorporated lessons learned into the toolkit
and refined the framework, data collection tools, and systems.
Figure 7. Data collection summary along the results chain
24 Digital Inclusion Social Impact Evaluation
This Digital Inclusion Impact Evaluation Toolkit features capacity building
guides, templates, and exercises for One Global Economy (OGE), its
international partners, and other Digital Inclusion organizations to better
define, plan and measure against its proposed outcomes (see Figure 8). OGE
is sharing this universal toolkit with templates designed to be customized for
each country/program with the collaboration of local partners, and enables
partners to adapt and help execute the evaluation in their country.
For a detailed list of the contents of the Impact Evaluation Toolkit, see
Figure 8. The Digital Inclusion
Impact Evaluation Toolkit
The evaluation team included OGE programmatic and leadership staff in
collaboration with an evaluation consultant who provided technical guidance
and assistance. The team began planning for the evaluation in the fall of 2013.
The Impact Evaluation methodology, including the Theory of Change, Results
Chain Model, Success Framework, indicators, and data collection methods
were designed and created over the course of five months, from October 2013
to February 2014.
Once the indicators were created and the methodology outlined, the team
began designing an end-user survey and internal monitoring tools in early 2014
and implementing them in the summer. The analysis of all data and writing of
the final report took place between August and September 2014 (see Figure 9).
The purpose of this impact evaluation was to be formative for continued
evaluation of impact. As such, this evaluation was carried out in only four
countries to test and improve the methodology. Moving forward, OGE will
continue to implement the routine reporting created as part of this evaluation
(monthly, training, and traffic reports) in all of its current and future country
programs and will implement subsequent surveys to continue to assess the
impact of its programs.
OGE chose India, South Africa, Mexico, and Egypt as the countries in which
to conduct this formative Impact Evaluation for a number of reasons. Firstly,
OGE had ongoing programs in each of those countries, which meant it had on-
the-ground partners who could help contact people who had taken trainings
and accessed content. Secondly, these countries represent a diverse array of
communities OGE serves, ranging from the densely urban areas of Mumbai
to very rural villages in the Mexican countryside with many gradations in
between. Thirdly, these four countries covered a wide array of different
programs that OGE employs.
26 Digital Inclusion Social Impact Evaluation
3.1 Results Chain Model and Success Formula
The Results Chain Model and Success Formula are program planning tools,
and served two functions in this evaluation. First, we used these tools to
retrospectively outline the theory of change and strategies for achieving the
desired outcomes of OGE’s programming overall. This was an essential step
to help develop a comprehensive set of indicators that would drive the data
collection methodology of the evaluation. Second, we field tested these tools
with country teams to gain consensus and improve the tools so that they could
be included in the Evaluation Toolkit as program planning tools – used in
all future OGE programs and adapted by other organizations when planning
The feedback from country teams in South Africa and Mexico was that
these were useful tools in looking at the program as a whole and better
understanding the relationship between program outputs and impact.
Managers in these countries were able to participate in a step-by-step process
of diagramming how these programs are designed to help their beneficiaries.
This clarified the strategies through which real change is affected and
highlighted the importance of their programmatic activities.
Of note the team in South Africa used an innovative approach to completing
the Success Formula: they completed the formula from the perspective of
one person who couldn’t find a job to figure out the best way to work with
them. This approach was incorporated into the directions for completing
Figure 9. Impact Evaluation Timeline
the Success Formula and results chain, but from a more holistic perspective:
completing the exercises several times over with different client avatars, each
with different problems and needs, as a needs assessment, program planning,
and evaluation tool. For example in Mexico, the team of student journalists
reflected on how articles on various locally relevant topics contributed to
community, personal, and social development, and how different formats
for disseminating global best practice information provided access to more
services and resources locally.
Overall, a challenge with these tools was getting program teams to be as specific
as possible about programmatic activity. Without guidance, local teams often
wrote blanket statements that were not particularly meaningful or measurable.
To help guide teams in the future, the evaluation team created a list of program
activities and indicators for them to include, adapt, or expand upon.
3.2 End-User Survey
SAMPLING STRATEGY AND LOCATION
The end-user survey is the most significant surveying OGE has ever undertaken
of the communities it serves. OGE posted links to the survey on the local
online portals in each country to try to get online users to participate and
actively followed-up with past program participants in communities to conduct
the survey in-person.
The survey was implemented in Mexico, Egypt, South Africa, and India. The
target population was anyone who had either visited a local online portal
or had attended a training course with an OGE curriculum. This is a very
demographically unspecific set of people, but that was necessary given the
diverse population that OGE serves.
A standardized survey questionnaire was uploaded to SurveyGizmo, a web-
based survey program that allows for skip logic, multiple choice, and write-in
responses. A link to the survey was posted to the digital platform in each of
the four countries, with translations in Arabic, Hindi, Spanish, and English
available. In addition, staff of OGE’s partners administered the questionnaire
in-person with hard copies to members of their community.
Data collection began on March 15 and continued through July 22, 2014.
It is impossible to overstate how critical OGE’s in-country partners were to
the success of this survey. OGE worked with Pratham InfoTech Foundation in
India; the Coptic Evangelical Organization of Social Services (CEOSS) in Egypt;
28 Digital Inclusion Social Impact Evaluation
Siyafunda in South Africa; and Universidad Autónoma del Estado de México
in Mexico. OGE works with these partners to implement its programs in each
country. Not only do these NGOs have a deep reach into the communities
where OGE works, but in many cases they also run computer or community
centers where OGE could get people from the community who had participated
in its programs to take the survey. The partners met with people in their
communities, conducted outreach, and encouraged people to take the survey.
DATA ENTRY, MANAGEMENT AND ANALYSIS
Most individual respondents completed the survey on SurveyGizmo. The hard
copies of the survey were directly entered into SurveyGizmo by members of
OGE’s partner NGOs to ensure standardized data between online and in-
The analysis plan used the Success Formula and Results Chain as a guide
to analyze the key performance indicators identified during the process of
creating those tools. For demographic and direct program impact variables,
data was compared between countries to see how the different programs
affected outcomes. For longer-term impact variables, data was compared
between those who did and did not participate in a training to see the long-
term effects of the training on key impact variables, particularly the impact of
training on taking informed action in key development areas. Furthermore, we
analyzed the data by time since training to uncover the longer-term impact of
the program and training.
Overall, this analysis was conducted with an eye towards understanding the
effect of content and training and how we can improve our programs to
achieve greater impact.
A criterion of p<0.05 was used to assess the statistical significance of
differences between groups. Differences with p<0.01 were noted as
3.3 Routine Reporting
A major goal of the Impact Evaluation was to build OGE’s capacity to
monitor and evaluate its programs. Therefore, as part of the evaluation, a
comprehensive routine reporting system was developed and implemented
(see Section 3.2 above).
OGE collaborated with its partners in each of the countries to create
the routine reporting templates, including monthly reporting forms, a
standardized training report form, and a traffic monitoring tool. The Monthly
Routine Reporting Form and the Training Report Form together create
a comprehensive internal monitoring system for OGE. The purpose of this
system is to:
• STANDARDIZE INFORMATION ACROSS COUNTRY PROGRAMS. The same
information will be collected from all of OGE’s programs using this form.
With standardized information, we can track programmatic activities and
progress organization-wide. Regular updates on program activities ensure
the history of a program is documented thoroughly.
• ENABLE “REAL-TIME” REPORTING. Activities are updated monthly as
they happen, rather than waiting until a report is due. If updates program
milestones are given on an on-going basis, the burden of giving information
for reports at the end of the period is reduced. Rather than going back and
asking country teams what happened during the reporting period, staff can
generate a report using the data that has been entered already.
• CONSOLIDATED SOURCE OF PROGRAM INFORMATION.
The Monthly Routine Reporting Form collects information on major activities
(e.g. trainings or workshops, computer centers opened, outreach activities, and
community activities), content developed for the digital platform, computer
center activity (e.g. users and center development activities), follow-up with
training participants, and stories of people innovating local solutions, quotes
and testimonials, and other notes. The Training Report Form is part of a larger
set of tools for designing and evaluating trainings, which include choosing
curricula and teaching objectives and post-training evaluations of skills
acquired. The Training Report Form reports on the content and objectives of
the training, the attendees, and the results of post-training assessments.
The reporting system utilizes a web-based survey design, Survey Gizmo, with
skip logic: answers to initial questions direct the respondent to subsequent
questions to complete information specific to that type of activity and allows
them to skip questions that are irrelevant.
Standardized monthly report form
submitted by the country team to
headquarters, including directions
and a description of how this
reporting fits into the larger impact
evaluation and OGE’s mission.
Training Report Forms
Standardized training report form
submitted by the country team
within one month of training to
headquarters, including directions
and a description of how this
reporting fits into the larger impact
evaluation and OGE’s mission.
30 Digital Inclusion Social Impact Evaluation
By including the local partners in the development of the reporting system,
OGE was able to integrate their feedback to make the system more accurate,
relevant, and feasible. For example, the South Africa Manager provided
helpful feedback for localizing the Monthly Reporting Form, to the Mobile
Entrepreneurs Program specific to South Africa, by suggesting that the form
integrate more skills in the checklists and that the form include a space for
local managers to provide testimonials from participants, which are gathered
regularly. Walking in-country managers and partners through their own
success formula and results chain often helps them gain a better perspective
on the work that they are doing.
Google Analytics allows OGE to track the information of digital platform users.
By tracking information using the Traffic Monitoring Tool, OGE can learn
which pages are viewed the most and for the longest amount of time in different
areas, where users come to the digital platforms from, and which topics resonate
the most. This allows OGE to give partner organizations accurate feedback
about articles written by community members, as well as pointing OGE towards
the issues that are most pertinent to different communities.
Traffic Monitoring Tool
A form to collect and consolidate
information from Google Analytics.
4.1 Google Analytics
TOTAL AND UNIQUE VIEWS
At the time of this report, OGE has had 10,756,356 sessions and 9,386,745
unique views on its digital platforms since 2005. As seen in Figure 10, unique
views have been steadily increasing yearly for OGE digital platforms overall.
Unique views on the Mexico digital platform have remained consistent since
July 2011 at an average of 450,000 unique views per year. The Egypt digital
platform, which started in 2013, had over 570,000 unique views that year.
AVERAGE NUMBER OF PAGE VIEWS
AND AVERAGE TIME SPENT ON THE PLATFORM
Two ways to measure online engagement are the number of pages viewed in
a session and the amount of time spent on the website. OGE monitors these
two metrics on all of its web properties using Google Analytics (Figure 11).
Figure 10. Unique views per
year by country and OGE overall
32 Digital Inclusion Social Impact Evaluation
In South Africa the number of pages visited per session has dropped a small
amount from just over two pages per session to about 1.8 pages per session
while the time spent on the website has increased from about 1:05 to 1:18.
There has been a similar trend in Mexico where page views decreased from 1.8
per session to 1.4 while the time on the sites has risen dramatically from 1:25 to
about 2 minutes. This signals a deeper engagement with the material.
India has trended oppositely, where page views have gone up from an average
of 2.85 to 3 and time spent on the web property has fallen from about 3:30 to
about 2:55. Though the time on the site has decreased, almost three minutes
spent on the site remains higher than the industry average of 10 to 20 seconds
(Nielsen 2011) and the “average of a little over one minute on a newspaper
website” (Grabowicz 2014).
Figure 11. Average number of pages viewed and time spent per session on the digital platform
As seen in Figure 12, health, education, and employment are consistently
among most accessed content. Because each portal is localized to its respective
region, programmatic efforts focus on relevant topics, and their significance to
the target population is often highlighted by trends in users’ activity as seen in
the diversity between countries in these results.
For example, South Africa shows the most views for citizenship-related content
compared to Mexico. In South Africa, the stakeholders’ engagement workshop
under the then new post-Apartheid democracy. Through developing content with
access local services, initiate processes for obtaining licenses and certificates, and
become informed and active citizens through participating in elections.
In other cases, top content indicates hot issues and can inform article creation for
the development of more specific content addressing those topics. For example,
soon after the launch of the Mexico digital platform, a review of Google Analytics
and comments from users on the platform showed that there was a demand for
more employment-focused articles. OGE, its local team, and student editors at
UAEM (partner University in Mexico) focused on developing articles aimed at
young people and job seekers, increasing the number of online resources that
help people understand how to apply for jobs online, learn interview skills, and
link them directly to Mexico-based online job-search engines. The demand for
livelihood-focused content addressing employment and education issues is also
illustrated by the high percentage of content viewed on women and teenagers,
which remains a top topic due to additional content OGE developed to guide
high school dropouts, single mothers, and adults through choosing the best
way to complete their education and earn degrees and certificates.
The structure of the program and its components in each individual country can
also explain why certain topics are more frequently trafficked over others. For
example, in Egypt, the goal of the project is to link young people to employment
opportunities. Content on the Egypt digital platform therefore largely focuses on
connecting individuals to resources addressing issues of financial literacy, income
generation, entrepreneurship, agribusiness, and best practices for microenterprise
development. In India, a recent program focused on implementing health camps
to teach basic sanitation practices, preventative healthcare, and adolescent health
at the partner centers. Many of the health camps have focused on sexual health
and serve young women, who lack access to such information. Top content
viewed on health in general, and articles on safe sex, and date rape drug awareness
are reflective of a need for adolescent health information.
See Appendix C for the top five articles by year for each country and OGE overall.
34 Digital Inclusion Social Impact Evaluation
MOBILE VS. DESKTOP USERS
The percentage of people accessing OGE’s digital platforms from mobile
devices steadily increased since 2011. In South Africa between 2013 and
2014, 53% of people accessing the digital platform did so via mobile devices.
Though the percent of mobile users are still low in Egypt (9%), India (22%),
and Mexico (24%), they are not insubstantial. All four countries are showing
substantial upward trends, with the percent of mobile users doubling in all
four countries in the last year. This trend has serious consideration for future
programs, which will be discussed later in this report.
Figure 12. Top content topics by country by year
Figure 13. Percent of all users accessing the digital platform with a mobile device by country by year
4.2 Survey Participants
OGE collected a total of 4,676 surveys between March 15 and July 22, 2014.
Incomplete surveys (n=308) and surveys from respondents living in countries
not included in the evaluation (n=8) were removed from the analysis. A total
of 4,364 surveys were included in the analysis: 2,407 from Egypt; 1,292 from
India; 491 from South Africa; and 174 from Mexico (see Figure 14).
The number of respondents in total and in each country is sufficient for OGE’s
purposes because the total is large enough to test for statistical significance and
each country’s total reflects the number of participants in OGE programs. For
instance, OGE works with CEOSS, one of the largest social service NGOs in
Egypt. The programs reach a lot of people and therefore the most surveys were
collected in that country. In contrast, OGE’s work in Mexico is in a few very
rural villages so the 174 completed surveys represent a very large number of the
participants in OGE’s programs there.
4.3 Background Characteristics of Respondents
Table B presents the demographics of survey respondents. This is a very
detailed and representative snapshot of OGE’s constituency, and it reflects
OGE’s focus on populations underrepresented in the economic mainstream,
particularly women, young people, and the unemployed.
Overall, 59% of the respondents were women; while the gender of respondents
varied significantly between countries the range was not large (56% in Mexico
to 65% in South Africa). See Spotlight: Gender for a detailed analysis of the
impact of OGE’s programs on women.
Figure 14. Data for analysis
36 Digital Inclusion Social Impact Evaluation
Of note, most respondents were between the ages of 18 and 34 at the time of
the survey (90%). This is partially because of the large amount of respondents
in Egypt (55% of the total survey) who were of that age (2,400 of the 2,407).
However, that trend held across the other countries as well, with 78% of Indian,
91% of Mexican, and 85% of South African respondents below the age of 30.
This reflects the emphasis OGE’s programs have on training young people
because youth around the world often lack job prospects and job skills, and
employment as a young adult is one of the best routes out of poverty. Young
people, who are more likely to have free time and less likely to have families,
are particularly able to take advantage of the job skills and entrepreneurial
programs that OGE offers.
Almost half of all Egyptians surveyed (48%) are unemployed even though 69%
have some college experience or have graduated from college. This speaks
to the difficult economic situation that many Egyptians face, but also to the
benefits of becoming as job-ready and as good at the soft skills of interviewing
and networking as possible. In India, despite a low unemployment rate, many
of the respondents were students (27%) or homemakers (25%). These are two
populations that can greatly benefit from digital literacy and job skills as well
as entrepreneurial training because of the opportunities they provide for small
business creation and joining the job force.
Male 1,010 (42.2%) 526 (40.8%) 76 (43.7%) 168 (35.3%) 1,780 (41.1%)
Female 1,382 (57.8%) 763 (59.2%) 98 (56.3%) 308 (64.7%) 2,551 (58.9%)
12-18 3 (0.1%) 2 (0.2%) 69 (39.7%) 2 (0.4%) 76 (1.7%)
18-24 1,376 (57.2%) 598 (46.4%) 66 (37.9%) 255 (52.3%) 2,295 (52.7%)
25-34 1,022 (42.5%) 406 (31.5%) 25 (14.4%) 157 (32.2%) 1,610 (37.0%)
35-44 4 (0.2%) 197 (15.3%) 9 (5.2%) 53 (10.9%) 263 (6.0%)
45-54 0 74 (5.7%) 4 (2.3%) 15 (3.1%) 93 (2.1%)
55-64 0 9 (0.7%) 0 6 (1.2%) 15 (0.3%)
65+ 0 3 (0.2%) 1 (0.6%) 0 4 (0.1%)
Married 594 (24.9%) 642 (50.3%) 25 (14.5%) 49 (10.2%) 1,310 (30.3%)
23 (1.0%) 23 (1.8%) 7 (4.0%) 19 (4.0%) 72 (1.7%)
Single 1,771 (74.1%) 612 (47.9%) 141 (81.5%) 413 (85.8%) 2,937 (68.0%)
Elementary 16 (0.7%) 246 (19.0%) 15 (8.6%) 2 (0.4%) 279 (6.4%)
Middle school 10 (0.4%) 323 (25.0%) 53 (30.5%) 9 (1.9%) 395 (9.1%)
Some high school 43 (1.8%) 214 (16.6%) 20 (11.5%) 54 (11.0%) 331 (7.6%)
High school grad 416 (17.3%) 232 (18.0%) 32 (18.4%) 305 (62.1%) 985 (22.6%)
Some college or
258 (10.7%) 173 (13.4%) 16 (9.2%) 60 (12.2%) 507 (11.6%)
College or technical
1,553 (64.5%) 97 (7.5%) 32 (18.4%) 61 (12.4%) 1,743 (39.9%)
Other 111 (4.6%) 7 (.5%) 6 (3.4%) 0 124 (2.8%)
Homemaker 118 (5.0%) 320 (24.8%) 14 (8.2%) 9 (1.9%) 461 (10.7%)
Retired 1 (0.1%) 5 (0.4%) 2 (1.2%) 0 8 (0.1%)
Self-employed 246 (10.4%) 88 (6.8%) 8 (4.7%) 44 (9.2%) 386 (9.0%)
Student 46 (2.0%) 349 (27.1%) 95 (55.6%) 145 (30.2%) 635 (14.8%)
Unemployed 1,139 (48.4%) 62 (4.8%) 9 (5.2%) 197 (41.0%) 1,407 (32.8%)
801 (34.1%) 465 (36.1%) 43 (25.1%) 85 (17.7%) 1,394 (32.5%)
* p<0.05 ** p<0.01
Table B. Background characteristics of respondents
38 Digital Inclusion Social Impact Evaluation
4.4 Training and Access
As seen in Figure 15, 3,562 people who took the survey (82%) participated in
one or more trainings by OGE or one of its partners between 2009 and 2014.
Most respondents reported participating in a training between 2012 and
2014, largely from respondents in Egypt and India.
The large numbers of people participating in trainings in Egypt and India is
mainly due to three factors: 1) Egypt and India were the largest number of
respondents to the survey, 2) the partners’ large reach in these countries,
and 3) the nature and scale of OGE’s programs in these countries.
In both India and Egypt, OGE works with larger, local NGOs, Pratham and
CEOSS, respectively, both of which have a greater reach than its partners
in Mexico. OGE implemented a large-scale, multi-year project in Egypt,
designed to train at least 1,200 youth. In India, OGE launched its training
curriculum at Pratham’s community technology centers in Pune, training
over 2,900 individuals in 2012, and built the capacity of 10 centers reaching
15 communities in Maharashtra State in 2013-2014 to implement a number
of training programs that served over 4,000 people. Siyafunda, OGE’s partner
in South Africa, also has a large network of community technology centers
located in mostly rural townships in Johannesburg. However, OGE only
works with about 5-8 of their centers per year, as the project is smaller in
scale. Finally, in Mexico, OGE works with three centers in very small, remote
villages, as well as small classes at local high schools in Toluca, in partnership
Missing training date from 204 respondents.
Figure 15. Training by
country by year
In Egypt and India, most respondents (92% and 93% respectively) had
participated in any training by OGE or one of its partners compared to half
the respondents in South Africa (62%) and a quarter in Mexico (29%). The
most common trainings were Career guidance and job hunting (73% of those
who participated in a training) and Community Connectors (66%).
It is important to note that the training programs in each country all focus
on digital literacy, technology adoption, and leadership skills. The leadership
focus varies in each country depending on the target population. In Mexico,
leadership skills taught are core soft and hard skills that can be exercised in
an academic or professional setting, or used to become actively involved in
community development initiatives. In Egypt and South Africa, there is an
emphasis on entrepreneurship and management, and in South Africa, India
and Egypt, on employment. Therefore, there exists great overlap in training
content from country to country, largely due to the fact that digital literacy
is very much connected to and has the impact to improve a number of other,
more specific skill sets pertaining to peoples’ livelihoods.
It can be inferred from these results and based on each country’s unique
program offerings that not every respondent indicated the training they
necessarily participated in. For example, the Mobile Entrepreneurs
Program, only offered in South Africa, consists of modules focused on
entrepreneurship, but also contains modules that teach career development
skills so that entrepreneurs also learn how to get jobs, if the choose, that
allow them to utilize their entrepreneurial skillset. Therefore, some of the
respondents may have said they had received training in career guidance
and other job-related skills if they participated in this training. In Mexico,
respondents who indicated they received career guidance and job training
could represent adults who participated in Community Connector-led
technology workshops at their local centers, which are aimed at parents,
many of which are first-time Internet users, who are interested in learning
how to look and apply for jobs online.
40 Digital Inclusion Social Impact Evaluation
Table C. Background characteristics of respondents
2,204 (91.6%) 1,205 (93.3%) 51 (29.3%) 306 (62.3%) 3,766 (86.3%)
Career guidance and
job hunting** 2,142 (97.2%) 483 (40.1%) 12 (23.5%) 104 (34.0%) 2,741 (72.8%)
1,822 (82.7%) 559 (46.4%) 32 (62.8%) 72 (25.5%) 2,485 (66.0%)
97 (4.4%) 29 (2.4%) 7 (13.7%) 112 (36.6%) 245 (6.5%)
0 224 (18.6%) 0 0 224 (6.0%)
0 166 (13.8%) 1 (2.0%) 7 (2.3%) 174 (4.6%)
Internet security and
IT adoption** 1 (0.1%) 86 (7.1%) 9 (17.7%) 7 (2.3%) 103 (2.7%)
10 (0.5%) 0 0 84 (27.5%) 94 (2.5%)
Professional and soft skills**
0 8 (0.6%) 4 (7.8%) 15 (4.9%) 27 (0.7%)
4 (0.2%) 0 6 (11.8%) 6 (2.0%) 16 (0.4%)
Other (not specified) 5 (0.2%) 116 (9.6%) 1 (2.0%) 5 (1.7%) 127 (3.4%)
None 220 (9.1%) 95 (7.4%) 123 (70.7%) 191 (38.9%) 629 (14.4%)
* p<0.05 ** p<0.01
The majority (71%) of respondents in Egypt reported having Internet
access at home; in the other countries Internet access in the home varied
from 14% in India to 41% in Mexico (Table D). In India, respondents most
commonly reported that they accessed the Internet at a CKC (84%) while
in Mexico almost half of the respondents reported accessing the Internet at
home (40%), school (43%), and at a CKC (37%) in addition to many using
telecenters (29%) and work (19%). In South Africa, the majority said they
accessed the Internet at a CKC (67%); while some reported accessing the
Internet at home (20%), school (15%), or at a telecenter (14%).
Where do you access the Internet?
1,692 (70.3%) 173 (13.4%) 70 (40.2%) 101 (20.6%) 2,036 (46.7%)
14 (0.6%) 14 (1.1%) 75 (43.1%) 77 (15.7%) 180 (4.1%)
171 (7.1%) 112 (8.7%) 33 (19.0%) 36 (7.3%) 352 (8.1%)
At a telecenter**
120 (5.0%) 8 (0.6%) 50 (28.7%) 67 (13.7%) 245 (5.6%)
At a CKC**
457 (19.0%) 1,086 (84.1%) 65 (37.4%) 329 (67.0%) 1,937 (44.4%)
On a mobile phone**
58 (2.4%) 155 (12.0%) 3 (1.7%) 12 (2.4%) 228 (5.2%)
Using a mobile
broadband USB stick** 20 (0.8%) 0 0 0 20 (0.5%)
At a library 0 0 0 1 (0.2%) 1 (0.1%)
access at home** 1,714 (71.2%) 183 (14.2%) 72 (41.4%) 138 (28.1%) 2,107 (48.3%)
* p<0.05 ** p<0.01
Table D. Background characteristics of respondents
4.5 Access Local Content
The high percentage of respondents who have accessed the digital platform
in Egypt and India is reflective of the reach of OGE’s partners in those
countries, as well as the high number of access points. Only 25% of
respondents in South Africa reported ever accessing the digital platform.
This number is somewhat surprising because OGE’s partner in South
Africa also has a significant reach and number of access points, however
this could be due to the small sample size taken from South Africa. If OGE
had surveyed a larger number of respondents from a greater variety of
townships where its access points are located, results might have shown
a higher percentage of people who have accessed the platform. The small
percentage of people who have ever accessed the digital platform in Mexico
is representative of the scale of OGE’s work and the number of access points
(three), which are located in small and rural towns.
42 Digital Inclusion Social Impact Evaluation
Table E. Read about topics on the digital platform
Ever accessed the
digital platform** 1,758 (73.0%) 1,114 (86.2%) 44 (25.3%) 124 (25.3%) 3,040 (69.7%)
317 (13.2%) 400 (31.0%) 14 (8.1%) 23 (4.7%) 754 (17.3%)
109 (4.5%) 165 (12.7%) 10 (5.8%) 26 (5.3%) 310 (7.1%)
1,046 (43.5%) 998 (77.2%) 26 (14.9%) 47 (9.6%) 2,117 (48.5%)
1,004 (41.7%) 906 (70.1%) 21 (12.1%) 58 (11.8%) 1,989 (45.6%)
to other people** 1,495 (85.0%) 325 (29.2%) 9 (17.7%) 74 (59.7%) 1,917 (63.1%)
* p<0.05 ** p<0.01
Figure 16, which shows the top three content areas by country, demonstrates
that content on the digital platform is tied to programmatic themes, but
also how content that the popularity of content is not always directly related
to training topics and may serve as key resources in supporting peoples’
For example, while the programmatic focus in OGE’s programs in Egypt
was employment and entrepreneurship, the program served some rural
communities where agribusinesses account for many of the small businesses.
Therefore it is unsurprising that agriculture was a popular topic among
people accessing the Egypt digital platform content. All three of India’s top
three content areas are reflective of training topics and services offered at
partner centers. In this case, the India results correlate to program objectives
focused on supporting people’s improvement in health, education, and
employment. In Mexico, there is no training program focused solely on
health, so the large percentage of people viewing health content could be
indicative of both the demand for and the value of localized information
on the topic. A key section of the Mexico digital platform is the Homework
Help section, which was developed to include links to online educational
resources such as tutorials and activities for over 100 concepts in math,
science, reading and other core subjects for primary, secondary, and high
schools students, aimed at guiding parents in helping their children improve
their academic achievement. According to Google Analytics, more than
a million users have accessed the Homework Help section, which could
contribute to education being among a top content area for Mexico. South
Africa’s program has a strong emphasis on employment and entrepreneurship,
which is also captured by the top content results.
All programs emphasize using and recommending others to the digital
platform for the purpose of accessing global best practice information and
community development-focused content on locally relevant issues and
topics. The digital inclusion ecosystem OGE has created with its partners—
access, content, and training—is designed to support word of mouth and
actions taken to increase technology adoption and usage of the digital
platform in the communities it serves.
Respondents were asked if they recommended articles on the digital platform
to others, and if so what topics they recommended. There was wide variation
between countries in both the frequency of recommending articles and
they type of content. Respondents from Egypt were most likely to report
recommending articles on the digital platform (85%), followed by South
Africa (60%) and Mexico (52%). Respondents from India were least likely to
report recommending articles on the digital platform to others. The topics
that were recommended mirrored the top content in each country.
4.6 Take Informed Action
The core of all of OGE’s programs is to help people to be better able to help
themselves by acquiring skills and gaining access to better information
via information technology to make informed decisions about their lives.
We measured this by asking not only what actions people took after they
attended a training or after they accessed content on the digital platform,
but if those actions improved their lives by gaining employment, starting a
business, earning more money, increasing their savings, and improving their
health and/or education.
Figure 16. Top three content
areas by country
Figure 17. Ever recommended
articles and top three topics
recommended on the digital
44 Digital Inclusion Social Impact Evaluation
Figure 18 shows the how often respondents in different countries report they
go online. It is very encouraging that more than half of respondents reported
that they go online at least once each day (orange bars in Figure 19). However,
respondents in India reported going online with much less frequency: only
29% reported going online daily and 43% reported going online only once
every few months (vs. less than 2% of respondents in the other countries). This
speaks to the need for computer centers and other points of access in India and
the benefit of online training and content in the other countries.
Figure 19 shows that only 29% of all respondents use email for personal
communication and 10% use it for business communication. This is a
critically important skill, particularly in the 21st
century economy. It is
possible that so few people use email for business because many of the
respondents are either unemployed or have a job that does not require
sending email. That said, the low numbers of people using email for personal
use are indicative of low levels of digital literacy given that so many of the
respondents reported going online regularly.
It is very important to have an online identity for finding and getting a job
as well as making the types of connections that can lead to professional
advancement. Email is one way to create an online identity and social
Figure 18. How often do you go online?
networking is another, which is why it is encouraging that almost two thirds
of respondents reported using online social networks (61%). This represents
a great opportunity to teach people how to use social networking to form ties
to their communities and to advance their professional lives, particularly in
Egypt where unemployment is high, Internet use is very regular, and many
respondents (43%) reported using the Internet to search and apply for jobs.
One reason for this high number is the OGE’s job-matching program where
Egyptians post their skills on an online database and employers post job
openings. Lastly, the data makes it clear that many people use the internet
to try to improve their lives by studying, researching, applying for jobs, and
social networking rather than just for frivolous purposes.
Less than 1% of respondents said they used the internet for downloading
music and watching movies online [data not shown].
Figure 19. How often do you go online?
** p <0.01
46 Digital Inclusion Social Impact Evaluation
4.7 Increased Opportunities in Income
USE IDENTITY TO PARTICIPATE IN THE LOCAL
AND GLOBAL ECONOMY
Over a quarter of the respondents reported having bank accounts and using
online banking, as seen in Table E (27%). However, there was wide variation
across countries: only 2% of respondents in Egypt reported having a bank
account compared to 70% in South Africa. However, of those who had bank
accounts, those from Egypt were much more likely to access their bank
account online (69% of those with a bank account, compared to 16% overall).
Very few respondents reported paying their bills online (5%).
Have a bank account**
55 (2.3%) 751 (58.1%) 39 (22.4%) 344 (70.1%) 1,189 (27.3%)
account online** 38 (69.1%) 56 (7.5%) 21 (53.9%) 71 (20.6%) 186 (15.6%)
Pay bills online**
74 (3.1%) 68 (5.3%) 25 (14.4%) 31 (6.3%) 198 (4.5%)
* p<0.05 ** p<0.01
START A BUSINESS
OGE administers business trainings that aim to help people start businesses
and employ best practices to maximize the chances that these businesses will
be successful. In these trainings people learn how to create an operational
budget, apply for loans and grants, and create marketing and business plans
amongst other activities.
Of the 245 respondents who said they had participated in one of OGE’s
business trainings, 46 reported that they started a business after training (19%).
Those who started a business were much more likely to have implemented
business best practices than those that did not, such as creating operational
budgets (57% vs. 19%), a business plan (70% vs. 44%), and a marketing plan
(52% vs. 19%). However, many respondents implemented these best practices
without starting a business. For example, 44% of respondents who didn’t start
a business after the business training stated that they created a business plan.
Since the majority of respondents attended a training within the last year, the
survey may have captured people in the process of starting a business. With
increased time since the training, future results may show a higher proportion
of trainees starting a business. [data not shown]
Table E. Read about topics on the digital platform