This document discusses measuring digital literacy and outcomes across different countries. It outlines a six step process for developing and evaluating digital inclusion interventions: 1) identify social challenges and desired outcomes, 2) identify excluded target groups, 3) identify how digital exclusion inhibits outcomes, 4) identify organizations for interventions, 5) provide resources, 6) evaluate if increased digital engagement improves well-being. It also discusses measuring digital skills, uses, and tangible outcomes at individual and societal levels across different demographic groups. Finally, it lists some collaborators for expanding such research globally.

1. Cross-national research:
Measuring digital literacy and
tangible outcomes
Ellen Helsper
@Ellenhel
e.j.helsper@lse.ac.uk

2. Idea
MACRO
(Culture & Society)
MICRO
(Individual)

3. Six steps from development to
implementation to evaluation
1) Identify the main social challenges and desired outcomes;
2) Identify socially excluded target groups;
3) Identify how digital exclusion inhibits achievement of the
desired outcomes for these target groups;
4) Identify organisations for interventions;
5) Provide resources;
6) Evaluate whether increased digital engagement has led to
improved well-being (i.e. SROI).
Recommendations for EU Digital Inclusion Policies
http://ec.europa.eu/social/BlobServlet?docId=12437&langId=en
Review of EU Digital Inclusion Policies
http://ec.europa.eu/social/BlobServlet?docId=11614&langId=en

4. Outcomes
Uses
Skills
What can
you do?
What do
you do?
What did you
get out of it?

5. Skills: How to ask
1) Not at all true of me
2) Not very true of me
3) Neither true nor untrue of me
4) Mostly true of me
5) Very true of me
Van Deursen, A.J.A.M., Helsper, E.J. & Eynon, R. (2014).
Measuring Digital Skills. From Digital Skills to Tangible
Outcomes project report.
Available at: www.oii.ox.ac.uk/research/projects/?id=112

6. Operational
I know how to
connect to a WIFI
network
I know how to
upload files
I know how to
adjust privacy
settings
I know how to
bookmark a
website
I know how to
use shortcut keys
I know how to
open a new tab
in my browser
I know where to
click to go to a
different webpage
I know how to
complete online
forms
I know how to
download/save a
photo I found online
I know how to
open
downloaded files

7. Information
Navigation
All the different
website layouts make
working with the
Internet difficult for me
I find it hard to find a
website I visited
before
I find the way in
which many
websites are
designed confusing
I find it hard to
decide what the best
keywords are to use
for online searches
Sometimes I end up
on websites without
knowing how I got
there
I get tired when
looking for
information online
I should take a
course on finding
information online
Sometimes I find it
hard to verify
information I have
retrieved

8. I am careful to make my
comments and behaviours
appropriate to the situation I
Social
I know how to change who I
share content with
find myself in online
I feel comfortable deciding
who to follow online
I know when I should and
shouldn’t share information
online
I know which information I
should and shouldn’t share
online
I know how to remove
friends from my contact lists

9. I would feel confident
putting video content
I have created online
Creative
I know which
apps/software are
safe to download
I am confident about
writing a comment on
a blog, website or
forum
I would feel
confident writing
and commenting
online
I know how to
design a website
I know which
different types of
licences apply to
online content
I know how to make
basic changes to the
content that others
have produced
I know how to
create something
new from existing
online images,
music or video

10. I know how to download
Mobile
I know how to
install apps on a
mobile device
apps to my mobile
device
I know how to keep
track of the costs of
mobile app use

11. 4.65
3.53
4.37
4.57
4.42
3.41
4.22
3.59
4.30
2.82
3.74
4.77
3.62
4.57
3.72
4.71
3.60
4.40
3.27
4.20
4.53
3.59
4.39
3.06
3.98
4.24
3.45
4.14
2.58
3.19
Operational*
Information
Navigation
Social*
Creative*
Mobile*
Male
Female
16-30 yrs
31-45 yrs
46-60 yrs
61+ yrs

12. 4.73
3.59
4.53
3.42
4.47
4.40
3.63
4.29
2.82
3.86
4.52
3.45
4.28
3.27
3.80
4.87
3.78
4.66
3.80
4.79
4.25
3.46
4.11
2.58
3.17
4.43
3.57
4.37
3.15
3.68
4.13
3.43
4.17
2.78
3.54
Mobile*
Creative*
Social*
Information
Navigation
Operational*
Occupation
Caretaker
Unable to work
Retired
Student
Unemployed
Employed - PT
Employed - FT
4.20
3.34
4.17
2.77
3.57
4.58
3.55
4.38
3.20
4.10
4.79
3.85
4.56
3.37
4.26
1 2 3 4 5
Mobile*
Creative*
Social*
Information
Navigation
Operational*
Education
Tertiary
Secondary
Primary

13. Problems
NL UK
M C S I O O C I O S N M
I know how to use a wide range of strategies when searching for information
I feel confident in my evaluation of whether a website can be trusted
I know how to turn my mobile phone off
I know how to make pop-ups disappear
If a technical problem occurs while I am using the Internet, I usually know how to fix
the problem
I know how to go to the previous page when browsing the Internet
I know how to use the refresh function
I know how to download files
I know some good ways to avoid computer viruses
I know how to use emoticons (e.g. smileys, emojis or text speak)
I generally compare different websites to decide if information is true
I carefully consider the information I find online
I know how to open a Web address directly without using a search engine like
Google
I tend to have no problems finding my way around a website
I am confident selecting search results
I normally look at more than the top three search results
It is easy for me to find information

14. Skills Use Outcomes

15. Tangible outcomes
Economic outcomes
Income: saved / made > products and earnings
Education: grades/degrees
Employment: productive/promotion/job
Property: bought/quality
Cultural outcomes
Belonging: Stronger sense of pride to belong to Gender, Ethnicity, Generation, Religion groups
Identity: Stronger sense of Gender, Ethnic, Generational or Religious identity
Social outcomes
Bonding: Personal networks
Bridging: Organised/civic networks/political networks
Personal outcomes
Psychological health: confidence/well being (happiness)
Physical health: Fitness/Health
Personality: Preferences/Attitudes

16. Outcomes: How to ask?
Quantity
Thinking about your online activities in the past year: how much do you agree
or disagree with the following statements?
e.g. I am fitter as a result of the online information, advice or programs /apps
I have used.
e.g. I got a certificate that I could not have gotten without the Internet.
Quality
Thinking about your online activities in the last year, how satisfied were you
with the following things?
e.g. The last online government service you accessed?
e.g. Your last contribution to an online discussion?

17. Going global
• Australia (Sharon Dane)
• Austin Texas (Joseph Straubhaar)
• LA & Chicago (Karen Mossberger)
• Europe (IPTS & EU Kids)
• Wales (Wisekids)