This document discusses testing and refining criteria to assess media literacy levels in Europe. It outlines the objectives to develop a tool to measure media literacy across age groups and countries. The approach includes reviewing literature, validating measures statistically, testing the tools, and providing recommendations. It identifies challenges in connecting individual and country-level measures and in measuring critical media analysis. The process of validation involves mapping indicators to data, identifying gaps and redundancies, and proposing a more focused set of variables. Recommendations include additional measures of credibility evaluation and critical understanding.

1. Testing and refining criteria to assess
media literacy levels
Dr. Monica Bulger, Oxford Internet Institute, University of Oxford
Kristian Pedersen, Danish Technological Institute
Cultures : médias, information et informatique une vision prospective
ENS Cachan, Paris, 18 March 2011

2. Purpose
Response to Audio-Visual Media Services
(AVMS) Directive, Article 33 (2007) that requires
media literacy levels for all member states to be
reported by December 2011 and every three
years after

3. Our consortium: Danish Technological Institute,
European Association of Viewers’ Interests,
Oxford Internet Institute, plus expert panel
A close follow-up to a previous study entitled
Study on Assessment Criteria for Media Literacy
Levels carried out by EAVI-led consortium
(2010)

4. Objectives
Develop tool for measuring media literacy levels
in all European countries across all age groups
Identify and clearly define tangible indicators
Inform actions for furthering media literacy policy

5. Our approach
Literature review
Statistical validation
Testing
Recommendations

6. What is media literacy?
Defined by AVMS as “skills, knowledge, and
understanding that allow consumers to use
media effectively and safely”
European Commission definition includes: ability
to access, analyze, and evaluate...images,
sounds, and messages...as well as to
communicate competently

7. What media are included in our
measures of media literacy?

14. Challenges:
Connect individual literacy levels with
aggregate country-level measures
Measure critical approach to media

15. Practical level challenges
Feasibility of implementing wishlist
Identifying priorities
Developing measures, where possible

16. Organizing framework: Media Literacy
From Study on Assessment Criteria for Media Literacy
Levels by EAVI-led consortium (2010)

17. Starting point: Critical
analysis of
existing framework
Review of previous work
Assess framework at conceptual and practical
levels
Identify tangible, achievable measures
Determine whether proposed criteria are
complete and will provide accurate results

18. Questions guiding critical assessment:
Does framework cover all aspects of media literacy
contained in the EC definition adopted as its base?
Should the framework cover other aspects of media
literacy as well, that is, is the EC definition too narrow?
Do the indicators proposed to operationalize the
framework actually measure the identified aspects of
media literacy?
Are the proposed indicators measurable as well as

19. Identified internal coherence as
potential problem. Inclusiveness
resulted in 59 indicators with a few
redundancies.

20. Identified relationships that needed further exploration

21. Review of existing literature evaluated
conceptual consistency in previous work

22. Review of existing literature evaluated
conceptual consistency in previous work

23. Review of existing literature evaluated
conceptual consistency in previous work

24. Next step: Statistical
validation of
existing framework
Sought empirical evidence for assumed
relationships
Attempted to connect framework’s 59
indicators to existing data
Identified feasible measures as well as gaps,
redundancies, and missing data

25. Process of statistical validation
Mapped each indicator to source
data

26. Media Literacy Framework
From Study on Assessment Criteria for Media Literacy
Levels by EAVI-led consortium (2010)

27. Mapped potential indicators to empirical data:

29. Process of statistical validation
Noted duplications
Assessed reliability and validity of each
variable as indicator of media literacy
Considered comparability of diverse datasets

30. Findings resulted in proposing a more
focused set of variables in an effort to
reduce redundancies and address gaps.

31. Missing from framework:
Credibility evaluation
How do people choose which messages/
sources to trust?
How do people determine believability of
media messages?
What behaviors demonstrate these practices?

32. Case of the Pacific Northwest Tree
Octopus
From Leu & Kinzer, University of Connecticut, USA

33. Evidence of existence?

34. Evidence of existence?

35. Sample question for credibility evaluation
When you go directly to a specific page how did you
originally come to know of that page?
--Mostly a page I found myself
--Mostly a page recommended by a friend or linked to
via an e-mail or blog

36. Sample question for credibility evaluation
When you visit new websites, which, if any of these
things do you do?
--Consider the overall look and feel of the site
--Compare the information across other sites
--Check whether the site is https
--Consider the qualifications or intent of the authors
--Ask others if they have visited the site

37. Findings additionally resulted in
recommendations for measures for
critical understanding.

38. Sample question for critical engagement
Do you believe there are differences in the way that the
same or related information is portrayed by the
different media below? We would like to know your
opinion even if you are not familiar with the particular
type of media.
--Different television channels?
--Different radio channels?
--Different newspapers?
--Different websites?

39. Sample question for critical engagement
When you notice such differences in the way that the
same or related information is portrayed by different
sources, what do you usually do …?
--Disregard or ignore them
--Try to compare with information found elsewhere (for
example, books, newspapers, another TV channel)
--Ask friends, family members, or other people for their
opinion
--I usually only consult one source

40. Sample question for critical engagement
Have you ever done something to prevent …?
--Receiving unsolicited e-mail, sometimes called SPAM
(for example, not posting your e-mail on web pages)
--Your computer being infected by viruses (for
example, installing a virus scan)
--Unpleasant experiences, such as receiving obscene
or otherwise unwanted e-mails (for example, installing
filters or blocking senders)

41. Limitations
Starting point: many aspects warrant further
exploration
Time constraints of survey limit scope (e.g.,
many environmental questions not addressed)

42. Implications
Inform future research
Provide framework for national measures
Address potential areas for education and
training

43. Thank you
Additional questions and comments can be
directed to monica.bulger@oii.ox.ac.uk and
kpe@dti.dk