The document discusses the BBC's approach to measuring and monitoring public value. It outlines the BBC's performance measurement framework, which consists of four headline measures - Reach, Quality, Impact, and Value. Various metrics are used to evaluate performance for each measure. The framework is designed to reflect the BBC's mission and purposes as a public service broadcaster in the UK.

1. Measuring and monitoring public
value - the BBC approach
Medientage München 2008
Shaun Day,
Chief Economist,
BBC Policy
Policy & Strategy Slide 1

2. Introduction Reach Quality Impact Value Framework PVT Data Sources
What is Public Value?
“Public Value is what
the public values…”
The Work Foundation 2006
Policy & Strategy Slide 2

3. Introduction Reach Quality Impact Value Framework PVT Data Sources
Introduction to Measuring Public Value
• ‘Building Public Value’ - published June 2004.
• It set out a number of proposals for measuring the public
value the BBC delivers, with four elements:
– The development of a public value test to assess new
services and changes to existing services
– The introduction of Service Licences as a tool for
accountability
– The commissioning of a large scale public value survey
every five years
– A new performance measurement framework.
Policy & Strategy Slide 3

4. Introduction Reach Quality Impact Value Framework PVT Data Sources
BBC Mission and Purposes
BBC Mission:
To enrich people's lives with
programmes and services that
inform, educate and entertain.
BBC Purposes:
• Sustaining citizenship and civil society
• Promoting education and learning
• Stimulating creativity and cultural excellence
• Representing the UK, its nations, regions and communities
• Bringing the UK to the world and the world to the UK
• Delivering to the public the benefit of emerging communications
technologies and services
Policy & Strategy Slide 4

5. Introduction Reach Quality Impact Value Framework PVT Data Sources
Performance Measurement Framework
RQIV
Reach • Quality • Impact • Value
Policy & Strategy Slide 5

6. Introduction Reach Quality Impact Value Framework PVT Data Sources
Measuring Public Value
• The performance measurement framework is designed to reflect the
purposes and characteristics of the BBC’s public service remit.
• Consists of four headline measures - the main drivers of public value:
Reach A measure of the BBC’s role of universal availability and equity –
delivering something for everyone
Quality A measure of the characteristics of public service broadcasting –
excellence, distinctiveness, originality and innovation
A measure of the characteristics of public service broadcasting
Impact for individuals and for citizens – the need to be memorable,
challenging and engaging, and supportive of the BBC’s purposes
A measure of our efficiency and effectiveness in delivering public
Value for Money
value. Also measures audience value.
Policy & Strategy Slide 6

7. Introduction Reach Quality Impact Value Framework PVT Data Sources
Reach
Reach is a measure of our remit to be universally available to
everyone in the UK
Example metrics Source
BARB (TV)/
Overall weekly 15 minute reach for TV and Radio
RAJAR (Radio)
BARB (TV)/
Share of audiences
RAJAR (Radio)
TNS New Media Tracker
Reach of online services
BARB
Reach of BBCi
Sage Metrics
Number of online Unique Users
Sage Metrics
Volume of online Live Streaming and Audio on Demand
BARB (TV)/
Reach of underserved audiences
RAJAR (Radio)/TNS
New Media Tracker
Various
Reach of new platforms (mobile, iPlayer, HD)
Policy & Strategy Slide 7

8. Introduction Reach Quality Impact Value Framework PVT Data Sources
Quality
Quality is a measure of the characteristics of public service
broadcasting – excellence, distinctiveness, originality and
innovation
Example metrics Source
PBTS
Overall approval of the BBC and BBC services (average out of
10, % scoring high (8-10))
Pulse (TV, Radio)
Appreciation (average AI, % scoring high (8-10)
Netpromoter (bbc.co.uk)
PBTS / Pulse /
Perceptions of quality (high quality, original, impartiality, etc.)
ad hoc surveys
QI tracker
Perceptions of High quality and innovation (average out of 10, %
scoring high (8-10))
PBTS
Channel best for (genre)…
Various
Awards (industry recognition)
Policy & Strategy Slide 8

9. Introduction Reach Quality Impact Value Framework PVT Data Sources
Impact
Impact is a measure of the characteristics of public service
broadcasting for individuals and for citizens – the need to be
memorable, challenging and engaging, and supportive of the
BBC’s purposes
Example metrics Source
QI tracker
Perceptions of challenging and engaging (average out of 10, %
scoring high (8-10))
Purpose Remit Survey
Perceptions of meeting BBC purposes
PBTS
Various
Impact beyond broadcast
Policy & Strategy Slide 9

10. Introduction Reach Quality Impact Value Framework PVT Data Sources
Value
Value for money is a measure of our efficiency and effectiveness
in delivering public value on behalf of the licence fee payer
Example metrics Source
BARB (TV)
Cost per user hour (TV, Radio)
RAJAR (Radio)
BBC Finance
Sage Metrics
Cost per unique user (bbc.co.uk)
BBC Finance
PBTS
Perceptions of value for money
Policy & Strategy Slide 10

11. Introduction Reach Quality Impact Value Framework PVT Data Sources
Framework for measuring performance in the BBC
Education & Representing World to UK, Emerging
Purposes Citizenship Creativity
Learning the UK UK to World Comms
Programme
and service goals Reach Quality Impact Value
Wkly Reach (BBC/Services) Approval (BBC/Services) Purpose performance Cost per viewer hour
Example Share Appreciation Index Perception metrics Cost per hour
measures Unique Users (online) Innovative content Perception of value
Live listening / AOD Impartiality Willingness to pay
Value Democratic Cultural Educational Social Global
Public Value
Policy & Strategy Slide 11

12. Introduction Reach Quality Impact Value Framework PVT Data Sources
How Performance is reported in the BBC
Screen grab of dashboard
Policy & Strategy Slide 12

13. Introduction Reach Quality Impact Value Framework PVT Data Sources
How Performance is reported in the BBC
Policy & Strategy Slide 13

14. Introduction Reach Quality Impact Value Framework PVT Data Sources
Public Value Test (PVT)
• PVT required to launch a new service or to amend an
existing service licence significantly
• BBC Trust balances Public Value against Market Impact
• Management submits evidence on Public Value using
RQIV framework
• Evidence submitted:
– Audience research (reach, quality and impact,
consumer and citizen value – willingness to pay)
– Reach Model
– Preliminary Market Impact Analysis
Policy & Strategy Slide 14

15. Introduction Reach Quality Impact Value Framework PVT Data Sources
Data sources
Overall BBC TV Radio Online
Reach Cross media BARB: reach Rajar: reach TNS: claimed
reach survey and share and share reach
Quality and • PBTS • PBTS • PBTS • PBTS
Impact • Purpose • Pulse • Pulse • Net
Remit Survey • QI Tracker • QI Tracker Promoter
• Purpose • Purpose • QI Tracker
Remit Survey Remit Survey
Value for • Household • Cost per • Cost per • Cost per
Money Value Survey Viewer Hour Listener Hour Unique User
• PBTS (CpVH) (CpLH) •(CpUU)
Policy & Strategy Slide 15

16. Introduction Reach Quality Impact Value Framework PVT Data Sources
Data sources
Usage Measurement
BARB: Broadcasters’ Audience Research Board
• Provides estimates of the number of people watching TV. This includes which channels and programmes are being
watched, at what time, and the type of people who are watching at any one time.
• Data is obtained from a panel of 5,100 homes (11,500 viewers), representative of the whole of the UK (24m TV
households).
• BARB provides television audience data on a minute-by-minute basis for any channel received within the UK. The BBC
tends to report 15 minute weekly reach for channels and overall, and 3 minute weekly reach for programmes.
• BARB uses an electronic meter - a black box, the size of a hardback book, which is put on top of the television set and
connected to it. Each person registers their presence in a room with a TV set on by pressing the button allocated to them
on the people meter handset.
• Some data is available the morning after transmission (the ‘overnights’). PVR/VCR play-back is incorporated within 7
days of transmission, and is referred to as consolidated data.
RAJAR: Radio Joint Audience Research
• Established in 1992 as a single audience measurement system for the UK radio industry – BBC and commercial. Owned
by the RadioCentre and BBC and run by Ipsos-RSL.
• Panel covers those aged 4+ but the industry-standard reporting is for those aged 15+.
• Manual system: mix of face to face interview, tick-box diary to record listening during week and self-completion
questionnaire.
• Covers all listening – at home, at work, et/c – unlike BARB which only covers home viewing. Provides some information
on listening via computers, MP3, etc but not levels of consumption on new platforms. Does not measure on-demand
listening (Radioplayer).
• Results are published on a quarterly basis.
Policy & Strategy Slide 16

17. Introduction Reach Quality Impact Value Framework PVT Data Sources
Data sources
Usage Measurement continued…
TNS New Media Tracker
• Omnibus Survey for a range of organisations including the BBC. Has been running since 2005 (prior to which the BBC
used a similar BMRB survey).
• Measures claimed reach of bbc.co.uk amongst adults (age 16+) in the UK in internet homes
Cross Media Reach survey
• A new survey that has been developed to be introduced in late 2008.
• Aims to measure claimed reach of all BBC services based on a detailed 7-day diary.
Household value
• New survey, due to be introduced in early 2008
• Aims to measure the level of consumer value delivered by the BBC to different audiences which are then placed into
segments based on behaviour rather than demographics
• “Value” is based on two axes:
• Received value: Minutes of consumption of all programmes x AI given to each minute = # of “value points”.
• Perceived value based on PBTS question: – “Do you think the licence fee is good value for money?”.
Policy & Strategy Slide 17

18. Introduction Reach Quality Impact Value Framework PVT Data Sources
Data sources
Attitudinal surveys
Pan BBC Tracking Study (PBTS)
• Longitudinal monthly tracking study run by TNS surveying c.700 adults (age 15+) each month
• Includes demographic and lifestyle indicators.
• Questions range from organisation-level, to channel, genre and programme level. Examples:
• Filter Q1: “Thinking about the BBC as a whole… on a scale of 1 to 10… please tell me your general impression of the
BBC.”
• Q15: “Which radio station provides the best sports news and coverage?”
Pulse
• Online survey run by GfK among a panel of 15,000, recruited to ensure even representation.
• Relatively new tool - established in 2005.
• TV data is used extensively by BBC Executive for scheduling and commissioning and in diagnosing strategic issues.
• Radio Pulse adopted in 2007; children’s panel has just been set up.
Quality & Impact tracker
• Created to report at service-level on delivery of the ‘content characteristics’ listed in the Agreement as requirements for all
BBC programmes and content.
• Face-to-face survey run by Ipsos MORI in two waves each year.
• Covers adults aged 15+ with statements for each characteristic, closely matched to those Ofcom uses in its PSB television
measurement work but adapted for use across radio and online.
• Gives comparisons with key competitor to each service.
Policy & Strategy Slide 18

19. Introduction Reach Quality Impact Value Framework PVT Data Sources
Data sources
Attitudinal surveys continued…
Purpose Remit Survey
• A BBC Trust-owned l survey.
• Will be based on 2007 Purpose Remit research and consultation.
• To be run bi-annually amongst representative sample of UK adults and will include demographic and lifestyle
indicators.
• Will measure audience perceptions of BBC’s delivery of most of its purpose priorities at a pan-BBC level (rather than
service or genre).
• Example questions:
• Q: “To what extent do you agree or disagree that the BBC has lots of fresh and new ideas.
• Q: “To what extent to you agree or disagree that the BBC helps me feel more involved in my local
communities”.
Net Promoter
• Regular surveys run by BBC assess whether users would recommend bbc.co.uk to their friends
• The net promoter score (NPS) is calculated as follows: NPS = % of Promoters - % of Detractors
Policy & Strategy Slide 19