Measuring public value - the BBC approach

2,267 views
2,063 views

Published on

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,267
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Measuring public value - the BBC approach

  1. 1. Measuring and monitoring public value - the BBC approach Medientage München 2008 Shaun Day, Chief Economist, BBC Policy Policy & Strategy Slide 1
  2. 2. Introduction Reach Quality Impact Value Framework PVT Data Sources What is Public Value? “Public Value is what the public values…” The Work Foundation 2006 Policy & Strategy Slide 2
  3. 3. Introduction Reach Quality Impact Value Framework PVT Data Sources Introduction to Measuring Public Value • ‘Building Public Value’ - published June 2004. • It set out a number of proposals for measuring the public value the BBC delivers, with four elements: – The development of a public value test to assess new services and changes to existing services – The introduction of Service Licences as a tool for accountability – The commissioning of a large scale public value survey every five years – A new performance measurement framework. Policy & Strategy Slide 3
  4. 4. Introduction Reach Quality Impact Value Framework PVT Data Sources BBC Mission and Purposes BBC Mission: To enrich people's lives with programmes and services that inform, educate and entertain. BBC Purposes: • Sustaining citizenship and civil society • Promoting education and learning • Stimulating creativity and cultural excellence • Representing the UK, its nations, regions and communities • Bringing the UK to the world and the world to the UK • Delivering to the public the benefit of emerging communications technologies and services Policy & Strategy Slide 4
  5. 5. Introduction Reach Quality Impact Value Framework PVT Data Sources Performance Measurement Framework RQIV Reach • Quality • Impact • Value Policy & Strategy Slide 5
  6. 6. Introduction Reach Quality Impact Value Framework PVT Data Sources Measuring Public Value • The performance measurement framework is designed to reflect the purposes and characteristics of the BBC’s public service remit. • Consists of four headline measures - the main drivers of public value: Reach A measure of the BBC’s role of universal availability and equity – delivering something for everyone Quality A measure of the characteristics of public service broadcasting – excellence, distinctiveness, originality and innovation A measure of the characteristics of public service broadcasting Impact for individuals and for citizens – the need to be memorable, challenging and engaging, and supportive of the BBC’s purposes A measure of our efficiency and effectiveness in delivering public Value for Money value. Also measures audience value. Policy & Strategy Slide 6
  7. 7. Introduction Reach Quality Impact Value Framework PVT Data Sources Reach Reach is a measure of our remit to be universally available to everyone in the UK Example metrics Source BARB (TV)/ Overall weekly 15 minute reach for TV and Radio RAJAR (Radio) BARB (TV)/ Share of audiences RAJAR (Radio) TNS New Media Tracker Reach of online services BARB Reach of BBCi Sage Metrics Number of online Unique Users Sage Metrics Volume of online Live Streaming and Audio on Demand BARB (TV)/ Reach of underserved audiences RAJAR (Radio)/TNS New Media Tracker Various Reach of new platforms (mobile, iPlayer, HD) Policy & Strategy Slide 7
  8. 8. Introduction Reach Quality Impact Value Framework PVT Data Sources Quality Quality is a measure of the characteristics of public service broadcasting – excellence, distinctiveness, originality and innovation Example metrics Source PBTS Overall approval of the BBC and BBC services (average out of 10, % scoring high (8-10)) Pulse (TV, Radio) Appreciation (average AI, % scoring high (8-10) Netpromoter (bbc.co.uk) PBTS / Pulse / Perceptions of quality (high quality, original, impartiality, etc.) ad hoc surveys QI tracker Perceptions of High quality and innovation (average out of 10, % scoring high (8-10)) PBTS Channel best for (genre)… Various Awards (industry recognition) Policy & Strategy Slide 8
  9. 9. Introduction Reach Quality Impact Value Framework PVT Data Sources Impact Impact is a measure of the characteristics of public service broadcasting for individuals and for citizens – the need to be memorable, challenging and engaging, and supportive of the BBC’s purposes Example metrics Source QI tracker Perceptions of challenging and engaging (average out of 10, % scoring high (8-10)) Purpose Remit Survey Perceptions of meeting BBC purposes PBTS Various Impact beyond broadcast Policy & Strategy Slide 9
  10. 10. Introduction Reach Quality Impact Value Framework PVT Data Sources Value Value for money is a measure of our efficiency and effectiveness in delivering public value on behalf of the licence fee payer Example metrics Source BARB (TV) Cost per user hour (TV, Radio) RAJAR (Radio) BBC Finance Sage Metrics Cost per unique user (bbc.co.uk) BBC Finance PBTS Perceptions of value for money Policy & Strategy Slide 10
  11. 11. Introduction Reach Quality Impact Value Framework PVT Data Sources Framework for measuring performance in the BBC Education & Representing World to UK, Emerging Purposes Citizenship Creativity Learning the UK UK to World Comms Programme and service goals Reach Quality Impact Value Wkly Reach (BBC/Services) Approval (BBC/Services) Purpose performance Cost per viewer hour Example Share Appreciation Index Perception metrics Cost per hour measures Unique Users (online) Innovative content Perception of value Live listening / AOD Impartiality Willingness to pay Value Democratic Cultural Educational Social Global Public Value Policy & Strategy Slide 11
  12. 12. Introduction Reach Quality Impact Value Framework PVT Data Sources How Performance is reported in the BBC Screen grab of dashboard Policy & Strategy Slide 12
  13. 13. Introduction Reach Quality Impact Value Framework PVT Data Sources How Performance is reported in the BBC Policy & Strategy Slide 13
  14. 14. Introduction Reach Quality Impact Value Framework PVT Data Sources Public Value Test (PVT) • PVT required to launch a new service or to amend an existing service licence significantly • BBC Trust balances Public Value against Market Impact • Management submits evidence on Public Value using RQIV framework • Evidence submitted: – Audience research (reach, quality and impact, consumer and citizen value – willingness to pay) – Reach Model – Preliminary Market Impact Analysis Policy & Strategy Slide 14
  15. 15. Introduction Reach Quality Impact Value Framework PVT Data Sources Data sources Overall BBC TV Radio Online Reach Cross media BARB: reach Rajar: reach TNS: claimed reach survey and share and share reach Quality and • PBTS • PBTS • PBTS • PBTS Impact • Purpose • Pulse • Pulse • Net Remit Survey • QI Tracker • QI Tracker Promoter • Purpose • Purpose • QI Tracker Remit Survey Remit Survey Value for • Household • Cost per • Cost per • Cost per Money Value Survey Viewer Hour Listener Hour Unique User • PBTS (CpVH) (CpLH) •(CpUU) Policy & Strategy Slide 15
  16. 16. Introduction Reach Quality Impact Value Framework PVT Data Sources Data sources Usage Measurement BARB: Broadcasters’ Audience Research Board • Provides estimates of the number of people watching TV. This includes which channels and programmes are being watched, at what time, and the type of people who are watching at any one time. • Data is obtained from a panel of 5,100 homes (11,500 viewers), representative of the whole of the UK (24m TV households). • BARB provides television audience data on a minute-by-minute basis for any channel received within the UK. The BBC tends to report 15 minute weekly reach for channels and overall, and 3 minute weekly reach for programmes. • BARB uses an electronic meter - a black box, the size of a hardback book, which is put on top of the television set and connected to it. Each person registers their presence in a room with a TV set on by pressing the button allocated to them on the people meter handset. • Some data is available the morning after transmission (the ‘overnights’). PVR/VCR play-back is incorporated within 7 days of transmission, and is referred to as consolidated data. RAJAR: Radio Joint Audience Research • Established in 1992 as a single audience measurement system for the UK radio industry – BBC and commercial. Owned by the RadioCentre and BBC and run by Ipsos-RSL. • Panel covers those aged 4+ but the industry-standard reporting is for those aged 15+. • Manual system: mix of face to face interview, tick-box diary to record listening during week and self-completion questionnaire. • Covers all listening – at home, at work, et/c – unlike BARB which only covers home viewing. Provides some information on listening via computers, MP3, etc but not levels of consumption on new platforms. Does not measure on-demand listening (Radioplayer). • Results are published on a quarterly basis. Policy & Strategy Slide 16
  17. 17. Introduction Reach Quality Impact Value Framework PVT Data Sources Data sources Usage Measurement continued… TNS New Media Tracker • Omnibus Survey for a range of organisations including the BBC. Has been running since 2005 (prior to which the BBC used a similar BMRB survey). • Measures claimed reach of bbc.co.uk amongst adults (age 16+) in the UK in internet homes Cross Media Reach survey • A new survey that has been developed to be introduced in late 2008. • Aims to measure claimed reach of all BBC services based on a detailed 7-day diary. Household value • New survey, due to be introduced in early 2008 • Aims to measure the level of consumer value delivered by the BBC to different audiences which are then placed into segments based on behaviour rather than demographics • “Value” is based on two axes: • Received value: Minutes of consumption of all programmes x AI given to each minute = # of “value points”. • Perceived value based on PBTS question: – “Do you think the licence fee is good value for money?”. Policy & Strategy Slide 17
  18. 18. Introduction Reach Quality Impact Value Framework PVT Data Sources Data sources Attitudinal surveys Pan BBC Tracking Study (PBTS) • Longitudinal monthly tracking study run by TNS surveying c.700 adults (age 15+) each month • Includes demographic and lifestyle indicators. • Questions range from organisation-level, to channel, genre and programme level. Examples: • Filter Q1: “Thinking about the BBC as a whole… on a scale of 1 to 10… please tell me your general impression of the BBC.” • Q15: “Which radio station provides the best sports news and coverage?” Pulse • Online survey run by GfK among a panel of 15,000, recruited to ensure even representation. • Relatively new tool - established in 2005. • TV data is used extensively by BBC Executive for scheduling and commissioning and in diagnosing strategic issues. • Radio Pulse adopted in 2007; children’s panel has just been set up. Quality & Impact tracker • Created to report at service-level on delivery of the ‘content characteristics’ listed in the Agreement as requirements for all BBC programmes and content. • Face-to-face survey run by Ipsos MORI in two waves each year. • Covers adults aged 15+ with statements for each characteristic, closely matched to those Ofcom uses in its PSB television measurement work but adapted for use across radio and online. • Gives comparisons with key competitor to each service. Policy & Strategy Slide 18
  19. 19. Introduction Reach Quality Impact Value Framework PVT Data Sources Data sources Attitudinal surveys continued… Purpose Remit Survey • A BBC Trust-owned l survey. • Will be based on 2007 Purpose Remit research and consultation. • To be run bi-annually amongst representative sample of UK adults and will include demographic and lifestyle indicators. • Will measure audience perceptions of BBC’s delivery of most of its purpose priorities at a pan-BBC level (rather than service or genre). • Example questions: • Q: “To what extent do you agree or disagree that the BBC has lots of fresh and new ideas. • Q: “To what extent to you agree or disagree that the BBC helps me feel more involved in my local communities”. Net Promoter • Regular surveys run by BBC assess whether users would recommend bbc.co.uk to their friends • The net promoter score (NPS) is calculated as follows: NPS = % of Promoters - % of Detractors Policy & Strategy Slide 19

×