• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Kittens are evil: Heresies in Public Policy
 

Kittens are evil: Heresies in Public Policy

on

  • 775 views

by Toby Lowe, Newcastle University Business School

by Toby Lowe, Newcastle University Business School

Statistics

Views

Total Views
775
Views on SlideShare
775
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Kittens are evil: Heresies in Public Policy Kittens are evil: Heresies in Public Policy Presentation Transcript

    • Kittens are Evil Heresies in Public Policy #KittensAreEvil
    • Language Outcomes-Based Performance Management “Outcomes Based Evaluation” “Outcomes/Results Based Accountability”TM “Results Based Management” “Payment by Results” = measure performance by the impact a person/team/organisation/project has in the world #KittensAreEvil
    • Research Findings • Measurement Problem: Outcomes don’t measure impact in people’s lives • Attribution Problem: Outcomes aren’t delivered by an organisation • OBPM distorts organisations’ priorities • OBPM undermines good frontline practice #KittensAreEvil
    • Measurement Problem: Outcomes Don’t Measure Impact #KittensAreEvil
    • “One clear and compelling answer to the question of "Why measure outcomes?" is: To see if programs really make a difference in the lives of people.” United Way of America #KittensAreEvil
    • The outcomes bible… “Outcomes Based Evaluation”, by Robert Schalock, 1995 #KittensAreEvil
    • How to measure an outcome • Delivery group/control group • In-depth qualitative research • Large scale quantitative research – designed by participants #KittensAreEvil
    • How to measure an outcome Minimum post-programme research time? 18 months #KittensAreEvil
    • What does get measured? Netten, A., Beadle-Brown, J., Caiels, J., Forder, J., Malley, J., Smith, N., Trukeschitz, B., Towers, A., Welch, E. and Windle, K. (2011) Adult Social Care Outcomes Toolkit v2.1: Main guidance, PSSRU Discussion Paper 2716/3, Personal Social Services Research Unit, University of Kent, Canterbury
    • What does get measured? Netten, A., Beadle-Brown, J., Caiels, J., Forder, J., Malley, J., Smith, N., Trukeschitz, B., Towers, A., Welch, E. and Windle, K. (2011) Adult Social Care Outcomes Toolkit v2.1: Main guidance, PSSRU Discussion Paper 2716/3, Personal Social Services Research Unit, University of Kent, Canterbury
    • ASCOT Control group? - No Qualitative techniques? - No Quantitative techniques? - Yes designed by service users? - No 18 month follow up? – Maybe #KittensAreEvil
    • Focusing Illusion “Nothing in life is quite as important as it seems to be while you’re thinking about it” Daniel Kahneman, Professor of Psychology and Public Affairs, Princeton University #KittensAreEvil
    • What does get measured? Quality Dimension Accountability Dimension #KittensAreEvil
    • #KittensAreEvil
    • #KittensAreEvil
    • The attribution problem: Outcomes aren’t delivered by organisations
    • What is an outcome?
    • Programme Logic Model Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and Program Planning, 2003
    • Programme Logic Model Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and Program Planning, 2003
    • Programme Logic Model Robert Schalock & Gordon Bonham “Measuring outcomes and managing for results”, Evaluation and Program Planning, 2003
    • What else is missing? Program participants? ? ?
    • “Outcomes are by definition results over which organizations do not have complete control” John Mayne, “Challenges and Lessons in Implementing Results-Based Management”, Evaluation, 2007
    • Theoretical problems: • Measurement problem: Outcomes don’t measure impact in people’s lives • Attribution problem: Outcomes aren’t delivered by an organisation The uncertainty principle in action?
    • What happens when people implement OBPM? What’s the evidence?
    • OBPM creates: •“Goal displacement” •“Creaming” •“Making the numbers” Burt Perrin “Effective Use and Misuse of Performance Measurement”, American Journal of Evaluation, 1998
    • Targets for results “frequently distort the direction of programs, diverting attention away from, rather than towards, what the program should be doing.” Burt Perrin, 1998
    • “Unintended consequences”: •focusing on those who are easiest to help •“difficult” clients are skipped in favor of the “easy” ones S van Thiel and F. L. Leeuw “The Performance Paradox in the Public Sector”, Public Performance and Management Review, 2002
    • “Ossification, a lack of innovation, tunnel vision and suboptimization” S van Thiel and F. L. Leeuw “The Performance Paradox in the Public Sector”, Public Performance and Management Review, 2002
    • “Target based performance management always creates ‘gaming’” (my emphasis) Bevan, G. and Hood, C. “What’s measured is what matters: targets and gaming in the English public health care system”, Public Administration, 2006
    • Triage “parks” disabled people on Work Programme Independent, Monday 28th January, 2013 “Work advisers 'pushing jobless into self-employment” BBC, 3rd February, 2013 “Private health contractor's staff told to cut 999 calls to meet targets” Guardian, Wednesday 23 January 2013 “NHS targets 'may have led to 1,200 deaths' in Mid-Staffordshire” Daily Telegraph, March 2009
    • “A4e employee forged signatures to boost job placement numbers” The Guardian, 6th March, 2012 “Serco gave NHS false data about its GP service 252 times” Guardian, Thursday 20 September 2012
    • Implementing outcomes approaches “Always results in gaming”: • Creaming/cherry picking (helping the easiest to help) • Targeting resources to produce data (teaching to the test) • Reclassifying results (pretending) • Making things up
    • Impact on frontline practice
    • Outcomes Based Accountability • Makes “it more difficult to engage with and build relationships with homeless and at risk young people” • Has significant impacts on the daily practice of workers • Reduces the time available to create a sense of belonging • Reduces the time to “develop young people’s life skills” Lynn Keevers (et al) “Made to Measure: Taming Practices with Results- based Accountability”, Organization Studies, 2012
    • What Social Workers do… • 86 per cent of time is system driven - filling in forms for accountability and discussing them with colleagues. • The 14 per cent of time spent face to face with a family member is not developmental. Hilary Cottom, Relational Welfare, 2011
    • Frontline practice = Reversal of relationship between worker/client From: how can I help you achieve your goals? To: how can you help me achieve my targets?
    • Models of Outcomes Based Performance Management
    • Payment by Results: Managing for Genuine Impact? 1. Accept you can’t do it 2. Define an outcome you can live with 3. Set targets for performance 5. Providers deliver activity 6. Gather outcomes data 7. Realise limits of influence 8. Game: •Cherry-pick •Teach to the test •Reclassify •Change practice to focus on data •Make up data 4. Providers plan activity to meet outcome targets
    • OBPM: Managing Performance for Genuine Attributability 1. Define performance you are responsible for 2. Set targets for performance 4. Deliver activity 5. Gather performance data 6. Act so as to improve performance data 3. Providers plan activity to meet performance targets
    • Summary • Outcomes don’t measure impact in people’s lives • Outcomes aren’t delivered by an organisation • OBPM distorts organisations’ priorities • OBPM undermines good frontline practice
    • If not outcomes, then what? • Bottom up is key – start from actual people’s needs • Deal with complexity: Put human judgement on the frontline • Trust & Transparency • Use social theory: measure change in social context and identity
    • Thanks for listening Toby Lowe E: tobylowe@hotmail.co.uk Twitter: @tobyjlowe