A Framework for Statistical Performance

258 views
190 views

Published on

Presentation at the Royal Statistical Society's International Conference, Newcastle, September 2013.

Paul Askew
speakingdata.org.uk
paul.askew@speakingdata.org.uk

A Framework Understating Statistical Performance

This paper presents a framework for understanding, managing and presenting statistical performance data. It provides a manageable, but multidimensional way of organising a range of possible analysis and assessment to provide a rounded and balanced picture for effective understanding and communication.

This is founded on a broad scope of assessment, based around three key analytical elements: trend, benchmark and target. That is the first multidimensional perspective. Each of those three key elements are then further disaggregated in a range of appropriate ways to provide a second, more disaggregate, multidimensional analytical assessment. There is then a third more tailored and specific multidimensional disaggregation to provide additional detailed statistical insight.

This draws on practical application of this framework in a range of sectors including health, policing and education, and across the public sector spectrum including Central Government, regulation and local service delivery, and in both strategic and operational environments.

This provides an overall framework to manage the communication and understanding of statistical performance, and focussed on public data. It provides a framework to build a balanced communication of statistical analysis and messages, and at the same time it provides the user and recipient of statistics and statistical analysis with a framework to both understand and question the scope and content of statistical communication.

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
258
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Framework for Statistical Performance

  1. 1. A Framework for Understanding Statistical Performance Paul Askew CONFERENCE 2-5 SEPTEMBER 2013 NEWCASTLE
  2. 2. Outline 1. Introduction 2. Framework – the “Why”  Operational Drivers  Current Strategic Drivers 3. Framework – the “How”  Macro level  Analytical level
  3. 3. 1. Introduction 1. Scope….A framework for  Managing Statistics about performance (rather than performance of statistical techniques) 2. Operational Origins • More about practical drivers and process • Utility….target setting, performance improvement 3. Distilling application and development across sectors…. • Criminal justice, regulation, education, health • It really matters….safety, housing, education….
  4. 4. 1. Introduction Operational Delivery Methodological Leadership
  5. 5. 1. Introduction 1. Scope….A framework for  Managing Statistics about performance (rather than performance of statistical techniques) 2. Operational Origins • More about practical drivers and process • Utility….target setting, performance improvement 3. Distilling application and development across sectors…. • Criminal justice, regulation, education, health • It really matters….safety, housing, education….
  6. 6. Outline 1. Introduction 2. Framework – the “Why”  Operational Drivers  Current Strategic Drivers 3. Framework – the “How”  Macro level  Analytical level
  7. 7. 2. Why - Operational Drivers 1. It actually matters to people – safety, home, education 2. Performance Regime – broad scope, high profile, deep drill down 3. “Multi-multi” dimensional – both of measures and assessments 4. Statistics meaning – datum, summary, technique 5. Targets - legal, audited, collaborative! 6. Performance Pantomime 7. Less about techniques, more about process 8. Operational Delivery – police, health, regulation…
  8. 8.  “Burglary is down compared to last month”  “Yes but it’s up compared the same month last year”  “Yes but it’s down overall for the financial year to date”  “Yes but its’ up for the calendar year so far”  “Yes but we’re still less better than our neighbours”  “Yes but they are reducing faster than we are this year”  “Yes but we’re still under (over) target”. etc………….
  9. 9. 2. Why - Operational Drivers 1. It actually matters to people – safety, home, education 2. Performance Regime – broad scope, high profile, deep drill down 3. “Multi-multi” dimensional – both of measures and assessments 4. Statistics meaning – datum, summary, technique, 5. Targets - legal, audited, collaborative! 6. Performance Pantomime 7. Less about techniques, more about process 8. Operational Delivery – police, health, regulation…
  10. 10. Smoothed Data or Real Data Smoothed Data Smoothed Data – 12 month rolling average This smoothed data is derived from any of these underlying raw data examples. Example Real Data Two month step Three month step Increasing Decreasing Decreasing convergence High and low Six month step Increasing convergence Highs and lows Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).
  11. 11. 2. Why - Current and Strategic Drivers 1. Data, Evidence, Decisions… Impact, Value. 2. Big & Open & Now data 3. Tactical vs. Strategic focus 4. Key Strategies…Communication emphasis - ONS, RSS… 5. Underlying Numeracy and statistical literacy 6. Policy Perception Gap 7. Data Science – Shakespeare review, Open Data, UKSA… 8. Austerity World - Effective (right thing) & Efficient (right way)
  12. 12. Data.gov…10K Scope - Detail - Volume
  13. 13. 2. Why - Current and Strategic Drivers 1. Data, Evidence, Decisions… Impact, Value. 2. Big & Open & Now data 3. Tactical vs. Strategic focus 4. Key Strategies…Communication emphasis - ONS, RSS… 5. Underlying numeracy and statistical literacy 6. Policy Perception Gap 7. Data Science – Shakespeare review, Open Data, UKSA… 8. Austerity World - Effective (right thing) & Efficient (right way)
  14. 14. Words Numbers
  15. 15. 2. Why - Current and Strategic Drivers 1. Data, Evidence, Decisions… Impact, Value. 2. Big & Open & Now data 3. Tactical vs. Strategic focus 4. Key Strategies…Communication emphasis - ONS, RSS… 5. Underlying Numeracy and statistical literacy 6. Policy Perception Gap 7. Data Science – Shakespeare review, Open Data, UKSA… 8. Austerity World - Effective (right thing) & Efficient (right way)
  16. 16. % Adults at GCSE+ Levels The numeracy challenge is big and getting bigger… • Literacy Improving while Numeracy declining Numeracy • 26% to 22% (7.5m adults) with GCSE+ • 17m adults at primary school level Skills for Life Survey 2011 (England) Department for Business Innovation and Skills
  17. 17. A Framework for Understanding Statistical Performance Paul Askew
  18. 18. 2. Why - Current and Strategic Drivers 1. Data, Evidence, Decisions… Impact, Value. 2. Big & Open & Now data 3. Tactical vs. Strategic focus 4. Key Strategies…Communication emphasis - ONS, RSS… 5. Underlying Numeracy and statistical literacy 6. Policy Perception Gap 7. Data Science – Shakespeare review, Open Data, UKSA… 8. Austerity World - Effective (right thing) & Efficient (right way)
  19. 19. Outline 1. Introduction 2. Framework – the “Why”  Operational Drivers  Current Strategic Drivers 3. Framework – the “How”  Macro level  Analytical level
  20. 20. 3. How - Macro DATA - inputs - INSIGHT ANALYSIS - outcomes - - process - PRODUCTS - outputs -
  21. 21. 1. Purpose 2. Requirements 3. Constraints DATA 4. Design 9. Entering 12. Storage Manage 5. Defiine 6. Specify 7. Collect 8. Record 1. Data Implement 2. Tools Analysis Strategy Synthesis Comms Cover the angles Stakeholders 1. Trend Graphics 2. Benchmark Time Periods Numbers Comparitors Time Periods Words 3. Target 3. Skills 4. Capacity 5. Question 6. Inclination Lift Pitch Summary Evidence PRODUCTS - outputs - ANALYSIS Keys Message - process -- - outcomes - 11. Validate - inputs - Plan INSIGHT 10. Process
  22. 22. OPEN 1. Purpose 2. Requirements 3. Constraints DATA 4. Design 9. Entering 6. Specify 7. Collect 8. Record 1. Data Implement 2. Tools Synthesis Comms Cover the angles Stakeholders 1. Trend Graphics 2. Benchmark Time Periods Numbers Comparitors Time Periods Words 3. Target 3. Skills 4. Capacity 5. Question 6. Inclination Lift Pitch Summary Evidence OPEN PRODUCTS - outputs - OPEN Analysis Strategy ANALYSIS Keys Message - process -- - outcomes - 12. Storage Manage 5. Defiine INSIGHT 11. Validate - inputs - Plan OPEN 10. Process
  23. 23. The factors: D T S C Q I d,t,s,c,q,i Data: Tools: Skills: Capacity: Question: Inclination: Right data? Enough of it? Good enough? Have any? Right ones? Have any? Right ones? How much? Realistic? Specific question to answer or issues to address Desire and drive to want to address the issues Relative weights
  24. 24. OPEN 1. Purpose 2. Requirements 3. Constraints DATA 4. Design 9. Entering 6. Specify 7. Collect 8. Record 1. Data Implement 2. Tools Synthesis Comms Cover the angles Stakeholders 1. Trend Graphics 2. Benchmark Time Periods Numbers Comparitors Time Periods Words 3. Target 3. Skills 4. Capacity 5. Question 6. Inclination Lift Pitch Summary Evidence OPEN PRODUCTS - outputs - OPEN Analysis Strategy ANALYSIS Keys Message - process -- - outcomes - 12. Storage Manage 5. Defiine INSIGHT 11. Validate - inputs - Plan OPEN 10. Process
  25. 25. 3. How – Analytical Level 0. Snapshot 1. Trend 2. Benchmark Time Periods Comparitors Time Periods 3. Target
  26. 26. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  27. 27. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  28. 28. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  29. 29. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  30. 30. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  31. 31. 3. How – Analytical Level 0. Snapshot - we have a number which is important to us 1. Trend - what’s happening to our measure over time 2. Benchmark – how this compares to others 2a. Trend for the comparison to others 3. Target – the trajectory for our measure 3a. – the comparison trajectory 0. 1. 2. 3.
  32. 32. 0. Snapshot – we have a number which is important to us Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  33. 33. 1. Trend – what’s happening over time Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  34. 34. 2. Benchmark – how this measures compares to others Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  35. 35. 2a. Trend for the comparison to others Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  36. 36. 3. Target - the trajectory for our measure Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  37. 37. 3a. Target - Trajectory for the comparison to others Value 160 140 120 100 80 60 40 20 0 t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+4 Time
  38. 38. Outline 1. Introduction 2. Framework – the “Why”  Operational Drivers  Current Strategic Drivers 3. Framework – the “How”  Macro  Analytical
  39. 39. A Framework for Understanding Statistical Performance Paul Askew Thank You CONFERENCE 2-5 SEPTEMBER 2013 NEWCASTLE

×