The document presents a framework for understanding statistical performance. It outlines operational and strategic drivers for managing statistics about performance. Operationally, performance is important to outcomes like safety and education. Strategically, there is more data and emphasis on using data for decisions. The framework provides a macro level overview of the data, analysis, insight, and product cycle. It also details an analytical level approach involving a snapshot, trend, benchmark, and target analysis to understand a measure over time and versus comparisons. The goal is to extract insight from data to improve performance.

1. A Framework for Understanding
Statistical Performance
Paul Askew
CONFERENCE
2-5 SEPTEMBER 2013
NEWCASTLE

2. Outline
1. Introduction
2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers
3. Framework – the “How”

Macro level

Analytical level

3. 1. Introduction
1. Scope….A framework for

Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
•
More about practical drivers and process
•
Utility….target setting, performance improvement
3. Distilling application and development across sectors….
•
Criminal justice, regulation, education, health
•
It really matters….safety, housing, education….

4. 1. Introduction
Operational
Delivery
Methodological
Leadership

5. 1. Introduction
1. Scope….A framework for

Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
•
More about practical drivers and process
•
Utility….target setting, performance improvement
3. Distilling application and development across sectors….
•
Criminal justice, regulation, education, health
•
It really matters….safety, housing, education….

7. Outline
1. Introduction
2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers
3. Framework – the “How”

Macro level

Analytical level

9. 2. Why - Operational Drivers
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…

10.  “Burglary is down compared to last month”
 “Yes but it’s up compared the same month last year”
 “Yes but it’s down overall for the financial year to date”
 “Yes but its’ up for the calendar year so far”
 “Yes but we’re still less better than our neighbours”
 “Yes but they are reducing faster than we are this year”
 “Yes but
we’re still under (over) target”.
etc………….

11. 2. Why - Operational Drivers
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique,
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…

12. Smoothed Data
or Real Data
Smoothed Data
Smoothed Data – 12 month rolling average
This smoothed data is derived
from any of these underlying
raw data examples.
Example Real Data
Two month step
Three month step
Increasing
Decreasing
Decreasing convergence
High and low
Six month step
Increasing convergence
Highs and lows
Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).

13. 2. Why - Current and Strategic Drivers
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)

14. Data.gov…10K
Scope - Detail - Volume

15. 2. Why - Current and Strategic Drivers
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)

16. Words
Numbers

17. 2. Why - Current and Strategic Drivers
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)

18. % Adults at GCSE+ Levels
The numeracy challenge is big and getting bigger…
• Literacy Improving
while Numeracy
declining
Numeracy
• 26% to 22% (7.5m
adults) with GCSE+
• 17m adults at
primary school level
Skills for Life Survey 2011 (England)
Department for Business Innovation and Skills

19. A Framework for Understanding
Statistical Performance
Paul Askew

20. 2. Why - Current and Strategic Drivers
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)

21. Outline
1. Introduction
2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers
3. Framework – the “How”

Macro level

Analytical level

22. 3. How - Macro
DATA
- inputs -
INSIGHT
ANALYSIS
- outcomes -
- process -
PRODUCTS
- outputs -

23. 1.
Purpose
2.
Requirements
3.
Constraints
DATA
4.
Design
9.
Entering
12.
Storage
Manage
5.
Defiine
6.
Specify
7.
Collect
8.
Record
1.
Data
Implement
2.
Tools
Analysis Strategy
Synthesis
Comms
Cover
the
angles
Stakeholders
1.
Trend
Graphics
2.
Benchmark
Time
Periods
Numbers
Comparitors
Time
Periods
Words
3.
Target
3.
Skills
4.
Capacity
5.
Question
6.
Inclination
Lift Pitch
Summary
Evidence
PRODUCTS
- outputs -
ANALYSIS
Keys
Message
- process --
- outcomes -
11.
Validate
- inputs -
Plan
INSIGHT
10.
Process

24. OPEN
1.
Purpose
2.
Requirements
3.
Constraints
DATA
4.
Design
9.
Entering
6.
Specify
7.
Collect
8.
Record
1.
Data
Implement
2.
Tools
Synthesis
Comms
Cover
the
angles
Stakeholders
1.
Trend
Graphics
2.
Benchmark
Time
Periods
Numbers
Comparitors
Time
Periods
Words
3.
Target
3.
Skills
4.
Capacity
5.
Question
6.
Inclination
Lift Pitch
Summary
Evidence
OPEN PRODUCTS
- outputs -
OPEN
Analysis Strategy
ANALYSIS
Keys
Message
- process --
- outcomes -
12.
Storage
Manage
5.
Defiine
INSIGHT
11.
Validate
- inputs -
Plan
OPEN
10.
Process

25. The factors:
D
T
S
C
Q
I
d,t,s,c,q,i
Data:
Tools:
Skills:
Capacity:
Question:
Inclination:
Right data? Enough of it? Good enough?
Have any? Right ones?
Have any? Right ones?
How much? Realistic?
Specific question to answer or issues to address
Desire and drive to want to address the issues
Relative weights

26. OPEN
1.
Purpose
2.
Requirements
3.
Constraints
DATA
4.
Design
9.
Entering
6.
Specify
7.
Collect
8.
Record
1.
Data
Implement
2.
Tools
Synthesis
Comms
Cover
the
angles
Stakeholders
1.
Trend
Graphics
2.
Benchmark
Time
Periods
Numbers
Comparitors
Time
Periods
Words
3.
Target
3.
Skills
4.
Capacity
5.
Question
6.
Inclination
Lift Pitch
Summary
Evidence
OPEN PRODUCTS
- outputs -
OPEN
Analysis Strategy
ANALYSIS
Keys
Message
- process --
- outcomes -
12.
Storage
Manage
5.
Defiine
INSIGHT
11.
Validate
- inputs -
Plan
OPEN
10.
Process

27. 3. How – Analytical Level
0.
Snapshot
1.
Trend
2.
Benchmark
Time
Periods
Comparitors
Time
Periods
3.
Target

28. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

29. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

30. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

31. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

32. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

33. 3. How – Analytical Level
0.
Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory
0.
1.
2.
3.

34. 0. Snapshot – we have a number which is important to us
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

35. 1. Trend – what’s happening over time
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

36. 2. Benchmark – how this measures compares to others
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

37. 2a. Trend for the comparison to others
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

38. 3. Target - the trajectory for our measure
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

39. 3a. Target - Trajectory for the comparison to others
Value
160
140
120
100
80
60
40
20
0
t-9
t-8
t-7
t-6
t-5
t-4
t-3
t=2
t-1 t=now t+1
t+2
t+3
t+4
Time

40. Outline
1. Introduction
2. Framework – the “Why”

Operational Drivers

Current Strategic Drivers
3. Framework – the “How”

Macro

Analytical

41. A Framework for Understanding
Statistical Performance
Paul Askew
Thank You
CONFERENCE
2-5 SEPTEMBER 2013
NEWCASTLE