This document provides an overview of a seminar on introducing measurement for improvement. The seminar agenda includes a welcome, introduction to the topic, and contact details. The presentation discusses using measurement to demonstrate whether improvement interventions are effective, provides examples of run charts to track data over time, and addresses challenges in measuring complex topics. Key points are that measurement for improvement can be kept simple, understanding baseline data is important, and capturing data over time can show whether unusual variation indicates an intervention worked. Resources for further information are also listed.

1. Introducing Measurement for Improvement
Welcome!
NHS Improving Quality
We will start shortly, but are waiting for people to join,
don’t worry if you can’t hear anything yet.
Before we start we will be going through some E-Seminar
housekeeping items, so that everyone can participate
fully in the online meeting
If you are having difficulties joining the meeting please
call Lynsey Ogilvie on 024 7662 7527.

2. AGENDA
1. Welcome, Introduction & Housekeeping
Jeri Hawkins Mental Health & Dementia Delivery Support Manager
2. Introducing Measurement for Improvement
Alison Crawford, Measurement & Evaluation Manager
3. Links and Contacts details

3. NHS Improving Quality
Introducing Measurement for Improvement
Welcome and Introduction
Jeri Hawkins
Mental Health & Dementia Delivery Support Manager
www.nhsiq.nhs.uk

4. • During the E-Seminar we will mute all delegate’s lines throughout
the presentation.
• If at other times you are in a noisy environment please mute your
line by pressing the mute button on your screen (this can be
found on the right hand side of the screen)
• If you would like to ask a question please use the raise hand
button (this can be found on the right hand side of the screen)
• This is an interactive session, please add your comments, and
thoughts into the chat box as we go through the presentation.
• At the Q&A session, type your question into the chat, or raise your
hand, we will un-mute all lines during the Q&A.
• If you are having any technical problems, send a message to the
Host via the chat panel or call Lynsey Ogilvie on 024 7662 7527.
• We will now start recording this Webinar

5. Alison Crawford
Measurement and Evaluation Manager
Improvement Capability
NHS Improving Quality
Introducing ‘Measurement for
Improvement’
See also
http://prezi.com/hjlmbaux8axf/?utm_campaign=share&u
tm_medium=copy&rc=ex0share

6. “You can’t fatten a cow
by weighing it”
(Palestinian proverb)
However, how else will a farmer
know when to send a cow to
market unless he measures it?

7. What is ‘Measurement for Improvement’?
Measurement - the size, length, or amount of something, as
established by measuring
Improvement - a thing that makes something better or is better
than something else
So measurement for improvement is the process we go
through to measure the things that we are trying to
improve, so that after we’ve made a change we can
demonstrate that’s it worked

8. The next hour…..
This session will……
• Work through a simple example of using data to show if a new
intervention works
• Practical considerations for real life situations
• Where to go next for more information and tools
This session won’t………..
• Delve into the theory of the statistics behind measurement for
improvement
• Cover capacity and demand

9. So why is a little bit scary…..?
• Target driven environments, which may encourage
counterproductive behaviour
• Endless submission of data, forms and measures into a black
hole
• The language is gobbledegook
• Something that ‘analysts’ or other people do
• Does it use complicated maths?
• Clinical trials need HUGE samples of patients
• Endless data collections, hard to find the right information
• Data is out of date

10. The good news….
Measuring things for improvement is probably one of the simpler
and most meaningful types of measurement!
• Doesn’t need large sample sizes (works on a ‘just enough’
principle)
• You choose the measure that’s relevant to the thing you are
trying to improve (no externally decided measures)
• Can be as simple as a count or percentage
• If you choose to collect your own data, it can be as up-to-date as
you want it to be
• Can be done with a simple line chart over time (but there are
more complex things too if you want to be extra clever)

12. Questions to ask
1. What is the aim of this
improvement exercise? Can I
distil this into a 2-minute elevator
pitch?
2. What exactly is the problem and
what is the size of it?
3. What sort of changes are ‘normal’
and how will we know if we’ve
made things better?
4. What does success look like, do
we have a specific target?

13. Gather data to understand the problem
Existing data
• National data collection
• Clinical audits
• Local patient administration systems
• Risk Management Systems
• Financial Systems
• Surveys / samples of patient records
New data
• Surveys
• New audit or data collection

14. Choosing measures
Choose measures based on
data you have available
Work out your ideal measure,
and then find data which fits
• Great if you know the data
well
• Might skip to a ‘proxy’
measure without looking for
a better fit data
• More thorough approach to
exploring the best measures
• Doesn’t assume you know
the data well
• Time consuming, needs
follow up research

15. Driver diagram
Aim or
Objective
Primary
Driver 1
Secondary
Driver 1
Primary
Driver 2
Secondary
Driver 2
Secondary
Driver 3
Secondary
Driver 4

16. Driver diagram
Lose 2 stone
in weight in 6
Months
Healthy
eating
Three calorie controlled
meals per day
Motivation
Fruit and veg snacks
only
Limit alcohol
Plan for social eating
Weekly weigh in
Daily exercise 20 mins
per day
Exercise
Rewards for milestones
Use pedometer to
measure steps
Weight
Calories
consumed
Units
consumed
Steps per day
Distance
travelled
Calories
burned
BMI
Waist
circumference
Treats
consumed

17. What does the raw data look like?

18. So we have data…..
5 people in Leeds West CCG were treated by the Early Intervention
Team in April 2014
But what does that mean?!!
• Is 5 a good or a bad number?
• Is this higher or lower than other similar CCGs?
• Is it normal for this number to fluctuate each month?
• Is there a number which represents ‘success’? (ie which the CEO
will be happy with, and you can say patients who need support
are getting it?)

19. 30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
Nationally 1.4% of people saw an EIT
In Leeds West this was 0.1%, which was low
compared to other CCGs across the country
Therefore we might infer that 5 people seeing
an EIT is too low, and could be improved
However, we might also check local
information about why that number might be
low, such as problems with this data source or
alternative ways people are being supported
that don’t fit this definition.
0 1000 2000 3000 4000 5000 6000 7000 8000

20. So what next?
Based on your local intelligence,
you need to plan in some
improvement activities, and
implement them one by one to
see if they make the situation
better.
That is, a sustained improvement
which doesn’t appear to be down
to chance or normal variation in
the data.
You can then do something which
makes the ‘improvement’
permanent or part of everyday
business (and look for other things
to improve…)

21. Run charts – where are we now?
20
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
• What’s your baseline, is it
steady?
• Does it reflect what frontline staff
think is happening?
• Is it a reliable source of data – ie
changes in the numbers not due
to problems with data collection,
such as who is on a shift,
changes in definition and so on.

22. Run charts – what will we be doing?
20
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
Referral pathway to EIT
mapped and revised to
be more efficient
Email and twitter
campaign to all staff

23. Run charts – where do we want to be?
20
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
Referral pathway to EIT
mapped and revised to
be more efficient
Email and twitter
The National level is 1.4% of people treatedc bamy pmaiegnn ttoa la hll estaalftfh
services saw an Early Intervention Team
Assuming the same proportion for Leeds West CCG
6090 x 1.4% = 85 referrals per month
Is 85 achievable, given the existing maximum of 5?
Is there capacity in the EIT to see 80 extra people in a month?
Do these planned actions expect to increase referrals to this level?
What does the CEO expect to achieve?

24. Run charts – what can we measure?
20
There might not always be a relevant source of data
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
Referral pathway to EIT
mapped and revised to
be more efficient
Email and twitter
campaign to all staff
published
1. Check with your local information team about sources of local
data they can recommend
2. Consider asking someone to add in collecting the data you need
to an existing collection
3. If you do need something new, make sure you get advice on
new surveys to make sure its valid and as easy as possible

25. Top tips for new surveys or audits
• Keep it short and sweet
• Survey questions, consider testing on a couple of people to
check they are understood and make sense
• Use a fixed way of collecting, such as a form or spreadsheet
• Write down any definitions you use, and share with people
doing the measuring
• If you are using equipment to measure, make sure it’s serviced
and calibrated to remove bias
• CAUTION – if you collect patient names get some advice on
storing and publishing the results
• Once you have determined that your interventions have
worked, STOP collecting data

26. Run charts – quick lesson in variation
20
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
Intervention
1
A shift: six or more consecutive
data points either all above or
below the median. Points on
the median do not count
towards or break a shift.
A trend: five or more
consecutive data points that are
either all increasing or
decreasing in value. If two
points are the same value ignore
one when counting.
Baseline
MEDIAN = 4
Has this
intervention made
an improvement?
Things that show normal variation –
• Temperature outside changes , during a day, a season or a year
• Number of visitors to a shop, by time of the day
• Number of cars driving on stretch of road, by time of day
• Number of crisps in a packet
• Height of men and women
• The time it takes to walk 100m
Known as common cause, this is the natural variation that we experience day to day
Things that show unusual variation –
• Temperature inside a home which has central heating or air conditioning
• Traffic on the road during le Tour de France
• Number of crisps in a bumper size packet of crisps (compared to a normal size packet)
• The time it takes to cycle 100m (compared to walking 100m)
Known as special cause, this is where something special or new has changed the normal passage of
events. In improvement we are looking for a positive ‘special cause’, much like adding central heating to a
house or increasing the size of a crisp packet

27. Run charts – lets look at some data
20
18
16
14
12
10
8
6
4
2
0
Patients visiting EIT
Intervention
1
A shift: six or more consecutive
data points either all above or
below the median. Points on
the median do not count
towards or break a shift.
A trend: five or more
consecutive data points that are
either all increasing or
decreasing in value. If two
points are the same value ignore
one when counting.
Baseline
MEDIAN = 4
Has this
intervention made
an improvement?

28. A shift: six or more consecutive
data points either all above or
below the median. Points on the
median do not count towards or
break a shift.
A trend: five or more consecutive
data points that are either all
increasing or decreasing in value. If
two points are the same value
ignore one when counting.
30
25
20
15
10
5
0
Patients visiting EIT
Baseline
Intervention
2
Intervention
1
MEDIAN = 5

29. More complex improvement challenges
• Politically sensitive
• Mortality
• Children’s services
• Vulnerable patients/clients
• Intangible and difficult to capture
• Social return on investment
• Experience of care
• Patient dignity
• Compassionate care
• Complex measures or indicators
• Mortality – Standardised Hospital
Mortality Indicator vs Hospital
Standardised Mortality Rate
What are your challenges?

30. Addressing challenges
• Accept no measure is perfect, it shows a snapshot of a situation.
You will need more than one measure or source of information
to tell the whole story
• Measure activity and new ‘stuff’ created as well as outcomes –
this will help show short term progress
• Involve users/stakeholders in choosing the desired outcomes
and setting the key measures of success
• Some measures are highly technical, just because the thing they
measure is complicated. Get specialist advice if you need it, also
compare against simpler measures (such as no of deaths)
• Use softer data, such as focus groups etc if needed. It all
contributes to telling the story!

31. Summary
1. Using measurement as part of your improvement work is
valuable and can be kept simple
2. Understand your existing or baseline data – does it represent
the real world you see? Ask questions
3. Capture data over time, to look for a shift or a trend which tells
you that something unusual is happening (hopefully your
planned improvement!)
4. Get help if you need it – analysts will find this sort of work very
interesting 

32. Useful resources
Institute for Healthcare Improvement – whiteboard series with Dr Bob Lloyd,
talking through the theory and tools of improvement
www.ihi.org
Quality Improvement Scotland – great eLearning tool, which includes
measurement for improvement modules
www.qihub.scot.nhs.uk/education-and-learning/qi-e-learning.aspx
NHS Institute – many products still available and relevant, such as the good
indicator guide and Mike Davidge’s videos (NB. The availability of this resource
at this web address may be time limited)
www.institute.nhs.uk
NHS IQ - let us know if you’d like a masterclass in a specific topic and we’ll host
something or create something to help
How to become an Improvement Measure Expert in 90 minutes – 12 activities
which will get you on the path to hands-on measurement
http://prezi.com/hjlmbaux8axf/?utm_campaign=share&utm_medium=copy&rc
=ex0share

33. CLOSE
THANK YOU FOR JOINING US TODAY
Any questions?
A link to the presentation will be sent to you via
email.
Please address additional questions or comments
to:
• Jeri.hawkins@nhsiq.nhs.uk
• Alison.crawford@nhsiq.nhs.uk