How to Measure Your Outcomes: A Simple and Scientific Approach - This session introduces the Life Skills Assessment Scale (LSAS). Published in an international scientific journal in 2014, this is the first standardized and validated scale to measure Life Skills in disadvantaged children and young people.
Dr. Fiona Kennedy, Consultant Clinical Psychologist
2. Working with
Dr. Dave Pearson
Consultant Clinical
Psychologist
Vishal Talreja
Cofounder & CEO
Dream A Dream
3. Why do we need to measure
outcomes?
Surely stories of success are enough?
Surely counting numbers of children reached is
enough?
Surely questionnaires are enough?
Any other ways we can measure outcomes?
4. Quantitative measures
Lies, darn lies and statistics!
A good measure will be standardised (see later)
There are existing standardised measures of, e.g.
educational achievement, personality, mental health….
But these are all standardised on Western populations
5. Because our young people are different
Adversity puts a strain on the body and the brain
Why do we need a new
scale?
6. Adversity
can include:
Abuse
• Abandonment
• War experiences
• Extreme poverty
• Poor care
• Poor nutrition
Children in life
skills programmes
will have
experienced
many of these
7. Malnutrition
not having enough to eat
and/or not being loved
Causes bodies and brains to develop
differently
8. Indian children..the facts
Hunger and Malnutrition (HUNGaMA) Survey Report (2011)
(Naandi Foundation, 2012)
reported up to 59% of children ’stunted’
Children in India (2012) Report
(Government of India Ministry of Statistics & Programme
Implementation)
in 2006 – approx. 48% children ‘stunted’
approx. 20% ‘wasted’
stunted=height for age wasted=weight for height
9.
10. Why does this matter?
Children in adversity have challenges
Thinking skills (understanding,
concentrating, remembering)
Relationship and attachment skills
Maturity may ‘collapse’ under pressure
Anxiety control skills
11. Life skills
These challenges appear in everyday life as life-skills
challenges
For example,
– interacting with others
– overcoming obstacles and solving problems
– taking initiative
– managing conflict
– understanding and following instructions
12. Meeting the challenge
Research shows:
With the right help, young people can
meet and overcome these challenges
But how do we know what is the right
help?
13. We need a new scale
Because the young people we work with
are a special group
Because they have challenges resulting
from adversity
Because they are in a developing world
context
14. The Life Skills Assessment
Scale: Measuring life skills of
disadvantaged children in the
developing world
Kennedy, F., Pearson, D., Brett Taylor, L. &
Talreja, V. (2014). Social Behavior and
Personality: An international journal, 42 (2),
197-210.
17. A good ruler is reliable
• Should give the same length every time it’s used
(test-retest)
• Should give the same length when I use it as when
you use it (inter-rater)
18. A good ruler discriminates
Should tell the
difference between
a long and a short
piece of cloth
(discriminant
validity)
19. Every inch matters
Each inch should contribute to the ruler: if I take an inch away
from a ruler, that should make a difference (internal
reliability)
20. An inch should be an inch
The world over, we
know what is an
inch: somewhere in
the world is a ‘gold
standard’ inch!
(normative group)
22. Developing scale items
then produced some items describing behaviour that could
be answered
by observers rating children and young people involved in
life skills programmes
23. Life Skills Assessment Scale
(LSAS)
interacting with others
overcoming problems and solving problems
taking initiative
managing conflict
understanding and following instructions
24. 5-point Likert-type scale:
1 (does not yet do),
2 (does with a lot of help)
3 (does with some help)
4 (does with a little help)
5 (does independently)
An overall score is calculated as the mean of all five skills.
25. Rater instructions
• Use judgment as to what a child might reasonably be
expected to manage at a given (chronological) age
• Note both chronological and apparent ages of the
child/young person.
• Note any difficulties resulting from the use of other than
first language
• Give examples of the observations on which ratings are
based (optional)
26. Developing the scale
Between 2009 and 2010 DaD employee raters
completed the Life Skills Assessment Scale (LSAS)
observing 1232 children and young people aged 8-16
years
from at-risk communities in Bangalore
as they participated in group activities
27. A Base for comparison
(norms)
This data was analysed to
produce normative scores
average score = 2.5
standard deviation = .75
for three age groups, 8-10 yrs,
11-13 yrs and 14-16
28. Does the scale hang
together?
The ruler: are all the inches
(items) part of the same ruler?
Yes!
☻
29. Does each inch do its bit?
If we take each inch (item) off
the ruler one at a time, does
the measurement go down?
Yes! ☻
30. Inter-rater reliability
Using a ruler: if I use the ruler
(scale) then you use the ruler to
measure the same thing, do we
get the same result?
Yes! ☻
31. Test-re-test reliability
Using a ruler: if I use the ruler
(scale) then use the ruler again
to measure the same thing, do I
get the same result?
Yes! ☻ ☻
32. Discriminant validity
Can the ruler (scale) tell the difference between a
long piece of cloth and a short piece of cloth?
comparing LSAS scores for adversity children
with scores for children from an Army Public
School
Yes! ☻
34. Age appropriateness
No significant differences in the scores for each of the three
age groups
Shows rater success in making age-appropriate judgments
E.g. 8-yr-old “please bring me the football”; 14-yr-old
“please collect all the footballs and put them away”.
35. Practical Considerations
The LSAS is scored by observer rating and has only five items
It is a very simple and practical assessment tool for use with
at-risk children
An explanatory session can be held or role-play scenarios
acted out with new raters so that any questions about its
administration can be raised.
36. Wider Use of the LSAS
Participants for this study from Bengaluru and surrounding
areas in India
The children came from a mixture of urban and rural
backgrounds
As this simple scale is child-centered, it could be used
worldwide with children from disadvantaged backgrounds
38. Why is this scale so simple?
Surely it should be complicated, long and difficult to be
good??!!
Seven years work to make the complex simple
One item represents many skills: e.g. managing conflict
E.g. fastening a button
39. Who can use it?
NGOs - Give feedback to stakeholders, inform programme
strategy and maximise effectiveness
Donors - Make better informed decisions about funding allocation
and support
Researchers - Advance the study of Life Skills
Clinicians - Use Life Skills as an outcome measure alongside
mental health measures and know it is appropriate for
disadvantaged Indian children
Programme developers - Check out programme performance at
an early stage
Carers - Keep track of the progress of children in your care
40. What can they use it for?
Measuring programme impact - Find out how effective interventions are at
increasing Life Skills among disadvantaged children in India
Getting a skills profile for an individual child - Look at five different Life
Skills to see relative strengths in an individual child
Comparing an individual’s scores with norms - See how one child compares
with the average scores for their age group
Allocating a child to the programme best for them - See how a child thrives
in different programmes and help them choose the best one for them
Comparing one programme with another - Discover which Life Skills are
best developed by a given programme
Feedback on progress - Let stakeholders (donors, parents, teachers, carers,
children) see progress visually
45. Allocating a child to the
programme best for them
For example, you may know
That your computer program is strong on developing
the understanding and following instructions lifeskill
Young Raju may need to develop his skills in this area
So you may suggest he joins your computer program
49. For example:
DaD All Program Impact 2013
1905 young people showed positive Life Skills development in one or more of the
five Life Skills Assessed.
3.145
2.915 3.008 2.848
3.26
3.749
3.421
3.621 3.439
3.804
Interaction
Score
Overcoming
difficulties
Taking Initiative Managing
Conflict
Understanding
& following
instructions
Baseline Endline
60. Feedback on progress
As well as using this scale to measure your own life
skills impact,
you can also prepare any or all of the above reports to
feed back to stakeholders of every sort