How to Measure Your Outcomes: A Simple and Scientific Approach - This session introduces the Life Skills Assessment Scale (LSAS). Published in an international scientific journal in 2014, this is the first standardized and validated scale to measure Life Skills in disadvantaged children and young people. Dr. Fiona Kennedy, Consultant Clinical Psychologist

1. Measuring Effectiveness
Doing what works
DR Fiona Kennedy
Consultant Clinical Psychologist

2. Working with
Dr. Dave Pearson
Consultant Clinical
Psychologist
Vishal Talreja
Cofounder & CEO
Dream A Dream

3. Why do we need to measure
outcomes?
 Surely stories of success are enough?
 Surely counting numbers of children reached is
enough?
 Surely questionnaires are enough?
 Any other ways we can measure outcomes?

4. Quantitative measures
 Lies, darn lies and statistics!
 A good measure will be standardised (see later)
 There are existing standardised measures of, e.g.
educational achievement, personality, mental health….
 But these are all standardised on Western populations

5.  Because our young people are different
 Adversity puts a strain on the body and the brain
Why do we need a new
scale?

6. Adversity
can include:
 Abuse
• Abandonment
• War experiences
• Extreme poverty
• Poor care
• Poor nutrition
 Children in life
skills programmes
will have
experienced
many of these

7. Malnutrition
 not having enough to eat
 and/or not being loved
 Causes bodies and brains to develop
differently

8. Indian children..the facts
 Hunger and Malnutrition (HUNGaMA) Survey Report (2011)
(Naandi Foundation, 2012)
reported up to 59% of children ’stunted’
 Children in India (2012) Report
(Government of India Ministry of Statistics & Programme
Implementation)
in 2006 – approx. 48% children ‘stunted’
approx. 20% ‘wasted’
stunted=height for age wasted=weight for height

10. Why does this matter?
 Children in adversity have challenges
 Thinking skills (understanding,
concentrating, remembering)
 Relationship and attachment skills
 Maturity may ‘collapse’ under pressure
 Anxiety control skills

11. Life skills
 These challenges appear in everyday life as life-skills
challenges
 For example,
– interacting with others
– overcoming obstacles and solving problems
– taking initiative
– managing conflict
– understanding and following instructions

12. Meeting the challenge
 Research shows:
 With the right help, young people can
meet and overcome these challenges
 But how do we know what is the right
help?

13. We need a new scale
 Because the young people we work with
are a special group
 Because they have challenges resulting
from adversity
 Because they are in a developing world
context

14. The Life Skills Assessment
Scale: Measuring life skills of
disadvantaged children in the
developing world
Kennedy, F., Pearson, D., Brett Taylor, L. &
Talreja, V. (2014). Social Behavior and
Personality: An international journal, 42 (2),
197-210.

15. Scale development

16. How to measure cloth

17. A good ruler is reliable
• Should give the same length every time it’s used
(test-retest)
• Should give the same length when I use it as when
you use it (inter-rater)

18. A good ruler discriminates
 Should tell the
difference between
a long and a short
piece of cloth
(discriminant
validity)

19. Every inch matters
 Each inch should contribute to the ruler: if I take an inch away
from a ruler, that should make a difference (internal
reliability)

20. An inch should be an inch
 The world over, we
know what is an
inch: somewhere in
the world is a ‘gold
standard’ inch!
(normative group)

21. First Steps
Literature review
World Health Organization (WHO,
1997) definition of life skills as a
globally influential framework.

22. Developing scale items
 then produced some items describing behaviour that could
be answered
 by observers rating children and young people involved in
life skills programmes

23. Life Skills Assessment Scale
(LSAS)
 interacting with others
 overcoming problems and solving problems
 taking initiative
 managing conflict
 understanding and following instructions

24. 5-point Likert-type scale:
 1 (does not yet do),
 2 (does with a lot of help)
 3 (does with some help)
 4 (does with a little help)
 5 (does independently)
 An overall score is calculated as the mean of all five skills.

25. Rater instructions
• Use judgment as to what a child might reasonably be
expected to manage at a given (chronological) age
• Note both chronological and apparent ages of the
child/young person.
• Note any difficulties resulting from the use of other than
first language
• Give examples of the observations on which ratings are
based (optional)

26. Developing the scale
 Between 2009 and 2010 DaD employee raters
 completed the Life Skills Assessment Scale (LSAS)
 observing 1232 children and young people aged 8-16
years
 from at-risk communities in Bangalore
 as they participated in group activities

27. A Base for comparison
(norms)
This data was analysed to
produce normative scores
 average score = 2.5
 standard deviation = .75
 for three age groups, 8-10 yrs,
11-13 yrs and 14-16

28. Does the scale hang
together?
The ruler: are all the inches
(items) part of the same ruler?
Yes!
☻

29. Does each inch do its bit?
If we take each inch (item) off
the ruler one at a time, does
the measurement go down?
Yes! ☻

30. Inter-rater reliability
Using a ruler: if I use the ruler
(scale) then you use the ruler to
measure the same thing, do we
get the same result?
Yes! ☻

31. Test-re-test reliability
Using a ruler: if I use the ruler
(scale) then use the ruler again
to measure the same thing, do I
get the same result?
Yes! ☻ ☻

32. Discriminant validity
 Can the ruler (scale) tell the difference between a
long piece of cloth and a short piece of cloth?
 comparing LSAS scores for adversity children
with scores for children from an Army Public
School
 Yes! ☻

33. Discussion
The statistical analysis indicates that the
scale can be used with confidence.
It is a reliable and valid measure of life
skills

34. Age appropriateness
 No significant differences in the scores for each of the three
age groups
 Shows rater success in making age-appropriate judgments
 E.g. 8-yr-old “please bring me the football”; 14-yr-old
“please collect all the footballs and put them away”.

35. Practical Considerations
 The LSAS is scored by observer rating and has only five items
 It is a very simple and practical assessment tool for use with
at-risk children
 An explanatory session can be held or role-play scenarios
acted out with new raters so that any questions about its
administration can be raised.

36. Wider Use of the LSAS
 Participants for this study from Bengaluru and surrounding
areas in India
 The children came from a mixture of urban and rural
backgrounds
 As this simple scale is child-centered, it could be used
worldwide with children from disadvantaged backgrounds

37. Measuring program impact

38. Why is this scale so simple?
 Surely it should be complicated, long and difficult to be
good??!!
 Seven years work to make the complex simple
 One item represents many skills: e.g. managing conflict
 E.g. fastening a button

39. Who can use it?
 NGOs - Give feedback to stakeholders, inform programme
strategy and maximise effectiveness
 Donors - Make better informed decisions about funding allocation
and support
 Researchers - Advance the study of Life Skills
 Clinicians - Use Life Skills as an outcome measure alongside
mental health measures and know it is appropriate for
disadvantaged Indian children
 Programme developers - Check out programme performance at
an early stage
 Carers - Keep track of the progress of children in your care

40. What can they use it for?
 Measuring programme impact - Find out how effective interventions are at
increasing Life Skills among disadvantaged children in India
 Getting a skills profile for an individual child - Look at five different Life
Skills to see relative strengths in an individual child
 Comparing an individual’s scores with norms - See how one child compares
with the average scores for their age group
 Allocating a child to the programme best for them - See how a child thrives
in different programmes and help them choose the best one for them
 Comparing one programme with another - Discover which Life Skills are
best developed by a given programme
 Feedback on progress - Let stakeholders (donors, parents, teachers, carers,
children) see progress visually

41. Getting a skills profile for an
individual child

42. For example: Raju
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
scores
scores

43. Comparing an individual’s
scores with norms

44. For example:
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
interacting ovecoming initiative conflict instructions over all
scores
norms

45. Allocating a child to the
programme best for them
 For example, you may know
 That your computer program is strong on developing
the understanding and following instructions lifeskill
 Young Raju may need to develop his skills in this area
 So you may suggest he joins your computer program

46. Measuring individual progress

47. For example: Raju
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
interac
ng
ovecom
ing
ini
a
ve
conflict
instruc
ons
overall
before
a er

48. Over all performance

49. For example:
DaD All Program Impact 2013
1905 young people showed positive Life Skills development in one or more of the
five Life Skills Assessed.
3.145
2.915 3.008 2.848
3.26
3.749
3.421
3.621 3.439
3.804
Interaction
Score
Overcoming
difficulties
Taking Initiative Managing
Conflict
Understanding
& following
instructions
Baseline Endline

50. Program performance

51. For example:

52. Comparing one programme
with another

53. For example: before vs after

54. Seeing which skills are
impacted by which program

55. For example: Mentoring:

56. Seeing how long a child
needs to attend

57. Longevity of Engagement

58. Seeing how often the child
needs to attend

59. Life Skills and Attendance

60. Feedback on progress
 As well as using this scale to measure your own life
skills impact,
 you can also prepare any or all of the above reports to
feed back to stakeholders of every sort

61. In pairs
 Discuss how you might use this scale in your own
setting

62. Feedback

63. Questions?

64. Goodbye!