Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety
•Download as PPTX, PDF•
2 likes•1,476 views
Dr Mary Salveron, Professor Leah Bromfield and Professor Fiona Arney Australian Centre for Child Protection
Recommended
Recommended
More Related Content
What's hot
What's hot (16)
Viewers also liked
Viewers also liked (20)
Similar to Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety
Similar to Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety (20)
More from BASPCAN
More from BASPCAN (20)
Recently uploaded
Recently uploaded (20)
Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety
- 1. Understanding the Signs of Safety Theory of Change and comparing outcomes for children pre and post Signs of Safety Dr Mary Salveron, Professor Leah Bromfield and Professor Fiona Arney British Association for the Prevention and Study of Child Abuse and Neglect (BASPCAN) 12-15th April 2015, University of Edinburgh
- 2. Overview • Signs of Safety Post-Doctoral Research Program • Evaluation approaches • Signs of Safety Theory of Change • Signs of Safety Administrative Data Study • Limitations and Discussion
- 3. Signs of Safety Post-Doctoral Research Program • 3 year research collaboration between ACCP and WA Department for Child Protection and Family Support • To examine the impact and implementation of the Signs of Safety child protection practice model/approach on children, parents and practitioners
- 4. Signs of Safety Post-Doctoral Research Program 2013 • Theory of Change • Implementation • Children’s Study 2014 • Practitioner Study • Administrative Data Study • Parent Study 2015 • Aboriginal Staff and Practitioner Study • Triangulation Paper
- 5. Signs of Safety • Home-grown, locally developed approach to child protection practice • Origins in Solution-focused brief therapy • Used in 12 countries and spreading across the world • Designed to provide child protection practitioners with straight forward methods and tools to tackle child abuse in partnership with parents and professionals/service providers • Limited evidence base
- 6. Administrative Data Study • Use Departmental data from ASSIST (known as ‘administrative data’) • Examine the impact of Signs of Safety on outcomes for children • Compare these to outcomes for children pre Signs of Safety • Essentially we want to know if Signs of Safety is making a difference at the highest level (substantiations, entry into care) (Department KPIs)
- 7. Hypotheses Compared to children pre Signs of Safety, there will be: 1. A decrease in the number of children coming into care of Department 2a. A decrease in proportion of children with substantiations following an unsubstantiated notification 2b. A decrease in proportion of children with substantiations following a substantiated notification 3. An increase in proportion of Safety and Wellbeing Assessments (SWA) outcomes proceeding to family support or child centred family support
- 8. Limitations • Retrospective study • Use of administrative data that was not designed for evaluative purposes • Outcomes are multiply determined and influenced by: – Changing policy and practice context – Evolving Signs of Safety child protection practice framework – Family and other service factors – Demands on the system and resourcing
- 9. Methodology • Retrospective (follow forward) historical control using Departmental administrative data from 3 consecutive years pre-Signs of Safety implementation (2005-2007) and 3 consecutive years post-Signs of Safety implementation (2011-2013) • First notifications
- 10. Cohort 1 – Pre SoS (2005-2007) Cohort 2 – Post SoS (2011- 2013) Number of Children (Pre-birth to 18 years) 3,011 7,562 Gender 1425 boys, 1565 girls 3383 boys, 3813 girls Aboriginality 854 Aboriginal, 1738 non- Aboriginal 419 not recorded 1602 Aboriginal, 1928 non- Aboriginal 4032 not recorded Notifications 4,340 12,003 Investigations/Safety and Wellbeing Assessments 2,391 (55% of notifications) 8,050 (67% of notifications) Substantiations 962 (40% of investigations) 2,160 (27% of investigations) Sample and Context
- 11. Findings Hypothesis 1: There will be decrease in the number of children coming into care of Department There was an increase in the number and rate of of children coming into care of Department at a population level Cohort 1 (pre-SoS) Cohort 2 (post-SoS) All children 2.06 per 1000 children in 2005 (347 children) 3.89 per 1000 children in 2011 (647 children) Aboriginal children 0.3 per 1000 children in 2005 (147 children) 0.49 per 1000 children in 2011 (268 children)
- 12. Findings Hypothesis 1: There will be decrease in the number of children coming into care of Department There was a decrease in the proportion of children who were notified who subsequently entered care Cohort 1 (Percentage of children who were first notified and came into care) Cohort 2 (Percentage of children who were first notified and came into care) All children 3011 children with a notification 367 who were first notified and came into care (12.1%) 7562 children with a notification 647 who were first notified and came into care (8.5%) Aboriginal children 3011 children with a notification 147 Aboriginal children who were first notified and came into care (4.8%) 7562 children with a notification 268 Aboriginal children who were first notified and came into care (3.5%)
- 13. Findings Hypothesis 2a: There will be a decrease in proportion of children with substantiations following an unsubstantiated notification There was an increase in the proportion of children with substantiations following an unsubstantiated notification between Cohort 1 and Cohort 2 Number of children with substantiations following an unsubstantiated notification Number of children with substantiations following an unsubstantiated notification All children 1,271 children with unsubstantiated notification 99 children (7.8%) had a substantiated notification 4,578 children with unsubstantiated notification 523 children (11.4%) had a substantiated notification Aboriginal children 370 children with unsubstantiated notification 44 children (11.9%) had a substantiated notification 976 children with unsubstantiated notification 181 children (18.5%) had a substantiated notification
- 14. Findings Proportion of all children with substantiations following an unubstantiated notification by primary harm type Primary Harm Type Cohort 1 Cohort 2 Emotional Abuse 5 out of 105 children (4.8%) 66 out of 1053 children (6.3%) Neglect 20 out of 296 children (6.8%) 114 out of 1067 children (10.7%) Physical Abuse 18 out of 492 children (3.7%) 19 out of 861 children (2.2%) Sexual Abuse 9 out of 441 children (2%) 35 out of 1324 children (2.6%)
- 15. Findings Proportion of Aboriginal children with substantiations following a unsubstantiated notification by primary harm type Primary Harm Type Cohort 1 Cohort 2 Emotional Abuse 5 out of 44 children (11.4%) 9 out of 142 children (13.4%) Neglect 7 out of 116 children (6%) 46 out of 193 children (23.8%) Physical Abuse 5 out of 119 children (4.2%) 2 out of 85 children (2.4%) Sexual Abuse 3 out of 109 children (2.8%) 4 out of 126 children (3.2%)
- 16. Findings Hypothesis 2b: There will be a decrease in proportion of children with substantiations following a substantiated notification There was an increase in the proportion of children with substantiations following an substantiated notification between Cohort 1 and Cohort 2 Number of children with substantiations following an unsubstantiated notification Number of children with substantiations following an unsubstantiated notification All children 854 children with substantiated notification 100 children (11.7%) had a substantiated notification 1,779 children with substantiated notification 319 children (17.9%) had a substantiated notification Aboriginal children 304 children with substantiated notification 52 children (17.1%) had a substantiated notification 501 children with substantiated notification 117 children (23.4%) had a substantiated notification
- 17. Findings Proportion of all children with substantiations following a substantiated notification by primary harm type Primary Harm Type Cohort 1 Cohort 2 Emotional Abuse 13 out of 112 children (11.6%) 59 out of 529 children (11.2%) Neglect 30 out of 276 children (10.9%) 82 out of 575 children (14.3%) Physical Abuse 15 out of 273 children (5.5%) 17 out of 315 children (5.4%) Sexual Abuse 4 out of 223 children (1.8%) 18 out of 379 children (4.7%)
- 18. Findings Proportion of Aboriginal children with substantiations following an substantiated notification by primary harm type Primary Harm Type Cohort 1 Cohort 2 Emotional Abuse 5 out of 49 children (10.2%) 19 out of 178 children (10.7%) Neglect 17 out of 115 children (14.8%) 39 out of 217 children (18%) Physical Abuse 9 out of 89 children (10.1%) 6 out of 69 children (8.7%) Sexual Abuse 3 out of 64 children (4.7%) 6 out of 46 children (13%)
- 19. Findings Hypothesis 3: There will be an increase in proportion of Safety and Wellbeing Assessments (SWA) outcomes proceeding to family support or child centred family support Hypothesis cannot be answered as no comparable fields for Cohort 1 and Cohort 2.
- 20. Limitations and understanding the context • Dramatic increase in children coming into contact with the Department • Policy and practice changes – Legislation • Entry in to out-of-home care – Early intervention services, placement prevention
- 21. OTHER IMPORTANT PEOPLE (Carers, Relatives, Friends, Non-Government Practitioners/Staff) ARE ENGAGED Figure 1: Signs of Safety Results Logic Children and young people are SAFE (long term sustained safety) Children and young people are SAFE from imminent/immediate harm 3 Future-focused | Working relationships based on partnership | SIGNS OF SAFETY PRINCIPLES | Critical Inquiry | Recognising what is working well Signs of Safety Tools for Children and Young People Signs of Safety Mapping (incl Three Columns) Creating Safety Networks Workforce Wellbeing/Resilience CHILDREN AND YOUNG PEOPLE ARE ENGAGED 2 PARENTS ARE ENGAGED 1 SHARED UNDERSTANDING SAFETY PLAN CASE PLAN SHARED AGREEMENT Children and Community Services Act 2004 (WA), Western Australia Child Protection Policy Context Children participateNoticing what works Families are socially connected SIGNS OF SAFETY DISCIPLINESSigns of Safety IMPLEMENTATION (e.g. training, coaching, leadership, appreciative inquiry) Signs of Safety practiced with fidelity Children’s perspective acts as circuit breaker 4
- 22. Key Messages • Importance of articulated theory of change – What is Signs of Safety intended to do? • Limitations of administrative data as evidence in a changing context • Problems related to the use of blunt indicators and directionality of hypotheses (surveillance effects, enhanced decision making) • Signs of Safety fidelity, implementation and dose • Best outcomes for children may not be measured by administrative data
- 23. Dr Mary Salveron Post-Doctoral Fellow (Signs of Safety) Australian Centre for Child Protection University of South Australia +61 8 83022980 / 0400 863 784 mary.salveron@unisa.edu.au www.unisa.edu.au/childprotection
Editor's Notes
- FIONA
- Many of you may know that Signs of Safety is a child protection practice framework/model implemented across Western Australia. Now in its seventh year of implementation, Signs of Safety was home-grown and locally developed back in the late 90s by Andrew Turnell and Steve Edwards but wasn’t taken up by the Department until 2008 with its origins based on solution-focused brief therapy and narrative therapy. To date SoS is utilised in over 12 different countries and over 100 jurisdictios across the world. SoS is designed to provide practitioners with straightforward methods to tackle child abuse in partnership with parents and professionals and service providers. While there is limited evidence of how it works and what the outcomes are for children, families and practitioners, there appears to be some promising elements to the approach. I will go through in more detail the implementation of Signs of Safety within the Department in the next presentation and the strategies they used utilising Roger’s diffusion of innovation theory and Fixsen’s implementation stages and drivers model.
- Evaluation best practice is pre-test/post-test with a comparison group Major Limitation: Historical Control Design is Retrospective Means we are limited to the information already collected across the whole time period (pre and post ASSIST)
- Linked to KPIs of the Department
- FIONA
- FIONA
- Aboriginal children 147/485829 x 1000 = 0.3 268/544493 x 1000 = 0.49
- Aboriginal children 147/3011 x 100 = 268/7562 x 100 =
- FINDINGS FOR ABORIGINAL CHILDREN The patterns for Aboriginal children were similar to the entire Cohorts, however the rates for neglect were much higher. For all Aboriginal children with an unsubstantiated notification (370 children in Cohort 1 and 976 children in Cohort 2), 11.9% of children in Cohort 1 (44 children) and 18.5% of children in Cohort 2 (181 children) had a subsequent notification.
- FINDINGS FOR ABORIGINAL CHILDREN The patterns for Aboriginal children were similar to the entire Cohorts, however the rates for neglect were much higher. For all Aboriginal children with an unsubstantiated notification (370 children in Cohort 1 and 976 children in Cohort 2), 11.9% of children in Cohort 1 (44 children) and 18.5% of children in Cohort 2 (181 children) had a subsequent notification.
- Mandatory Reporting Changes in data reporting, recording
- As represented visually on this slide and read from bottom up), the Master Signs of Safety Results Logic situates Signs of Safety in the Western Australian legislative and practice context. It shows the disciplines and principles (the how) along with the Signs of Safety Mapping (the what), as the core elements of the approach, which are supported by other Signs of Safety tools and techniques. Where these approaches are implemented with fidelity, the theory suggests that they feed into the inter-related and mutually reinforcing casework objectives of developing safety plans and case plans and reaching shared understanding and shared agreement. These casework objectives are influenced by the extent to which parents, children and other important people are ‘engaged’. Additional logic models (denoted by blue stars numbered one to three) have also been created to further expound the notion of engagement and the way in which this interfaces with the casework objectives for each of these groups. Along the top of this diagram are the outcomes: Children and young people are safe from imminent harm and Children and young people are SAFE. The arrow leading from children and young people are safe from imminent harm back to the inputs depict the cyclical notion of assessment and intervention that may be required in order to shift from short-term to long-term safety. Finally, in recognition of the broader aspects of Signs of Safety and the anticipated benefits for the workforce , the bottom left hand corner includes the diagram of the key implementation expectations and the dual and mutually reinforcing outcomes of fidelity to the approach and workforce wellbeing/resilience. The development of the results logic and theory of change that underpinned Signs of Safety provided us with the evaluation components essential for developing the program of research. While we knew there needed to be research that encompassed outcomes for children, parents, practitioners and other important stakeholders, the outcomes (not perceptions) were more clearly defined with the theory of change. The logic gave us an understanding of the relationship between the various outcomes and components such as the relationship between fidelity and workforce wellbeing, fidelity and safety outcomes for children, children and parent engagement, parent engagement and safety outcomes. In addition, the logic also provided more efficiency so as not to burden participants (so in undertaking the research we could ask participants the right and relevant questions) Road map for evaluation Deeper understanding of Signs of Safety practice approach Shared experience Went on journey with us, to help us as evaluators Theory of change significant in articulating assumptions of how a program is meant to work Helpful for evaluators to understand and provide guidance to what we need to measure Adds rigour to evaluation Many spin offs – not just research but collaboration and an analysis of gaps and what needs to change or be better understood
- Administrative data system – need to build data system to answer evaluation questions and clinically relevant Directionality – engagement of families – do they make changes (behaviour change) and report to Department Achieving positive results in child welfare generally requires a team effort and multiple interventions. Thus, it may never be possible for child welfare and other community service providers to pinpoint any one particular strategy that impact on children entering care. This is why it is essential to take a multiple strategy approach to careful implement practice methods, successfully engage families and have equitable access to services.