What they say and what they really mean – look at the budgets to see what funders really want – though often managers are confused too. Approach of a policy analyst – like whitewater rafting – not an advocate; work within constraints. There’s always something to work with.
CIC Report on Plans and Priorities 2011-2012 www.tbs-sct.gc.ca/rpp/2011-2012/inst/imc/imc01-eng.asp#mm NOTE5. The four population groups are: visible minority and foreign-born; non-visible minority and foreign-born; visible minority and Canadian-born; and non-visible minority and Canadian-born.Also www.tbs-sct.gc.ca/rpp/2011-2012/inst/imc/imc-eng.pdf
This format is based on the CIC Ontario Region format for proposals in 2010 . It may have changed since then. VALID LOGIC MODELS ARE NOT DEVELOPED ON FLIPCHARTS IN BRAINSTORMING SESSIONS!
Standardized school tests and baby rats – but we need to use something. Unobtrusive measures may have ‘dead grandmother’ effect. It’s not possible to be neutral when collecting high-stakes data; methodology must ensure high quality data (e.g., audits, objective studies using double-blind designs), though careful observations by participants must be used to develop the hypotheses.According to recent meta-analysis of survey research, about 1/3 of scientists admit to questionable research practices involving an intention to deceive. Deliberate falsification about 2%, other forms of misconduct much higher. We all need to acknowledge this problem and ensure there are built-in safeguards. (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0005738)
Working with Canadian Centre for Financial Literacy (CCFL), a program of SEDI (Social and Enterprise Development Innovations), which is developing a needs assessment tool for settlement agencies.
Personal communication, CIC policy analyst, March 2012
A5 measuring settlement outcomes tied to cic's logic model
Measuring Settlement Outcomes Tied to CIC’s Logic Model 2012 OCASI ConferenceDr. Gillian Kerrgillian@gkerr.com 1
Agenda• The context and background of CIC’s logic model• How CIC’s policy goals should affect the way that you design and deliver services• How policy goals should affect the way that you measure and communicate the impact of your servicesQUESTIONS:• Any specific topics you want me to cover?• Who is involved in CIC’s Settlement and Immigration Renewal Exercise? (correct me when I’m wrong)• Who works in the federal or provincial government? 2
Key underlying concepts• A budget is ideology stripped of rhetoric. To understand policy, look at where the money is going.• Funders are not rational, and neither are agencies, so don’t believe policy analysts.• The most predictable impact of ‘pay for performance’ is increased cheating. It takes years to get it right. 3
Settlement Program Logic Model – 2010 Policy Development, Program Design and Management Settlement Services Language Information & Program Policy and Program Program Implementation and Needs Assessment Learning & Employment-related Support Services Awareness Community ConnectionsComponents Development Management and Referrals Skills Services Services Development • Strategic plans • Operational plans • Initial and on-going Enabling services: • Information • Language • Labour market • Individual and needs assessments – Childminding products training bridging community-level • Policy, priorities, standar • Program delivery materials and – Transportation ds and outcomes (including language • Other skills/ • Job search skills bridging, e.g.: tools assistance • Orientation assessments) – Provisions for life-skills training – Host/ mentor matches • Performance • Functional guidance & training sessions • Referrals to CIC- disabilities training – Volunteers engaged trained measurement strategy • Promotion • Labour market and supported and national reports • Data collection and funded and Other support and outreach information regional/local/SPO reports community • Cultural awareness, anti- Outputs • Horizontal coordination services: • Workplace racism, and welcoming • Regional, local and SPO settlement • PT consultations services – POE reception services orientation communities services coordination – Translation • Research analysis/ and – Interpretation reports • Service delivery capacity building – Settlement/ crisis • Best practices and info sharing counselling • Funding allocation • Contribution agreements E 6. Clients, service providers and CIC are aware of 1. Policies and programming align with departmental and newcomer settlement government priorities 8. Target population is aware of CIC settlement services needsImmediate 2. Program models are evidenced-based, informed by 9. Timely, useful and appropriate CIC settlement services are available in the Official Language of choice (in 7. Referrals and accordance with the Official Languages Act and Policy) Outcomes stakeholder input and address the barriers & needs of both personalized settlement newcomers and communities plans are based on 10. Clients obtain the CIC settlement services they need to deal with settlement issues as they emerge 3. Standards, tools, resources and program coordination support assessed settlement the effective delivery of services) needs 4. Services are efficiently delivered 5. Provision of settlement services across Canada that achieve A B C D comparable outcomes 17. Clients are connected to the 15. Clients have knowledge broader community and social 11. Clients have timely, useful and 13. Clients have the official of the Canadian work networks accurate information needed to make language skills needed to environment and are informed settlement decisions function in Canadian 18. Program participants are awareIntermediate connected to local 12. Clients understand life in Canada society of newcomers’ needs and Outcomes labour markets including laws, rights, responsibilities 14. Clients have the skills/life- contributions and are engaged in 16. Clients have the skills to newcomer settlement and how to access community skills needed to function in find and apply for Key Outcomes resources Canadian society employment Expected A - Orientation Results B - Language/Skills A - Orientation C - Labour Market Access Ultimate 19. Newcomers find employment commensurate with their skills and experience Outcomes DB Community Connection - - Language/Skills 20. Newcomers enjoy their rights and act on their responsibilities in Canadian society E - Development and Capacity Building C - Labour Market Access 21. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian society D – Welcoming Communities 22. Newcomers contribute to the economic, social and cultural development needs of Canada (in PAA) CIC Strategic E - Program and Policy Development Outcome 3 23. Successful integration of newcomers into society and the promotion of Canadian Citizenship
Logic models (1)• A logic model shows the links between your daily activities and your long term outcomes Short term Long term Activities outcomes outcomes• Explains to funders why you are doing what you’re doing• Highly simplified summary of what you are trying to achieve• Helps to plan your programs to achieve your goals 8
Case study: Change to Citizenship Regulations Proposed amendments to Citizenship Regulations (which govern implementation of Citizenship Act) require objective evidence of English or French language ability to be included with applications. The required level is set to be equivalent to the Canadian Language Benchmark level 4: “The proposed amendment to section 14 of the Regulations uses CLB/NCLC benchmarks which are recognized as the official Canadian standards for describing, measuring and recognizing the language proficiency of adult immigrants in both English and French.” • The indicator operationalizes the policy. • SPOs are not expected to create valid language tests to measure policy outcomes. • SPOs should be involved in creating good indicators to protect legitimate interests of newcomers & SPOs 10
Campbell’s Law for Indicators"The more any quantitative social indicator isused for social decision-making, the moresubject it will be to corruption pressures and themore apt it will be to distort and corrupt thesocial processes it is intended to monitor.“Donald T. Campbell, Assessing the Impact of Planned Social Change, 1976. 12
Case study: Developing asettlement needs assessment toolIn CIC logic model, settlement needs assessment comes first 13
What would you be measuring if you took the logic model seriously? What would you see if you were delivering services according to the logic model? 14
Does CIC care about the logic model? Before investing in new measures, are they valued by the funder? Relevant CIC Planning Highlights for Settlement Programs – 2011-2012: • Complete a review of the Settlement Program to include recommendations on priorities, service delivery mechanisms, performance measurement strategy and governance structure. • Collaborate with provinces and territories to develop an evidence base on settlement outcomes for immigrants and draft a pan-Canadian framework for settlement outcomes for consideration by Ministers. • Plan and implement the methodology, research approach and launch of the Settlement Program evaluation. • Complete implementation of action items derived from evaluations of the Official Languages Program, the Language Instruction for Newcomers to Canada program, the Immigrant Settlement and Adaptation Program, the Host Program and the Welcoming Communities Initiative, as well as the audit of the administration of settlement programs. • Complete field testing of the Portfolio-Based Language Assessment and launch a pilot project to test the delivery infrastructure required for a new national test of language achievement. The objectives will be to provide newcomers with a language credential and improve CIC’s ability to evaluate the impact and outcomes of language programs. • Implement a standardized approach to newcomer needs assessment and associated settlement plans across service delivery organizations, and ensure collection of baseline data. • Include a website on best practices in settlement to share evidence-based practices among service delivery organizations across the country. • Expand local partnerships and pilot projects from Ontario to other parts of Canada, focusing on larger urban centres. From CIC Report on Plans and Priorities 2011-2012 www.tbs-sct.gc.ca/rpp/2011-2012/inst/imc/imc- eng.pdf
Are current program policy initiatives consistent with logic model? Settlement Information Renewal Exercise (SIRE) CIC is committed to improving Needs Assessment & Referral Services (NARS) in 3 ways: • Establishing data collection and reporting protocol to collect baseline data on settlement needs (iCARE’s first module is NARS) • Developing settlement planning tools for newcomers (e.g., Living in Canada site) • Developing guidance and standards for CIC staff and SPOs Conclusion: Yes, it looks safe to rely on the logic model, and it’s a good time to influence CIC tool development. 16
Consider cooperative development• SPOs reluctant to share resources with colleagues, resulting in a whole lot of half-baked tools that turn well-intentioned staff into thieves or hoarders and prevent improvement.• Use open-sourced/open-access approach to create valid and legitimate tools that are consistent with sector’s values and outcomes.• Work with CIC policy analysts to co-develop - they love to be appreciated and it will result in more integration.• Use other initiatives to lever sector resources – e.g., Canadian Centre for Financial Literacy’s settlement needs assessment tool. 18