CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Compliance Framework


Published on

This is a paper presentated to CES Toronto 2013. It describes a case study of how a management process is being used to embed effective design of regulatory compliance approaches into practise, and how it makes space for monitoring and evaluation. It highlights the importance of team formation in bringing expertise toegether from across the regulatory system (standard setting, monitoring and behaviour change). It also highlights some of the lessons from the initial application of the framework

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Compliance Framework

  1. 1. Risk Based Compliance Framework as anApproach to Engage Practitioners inEvaluationCanadian Evaluation Society Conference 2013Mark WhiteOntario Ministry of the Environment
  2. 2. Purpose• Describe the Risk Based Compliance Framework• A case study in the use of a management process toengage practitioners in evaluation• Highlight our learning relevant to the themes ofengagement across boundaries and embeddingevaluation in the organization• It has to be needed... now!• Familiarity breeds... engagement• It helps to be good enough, not perfect• Adapt to the environment• Providing a trigger for evaluation2 Ministry of the Environment
  3. 3. 3Risk Based Compliance Framework: OverviewContext•A healthy environment for a strong Ontario•Compliance approach = a combination oftools to monitor compliance and to influencebehaviourAim•To develop compliance approaches thatensure efficient and effective delivery ofregulatory objectivesKey Features•Integrates expertise across organizationalboundaries•Systematically considers influencesBenefits•Focus on evidence based decisions•Focus on communicating the benefits wedeliver3 Ministry of the Environment
  4. 4. Risk based Compliance Framework: Steps4
  5. 5. The Risk based Compliance Framework and Evaluation5 Ministry of the Environment
  6. 6. Performance Monitoring6 Ministry of the Environment• Performance Monitoringunderpinned by logic model A conceptual framework formingthe basis for a managementscorecard Intuitive way of linking activitiesto regulatory objectives Tailored for each complianceapproach• Challenges Outcome measures Limited evidence base Cost effectiveness
  7. 7. Engaging Practitioners in Evaluation7 Ministry of the Environment
  8. 8. Theme 1: It has to be needed… now!8 Ministry of the Environment
  9. 9. Theme 2: Familiarity breeds… Engagement•Relationship between evaluativeprocess and context is importantfor engagement Familiarity with concepts Familiarity with tasks Fit with process•Engagement on easier tasksprovides basis for more complexevaluative tasks9 Ministry of the Environment
  10. 10. Theme 3: Good enough, not perfect10** Haynes, L. et al. (2012). Test, Learn, Adapt: Developing Public Policywith Randomised Controlled Trials, Cabinet Office. Behavioural Insights• Working together to improvethe framework Improves the product Improves engagement• Pilot project Test on a ‘Real’ problem Engages project team Engages management• Awareness raising as anopportunity to learn andadapt
  11. 11. Theme 4: Adapt to the Environment11 Ministry of the EnvironmentDarwin’s finches**Wikimedia Commons (PublicDomain)• Adapting the framework tomeet the needs of thespecific circumstances Subject matter Tailoring the framework to context Importance of understandinginterdependencies betweencompliance tools Limitations in evidence base Users Integration of different subjectmatter expertise Decision required Most effective & efficientcombination of approaches acrosscompliance program
  12. 12. Theme 5: Providing a ‘Trigger’ for Evaluation12 Ministry of the Environment• ‘Trigger’ points forevaluation are important Using a managementprocess to ensure evaluationis considered in a timelyfashion Consideration of what is goodpractice Development of Monitoring andEvaluation Plan prior toapproval
  13. 13. Conclusion Evaluation across Boundaries– Functional & Professional Engaging Practitioners in Evaluation– Management process:o Driven by need to promote effectivenesso Approach tailored to the circumstances– Evaluation:o Design Effectivenesso Implementation Effectivenesso Outcome Effectiveness13 Ministry of the EnvironmentQuestions?mark.white@ontario.ca