Recommended
More Related Content
What's hot
What's hot (12)
Similar to MFLN Evaluation and Reporting Launch
Similar to MFLN Evaluation and Reporting Launch (20)
Recently uploaded
Recently uploaded (20)
MFLN Evaluation and Reporting Launch
- 2. Developmental Evaluation “DE supports innovation development to guide adaptation to emergent and dynamic realities in complex environments.” — MQP Goal: support project, program, and organizational development with timely feedback
- 3. Core DE Question What is getting developed and what are the implications of what gets developed? Complexity theory of change: Bring people together who are knowledgeable and committed and they will self-organize, take action, and work together to create movement, innovation, and change.
- 4. Key Characteristics of DE
- 6. Focus on development Development is when people are changing what they are doing, and the very nature of the standards are also changing. Standards are adapting to changing conditions. NOT the same as continuous improvement (formative eval or accountability) – Quality improvement is helping programs meet standards that have been set.
- 7. Complex and dynamic environment Multiple leadership nodes Multiple paths forward
- 8. Rapid and continual feedback Ask evaluative questions Apply evaluative logic Gather and report evaluative data Evaluation becomes part of
- 9. Developmental Evaluator Works collaboratively with innovators to conceptualize, design, and test new approaches in a long-term, ongoing process of adaptation, intentional change, and development.
- 10. Evaluation for MFLN • Ongoing development • Innovation in learning – Internal practice – External product • Reflection and action
- 11. DE for MFLN
- 12. MFLN: Focus on development Programming has been established… – Identify guiding principles that inform ongoing development – Establishes track record, reliability, and reputation ... BUT is not necessarily fixed – Innovate, innovate, innovate – Internal adaptability to meet changing external standards and conditions
- 14. Cooperative Agreement “An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on- time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi- directionally. This often results in
- 15. Constant evaluative feedback and discussion Monthly reports (internal, DoD) Quarterly reports (NIFA) Annual reports (internal, DoD, NIFA) Webinar evaluation reports (CAs) Quarterly webinar reports (internal, DoD) Weekly leadership team meetings Instant messaging Hangouts E-mail Google docs Twitter Phone
- 16. Internal evaluator Member of leadership team Elucidates innovation and adaptation processes Tracks implications and results Facilitates data-based decision-making
- 17. DE for MFLN: What It Looks Like Continuous adaptation
- 18. DE for MFLN: What It Looks Like Continuous adaptationSocial, political, economic, technological, and demographic patterns
- 19. DE for MFLN: What It Looks Like Continuous adaptationSocial, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences
- 20. DE for MFLN: What It Looks Like Continuous adaptationSocial, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment
- 21. DE for MFLN: What It Looks Like Continuous adaptationSocial, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment eXtension
- 22. DE for MFLN: What It Looks Like Continuous adaptationSocial, political, economic, technological, and demographic patterns Emergent developments/needs of our target audiences Cooperative agreement environment eXtension DoD
- 23. DE for MFLN: What It Looks Like (not this. . . .)
- 24. DE for MFLN: What It Looks Like (but this. .
- 25. Inspired by Jeff Conklin, cognexus.org Time Complex development situations are ones in which this… 25 Michael Quinn Patton AEA eStudy webinar 2014
- 26. Michael Quinn Patton AEA eStudy webinar 2014 26 And this…
- 27. Turns out to be this… Time 27 Michael Quinn Patton AEA eStudy webinar 2014
- 28. Michael Quinn Patton AEA eStudy webinar 2014 28 …looks like this
- 30. DE for MFLN: What It Looks Like Inductive
- 31. DE for MFLN: What It Looks Like Inductive Chaotic and messy
- 32. DE for MFLN: What It Looks Like Inductive Chaotic and messy Embrace forks in the road
- 33. DE for MFLN: What It Looks Like Inductive Chaotic and messy Embrace forks in the road Developmental moments Track what’s going on Understand what’s going on Adapt to what’s going on
- 34. DE for MFLN: What It Looks Like Everyone is an evaluator
- 35. DE Successes for MFLN
- 36. DE Successes for MFLN Very responsive to our partners’ programmatic requests and innovations
- 37. DE Successes for MFLN Very responsive to our partners’ programmatic requests and innovations You. Here. Now.
- 38. DE Successes for MFLN Very responsive to our partners’ programmatic requests and innovations You. Here. Now. Programming
- 39. DE Successes for MFLN Very responsive to our partners’ programmatic requests and innovations You. Here. Now. Programming “Walk the talk”: model use of social media, collaborative learning, personal learning networks
- 40. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious
- 41. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings
- 42. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings IRB challenges
- 43. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings IRB challenges Daily, evaluative thinking
- 44. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings IRB challenges Daily, evaluative thinking Platform delivery limitations
- 45. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings IRB challenges Daily, evaluative thinking Platform delivery limitations Military culture
- 46. DE Challenges for MFLN Innovation is challenging, slow, and can be contentious Moving from program improvement mode to redesign based on evaluation findings IRB challenges Daily, evaluative thinking Platform delivery limitations Military culture Scaling up
- 47. DE Features for MFLN: Nodal Evaluations Webinars Social media Focus groups Deliverables/plans of work • Why as much as what Documentation • Change, innovation, decision-making
- 48. DE Features for MFLN: Reporting From Concentration Areas • PIs – Monthly via Google forms: narrative updates, pubs, presentations, challenges • Social Media Specialists – Monthly via Google forms: blogs, AaE, CEUs – Webinar evaluation reports
- 49. DE Features for MFLN: Reporting To CAs: • Monthly social media from Sprout Social • Quarterly webinar data reports Internal/External: • Monthly (internal/DoD) • Quarterly (NIFA) • Quarterly webinar (internal/DoD) • Annual (internal/D0D/NIFA)
- 50. DE Features for MFLN: Reflective Discussion CAs: • Monthly social media specialists meetings – NetLit, processes, evaluation, communications Leadership: • Weekly meetings PIs: • Monthly meetings
- 51. DE Features for MFLN: Learning and Working Paradigms Leadership Reflection Transparency Collaboration Action Innovation
Editor's Notes
- DE is not ongoing formative eval DE supports and documents development
- DE is not ongoing formative eval DE supports and documents development
- Focus on development, which is not the same as continuous improvement (formative eval) or accountability Improvement vs. development Quality improvement is helping programs meet standards that have been set. Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
- Focus on development, which is not the same as continuous improvement (formative eval) or accountability Improvement vs. development Quality improvement is helping programs meet standards that have been set. Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
- Focus on development, which is not the same as continuous improvement (formative eval) or accountability Improvement vs. development Quality improvement is helping programs meet standards that have been set. Developmental evaluation is when people are changing what they are doing and the very nature of the standards are also changing. Standards are adapting to changing conditions DE: standards themselves and what it means to meet them and how to talk about them and how to apply them is being developed and changed in relationship to a changing world; very different from having a predetermined set of quality criteria that a program has to meet.
- Cooperative agreement: NIFA, DoD, and UIUC Different than traditional grant, which is when you articulate your research goals and deliverables before receiving your funding; instead, we receive funding based on a loosely defined set of criteria (plans of work) An award similar to a grant, but I which the sponsor’s staff may be actively involved in proposal preparation, and anticipates having substantial involvement in research activities once the award has been made. Can you tell me if the PF VLE was a result of that responsiveness? I'm looking for examples of how our plans of work can change during the year based on DoD conversations and request. I can't remember, but I thought the VLE was an example of that--not planned for 2014 but instead requested after the year started. Yes, this is a great example. DoD thought is was coming at some point - we had talked about it in the past. They felt that it was time, as it met a quickly evolving need from their end. They asked and stressed how important it would be to accomplish this at this time and we went back to our PF team and they were open, excited, and offered a plan for how they could do it within their budget (not adding but rearranging and deleting other deliverables). After PF’s new plan was discussed with DoD, all were happy to move forward. Another example of how this cooperative agreement has worked from our end (we instigated a suggested change) was when I began to hear more and more about how military caregiving was an increasing concern from those Anne and I talked with when we were at conferences (especially military conferences). We were not hearing this from MC&FP and so I did some homework on my end, began to have these conversations with Betsy and Eddy, and suggested that we should consider expanding in this area. They took this idea back to their offices and partners and then came back to us with an approval to move forward. We did this within our 2010 award at the end when we needed to re-capture funds and construct a re-budget. We had about $40K to play with and conducted a 5 month pilot. MC&FP liked it that it was fully funded for 2012 (and of course has been ever since with great success). So it goes both ways, they ask and we deliver - we suggest and they agree. It’s been great. There are other examples if you need more, just let me know. eXtension: emergent and dynamic organization both tied to and simultaneously oriented away from Cooperative Extension.
- .
- We
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Real time innovation is challenging --We’re very good at innovating and request of our partners; not as nimble for us to innovate on our own (still top down in that way); --PIs bought out for only a small portion of time; innovation and change is slow, may not have everyone’s buy-in Challenge: We were really in program implementation/improvement mode for the first four years. How do you introduce the possibility that you are no longer in program improvement mode but in fact that program needs to be redesigned? How do you prepare the people you’re working with for the possibility that you are no longer in program improvement mode but rather that you need to begin adapting and changing to findings of formative evaluation? Difficult to make changes quickly to eval and research --DoD looking for all kinds of information, but IRB is a challenge (18 month IRB process for working with contractors; hard to be responsive and provide real-time feedback) Evaluation is constantly “in your face” --Getting everyone to think evaluatively is challenging Platform limitations for reaching people on bases; --Limits innovation Military culture (private, formal); Doubling in size in 2015—lots of work to get done and most days are about just getting through the day
- Lots of