• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Ces13 roles pdf
 

Ces13 roles pdf

on

  • 231 views

Presentation given at #CESToronto2013. Abstract: Developmental evaluation (DE) supports social innovation and program development by guiding program adaptation to emergent and dynamic social realities ...

Presentation given at #CESToronto2013. Abstract: Developmental evaluation (DE) supports social innovation and program development by guiding program adaptation to emergent and dynamic social realities (Patton, 2011; Preskill & Beer, 2012). To that end, the developmental evaluator is expected to draw on a multitude of skills and to take on multiple roles depending on the situational demands of the DE. Beyond that little guidance is provided in the literature to guide evaluation practice amidst the decidedly complex and turbulent space of social innovation.

This presentation explores the contextual demands made of the evaluator and the emergence of the different roles taken on by the developmental evaluator to enable social innovation in a case of DE. The ‘preformative development’ of this social innovation saw the integration of microblogging into teacher education. Analysis revealed four roles assumed by the developmental evaluator when innovating: a) evaluator, b) facilitator of learning, c) project manager, and d) innovation thinker.

Statistics

Views

Total Views
231
Views on SlideShare
135
Embed Views
96

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 96

http://chiyanlam.com 95
http://translate.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Ces13 roles pdf Ces13 roles pdf Presentation Transcript

    • ChiYan Lam, MEdCES 2013Insights on DevelopmentalEvaluator Roles duringInnovation: A Case Study@chiyanlamJune 11, 2013Assessment and Evaluation Group, Queen’s UniversitySlides available now at www.chiyanlam.com1Monday, 10 June, 13
    • Acknowledgements2Monday, 10 June, 13
    • 3Monday, 10 June, 13
    • ImpersonalHard to FocusPassive4Monday, 10 June, 13
    • ✓Learn from peers✓Reflect on priorexperiences✓Meaning-making✓Active constructionof knowledge5Monday, 10 June, 13
    • Assessment Pilot Initiative• Barriers to Teaching and Learning: $, time,space• Interested in integrating Web technologiesto Teacher Education to open up possibilities• The thinking was that assessment learningrequires learners to actively engage withpeers and challenge their ownexperiences and conceptions ofassessment.66Monday, 10 June, 13
    • Dilemma• Barriers to Teaching and Learning: $, time, space.• PRACTICUM = out of sight, out of touch.• Instructors became interested in integrating Webtechnologies to Teacher Education to open uppossibilities• The thinking was that assessment learningrequires learners to actively engage with peersand challenge their own experiences andconceptions of assessment.77Monday, 10 June, 13
    • So what happened...?• 22 teacher candidates participated in a hybrid, blendedlearning pilot. They tweeted about their ownexperiences around trying to put into practicecontemporary notions of assessment.• Guided by the script:“Think Tweet Share”• Developmental evaluation guided this exploration,between the instructors, evaluator, and teacher candidatesas a collective in this participatory learning experience.• DE became integrated; Program became agile andresponsive by design88Monday, 10 June, 13
    • 99Monday, 10 June, 13
    • Research Purposeto learn about the capacity of developmentalevaluation to support innovationdevelopment.(from nothing to something)1010Monday, 10 June, 13
    • Research QuestionWhat insights, if any, can be drawn from thisdevelopment about the roles and theresponsibilities of the developmentalevaluator?1111Monday, 10 June, 13
    • Developmental Evaluationin 2011• takes on a responsive,collaborative, adaptiveorientation to evaluation• complexity concepts• systems thinking• social innovation12Monday, 10 June, 13
    • Developmental EvaluationDE supports innovation development toguide adaptation to emergent anddynamic realities in complexenvironmentsDE brings to innovation and adaptationthe processes of:• asking evaluative questions• applying evaluation logic• gathering and reporting evaldata to inform support project/program/product, and/ororganizational development in realtime.Thus, feedback is rapid.Evaluator works collaboratively withsocial innovators to conceptualize,design, and test new approaches inlong-term, ongoing process ofadaptation, intentional change anddevelopment.Primary functions of evaluator:• elucidate the innovation andadaptation processes• track their implications and results• facilitate ongoing, real-timedata-based decision-makingin the developmental process.(Patton, 1994, 2011)13Monday, 10 June, 13
    • Developmental EvaluationDE supports innovation developmentto guide adaptation to emergent anddynamic realities in complexenvironmentsDE brings to innovation and adaptationthe processes of:• asking evaluative questions• applying evaluation logic• gathering and reporting eval datato inform support project/program/product, and/ororganizational development in realtime.Thus, feedback is rapid.Evaluator works collaboratively withsocial innovators to conceptualize,design, and test new approaches inlong-term, ongoing process ofadaptation, intentional change anddevelopment.Primary functions of evaluator:• elucidate the innovation andadaptation processes• track their implications and results• facilitate ongoing, real-time data-based decision-making in thedevelopmental process.(Patton, 1994, 2011)ImprovementvsDevelopment .14Monday, 10 June, 13
    • DevelopmentalEvaluationisrealitytesting.15Monday, 10 June, 13
    • Practicing DE• Adapative to context, agile in methods,responsive to needs• evaluative thinking - critical thinking• bricoleur• “purpose-and-relationship-driven not[research] method driven”(Patton, 2011, p.288)16Monday, 10 June, 13
    • Five Purposes and Uses1. Ongoing development in adapting program, strategy,policy, etc.2. Adapting effective principles to a local context3. Developing a rapid response4. Preformative development of a potentially broad-impact, scalable innovation5. Major systems change and cross-scale developmentalevaluation17(Patton, 2011, p. 194)Five Uses of DE17Monday, 10 June, 13
    • Evaluator Roles18Neutral, detached social scientistJudgeProgram AdvocateConsultantChange agentTechnical expertFacilitator of organizational inquirySupport Decision MakingCollaboratorEmpower othersTextAuditor18Monday, 10 June, 13
    • Role Conceptualization• Depends on...• Evaluation methods• Evaluation models• evaluator relationship withstakeholders• activities!? (Skolits, Morrow, & Burr,2009)1919Monday, 10 June, 13
    • Why Roles Matter• Roles we adopt determine behaviours anddegree of contribution within an eval.• Has implications for:• how we conduct evaluation (processes)• what is considered acceptable behaviour(practices)• training2020Monday, 10 June, 13
    • DE Roles - Bricoleur• Patton, 1994, 2011• Collaborator• Evaluator• Support decision-making• Documentarian• Elucidate innovation and adaptation...• .....?2121Monday, 10 June, 13
    • DE Roles - DE Primer• Coach to the organization, and to theinnovation process• Collaborator2222Monday, 10 June, 13
    • DE Roles - DE 201• Analyst (strategic thinking, patterrecognition)• Relationship Builder• Servant Leader• Facilitator of Learning23(Dozois, Langlois & Blanchet-Cohen, 2010)23Monday, 10 June, 13
    • Method & Methodology• Qualitative Case Study• understanding the intricacies into thephenomenon and the context• Case is a “specific, unique, boundedsystem” (Stake, 2005, p. 436).• Understanding the system’s activity, and itsfunction and interactions.• Qualitative research to describe, understand, andinfer meaning.2424Monday, 10 June, 13
    • Data Sources• Three pillars of data1. Program development records2. Development Artifacts3. Interviews with clients on the significanceof various DE episodes2525Monday, 10 June, 13
    • Data Analysis1. Reconstructing evidentiary base2. Identifying developmental episodes3. Coding for developmental moments4. Time-series analysis2626Monday, 10 June, 13
    • 27!27Monday, 10 June, 13
    • Findings - DE Rolesduring Innovation1. Manager2. Facilitator of learning3. Evaluator4. Innovation thinker2828Monday, 10 June, 13
    • DE Role - Manager• Manage the project• resources, personnel• Manage the progress• timeline, deadlines,milestones• Rally the troop• keep on taskhttp://civilengineerlink.com/wp-content/uploads/2011/02/project-managers.jpg29Monday, 10 June, 13
    • DE Role - Evaluator• Report on usage• shortened reporting cycle• monitor the conversationand report on substantivethemes• facilitates data-baseddecision-making• Systematic inquiry - renderinformation (e.g. via lit review)to inform program decisionshttp://freshspectrum.com/freshspectrum-in-the-wild-12-illustrations/30Monday, 10 June, 13
    • DE Role - FacilitateLearning• Meaning-making of data• made difficult due to noprior framework• had to be appropriate givencontext• To evaluate requires deepfamiliarization with a particularsubject --> makes you a quasi-expert.• creates opportunity to helpdevelop capacity in others31Monday, 10 June, 13
    • DE Role - InnovationThinker• Innovation is likeentrepreneurship. Goodentrepreneur has goodbusiness acumen. Knowsthe ‘business’• Evaluators working insocial innovation needsto know innovation .• Process of innovation isoften non-linear.http://seec.schulich.yorku.ca/ContentResources/LibraryFiles/innovations.jpg32Monday, 10 June, 13
    • ThankYou!Let’s Connect!@chiyanlamchi.lam@QueensU.ca33Monday, 10 June, 13