Designing Evaluations for Developing Digital Literacies Support Webinar 13 april 2012

Evaluation support webinar – 13th April 2012 Chair: Paul Bailey Facilitator: Jay Dempster Session plan: 1.00 – 2.30 pm1. How the programme is supporting evaluation 3 critical factors for projects, “aspiractions” for effective evaluations2. From baselining to evaluation where we‟re at, approaches being used, your documentation3. Evaluation reality check => how clear are you? stakeholder engagement, progress in thinking, different facets of your evaluation4. Developing DL indicators & measures (indiv, group, org)5. Q&A / further support needs

How the programme is supporting evaluation

Evaluation aims, scope & focusEvaluation is measuring success in a systematic and objective way. Evaluation focuses on whether the project was effective, achieved its objectives, and the outcomes had an impact.

Evaluation aims, scope & focus Evaluation is measuring success in a systematic and objective way. Evaluation focuses on whether the project was effective, achieved its objectives, and the outcomes had an impact.1. Baseline2. Project processes3. Progress against plan/activities4. Understanding/lessons learned5. Value: tangible benefits, outcomes & impact6. Deliverables: outputs, dissemination

How projects are approaching evaluation

How projects are documenting evaluationFORMALLY: INFORMALLY: Project aims & objectives Critical success factors  Data management Anticipated impact  Journaling/blogging areas/descriptions  Wiki Outputs & outcomes  Project meetings/notes Detailed evaluation plan  Steering/prof. groups Baseline report ? 3 x Interim reports Final report

Building out the evaluationEvaluation Indicators Baseline Source of CollectionQuestions information method ScheduleWhat do you How will What is the Where can How will the When, wherewant to know? you know? situation prior this data be Evidence data and who will(for each to any obtained? be gathered? gather theaspect/ activity? data?stakeholder)the key questions measure of survey, interview, timing,for each objective success – observation, task/peoplethat will need to be indicators that focus group etc allocationevaluated something has been achievedAdapted from JISC Six Steps Evaluation Handbook& used at programme start up meeting – aligns with DL „framework‟ 6 aspects

Evaluation check list/critique• Baselining - How has this review & stakeholder engagement helped you to gain a shared sense of what digital literacies mean, how they are currently being developed & supported, and what the key questions are for your project/institution/partners?• Contextual framework – What is emerging in terms of a common focus and common language for how you will define and measure effectiveness & progress in digital literacies? (contributing towards what the programme is learning collectively)• Triangulating – How you are cross-checking evidence by using a variety of types & sources of data/stakeholder input, review/responding to emerging findings?• Key indicators – What kinds of descriptors/measures/rubrics are you (and/or stakeholders) using, which are the most useful, for baseline/impact indicators of ‟success‟?• Project effectiveness – How are you building in monitoring, reflection, QA of project processes, progress/emerging outcomes, capturing the unexpected?• Feasibility – Is your evaluation robust/realistic given timescale, resources, skills etc?

Project requirementsThree critical factors:1. The QUALITY of your reports is dependent pretty much on theROBUSTNESS of your evaluation processes.2. You should work with the programme support team (E & S & CFs)and your key stakeholder/steering groups or prof bodies) to help youCAPTURE & EVIDENCE what you are LEARNING/ACHIEVING fromyour project (early on & final outcomes).3. To COLLATE & REPORT your findings, all of your project outputsneed to be ORGANISED & EASY TO FIND (on your website and/or prog wiki).

“Aspirations/actions”for effective evaluations

Aspirations for evaluation ... what we aim to produce Clear, coherent set of questions Questions that span/address stakeholder interests Evidence that answers your key questions! Having „evaluation‟ questions, as distinct from „research‟ questions Cohesive data analysis ACROSS a set of evaluation activities Incisive reports & actionable findings

Actions for evaluation ... what we try to avoid (adapted from Jane Davidson)1. The „NO Questions‟ Evaluation2. The „NO Answers‟ Evaluation

Actions for evaluation ... what we try to avoid3. The „Research Questions‟ Evaluation4. The „Disconnected‟ Evaluation

“Aspiractions” for evaluation ... ‘reality-check’ list1. Do your questions constitute „big picture‟ outcomes?2. Are your questions „evaluative‟ in nature? (‘What’s so?’ but also ‘So what?’ and ‘What next?’)3. Will your answers/findings be direct, explicit & „actionable‟?

From baselining to evaluation Where are we at? How clear can you get, really? Evaluation reality check ...

10 baseline outcomes useful for evaluation1) contextualise „big picture‟ evaluation questions2) check back against the „why and how‟ of project achievements3) review stakeholders‟ shared/distinct meanings, sense of „success‟4) develop attributes/terminologies/taxonomies (project, programme)5) link to/expand skills/abilities/competencies/national frameworks6) map questions & intended benefits/impacts to DL framework focus areas & project beneficiaries (indiv, group, org)7) provide a clear structure for collecting evidence8) confirm methodologies/techniques & measures/indicators9) revise plan (as required) for resources, timings, data analysis (coding)10) explore appropriate reporting/dissemination formats for different stakeholder groups

From Baselining to EvaluationWhat emerged/changed?  interim findings1. How did the baseline help establish any shared meanings across the project and its context? (e.g. terminologies, assumptions made etc.)2. What useful descriptors/rubrics/key indicators are emerging? (for how you define and measure effectiveness & progress in digital literacies – people, groups, organisationally)3. What methodologies are you using to gather and analyse data? And how will you continue to triangulate/cross-check across your stakeholders?4. What challenges/problems (if any) are you coming up against (and resolving)?

How clear can you get, really? Can you rehearse your ‘evidence interpretation’ BEFORE you collect it? Overview of your project purpose and objectives, sufficient to get a flavour/reminder what it is about Summary of your key stakeholder interests in the project & evidence of its outcomes Your key evaluation questions & their ‘success’ indicators (what would constitute an ‘excellent’ outcome in key areas of impact, as opposed to good, mediocre, inadequate, minimally effective, below target, etc.) Evaluation measures & methodsArguments for your methodology including any novel/specific featuresWays to consult on/collect evidence around ‘benefit’ -‘value’ -‘worth’ - ‘impact’

EvaluationReality Check

Progress in thinking/planning?Q1. Briefly describe your approach to evaluating your DL project.Q2. In what ways have you used the DL synthesis framework or evaluation templates offered by the programme, and/or created your own tools/approaches?Q3. What are your big picture questions in relation to your intended changes/enhancements?Q4. How far along are you in identifying what success looks like in relation to those intended outcomes?Q5. How well do you feel your evaluation activities integrate with other project activities and gather data about uptake and engagement levels? across your stakeholders?

Interim reflections/revisionsA. How has initial understanding of the context & nature of your project’sintervention impacted your evaluation strategy and methodologies?  Macro „institutional‟ level?  Micro „individual/group‟ level?B. What are your preferred evaluation models & methods of datacollection, and why did you select these? 1. How have these changed (if at all) after baselining pilots/experiences? 2. Does this approach support one-off sets of findings and/or emerging evidence of innovation/change over time (addressing complexity & „big picture‟ questions)? 3. Who are the active/reflective participants (internal agents/external stakeholders) in your project‟s development and how are you involving them in your evaluation? 4. How do interim findings feed into dissemination/revisions to intentions and lines of enquiry?

Support needs Your evaluation plans Timing Factor to Questions to Method(s) Measure of Evaluate Address Success1. Have you considered the different facets of your evaluation? Could a logic model be useful?2. How well are your evaluation activities clearly identified, structured, integrated, resourced & scheduled into your project plan timeline (v.v. other activities, milestones)?3. What are the issues /needs /gaps /questions arising for you?

Designing Evaluations for Developing Digital Literacies Support Webinar 13 april 2012

by Jay Allyson Dempster on May 28, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 131

Statistics

Designing Evaluations for Developing Digital Literacies Support Webinar 13 april 2012 Presentation Transcript