JISC BCE Programme – ‘Evaluation Phase 1’ Evaluation Framework – Implications for BCE Projects A Presentation by: Dr. Steve Johnson WM Enterprise 05.03.2009

Introduction WM Enterprise (WME) is a specialist economic development & regeneration consultancy JISC has commissioned WME to undertake ‘Evaluation Phase 1’ (work stream 6) of the Business and Community Engagement (BCE) programme, to establish an evaluation framework for the programme and its 6 constituent work streams WME Project Team: Project Director – Dr. Steve Johnson Project Manager – Ruth Winterton Research Consultant – Helen Blakeley

WM Enterprise (WME) is a specialist economic development & regeneration consultancy

JISC has commissioned WME to undertake ‘Evaluation Phase 1’ (work stream 6) of the Business and Community Engagement (BCE) programme, to establish an evaluation framework for the programme and its 6 constituent work streams

WME Project Team:

Project Director – Dr. Steve Johnson

Project Manager – Ruth Winterton

Research Consultant – Helen Blakeley

Elements of the Evaluation Framework Capital Evaluation Framework JISC Evaluation Handbook Attribution Sustainability Practical solutions Identification of good practice Quality Flexibility Short, medium & long-term measures Programme, work stream and project level evaluation Strategic & operational levels Formative & summative Elements to be included within the Framework

BCE Programme - Logic Chain Rationale “ In the ‘knowledge economy,’ BCE makes a key contribution to innovation, competitive advantage, skills enhancement workforce development and social cohesion.” High-level Aims Aims: To enhance institutional efficiency, effectiveness and opportunities in business and community activities. To improve access to institutions’ knowledge assets for business and community orgainsations. Inputs & Activities Financial Resources: c.£2 million per annum up to 2011 Participation from public & private sectors and social, cultural and civic arenas. Short-term Outputs Range of outputs across 6 work streams, including: New IT tools and collaborations to support BCE; Tailored workshops and training sessions; Dissemination and networking events; and Pilot projects and trials of various innovative IT solutions. Medium-term Outcomes (Benefits) Improved ICT to support BCE Wider engagement in BCE activities; Productivity enhancement; Innovation and enterprise; Employer engagement; and New and improved relationships between institutions and external BCE partners. Long-term Impacts More effective BCE activities; Enhanced institutional efficiency, effectiveness and opportunities in BCE activities; Improved access to institutions’ knowledge assets for BC organisations; and Contributions to more; highly skilled workforce; competitive economy; and cohesive, knowledge-based society.

High-level Aims

Aims:

To enhance institutional efficiency, effectiveness and opportunities in business and community activities.

To improve access to institutions’ knowledge assets for business and community orgainsations.

Short-term Outputs

Range of outputs across 6 work streams, including:

New IT tools and collaborations to support BCE;

Tailored workshops and training sessions;

Dissemination and networking events; and

Pilot projects and trials of various innovative IT solutions.

Medium-term Outcomes

(Benefits)

Improved ICT to support BCE

Wider engagement in BCE activities;

Productivity enhancement;

Innovation and enterprise;

Employer engagement; and

New and improved relationships between institutions and external BCE partners.

Long-term Impacts

More effective BCE activities;

Enhanced institutional efficiency, effectiveness and opportunities in BCE activities;

Improved access to institutions’ knowledge assets for BC organisations; and

Contributions to more; highly skilled workforce; competitive economy; and cohesive, knowledge-based society.

Key Outcomes and Impacts Our document review and consultations have identified that the impact of the BCE programme falls into 5 main categories : High-level strategy and policy impact Impact on JISC (Impact on) institutional strategies (Impact on) institutional BCE activities Wider BCE impacts All aspects of the BCE programme contribute to the higher-level aims and objectives of the programme (and to the wider knowledge transfer/innovation/BCE agenda). However, each work stream/project addresses specific areas of the agenda. Therefore, on a work stream and project level , some of the above categories are more directly relevant to specific work stream benefits/project aims and objectives than others. The framework recognises the above and sets out our initial view of the nature of impacts on a work stream and project level where: Indirect impact Direct and indirect impact Direct impact

Our document review and consultations have identified that the impact of the BCE programme falls into 5 main categories :

High-level strategy and policy impact

Impact on JISC

(Impact on) institutional strategies

(Impact on) institutional BCE activities

Wider BCE impacts

All aspects of the BCE programme contribute to the higher-level aims and objectives of the programme (and to the wider knowledge transfer/innovation/BCE agenda).

However, each work stream/project addresses specific areas of the agenda. Therefore, on a work stream and project level , some of the above categories are more directly relevant to specific work stream benefits/project aims and objectives than others.

The framework recognises the above and sets out our initial view of the nature of impacts on a work stream and project level where:

Indirect impact

Direct and indirect impact

Direct impact

Evaluation Framework – Programme & Work Stream Level High-level Strategy and Policy Impact Type of Impact Reach and impact Spatial Type of institution Contribution to policy development Local level Regional level National level Work stream 6 Work stream 5 Work stream 4 Work stream 3 Work stream 2 Work stream 1 Outcome/ Impact

Evaluation Framework – Programme & Work Stream Level Impact on JISC Type of Impact Impact on JISC strategy/activity Awareness of JISC JISC-wide coherence Work stream 6 Work stream 5 Work stream 4 Work stream 3 Work stream 2 Work stream 1 Outcome/ Impact

Evaluation Framework – Programme & Work Stream Level (Impact on) Institutional Strategies Type of Impact Role of IT in institutional strategies Collaboration between institutions Institutional joined-up thinking Impact on institutional strategies Work stream 6 Work stream 5 Work stream 4 Work stream 3 Work stream 2 Work stream 1 Outcome/ Impact

Evaluation Framework – Programme & Work Stream Level (Impact on) Institutional BCE Activities Type of Impact Take-up of BCE products Increased BCE income Engagement in partnership and network activity Work stream 6 Work stream 5 Work stream 4 Work stream 3 Work stream 2 Work stream 1 Outcome/ Impact

Evaluation Framework – Programme & Work Stream Level Wider BCE Impacts Type of Impact Improved quality and quantity of work-based and community education Improving the quality of services offered to employers, communities, third sector groups and learners Work stream 6 Work stream 5 Work stream 4 Work stream 3 Work stream 2 Work stream 1 Outcome/ Impact

Evaluation Framework: Project Level TYPE OF IMPACT OUTCOME/IMPACT Project-level evaluation: focus on BENEFITS Measurement against specific project aims/objectives Improved quality and quantity of work-based and community education Improving the quality of services offered to employers, community, third sector groups and learners Wider BCE Impacts Take-up of BCE products Increased BCE income Engagement in partnership and network activity Institutional BCE Activities Collaboration between institutions Role of IT in institutional strategies Institutional joined-up thinking Impact on institutional strategies Institutional strategies Awareness of JISC Impact on JISC strategy/activity JISC-wide coherence Impact on JISC Reach and impact (spatial, type of institution) Contribution to policy development (local, regional, national level) To be assessed on project-by-project basis, using Evaluation Framework as guide To feed into overall strategic evaluation of JISC BCE High-level strategy and policy impact

BCE Evaluation When? Evaluation should be ongoing process … starting as early as possible … and built into programme and project planning Early activity should be primarily formative … with focused activity from April/May until Oct/Nov 2009 … informing project delivery on ongoing basis Regular feedback/review throughout timescale of programme … at least every 6 months through workshops or other appropriate mechanisms Final summative phase in 2011 … with focus on outcomes, impacts, attribution, value for money etc

Evaluation should be ongoing process

… starting as early as possible

… and built into programme and project planning

Early activity should be primarily formative

… with focused activity from April/May until Oct/Nov 2009

… informing project delivery on ongoing basis

Regular feedback/review throughout timescale of programme

… at least every 6 months through workshops or other appropriate mechanisms

Final summative phase in 2011

… with focus on outcomes, impacts, attribution, value for money etc

BCE Evaluation Who and How? Overall strategic evaluation should be undertaken by an independent, external body … acting as ‘ critical friend ’; providing ‘supportive challenge’ … working within agreed Evaluation Framework Programme, workstream and projects should all have evaluation built into ongoing activities as key priority … using external resources if/when appropriate … with active and ongoing involvement of key stakeholders e.g. government agencies, funding councils, BCE organisations, employer and community bodies, innovation bodies etc.

Overall strategic evaluation should be undertaken by an independent, external body

… acting as ‘ critical friend ’; providing ‘supportive challenge’

… working within agreed Evaluation Framework

Programme, workstream and projects should all have evaluation built into ongoing activities as key priority

… using external resources if/when appropriate

… with active and ongoing involvement of key stakeholders e.g. government agencies, funding councils, BCE organisations, employer and community bodies, innovation bodies etc.

WM Enterprise Bradford Office The Design Exchange 34 Peckover Street Little Germany Bradford BD1 5BD t: 01274 743600 e: [email_address] w: www.wm-enterprise.co.uk  Birmingham  London  Merseyside  Bradford  Surrey