http://tel.ioe.ac.uk/tel-seminars/teldi2012/TEL Digital Inclusion ConferenceJanuary 17th 2012Sheffield
Making a difference: How can we demonstrate the impact of learning technology research?
Making a difference: How canwe demonstrate the impact oflearning technology research?Professor Jane Seale
Overview• Provide an overview of the different ways that funders such as ESRC and HEFCE define and conceptualise “impact” and “users of research”;• Offer my personal interpretations, with examples, of how learning technology research might demonstrate impact;• Facilitate debate and reflection regarding how well placed current UK learning technology research is to meet the challenges of demonstrating impact.
Defining impact• Broadly speaking: – Where research has a transformative and beneficial effect on people, groups, organisations, communities – Evidence for this impact goes beyond citations in academic journals • Impact on “real-world” users- through user engagement
Separating impact of e-learning from theimpact of e-learning research E-learning has had an impact• We “know” that e- on pedagogy- it is learning can be motivational, teachers have transformative been transformed• How can we demonstrate the impact of research that reveals this transformation?
What claims for impact can the e-learning research community make?• Conceptual – Diana Laurillard (2002) Rethinking university teaching: a conversational framework • 4350 Google Scholar citations• Practical – Gilly Salmon (2003) E-moderating: The key to teaching and learning online • 2676 Google Scholar citations• Can “we” prove that government, universities, lecturers have transformed the way we teach using technologies as a result of these publications (through changes in policies, standards, practice, systems, resources etc.) ? – How??
What claims for impact can I personally make? A personal reflection (1)• Jane K Seale, Alan J Cann (2000) Reflection on-line or off-line: the role of learning technologies in encouraging students to reflect: Computers & Education, 34, 3-4, 309-320• Empirical paper: one of my papers submitted to RAE 2001 as part of Kings College submission – Comparison of two individual teaching practices• Dissemination activity: 2 conferences, Alans‟ personal web page, 2nd related journal paper• Impact ?? – CAL99: positive reaction regarding “honesty” in admitting failure – CTI Biology Virtual Conference 1997- voted best paper – My most cited paper: 74 Google citations (all academic journals and conference papers)
A personal reflection (2)• Phipps, L., Sutherland, A and Seale, J (eds). Access All Areas: disability, technology and learning. ALT/JISC/TechDis. (2002) – An edited pamphlet aimed at practitioners- review/description of accessibility law and implications for practice – Not submitted to RAE 2008 by Southampton University• Dissemination (impact)activity: freely available online through TechDis , ILT, ALT• Impact?? – 6th most cited publication in Google Scholar (mixture of academic and policy related citations) • Equality Challenge Unit (2006) Disability legislation: practical guidelines for academics.. • Welsh Assembly Government (2005) Disability Issues for Post-16 Learning Provision. • Cited in many new lecturer programme reading lists
Personal reflections (3)• LEXDIS- JISC funded project 2008• Students and academic staff as partners in the project- user engagement• Dissemination (impact) activity: – Project website, 3 journal articles (empirical) 2 newsletter articles, 1 book chapter, 5 conference papers, workshops, desk-side 1-to1• Impact?? – One whole page spread in TES – Students still contributing to the website 4 years after end of project – Winner of 2009 IMS Learning Impact Award
Which of these three projects had themost “impact”?• I personally value each of these 3 pieces of work for different reasons in relation to their impact on academia, policy, & students• Link between impact and quality potentially complex• Only the last project had anything like a concrete strategy for user engagement at the beginning of the project
1. How do funders such asESRC and HEFCE define andconceptualise “impact”?
Impact and Excellence• Both HEFCE and ESRC assert that you can‟t have impact without excellence – Excellent research underpins excellent impact
Impact is about showcasing andexploitation• HEFCE: The REF will – Showcase the success of UK research in contributing to the economy and society – Encourage more effective dissemination, application and exploitation of research
Impact is about justification ofvalue and worth:• ESRC: – “In recent years, the government has placed increasing emphasis on the need for evidence of economic and social returns from its investment in research. – By ensuring that ESRC-funded research makes the biggest possible impact on policy and practice, and improving how we measure and capture this, we are better able to support the case for research funding. – Impact helps to demonstrate that social science is important – that it is worth investing in and worth using.”
Looking forwards and lookingbackwards• ESRC : when bidding: accounting for future impact – ESRC/RCUK: applicants are asked to give an account of the anticipated future effects of their research on potential non-academic users• HEFCE REF: accounting for past impact
HEFCE (REF) definitions• For the purposes of the REF impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia• Having an effect on: – the activity, attitudes, awareness, behaviour, capacity, opportunity performance, policy, practice, process or understanding – of an audience, beneficiary, community, constituency, organisation or individuals – in any geographic location, whether locally, regionally, nationally or internationally• Impact on academic knowledge excluded• Impacts on students, teaching or other activities within submitting HEI excluded
REF impact assessment: case studies• Provision of “case studies” in which starting point is 2* paper(s) and the HEI provides a case for: – Impact of the research described in the paper(s) – What the HEI has done to facilitate impact of the research• 20% of overall assessment score• 2 case studies per 15 FTE (6 for 45+ FTE)• Jan 1st 1993- 31st December 2013
Impact Criteria (drawn from Panel C interpretation)• Reach: understood in terms of the extent and diversity of the communities, environments, individuals, organisations or any other beneficiaries that have benefitted or have been affected – Reach will not be assessed in purely geographic terms, nor in terms of absolute numbers of beneficiaries, but rather in terms of the extent to which the potential number or group of beneficiaries have been affected• Significance: The degree to which the impact has enriched, influenced, informed or changed policies, opportunities, perspectives or practices of communities, individuals or organisations• Panels B & C will assess the two criteria on a holistic basis
Type ExamplesImpacts on creativity, Informed public or political debateculture and society Improved welfare, equality, social justice, Production of cultural artefacts (films, TV)Economic, commercial, Development of new or improved products or processesorganisational impacts Developing alternative economic models Changed approach to management of resources resulting in improved deliveryImpacts on environments Changed resource management practicesHealth and welfare Influence on CPD/improved training standardsimpacts (incl. protection Improved provision or access to servicesor advocacy of rights or Improved health and welfare outcomes (must includeinterests) educational outcomes!)Impacts on practitioners Development of resources to enhance professional practiceand professional services Use of research findings to define best practice or in conduct of practice Influence on professional standards, guidance or training Practitioner debate has been informed or stimulated by research findingsImpacts on public policy, Influencing the work of NGO or commercial organisationslaw and services Policy debate stimulated by research evidence Improved public understanding of social issues/challenging conventional wisdom
What counts as evidence?• Qualitative or quantitative for Panel C• For Panel B- wherever possible quantitative should be included
Example indicators of impact• Citations – Citation in a public discussion, consultation document or judgement – Citation by journalists, broadcasters or social media – Citation by international bodies such as UNESCO, IMF – Evidence of citation in policy, regulatory, strategy, practice documents• Measures – Number of visitors, audience, participants – Measures of improved inclusion, welfare or equality – Service satisfaction measures – Outcome measures – Quantitative data relating to cost-effectiveness or organisational performance
Examples of impact indicators• Debate – Evidence of debate among practitioners leading to developments in attitudes or behaviours – Public debate in the media – Parliamentary or other democratic debate• Use – Use in scrutiny or audit processes such as Select Committees – Incorporation in training or CPD material• Reviews and documentation – Media reviews – Independent documentary evidence of influence on guidelines, legislation, regulation, policy or standards – Documented change to professional standards or behaviour
Evidencing impact• Each case study must explain how the research led to or contributed to the impact and include appropriate sources of information external to the HEI to corroborate these claims
RCUK/ESRC definitions of impact• “Research Councils UK (RCUK) defines research impact as the demonstrable contribution that excellent research makes to society and the economy.• Research impact embraces all the diverse ways that research-related skills benefit individuals, organisations and nations. These include: – fostering global economic performance, and specifically the economic competitiveness of the United Kingdom – increasing the effectiveness of public services and policy – enhancing quality of life, health and creative output.• A key aspect of this definition of research impact is that impact must be demonstrable. It is not enough just to focus on activities and outputs that promote research impact, such as staging a conference or publishing a report. – You must be able to provide evidence of research impact, for example, that it has been taken up and used by policy makers, and practitioners, has led to improvements in services or business.”
ESRC Types of research impact• Academic impact is the demonstrable contribution that excellent social and economic research makes to scientific advances, across and within disciplines, including significant advances in understanding, method, theory and application.• Economic and societal impact is the demonstrable contribution that excellent social and economic research makes to society and the economy, of benefit to individuals, organisations and nations.• The impact of social science research can be categorised as: – Instrumental: influencing the development of policy, practice or service provision, shaping legislation, altering behaviour – Conceptual: contributing to the understanding of policy issues, reframing debates – Capacity building: through technical and personal skill development.
Planning research impact • “To plan impact effectively you need to: – identify your key stakeholders, for example, other researchers; public sector; business/industry – Identify how they will benefit from your research – types of impact might include: improving social welfare/public services; influencing public policy; contributing to operational/organisational change – Identify how you will ensure they have the opportunity to benefit, for example through organising public events; conferences; interaction with the media. – For practical guidance on planning research impact, see the information on developing an impact strategy”http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-toolkit/
ESRC Impact strategy: objectives• “What are the likely outcomes of this research?• Who will benefit from this research?• How will they benefit from this research?• How can you involve potential beneficiaries in this research?• How will you know if it has made a difference?”
ESRC Impact strategy: Messages• “Clear, succinct messages that summarise your research.• Over-arching messages that can be used while it is underway, or specific messages that relate to particular parts of the project.• Using different formats – a media release – a report – a research briefing – a newspaper article – a website page.• Think in advance about stories, case studies and packages of information that will bring your project to life for key audiences.• Creating a brand: distinct identity – your project name – a strapline or one-line description of the project – a logo or visual mark, – the application of your brand across a range of materials – Ensure your brand reflects the overall values and objectives of your research and your impact plan.”
ESRC Impact strategy: targetingaudiences• “It is vital to know who your key audiences are. – Think beyond the obvious organisations and individuals.• Prioritising your audiences – Since time and money are limited, it is useful to rank each of your potential audiences and user groups according to their importance and influence relative to your strategy. – A good question to ask is: if you had half the money and half the time to spend on your strategy, which of your potential audiences would you focus on?”
ESRC guidance on tools and channels• Branding• Events• Interactive media: websites, social media• Media relations• Networking• Public affairs: influencing policy makers• Publications• Public engagement: shape research agenda, steer project, share results, raise awareness
Evaluating impactEvaluating economic and capacity building impact probably quitestraight forward. For example, for capacity building could apply thisESRC advice:• “Quantitative data– provide measures, for example of how many people attended the event and what they thought of it.• Qualitative data – seek to illuminate individual experiences and provide additional subjective context to the evaluation. These data explore the participants‟ experience in more depth than quantitative data.• Observational data – exploring how people participate in an event can be illuminating. Did they participate in all events/activities? Were some aspects more popular than others? How did people interact with your website or display? A clear idea of what you are looking for is important when structuring observational data.• But, is this measuring user engagement and can this be legitimately used as a proxy for impact”
Evaluating impact• Evaluating impact of conceptual and related impacts harder and ESRC is addressing this• It is commissioning case studies on methods for capturing the more diffuse types of social science impact – e.g. mapping conceptual developments (such as changes in thinking, debate, culture and direction) – „people-flow‟ impacts across the researcher/user interface•
Implications for e-learning research • If we want to defend or protect e-learning research then we have to demonstrate its public value
2. How can learningtechnology researchdemonstrate impact?
TLRP TEL as an example• TEL and impact• “The TEL programme is keenly aware of the need to demonstrate impact beyond academia. A number of projects have already begun to demonstrate how their TEL funded research could have a practical, positive impact on such areas as: – relieving teacher‟s time and paperwork constraints – addressing difficult subjects in schools in different ways – providing better opportunities for disadvantaged people to develop skills through interacting with technology – showing how technology can be used in learning environments to save time and money whilst providing improved learner outcomes.
Innovations in translation• http://tel.ioe.ac.uk/category/impact – Exhibitions – Prizes – Practitioner workshops – Media articles – Journalist informed tweets – Videos – Press releases – Breakfast conference with MP‟s
Haptel as a case in point • Clear messages about: • Cost effectiveness • Evidence that students do just as well with haptics • User engagement- clinical dentists on the team • Vision for implications for future
Haptel as a case in point • Participated in an evening of demonstrations of the latest medical technology for the general public at the Science Museum
3. Facilitate debate andreflection regarding how wellplaced current UK learningtechnology research is to meetthe challenges of demonstratingimpact.
An impact SWOT analysis for e-learning researchSWOT ExamplesStrengths We have a detailed knowledge and insight into one key user group- practitioners.Weaknesses Poor relationship with government-policy naivety Tendency to follow technological trends rather than set themOpportunities Methodologies suited to user-engagement Knowledge transferThreats The culture of the lone e-learning “champion” researching own practice The memory of the failed UKEU- the elephant in the room
Strengths• We have a detailed knowledge and insight into one key user group- practitioners – We inhabit their world because more often than not we are “one of them” – Ideally placed to evidence impact on practice and inform practitioner debates, practice, standards…(but we have to think beyond our own institutions to do this)Impacts on Development of resources to enhance professionalpractitioners and practiceprofessional services Use of research findings to define best practice or in conduct of practice Influence on professional standards, guidance or training Practitioner debate has been informed or stimulated by research findings
Weakness• Very little e-learning research(ers) speak to government – What e-learning report/papers have been cited in key government education policy documents and debate? – Web science on the other had/have the ear of government – TEL/ALT working hard to cement a good relationship with BIS on our behalf • But we have to engage too….. • To do this we have to re-conceptualise who we think our end-users are
Weakness• Tendency to follow technological trends rather than set them: – We apply the tools around us to educational contexts (e.g. web 2.0 tools) whilst there is some innovation in this (applying existing tools to new situations) but couldn‟t we be doing so much more?• Naïve and ill-prepared in our approach to commercialisation of technological developments
Opportunities: methodologies suited touser engagement• Channels of engagement- e.g. social media• Process of engagement- participatory action research, partnerships with industry
Methodologically, impact requires us to stepout of our comfort zones and make “contactwith strangers”• Brewer (2010) Impact requires the discipline to reconsider its own „strangers‟, those with whom research relationships came to appear unusual and odd […] more obviously, policy-makers and civil society groups [..]• Impact has accelerated contact with „strangers‟ and is disquieting for this reason, but the long-established tradition of action research, participatory forms of research, or research sensitive to respondents‟ sense of research fatigue, are forms that enhance people‟s participation in the design and conduct of the research precisely in order to increase its impact
Opportunities: Knowledge transfer• Can our work translate to related fields?• Examples: – Work based training- internships (current political hot potato with introduction of 9K fees) – VLE design (why shouldn‟t we shape what Blackboard does for once?) – Distance learning design and delivery for military personnel on duty
Threats: burned fingers and longmemories• UKEU and long memories – A company and website set up by HEFCE that promoted online degrees from UK universities. UKeU delivered courses over a learning environment developed by Sun Microsystems UK. – In 2004 it was announced that the project should be wound up as it was a failure having cost £50 m and recruited only 900 students – E-learning a victim of its own “hype”
Threats: Lone Ranger syndrome• Brewer (2010)• Impact disturbed the comfortable rules of the game by which research reputations and resources were established – Marks the demise of the lone researcher model of social research in the 21st century. – Promotes engagement with real world problems
Threats: Lone Ranger Syndrome• We need more than ever to develop research teams that possess the wide range of skills required to conduct research AND plan, implement and evaluate an impact plan – Researching own practice, in own institution through personal interest and enthusiasm, with limited skill-set no longer “cuts it”