Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Surveying administrative innovations in tertiary education: experience from Australia and NZ


Published on

This presentation was given by Anthony Arundel at the Public Conference “Innovation in education : What has changed in the classroom in the past decade?”.

Measuring innovation in education and understanding how it works is essential to improve the quality of the education sector. Monitoring systematically how pedagogical practices evolve would considerably increase the international education knowledge base. We need to examine whether, and how, practices are changing within classrooms and educational organisations and how students use learning resources. We should know much more about how teachers change their professional development practices, how schools change their ways to relate to parents, and, more generally, to what extent change and innovation are linked to better educational outcomes. This would help policy makers to better target interventions and resources, and get quick feedback on whether reforms do change educational practices as expected. This would enable us to better understand the role of innovation in education.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Surveying administrative innovations in tertiary education: experience from Australia and NZ

  1. 1. Anthony Arundel (Dominique Bowen Butchart & Sarah Gatenby-Clark) Surveying administrative innovations in tertiary education: experience from Australia and NZ OECD 2019 Study funded by the LH Martin Institute, University of Melbourne
  2. 2. Online and mailed survey OECD 2019 • 39 Australian universities • 6 New Zealand universities
  3. 3. Reasons for this survey • Experiment with measuring innovations developed by administrative staff in universities – Builds on recent research on measuring innovation in the public sector • Collect data of value for benchmarking (across universities and over time) – Methods that managers use to innovate – Outcomes and obstacles OECD 2019
  4. 4. Comparison with OECD “Measuring Innovation in Education” OECD AUS-NZ Survey Compare innovation in education to other sectors Main interest is comparing innovation in universities to other public sector organizations Identify use of specified educational practices over time Innovations defined by type only (service, etc), asks about a most important innovation Construct metrics to evaluate effect of educational practices on outcomes Focuses on the process for administrative innovations: how innovations are developed, plus outcomes and obstacles Evaluate drivers of innovation in educational practices (not yet) Drivers plus the effect of a ‘inclusive innovation culture’ Develop an appropriate methodology (not yet) Part of ongoing work on measuring innovation in the public sector
  5. 5. OECD 2019
  6. 6. Survey methods • Questions cognitively tested in 13 face-to-face interviews with Senior Managers at seven universities. • Questionnaire sent to 1,516 senior managers in 10 functional areas (Library services, governance, IT services, etc.) – Not sent to senior executive or middle level managers • 573 respondents (37.8% response rate) • Responses from all targeted universities (45 in total) • Questions refer to the respondent’s “area of responsibility” • Reference period of two years. OECD 2019
  7. 7. + Innovation culture Revisions - Risk aversion OECD 2019
  8. 8. OECD 2019 Function of the unit General information Innovation environment Innovation methods Use of design thinking methods Two year reference period for all questions
  9. 9. OECD 2019 Most important innovation Abandoned or under-performing innovation Obstacles to innovation Most important innovation = greatest expected impacts on the respondent’s area of responsibility, university, students, or staff
  10. 10. Percent respondents reporting innovations by type 28% 48% 50% 71% 91% 91% 0% 20% 40% 60% 80% 100% Marke ng / communica on Organisa onal Services / products Processes Any innova on Any innova on (Australian public sector) OECD 2019
  11. 11. Examples of the most important innovations • New form of therapy for university students. • Enrichment program for high-achieving high school students. • Customized website to provide career development strategies to international students. • Online suite of resources to assist students in managing social media. • Mobile app to allow students to manage their courses, lectures and tutorials from a smartphone. OECD 2019
  12. 12. OECD 2019 Source of the idea for the most important innovation, percent of respondents 3 6 6 24 30 31 0 5 10 15 20 25 30 35 Academics or students at your university Consultants, businesses or other universi es Other Staff that report to the respondent or other managers Respondent VC, Council or Senior Execu ve
  13. 13. ‘Percent respondents giving a ‘high importance’ rating to innovation drivers OECD 2019
  14. 14. OECD 2019 Methods to support innovation
  15. 15. Widespread use of ‘good practice’ innovation methods • 52% of respondent’s staff involved in brainstorming meetings to develop ideas for innovations. • 61% of respondents delegate responsibility for an innovation to an individual. • 73% of respondents report collaborating on their most important innovation. • Majority of respondents use design-thinking methods. OECD 2019
  16. 16. Percent respondents reporting use of design-thinking methods Most of these methods involve ‘co-creation’ with end users of the innovation
  17. 17. OECD 2019 Does your workplace have an inclusive innovation culture?
  18. 18. OECD 2019 49% of respondents agree that their “Senior Executive support a positive innovation culture that includes all staff.” (inclusive innovation culture)
  19. 19. OECD 2019 Percent respondents by university agreeing that their Senior Executive supports an inclusive innovation culture
  20. 20. Share of staff involved in three innovation support methods by agreement with an inclusive innovation culture OECD 2019 41% 39% 46% 51% 52% 58% 0% 10% 20% 30% 40% 50% 60% Working groups to develop or implement an innova on Training for how to use an innova on Mee ngs to brainstorm ideas for innova ons Agree that there is an inclusive innova on culture Disagree
  21. 21. Correlation between an inclusive innovation culture & the use of design thinking methods Design-thinking method Odds ratio p Conduct project user or focus groups 1.9 .029 Surveys of potential users 2.2 .004 “Ease-of-use” surveys 2.7 .001 Pilot tests of an innovation 1.9 .033 Post-implementation studies to identify problems 2.1 .012 OECD 2019 Results from logistic regressions that control for innovation type, reasons for innovating, restructuring, number of staff, and function Comparison between ‘agree’ with an inclusive culture versus ‘disagree. Evidence for a dose-response effect for all methods.
  22. 22. OECD 2019 Outcomes (can be positive or negative) 1. University’s brand or reputation 2. Simpler or faster processes 3. Increase in revenue 4. Employee working conditions 5. Student experience 6. Teaching and learning 7. Research 8. Reduction in costs (Results for positive benefits problematic – lack of time?)
  23. 23. Correlation between an inclusive innovation culture and “major positive effects” from the most important innovation Positive effect on: Odds ratio p Simpler or faster processes 1.9 .08 Employee working conditions - ns Student experience 2.2 .099 Reduction in costs - ns University’s brand or reputation .38 .017 OECD 2019 Results from logistic regressions that control for use of design thinking methods, number of staff, source of the idea, type of innovation. High number of ‘too early to tell’ and ‘not relevant responses’
  24. 24. OECD 2019 The use of design-thinking methods and collaboration strongly increase the probability that the most important innovation is a novel process or service
  25. 25. OECD 2019 What doesn’t work? • Abandoned or underperforming innovations • Presence of resource obstacles • Negative outcomes for the most important innovation
  26. 26. OECD 2019 The absence of an inclusive innovation culture doubles the probability of an abandoned or underperforming innovation A budget cut increases the probability of abandonment or underperformance by 60%.
  27. 27. The odds of reporting each of three obstacles that are measures of a lack of resources (skills, funding and time) decreases substantially in the presence of an inclusive culture (Odds of 0.32, 0.24 & 0.12). OECD 2019
  28. 28. OECD 2019 The main factor increasing the reporting of all resource obstacles is when innovation is driven by a crisis requiring an urgent response (Odds of 2.2 to 2.8).
  29. 29. Factors correlated with one or more negative effects from the most important innovation (MII) • The absence an inclusive culture increases the odds of a negative effect from the MII by 2.5 times. • When the idea for the MII is obtained from the Senior executive versus the respondent, the odds of a negative effect is increased by 1.9 times. – (respondent better informed or tries harder?) OECD 2019
  30. 30. Conclusions • We can measure administrative innovations, innovation activities and obstacles reported by university managers. – Beneficial outputs are harder to measure (insufficient time?) – Problem of self-reported outcomes • What works: collaboration, inclusive culture, design-thinking methods, other innovation support methods, sufficient resources – Similar success factors as in the private sector and in other public sector innovation surveysOECD 2019
  31. 31. Conclusions: An inclusive innovation culture – Large impact on the use of ‘best practice’ innovation support methods such as design- thinking. – Substantially decreases the probability of an abandoned or under-performing innovation and negative effects from a most important innovation. – No effect on novel innovations. – Positive but not robust effect on a few beneficial outcomes of the most important innovation. OECD 2019
  32. 32. Further information University survey results: %20ARUNDEL%20innovation%20in%20universities.pdf Overview of measuring innovation in the public sector: Arundel A, Bloch C, Ferguson B. Advancing innovation in the public sector: aligning innovation measurement with policy goals, Research Policy, 2918, (Open access) OECD 2019