Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next

Share

NCSEHE webinar: Indigenous Perspectives on Evaluation in Indigenous Higher Education

Webinar hosted by James Smith and Kim Robertson puts a spotlight on data sovereignty and the importance of listening to Indigenous perspectives on evaluation in Indigenous higher education.

  • Be the first to like this

NCSEHE webinar: Indigenous Perspectives on Evaluation in Indigenous Higher Education

  1. 1. CRICOS Provider Code 00301J TOWARDS DATA SOVEREIGNTY: National Centre for Student Equity in Higher Education (NCSEHE) 6 April 2018 A NATIONAL CONVERSATION ABOUT STRENGTHENING EVALUATION IN INDIGENOUS HIGHER EDUCATION CONTEXTS IN AUSTRALIA
  2. 2. Building Legacy and Capacity workshop series Setting the scene The logic behind the concept: • NCSEHE Board: strategic project to build legacy beyond the current funding period and leverage the NCSEHE funded research. • Deepening some of the insights generated during the 10 Conversations at the NCSEHE 2016 National Forum. • Developing a format that enables productive conversations about complex issues. • 2017 workshops focused on career education in low SES and regional/remote schools as well as supporting regional and remote students in higher education. The objectives of the workshops are to: • Define a collective knowledge base informed by research and practice. • Engage in strategic and action planning to guide institutional practice and future research. • Develop evidence-informed policy advice.
  3. 3. Six high-level questions frames the group discussion: 1. What do we know about the nature of the problem? 2. What does success look like? 3. What do we know from current practice and research: What works? What doesn’t work? Why and why not? 4. What principles underpin good practice and why? 5. How could the Australian Government, State Governments and Universities, better support effective approaches? 6. What are the gaps in knowledge to promote positive change? Building Legacy and Capacity workshop series Structure of the workshop
  4. 4. Expert Workshops: • 10-12 subject matter experts approaching the topic from difference perspectives: researchers, equity practitioners, policy makers and community partners. • Advance a national conversation at the intersection of equity research, practice and policy for the benefit of the sector. Dissemination of insights generated during the workshop: • Webinars. • Publication: extended version of pre-reading document, including Good Practice Guide and recommendations for policy and research. • Professional illustration to complement text. • Presentations at the EPHEA Conference and to the Department of Education and Training. Building Legacy and Capacity workshop series Approach: Expert workshops and multi-modal discussion
  5. 5. Peter Anderson Queensland University of Technology and WIRA Jason Brailey RMIT Nathan Cassidy Universities Australia Bronwyn Fredericks Central Queensland University Kathryn Gilbey University of Southern Queensland Cheryl Godwell Charles Darwin University and NATSIHEC Kim Grey Department of Prime Minister and Cabinet Leanne Holt Macquarie University and NATSIHEC Maria Raciti University of the Sunshine Coast Kim Robertson Charles Darwin University (Workshop Co-Facilitator) James Smith Menzies School of Health Research (Workshop Co-Facilitator) Karen Treloar TEQSA Sue Trinidad NCSEHE Nadine Zacharias NCSEHE (Workshop Series Co-ordinator) Towards Data Sovereignty: A national conversation about strengthening evaluation in Indigenous higher education contexts in Australia Workshop 3 participants
  6. 6. Acknowledgement of Country We would like to acknowledge the Larrakia people, upon whose land we present from today. We also acknowledge the lands from which participants join us today. We pay our respects to Aboriginal Elders past, present and emerging from these lands, and we extend this acknowledgement to all other Aboriginal and Torres Strait Islander people here with us today.
  7. 7. A SNAPSHOT OF WHAT WE KNOW ABOUT EVALUATION IN INDIGENOUS HIGHER EDUCATION
  8. 8. What do we know about evaluation in Indigenous higher education? While considerable data was available through departmental program-based reporting to monitor progress, there was not always sufficient evidence to assess the overall success or otherwise of specific programs. In some cases, there were no independent evaluations of programs for the Panel to draw on. (Behrendt et al 2012, p154)
  9. 9. Why is this important? There are unique considerations in the Indigenous HE space: • On Stony Ground — emphasised importance of Indigenous governance and participation in Australian universities (Moreton-Robinson et al 2011). • Behrendt Review (2012) — recommended the development of an Indigenous HE monitoring and evaluation framework. • HEPPP 2014 NPP Grant — ‘building an evidence base about Indigenous pathways and transitions into HE’. • Aboriginal and Torres Strait Islander Higher Education Advisory Council (2016) — recommended development of an Indigenous HE Performance Framework. • Release of the Universities Australia Indigenous Strategy 2017-2020.
  10. 10. What else do we know? Key points of interest: • High level of scrutiny of evaluation in Indigenous affairs contexts. • Indigenous Advancement Strategy evaluation framework developed. • Existing models — e.g. Aboriginal and Torres Strait Islander Health Performance Framework? • Fragmentation of Indigenous policies and programs in higher ed at national level. • Equity vs Indigenous. • Indigenous leadership, standpoints, knowledges and cultural standards are important. • Emergence of data sovereignty discussions nationally and globally.
  11. 11. Key points of interest: • Collecting baseline data from all Australian universities in 2018. • Strategy is aspirational. • KPIs are the link to executive attention. • Indigenous workforce issues are urgent! Generation about to retire: Indigenous expertise will be lost. • Document is the thorn in the side of the exec to get attention and keep the agenda going. • Indigenous people don’t just want to sit in Indigenous studies any more. Want to be across the institution. • You can’t collapse reporting to quantitative measures, but needs qual reporting as well. • Resources a challenge: Indigenous areas are asking for their fair share (i.e. not just Indigenous supplementary $). Reporting against the Universities Australia Indigenous Strategy 2017-2020
  12. 12. 1. Growing Indigenous leadership. 2. Addressing white privilege and power. 3. Valuing Indigenous knowledges and prioritising Indigenous epistemologies. 4. Incentivising cultural competence. 5. Embracing political challenges as opportunities. 6. Recognising sovereign rights. 7. Increasing funding and resources. 8. Leading innovative policy development, implementation and reform. 9. Investing in strategy development. 10. Investing in cultural transformation, change and quality improvement. 11. Improving Indigenous student outcomes. 12. Promoting cultural standards and accreditation. 13. Reframing curricula to explicitly incorporate Indigenous knowledges and practices. 14. Investing in an Indigenous workforce. Enablers of, and drivers for, strengthening evaluation 2017 NCSEHE Equity Fellowship
  13. 13. Yarning about evaluation • Drivers/enablers not unique to evaluation — but also common to other areas of university business. • Data sovereignty: Indigenous control and decision-making at all stages. • Executive decision making: Indigenous people not always at the table. Indigenous PVCs often not directly involved in important decisions. • Acknowledge difference between ‘equity’ and ‘sovereign rights’. • Cultural experts should guide evaluation work. There is an opportunity to build the evaluation capacity/skills of the Indigenous HE workforce. • Evaluation often perceived as the ‘poor cousin’ — program/intervention planning and implementation given priority. • There are very complex lives behind the numbers. Stories and other forms of qualitative evidence can assist. Context is important!!! • Combining different information (e.g. ISSP + UA) increases the capacity to develop more detailed institutional snapshots. What is the problem?
  14. 14. Yarning about evaluation What is the problem? • Segregation of ‘black business’ — often not seen as core work. Whole of University approach is required. • We need evaluation tools that measure commitment, not just compliance. • What’s the end game? What will be the outcome of the data collection? • Do we need a repeat of On Stony Ground? Could be used to see what has changed? What has worked? What hasn’t? • Cross-cutting evaluation: how do we get different evaluations to inform each other? • Equity and Indigenous. • Different education sectors (e.g. early years, schools and universities). • Other sectors (e.g. health). • How do we evaluate the contribution that Indigenous people are making to the mainstream HE agenda?
  15. 15. DEFINING SUCCESS AND GOOD PRACTICE PRINCIPLES
  16. 16. • ‘Success’ is subjective — trying to define success often generates more questions than answers. • Indigenous epistemologies and ontologies need to be valued. These should be reflected through Indigenous methodologies, methods and in the ‘evaluation terminology’. • Proximity to influence is important. Indigenous people need to be involved in deciding what gets evaluated, how and why. Indigenous leadership should underpin institutional evaluation. • Evaluation design is important. This should start early with the end goal in mind. Expected impacts and outcomes can be mapped out over time. • Evaluation should be forward looking and inform changes in practice. • Better longitudinal data would be useful to frame future strategy development. Defining success Meaningful evaluations of Indigenous higher education
  17. 17. • Collecting data: we need to collect data, and build data systems, that correspond to the identified needs and aspirations of Indigenous students. • Using data: we need clear progress markers that serve a clear purpose at institutional and national levels. • Indigenous student voices are privileged in program design and evaluation (i.e. student- centred approach). • Summative evaluations should not be used punitively by institutions (i.e. to cut programs). Rather they should inform quality improvement. • Tensions between compliance reporting and impact/outcome evaluation are resolved by, for example, placing a greater focus on program efficacy and effectiveness. • Benchmarking against mainstream indicators is not always useful because it can perpetuate a deficit discourse. Indigenous specific benchmarks would be a better alternative. • There should be integrated funding models and reporting requirements across institutions and Government. Defining success Meaningful evaluations of Indigenous higher education
  18. 18. 1. Recognise sovereign rights (who speaks for whom?). 2. Demonstrate and celebrate Indigenous expertise, diversity, excellence and difference. 3. Commit to systems and continuous quality improvement. 4. Foster Indigenous leadership within and outside of the sector. 5. Clearly define the purpose of evaluation (Indigenous self-actualisation). 6. Recognise cross-cutting nature of Indigenous evaluation. 7. Focus on use and impact. 8. Build and design evaluation expertise among Indigenous and non-Indigenous HE staff. 9. Adopt a forward focus. 10. Ensure evaluation methodologies are ‘fit-for-purpose’ (adaptive and acknowledging diverse goals of Indigenous HE). 11. ‘Do no (more) harm’ (evaluation not always a safe space; be careful what you ask; resistance against power sharing with Indigenous HE stakeholders is real). 12. Strive for innovation that builds on Indigenous knowledges/practices. Good practice principles
  19. 19. Recommendations for future practice 1. Create a structure of responsibility and accountability that supports a whole-of- university approach. 2. Leverage the current policy shift from access/recruitment to retention/completion to change the balance between funding, the institution and students (and their supporters). 3. Put Indigenous consciousness and praxis at the centre of decision making to facilitate agency within the institution. 4. Leverage expertise of Indigenous staff across the sector, rather than tie them to an institution. 5. Hold a strategically focused Indigenous higher education conference. 6. Universities could consider making a ‘contribution’ to NATSIHEC to build capacity, disseminate findings, advance major projects, and provide regular communication.
  20. 20. Recommendations for policy 1. Pay greater attention to recommendations from past reports (i.e. development of a performance, monitoring and evaluation framework/strategy). 2. Bring together Indigenous higher education policy making, programming and funding into the same Australian Government Department to increase cohesion. 3. Enhance cross-portfolio communication and direction, including across the silos of the Indigenous Advancement Strategy (IAS). 4. Generate actionable insights from annual reports to government: review, synthesise, feed back to the sector and all key stakeholders, incl. TEQSA. 5. Consider a legislated commitment to Indigenous higher education in university legislation as is the practice in Victoria. The founding Acts of Victorian universities include the following statement: “The objects of the University include: (i) to use its expertise and resources to involve Aboriginal and Torres Strait Islander people of Australia in its teaching, learning, research and advancement of knowledge activities and thereby contribute to — (ii) realising Aboriginal and Torres Strait Islander aspirations; and (iii) the safeguarding of the ancient and rich Aboriginal and Torres Strait Islander cultural heritage”. Government
  21. 21. Recommendations for policy 1. Higher Education institutions to take Indigenous workforce issues much more seriously. Urgent action required! 2. Ensure Indigenous people are involved in decision-making at all levels of Institutions 1. Continued consultation and partnership development between UA and NATSIHEC to input into measures of Indigenous evaluation 2. NATSIHEC to provide guidance to TEQSA on what the expectations are to support Indigenous evaluation (particularly in relation to national standards and accreditation): conversation to commence under the current MOU 3. Review the ways in which TEQSA chooses ‘experts’: how does it recognise Indigeneity and Indigenous expertise? Other Players – NATSIHEC/TEQSA/UA Institutions
  22. 22. Questions & Answers Please raise your hand in the control panel
  23. 23. Q1. Are there any examples of Indigenous methods/methodologies in evaluation? Do these conflict with any requirements under the governance of universities as public institutions? • Lots of texts about Indigenous methods/methodologies - although primarily relating to Indigenous research not necessarily evaluation (e.g. Linda Tuhiwai Smith — NZ; Maggie Kovach — Canada). • Some emerging work relating to developmental evaluation (e.g. Michael Quinn Patton — US; Nan Wehipeihana — NZ) and realist evaluation (e.g. Gill Westhorp and Emma Wiliams — Aus) applied to Indigenous affairs contexts (but seldom applied in higher education). • Qualitative methods such as yarning, storytelling, use of narratives and metaphors are considered important Indigenous methods in research and evaluation. They are currently under-utilised in Indigenous higher education. Watch-out for the release of Professor James Smith’s Equity Fellowship Report (it raises various aspects of this topic). • Quantitative data is also important, but best combined with qualitative methods to provide a greater level of context. • No perceived conflicts with governance requirements of universities as public institutions. • Keep an eye out on data governance/sovereignty discussions happening nationally and globally.
  24. 24. • Absolutely!!! ...but also important to consider local context. • There are research projects across Australia (both institutional and national in focus) examining the Indigenous student experience in different ways, and at different junctures. See for example: • OLT project being led by Prof Denise Wood at CQU in conjunction with JCU, UniSA, UoN and CDU — report to be released after June 2018. • Book and journal issue on pathways into higher education for remote students (perhaps focused more on outreach programs for prospective students) — https://www.springer.com/gp/book/9789811040610; and http://www.cdu.edu.au/sites/default/files/the-northern- institute/cdu_ni_learning_communities_journal_2015_17_read-online.pdf. • NCSEHE project led by Prof James Smith examining Indigenous VET to HE pathways — https://www.ncsehe.edu.au/identifying-strategies-for-improving-vet-to-he-transitions-for- indigenous-learners/. • NCSEHE project led by Dr Jack Frawley on Indigenous self-efficacy research — https://www.ncsehe.edu.au/mediareleases/motivations-for-success-strengthening-self-efficacy-to- improve-indigenous-student-outcomes/. • NCSEHE project being led by Ms Fiona Shalley examining Indigenous student completions in two regional universities — https://www.ncsehe.edu.au/2017-research-grants-program-projects/. • There is evidence of Indigenous students being successfully involved in student survey design and implementation in Canada — e.g. Western University, University of Saskatchewan. Q2. Is there value in a sector-wide approach to capturing Indigenous student voice for the purpose of benchmarking Indigenous student experience (as opposed to local university level data collection)?
  25. 25. • Continue to collaborate with Indigenous colleagues and peak Indigenous bodies to evaluate this work. • NATSIHEC is leading some work relating to STEM…the report should be released shortly. • Please publish outcomes of this work (there is very little quality evaluation evidence in this space…despite increased attention being paid to Indigenous students and STEM – see for example https://www.pmc.gov.au/news-centre/indigenous-affairs/funding-applications- invited-deliver-indigenous-girls-stem-academy. • There is no set rule about how to measure this, but think creatively and well done on what you have achieved so far!!! Q3. [We] are part of a grant which brings together some Indigenous people and some mathematicians and statisticians, using Indigenous pedagogies in mathematics and statistics classes. Some of those classes have few or no indigenous students enrolled. Do you have advice on how to evaluate the effectiveness of this strategy, given so few Indigenous students directly affected (though many more might be eventually, since many of the maths students are teaching students). We do acknowledge the importance of also affecting non-Indigenous students. We would love advice how to measure, and what to measure.
  26. 26. • Increase the Indigenous workforce (this needs urgent attention). This needs to occur across the university not just in ‘Indigenous offices/centres/areas’. • Increase Indigenous leadership and governance across all areas of the university. • Strategically use the skills and expertise of Indigenous academics and the growing number of HDR students. • Involve Indigenous students as part of the strategy development. • Watch out for the upcoming NATSIHEC report which has a section on WOU approaches. • Evaluate how these approaches are working (i.e. good quality process and impact evaluations). • Encourage the sector to share stories about what is working and why (note: some universities appear to be reluctant to share evidence outside of their own institution). Q4. What were the practical strategies for creating a 'whole of university' approach within tertiary institutions. National Centre for Indigenous Studies at ANU would be highly supportive of the idea to hold a national conference on these issues.
  27. 27. • Absolutely!!! • There are some good examples about internet being used in creative ways to engage Indigenous students/communities. • Regional universities have developed solid expertise in this area. • Check-out the Broadband for the Bush Alliance — http://broadbandforthebush.com.au/. • There hasn’t been a great deal of thinking about how to embed internet into evaluation processes/methods in Indigenous higher education contexts (usually its been approached from a program implementation perspective). Innovation is required in this regard! Q5. Is there a role for the Internet, as a way to get the university out beyond the campus? Have in mind Philip Townsend's work on Mobile Devices for Aboriginal Pre-service Teachers http://blog.highereducationwhisperer.com/2018/04/mobile-devices- for-aboriginal-and.html
  28. 28. Professor James Smith & Ms Kim Robertson C/- Office of the Pro Vice Chancellor — Indigenous Leadership Charles Darwin University james.smith@menzies.edu.au kim.robertson@cdu.edu.au
  29. 29. Website: ncsehe.edu.au Email: ncsehe@curtin.edu.au Twitter: @NCSEHE Google+: NcseheEduAu Facebook: National Centre for Student Equity in Higher Education THANK YOU

Webinar hosted by James Smith and Kim Robertson puts a spotlight on data sovereignty and the importance of listening to Indigenous perspectives on evaluation in Indigenous higher education.

Views

Total views

358

On Slideshare

0

From embeds

0

Number of embeds

235

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×