Learning analytics and the learning and teaching journey | Prof Deborah West (Flinders University) & Bill Searle (CDU) | TLCANZ17
•Download as PPTX, PDF•
1 like•175 views
Much work has been done across the sector in relation to learning analytics including the implementation of Analytics for Learn as well as Pyramid and SQL reporting. This work has provided us with data around learning and teaching interactions at various levels and in different contexts. From this data reports are generated that can be used in a variety of ways including to address issues of retention, assist with student success, support teaching practice and facilitate curriculum improvement . However, many academics are not quite sure of what is available, what it can be used for or the timing around usage. This can present a range of challenges including the under-utilisation of reports that are available, inappropriate use of reports or a sense that reports are not very useful. One way that we are tackling these challenges at Charles Darwin University it to conceptualise the reports within the framework of the learning and teaching journey. This includes a variety of perspectives from the student journey to the curriculum lifecycle. This also provides the opportunity to consider the relevance of reports to different learning and teaching contexts and approaches. This session will present our framework highlighting recommended time frames and applications for various reports as well as drawing attention to both the benefits and limitations of the approach.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Learning analytics and the learning and teaching journey | Prof Deborah West (Flinders University) & Bill Searle (CDU) | TLCANZ17
Similar to Learning analytics and the learning and teaching journey | Prof Deborah West (Flinders University) & Bill Searle (CDU) | TLCANZ17 (20)
More from Blackboard APAC
More from Blackboard APAC (20)
Recently uploaded
Recently uploaded (20)
Learning analytics and the learning and teaching journey | Prof Deborah West (Flinders University) & Bill Searle (CDU) | TLCANZ17
- 1. Learning analytics and the learning and teaching journey Professor Deborah West Pro Vice-Chancellor (Learning and Teaching Innovation) Flinders University Formerly, Director Office of Learning & Teaching CDU Bill Searle Manager, Learning Technologies CDU www.cdu.edu.au @CDUni
- 2. 2 This is a story of a complex university, with limited resources, trying to bring about radical change by leveraging the emerging power of analytics. 1. About Charles Darwin University 2. The Analytics Project 3. Current Usage of Analytics 4. Linking Analytics to the learning and teaching journey Presentation Outline
- 3. About Charles Darwin University • 24,000 students (HE and VET) • Multi-campus; multi-state; multi- modal • Bb Learn, Bb Collaborate, Bb Mobile, Bb Analytics, BbSS… • 75% HE students wholly online • 2nd fastest growing university in Australia • Top 2 for graduate employment & starting salaries
- 4. About Charles Darwin University – radical change • Need to improve retention • Need to better empower academic staff • Need to boost student & staff presence and community • Need to engage students all the time – not just when staff and peers are available • And thereby improve student learning outcomes
- 5. 5 The project was semi- structured as there were no pre- determined outcomes. Bb off-the-shelf reports provided an easy start. But user buy-in was the key! Project Sponsor Project Team – academic and technical Launch – initial pitch and brainstorming Consultancy – 744hrs, onsite consultations and training, report development Implementation – exploratory, no initial obligations Analytics Project
- 6. 6 Investment Consultancy Costs In-house Costs $ Time Prediction: Analytics investment is not going to stabilise or reduce in the near future. It’s going to become a big deal.
- 7. 7 • Small university with limited resources. • Bb offers quick points on the board. • Ease of use & maintenance. • Professional development & support. • But CDU is developing in-house data warehouse and BI system - because we believe the intelligence is too limited until it draws from the maximum digital footprint possible. Why Bb A4L?
- 8. 8 Journey Timeline Kick-Off Project Team Contract Stakeholders Infrastructure No fixed agenda Business Intelligence On-site visits Report Customisation First Pyramid Reports - incl. School Dashboards First SQL Reports Strategy Adoption Conversations Is Analytics being used? What do academics want? What is possible? Case Studies Customisation Infrastructure Bespoke reports Support & Training First Reports “Live” OLT Research Project 14-15 Application Use Cases Further Customisation VET Reports Upgrade Bb Community Site IRU-MRUN Project 16-17 Next Steps Build Praxis Community Student Dashboards (IRU) Actions & Interventions Multiple Datasets Professional Development 2013 2014 2015 2016 2017 2018
- 9. 9 Current Usage of Analytics
- 11. 11 Customisation and Bespoke Reports Grade Performance Time & Date Who’s in my Class? Counts
- 12. 12 Suite of Reports Bb Learn Analytics • Student Snapshot • Student at a Glance • Unit at a Glance • Activity & Grade Scatter Plot • Activity Matrix Report • VET Site at a Glance School & Faculty Reports (via SQL) • Grade Centre Exception • Log in Exception • Submission Details School Dashboard & Scorecards (Pyramid) • Overview School Metrics • Activity across Semester • Items in Units • Demographics • Student activity by Program
- 13. 13 Report Accesses over past 12 months 0 500 1000 1500 2000 Student Snapshot Student At A Glance Unit At A Glance Activity Matrix Activity & Grade Scatter Plot Source: Analytics On Analytics SSRS Report 1124 Unit Offerings
- 14. 14 Survey of Staff Rating of Report Utility, with Comments 0 1 2 3 4 5 Unit At A Glance Student At A Glance CDU Institution A Institution B 0 5 10 15 20 25 Usefulness / Purpose Design of visualisation Data Related Unit At A Glance Student At A Glance Source: D West et al, IRU/MRUN Analytics research project, in progress (n=48)
- 15. 15 Usage of Analytics Recent project/focus groups suggest: Many academics are not aware of what is available to them… Under-utilisation of reports (some reports used less, pockets of academics who don’t know about it / can’t see the relevance We asked ourselves? Did we overestimate the knowledge/interest in using analytics? How can we encourage use?
- 16. L&T Review Plan Aligning Analytics Reports to the Curriculum Lifecycle 1. Student Snapshot (overview of the class) 2. Student at a Glance (accesses & interactions) 3. Activity & Grade Scatterplot (student performance) 4. Unit at a Glance (unit design analysis) 5. Pyramid Reports – School (unit comparisons) 1 2 3 4 5 6 6. Pyramid Reports – School (unit preparation)
- 17. L&T Review Plan Aligning Analytics Reports to the Curriculum Lifecycle 1. Prior to teaching starts & 1st 2 week of teaching. Set up retention rules & conditions 2. During teaching cycle. Monitor activity, engagement and assessment. 3. At the end of teaching. Grading, moderation and student disputes 4. Review of site design. analysis of tool usage, utilities and items 5. Analysis of site compared with the School. Unit comparisons, assessment, items & tools 1 2 3 4 5 6 6. Site preparation. How has the site changed over time?
- 18. Aligning Analytics Reports to the Learning & Teaching Cycle 1. Who is in my class? Previous cohort performance? Student Snapshot Report & School Dashboard 2. Monitor accesses, engagement, activity & early assessment tasks Student At A Glance & Activity Matrix 3. Who is at risk? Exceptions & submissions details Grade Scatter Plots & Exception reports Student At A Glance & Activity Matrix 4. On track? – Monitor activity & progress Unit At A Glance & Student At A Glance Activity Matrix & Grade Scatter Plots Exception reports 5. Monitor assessment activities Exception reports & Grade performance 1 2 3 4 5 6 Weeks 1- 3 Weeks 3-6 Weeks 6-9 Weeks 9- 12 Weeks 12- 15 Week 16 6. Grading & Moderation Student At A Glance & Activity Matrix
- 19. Aligning Analytics with other Lifecycles 1 Curriculum Lifecycle Student Lifecycle Staff Lifecycle L&T Review Plan Potential Onboarding WorkPromotion Exit AlumnusPotential Admissions Study Senior Study Graduation / Exit Alumnus
- 20. Acknowledgements and Questions 1 Acknowledgements Questions? Slides in this presentation have been re-purposed from a presentation at Bb World 2017 by Professor Martin Carroll & Bill Searle. IRU / MRUN Project 2017 “The Use of Learning Analytics to Support Improvements in Teaching Practice” Any questions?
- 21. Thank You 1
Editor's Notes
- Welcome and Introductions
- Today we are going to introduce you to our University – Charles Darwin University We are going to provide an overview of our Analytics Project Tell you about the use of analytics at CDU and provide some insight into its perceived usefulness And to close we are going to provide a framework for linking Analytics to the learning and teaching journey
- We have 24,000 students enrolments and are a dual sectorUniversity, (offering studies in Higher Education & Vocational Education known as VET). We are multi state (operating in dispersed locations) and multi modal supporting External and Internal students). We have a Unique context. Our main campus is located in the Top End of Northern Australia in Darwin with campuses and centers dispersed across the Northern Territory and Australia. We license a suite of Blackboard Products to support online learning and teaching. Bb Learn Bb Collaborate Bb Mobile Suite Analytics for Learn Managed Hosted (Sydney data Centre) With support services provided by Blackboard Student Services We have a large digital footprint with 75% of students studying wholly online. CDU is the 2nd fastest growing university in Australia And we rank in the top 2 Universities for graduate employment and starting salaries in Australia.
- We wanted to inspire radical change at the University. Need to improve retention (common among us all!) Provide staff with the tools and utilities to make key decisions and empower then to do so! Have a desire to boost the student and staff presence – Analytics provides insight to achieve this. Student engagement is critical and not just when staff and peers are available Student outcomes and success is at the forefront of our objectives….. Analytics presented opportunities to achieve this…
- Project was designed using a project management methodology Project Sponsorship and ownership sat with the PVC Education & Student Success Project team involved representatives from key stakeholder groups including academic & technical staff University-wide Retention Project was operating in parallel to LA Project & was used & was drawn on to feed into the LA Project Contracted Bb Consulting Services to help drive the implementation through onsite consultations, awareness and training for staff, technical set up and report development…. Launch provided opportunities to pitch LA to the broader academic community, brainstorm and provide feedback No obligation to use, was not mandated, but use was encouraged.
- Consultancy investment costs re-occur over time and are subject to several considerations; Extent of customisation Upgrades or changes to technology Evolution of adoption and use Changes to data sources Changes to organization structure University Investment Starts off small but gradually increases over time to plateau But as the chart shows can change over time as adoption grows – Can go one way or the other?
- Why Bb A4L? Small university with limited resources Limited recruitment pool with high staff turnover Bb offered quick wins and could get us there in the least amount of time Bb brought expertise Ease of use - Integrated with LMS – support and maintenance of the technology and platform Professional development, training and support to the academic & professional staff
- Started our journey in 2013 with no fixed agenda. But paid attention to stakeholder engagement & consultation Technical infrastructure and installation 2014 Went live with Bb Learn integrated report suite Scoped requirements for be-spoke reports and customization (E.G. Swapped out one report and replaced with commissioned Student Snapshot Report) Focus on staff development support and training CDU led -OLT National Research Project “Lets talk learning analytics for student retention” 2015 Further customization to reflect institutional context Launched a suite of BI reports using Pyramid Analytics Be-spoke School dashboards and scorecards SQL Reporting services reporting available to support staff In 2016 The focus was the applying LA reports to learning & teaching to build use cases Commissioned a be-spoke report for Vocational Education & Training Further customizations to reflect context Technical upgrade of the platform – new report design (Ultra look and feel) Established Bb Community Site IRU-MRUN Project 2017 The focus this year has been on the development of a LA strategy to: Broaden adoption Establish policy and processes – data stewardship Socialise the conversation about LA Determine and understand the use and usefulness We asked ourselves What do academics want? And what are the opportunities? The vision for the next 12 months will be around: Build Praxis Community – community of practice and research output Student Dashboards (IRU Project) How can we best support actions and interventions Incorporate multiple data-sets Re-new professional development and promotion
- Pause – Break Slide
- Analytics Community Site Built using the Bb Community option for Bb Learn – so sits in context with the LMS Provides a range of resources, guides, help files. Includes descriptions of each report, possible use case and data dictionary Is the gateway for academics to gain access to Pyramid Analytics School Dashboards
- Customisation & Bespoke Reports Customization focus: Grade Performance by – Enrolment Type (INTERNAL / EXTERNAL) Course Access Time spent Interactions Time and Date What items have been accessed? What is the date of First Access and Last Access And how long have they spent on the item? Who is in My Class Student Snapshot report Available to academic before commencement of teaching to gain insight into the class Highlights key student demographics E.G. Male / Female / International / Domestic / Age Range / Non-English Speaking Background GPA % of program completion Course and Program attributes E.G. Degree Program Basis of Admission Load for the semester Counts A measurement of the number of items in the course and how often that item is accessed by a student.
- Suite of Reports Bb Learn Analytics: Student Snapshot Report Student at a Glance Unit at a Glance Grade Scatter Plots Grade and activity Matrix VET Site at a Glance SQL Reports Rolled up to a school faculty level Offer drill through to unit / course level Exception reports Log In Grade Performance Submissions & Posts Pyramid Analytics School Dashboards and Scorecards Over of the Semester compared to previous offering % Of item accessed School Item use by Content / Tools and Assessment Discussion Board Summaries (usage and character counts) Access patterns by: Time Bands / Days of the week Interactions and engagement Grade Performance Time spent Student demographics by School Student retention (grade performance & engagement) Degree / Program level report to gain insight into all course enrolments for the semester
- Report Accesses over the last 12 months: So is Learning Analytics used? Like most clients we wanted to know if LA was being used by the academic community? We have a report available via SQL Reporting Services that draw's on data retained in access logs. This has been visualized to offer insight into the use of the Bb Learn Integrated Reports. Over the last 12 months (July 2016 –July 2017) the Student Snapshot Report is most commonly accessed. This report is used before the commencement of teaching to gain an understanding of the class. This is followed by the Student at a Glance Report. To our surprise the runs of Student at a Glance have increased over the last 12 months, this report compares an individual student to the class and highlights accesses, interactions & engagement, time spent and submissions and posts. It also includes Bb Learn Grade Centre for the student in comparison to the class. Staff have reported this to be most useful in terms of classroom analytics than the Unit at a Glance Report.
- Survey of Staff rating of report utility, with comments In June this year we concluded a project with the IRU (Innovative Research Universities) a consortium of Australian Universities and the Malaysian Research University Network a consortium of Malaysian Universities exploring "The use of Learning Analytics to support improvements in teaching practice" . The preliminary findings form focus group discussions where we asked academic to rank Student as a Glance and Unit a Glance Reports: The first chart Report shows the mean scores on a scale of 1-5 with 1 being not useful to 5 being very useful compared across 3 universities. Student at a Glance was ranked higher by all participants The second chart shows the number of comments made against each report related to Learning and Teaching Style Data Analysis Training And usefulness/purpose. Staff reported that: Student At a Glance was most useful and fit for purpose Did not require much support or training as it was self evident While Unit At a Glance was seen as useful several comments were made about the comparisons used in the data analysis.
- Could reference previous LA project findings – not much has changed Enter aligning to the curriculum lifecycle and L&T lifecycle – other cycles?
- To encourage the broader adoption of learning analytics across the academic community we aim to align analytics with the Curriculum Life Cycle Following the cycle of Learn & Teach / Review and Plan it was evident that specific reports provide insight at specific time. Preparation for teaching – Overview of the class – Student Snapshot Once teaching commences – accesses and interactions – Student at a Glance Monitoring student performance and progress (grading & moderation– Activity & Grade scatter plot Review of unit design and analysis – Unit at a Glance Plan - Comparison across the school – Pyramid Reports What tools are used? Student performance, interaction & engagement Plan – Unit preparation – Pyramid Reports Degree / Program analysis Correlate to final grade Compare against accesses / time / interaction & engagement metrics Understand impact across a school (Degree / Program)
- Following the cycle of Learn & Teach / Review and Plan it was evident that specific reports not only provide insight but can be used to support the Curriculum Lifecycle The slide drill into the Curriculum Lifecyle Learning & Teaching Prior to teaching starts – Set up retention rules and conditions drawing on insight from: LA from previous offering and the Student Snapshot Report During the Teaching Cycle - Monitor engagement, activity, accesses and assessment using LA reports At the end of the teaching cycle – Draw on LA to inform key decisions to support grading moderation, student disputes and grade boundaries In the review process use LA to analyse tools, items, content, assessment to understand use during teaching and inform site design. Draw on reports from Pyramid Analytics that offer insight across several units, school or program level And finally in planning stage use reports from Pyramid Analytics that provide longitudinal data about how the site has changed over time.
- To encourage the broader adoption of learning analytics across the academic community we aim to align analytics with the Learning & Teaching Life Cycle Following the teaching cycle in 3 week blocks and thinking about key semester dates: First 2 weeks for enrolments Census date for withdrawing Assessment/key unit dates (generally in the core teaching period) End of semester (grading & moderation)
- There is other work to do to align Learning Analytics with the broader Student Lifecycle and the Staff Lifecycle.
- Slides in this presentation have been re-purposed from a presentation at Bb World 2017 by Professor Martin Carroll & Bill Searle. IRU / MRUN Project 2017 “The Use of Learning Analytics to Support Improvements in Teaching Practice” Q&A