Successfully reported this slideshow.

CEMCA EdTech Notes: Learning Analytics for Open and Distance Education


Published on

Published in: Education, Technology
  • Be the first to comment

CEMCA EdTech Notes: Learning Analytics for Open and Distance Education

  1. 1. 1CEMCA EdTech Notes A topical start-up guide series on emerging topics on Educational Media and Technology
  2. 2. 2 CEMCA EdTech Notes IntroductionIntroduction L earning analytics make use of large datasets in order to improve learning and the environments in which it takes place. Students and educators can use these analytics to review work that has been done in the past, to support current study and to access recommendations for future activities. In the context of open and distance education, analytics are able to draw on information about online learning activity in order to support teachers and to help guide learners. Many of the major learning management systems (LMSs) used to support education worldwide currently have basic-level learning analytics built into them, and new tools are currently under development that will expand the use of analytics within open and distance education. Context The recent development of learning analytics is associated with several factors (Ferguson, 2012). The first of these is the increasing availability of ‘big data’ – datasets so complex that they require new approaches to management and analysis. Within education, particularly within online distance learning, Learning Management Systems (LMSs) collect and store enormous amounts of administrative, systems, personal and academic information. Institutions running these systems have an interest in finding ways of extracting value from their big sets of learning-related data. At the same time, access to devices with Internet access has grown and there has been an increase in the take-up of online learning. This offers many benefits for learners and extends the availability of expertise and resources, but also poses challenges. Teachers who lack the visual cues that are available in a face-to-face classroom may struggle to recognise when online students need more challenge, when they are confused, overwhelmed or failing to engage. Learners, particularly those working alone with open educational resources, can find themselves adrift in a sea of information. Both learners and teachers need tools that can help them to optimise opportunities for learning. In the wider context, educational institutions and regions are increasingly seeking to measure, compare and improve educational performance. Governments and local administrative bodies are looking for ways of identifying best practice in order to optimise learning and educational results at national and international levels. There is therefore a demand at national level, at local level and at the individual level of students and teachers for analytics that can not only make sense of the increasing amount of educational data but can also use it to support learning and teaching. Learning Analytics These various interest groups have made different demands on the big educational datasets that are now available. This has resulted in the development of several related areas, with overlapping interests and similar concerns. Broadly speaking, Educational Data Mining focuses on managing and making sense of data, while Academic Analytics deal with institutional and national priorities such as retention and success rates. Learning Analytics are defined by the Society for Learning Analytics Research (SoLAR) as ‘the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs’. A key element of analytics is that they are not primarily concerned with reporting activity or with theoretical insights. Instead, they focus on the provision of ‘actionable intelligence’ that can provoke or encourage practical action. In an educational setting, analytics allow learners and educators ‘to increase the degree to which our choices are based on evidence rather than myth, prejudice or anecdote’ (Cooper, 2012). A review of key learning analytics tools (Dyckhoff, Lukarov, Muslim, Chatti, & Schroeder, 2013) has Teachers who lack the visual cues that are available in a face-to-face classroom may struggle to recognise when online students need more challenge, when they are confused, overwhelmed or failing to engage
  3. 3. 3CEMCA EdTech Notes Learning Analytics in Use The range of available learning analytics tools and approaches is constantly expanding. Here, three examples are used to give a flavour of how learning analytics are currently being used. For a more detailed overview of the field, see the freely available report on analytics and infrastructure produced by JISC CETIS (Kraan & Sherlock, 2013). Course Signals at Purdue University The Signals project, developed over a decade at Purdue University, applies statistical tests to large highlighted ways in which they can be used by educators and learners: Educators can use learning analytics to • monitor the learning process • explore student data • identify problems • discover patterns • find early indicators for success, poor marks or drop-out • assess usefulness of learning materials • increase awareness, reflect and self reflect • increase understanding of learning environments • intervene, supervise, advise and assist, and • improve teaching, resources and the environment. Learners can use learning analytics to • monitor their own activities, interactions and learning process • compare their activity with that of others • increase awareness, reflect and self reflect • improve discussion participation, learning behaviour and performance • become better learners, and • learn. Figure 1: Course Signals in use at Purdue University, USA A ‘traffic signal’ status display tells students when things appear to be going well (green), when the system has detected some cause for concern (amber) and when they have been classified as at high risk (red) datasets in order to predict, while courses are in progress, which students are in danger of falling behind. A ‘traffic signal’ status display tells students when things appear to be going well (green), when the system has detected some cause for concern (amber) and when they have been classified as at high risk (red). The signal colour is associated with advice on action that the students can take to get back on track. The university reports improved retention levels when Course Signals is used. Students in experimental groups seek help earlier than those in a control group, the percentage of students awarded A and B grades rises, and the percentage awarded D and F grades falls (Pistilli & Arnold, 2012). Social Networks Adapting Pedagogical Practice A challenge within distance learning is to monitor and support activity within online discussion forums. The freely available SNAPP tool was developed to enable educators to visualise the development of relationships within these environments. SNAPP makes it clear
  4. 4. 4 CEMCA EdTech Notes Mini Case Dr Patel teaches Arts 101, an online course at a large Canadian university. Arts 101 has been designed to be a collaborative course, with the 200 students guided by tutors as they engage in discussion and build a shared understanding of course material. However, feedback from previous years suggests that some students struggle to engage in the discussion forums. In a face-to-face setting it would be relatively simple to identify and support students who are isolated and unengaged. In the standard forum view, it is difficult to do this without looking at the contributions of every student individually. Dr Patel installs the freely available SNAPP tool, developed for use with standard learning management systems. This allows her to visualise her forums as networks in which individuals are represented as nodes, linked by lines representing the connections they have made as they have responded to discussions. This view shows potential problems, allowing her to take action to support learning. One group is in a characteristic ‘wagon wheel’ formation. At the hub of this network is the teacher – her connections radiate out in all directions, but the students are not talking to anyone else. At the start of a course, this is a typical formation, but by this time students should be engaging in discussion with each other. Changing behaviour here is easy, a prompt from the tutor encourages students to engage more widely. Some isolated nodes in the main group are of more concern. These are students who have posted in the forum but have received no replies, so they appear as individuals with no connections. Clicking on these nodes provides a link to basic data. Worryingly, many of these students are receiving low grades, and are among those most in need of support. A check with the course team reveals these students are asking basic factual questions and tutors are leaving it to other learners to respond. The problem is that other learners are not responding. A two-pronged approach works here – the tutors identify and work with the isolated students, and also explain to all participants the advantages of answering questions and addressing confusion. Later in the course, the network has taken on a different form, with two large groups that are not talking to each other. Students focused on a history assignment are not engaging with others who are focused on literature. This lack of links means that the two groups are not sharing ideas and information. Again, the SNAPP visualisation allows Dr Patel to identify the problem and she is able to establish links between the two groups. when students have posted but received no reply, or when cliques are forming and dominating a forum. It shows when a teacher is taking the lead, when groups are working together, or if a teacher is working only with specific groups. Being able to see these representations in real time, rather than when the course has finished, makes it possible to take action. Where appropriate, isolated students can be supported and encouraged, groups reorganized, and educator effort refocused (Bakharia & Dawson, 2011). Figure 2: Interpreting a SNAPP network diagram of student interactions Figure 3: Comparison of threaded forum views and SNAPP visualisations of forum activity
  5. 5. 5CEMCA EdTech Notes Making Effective Use of Learning Analytics Tools are not the only key element of learning analytics. They form part of a system in which human decisions and actions are also of key importance. Without attention to context, to people and to learning design, learning analytics cannot function effectively. Context includes the problems that can arise when a large-scale learning analytics system undertakes real-time analysis of varied types of data in order to produce reports. Unless the system is confined to an LMS, data it employs are likely to be generated from systems that were not designed with a view to such integration and are unlikely to offer data consistency. In addition, while using data from different live operational systems, analytics must not limit the operational effectiveness of any of these systems. This may necessitate the development of a data warehouse where data can be brought together and processed without interfering with activity on the live system. In order to use a learning analytics system effectively, people require support. Staff will need to be able to select and interpret suitable metrics and to engage in good quality dialogue about how and why these are used. As learners and teachers are likely to have limited experience of data visualisation, this needs to be considered when Identifying and Supporting Students ‘At-Risk’ Students In distance education it is sometimes difficult to find and support students who are struggling on a course. The JISC-funded RETAIN project at the UK Open University uses online data to predict which students are at risk of dropping out. The project found that variability amongst learners makes it difficult to build the profile of an average student. Instead, the system looks for individuals who change their patterns of activity. Online behaviour of learners is usually consistent until they hit a problem. At that point, they may reduce usage because they are discouraged, or increase usage because they are searching for a solution. Analysis of their activity can be presented to tutors in a dashboard, where it is possible to drill down for more data, and to record action taken (Wolff & Zdrahal, 2012). Figure 4: Identifying at-risk students. RETAIN analytics based on engagement, submission and grade for tutor-marked assessments (TMAs)
  6. 6. 6 CEMCA EdTech Notes Challenges Although the use of student data to help with the achievement of desired learning outcomes is a valuable activity, it raises problems in terms of approaches to teaching and learning and in terms of ethics and data management. Analytics cannot be understood simply as tools to support teaching and learning, they carry with them assumptions about how learning takes place and about how it can be managed. In countries such as the UK and Australia, the use of analytics related to national school tests has been associated with state management of education and with enhanced regulation of the teaching and learning environment. There is always a danger that analytics will focus attention on the achievement of a constrained set of measurable criteria, and will distract attention from other aspects of learning. Educators need to consider and be aware of the models of teaching and learning that underpin any set of analytics, and the categories of action and interaction prioritised by those models (Griffiths, 2013). They also need to be aware of ethical issues associated with learning analytics. These include consideration of how data will be collected and stored, how it will be classified and managed, how and when it will be anonymised, and how it will be interpreted and presented. Educational institutions are likely to need a comprehensive data governance structure, drawn up with reference to local legislation, to help them to deal with these issues (Slade & Prinsloo, 2013). Procedures will be required that will discourage people from providing false data in order to game the system, and that will help educators and students challenge findings if it appears that analytic tools are producing biased results or reinforcing stereotypes. In addition, institutions should always set time aside to test that every predictive model employed is capable of generating useful and valid predictive insights in the local context. Being aware of potential problems and planning to reduce them and deal with them is key to the successful use of learning analytics. designing dashboards and developing training programmes. The emotional elements of assessment and feedback must also be taken into account, developing effective strategies for sharing data with learners in ways that are both supportive and sensitive. Learning design is the process of putting in place practices, activities, resources and tools that will allow learners to achieve specified learning outcomes in a given situation. This helps interpretation of the data that learner environments generate by linking the activities of learners with the outcomes they should be working to achieve (Lockyer, Heathcote, & Dawson, 2013). It also helps to move the analytic focus away from simplistic presentation of data relating to click rates and test results towards learning and teaching concerns – ‘Which elements are my students struggling with?’ ‘What misconceptions have they shown? ‘How can analytics help us to achieve the desired learning outcomes?’ Procedures will be requiredthatwill discourage people from providing false data in order to game the system, and that will help educators and students challenge findings if it appears that analytic tools are producingbiased results or reinforcing stereotypes
  7. 7. 7CEMCA EdTech Notes Finding Out More The Society for Learning Analytics Research (SoLAR) organises both national and international conferences, including the annual Learning Analytics and Knowledge conference (LAK); publishes the Journal of Learning Analytics; and runs an online doctoral school (SoLAR Storm) for students around the world. The CETIS Analytics Series of reports, available free of charge online, explores key issues around the strategic advantages and insights that analytics bring to the education sector. The Educause Library brings together analytics reports and case studies analytics The Future To date, a major focus of learning analytics has been on their use in LMSs, where large amounts of student data are readily available. This has proved fruitful in universities running such systems, in countries where regular and reliable Internet access is easily available to students. In countries where these conditions are not met, learning analytics are more difficult to implement (Faridhan, 2012). New developments in learning analytics are reported in the proceedings of the annual international Learning Analytics and Knowledge conference organised by SoLAR. Work is currently in progress to develop analytics that can be used on massive open online courses (MOOCs). This form of open learning has recently seen huge international engagement, and is capable of providing structured resources and discussion to tens of thousands of learners on a single course. MOOC analytics offer the potential of providing support and recommendations to learners who do not have access to a local educational institution or who only have intermittent Internet access. Other innovations currently under development are recommender systems and evidence hubs that will help learners to search online educational resources quickly and effectively. Data from handheld devices is being used to underpin apps that can support learning and can offer suggestions Summary Learning analytics make use of ‘big data’ in order to understand and develop both learning and the contexts in which it takes place. They focus on ‘actionable intelligence’ _ information, recommendations and prompts that can be used to increase student engagement and learning. This is a relatively new field, but it draws on extensive work in related areas. As a result, tools and methods are already available that can support both learners and educators. The majority of current tools are best suited to students and teachers with reliable access to an online LMS. Tools currently under development will provide more support for learners in other contexts, including those engaged in open education, those with limited access to online environments and those learning with the help of mobile phones. and prompts via mobile phones. On every continent, researchers and educators are working to develop analytics that are sensitive to local needs and context. Other innovations currentlyunder development are recommender systems and evidence hubs that will help learners to search online educational resources quicklyand effectively
  8. 8. 8 CEMCA EdTech Notes References Bakharia, A., & Dawson, S. (2011). SNAPP: A Bird’s-Eye View of Temporal Participant Interaction. Paper presented at the LAK11: 1st International Conference on Learning Analytics and Knowledge (27 February – 1 March), Banff, Canada. Cooper, A. (2012). What Is Analytics? Definition and Essential Characteristics. Bolton: JISC CETIS. Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at the LAK13: 3rd International Conference on Learning Analytics & Knowledge (8 – 12 April), Leuven, Belgium. Faridhan, Y. E. (2012). Exploring Data from Existing Systems That Are Useful in Assisting Student Learning: An Indonesian Perspective. Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning (IJTEL), 4(5/6), 304-317. Griffiths, D. (2013). The Implications of Analytics for Teaching Practice in Higher Education. Bolton: JISC CETIS. Kraan, W., & Sherlock, D. (2013). Analytics Tools and Infrastructure. Bolton: JISC CETIS. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: aligning learning analytics with learning design. American Behavioral Scientist, 57(5), doi: 10.1177/0002764213479367. Pistilli, M., & Arnold, K. (2012). Course Signals at Purdue: Using Learning Analytics To Increase Student Success. Paper presented at the LAK12: 2nd International Conference on Learning Analytics and Knowledge (30 April – 2 May), Vancouver, Canada. Slade, S., & Prinsloo, P. (2013). Learning analytics: ethical Issues and dilemmas. American Behavioral Scientist, 57(5), doi: 10.1177/0002764213479366. Wolff, A., & Zdrahal, Z. (2012). Improving retention by identifying and supporting ‘at-risk’ students, EDUCAUSE Review Online. Retrieved from and-supporting-risk-students. CEMCA EdTech Notes is a topical start-up guide series on emerging topics in the field of educational media and technology. New titles are published every year. Series Editor: Sanjaya Mishra Designer: Sabyasachi Panja Copyright © CEMCA, 2013. CEMCA EdTech Notes: Learning Analytics for Open and Distance Education, is made available under a Creative Commons Attribution 3.0 License (international): Views expressed in the CEMCA EdTech Notes are that of the author, and do not necessarily reflect the views of CEMCA/ COL. All products and services mentioned are owned by their Dr. Rebecca Ferguson is Research Fellow in the Institute of Educational Technology, The Open University, UK. She works at the intersection of complex social and technological change and cross-disciplinary research into Educational Futures. She fosters’ Open University research that is (re)conceptualising contemporary and next-generation educational practices and technologies. She can be reached at Rebecca[dot]Ferguson[at]open[dot]ac[dot]uk respective copyrights holders, and mere presentation in the publication does not mean endorsement by CEMCA/COL. CEMCA in an international organization established by the Commonwealth of Learning, Vancouver, Canada to promote the meaningful, relevant and appropriate use of ICTs to serve the educational and training needs of Commonwealth member states of Asia. CEMCA receives diplomatic privileges and immunities in India under section 3 of the United Nations (privileges and immunities) Act, 1947. Printed and published by Mr. R. Thyagarajan, Head (Administration and Finance), CEMCA, 13/14 Sarv Priya Vihar, New Delhi 110016, INDIA. Website: