Students' perspectives on learning analytics dashboards
•Download as PPTX, PDF•
3 likes•481 views
Presented at ALT Sept 2017
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Students' perspectives on learning analytics dashboards
Similar to Students' perspectives on learning analytics dashboards (20)
Recently uploaded
Recently uploaded (20)
Students' perspectives on learning analytics dashboards
- 1. Students’ perspectives on learning analytic dashboards Dr Liz Bennett and Dr Sue Folley University of Huddersfield ALT-C Sept 2017
- 2. Rationale For The Study 2 Learning Analytics is a topical area with claims that it can improve student engagement, attainment and retention – but few HEIs are currently using student- facing data/dashboards The research in this area tends to focus on the technical aspects collection and analysis of data, rather than students’ response/perspective. This research aims to address this gap
- 3. Aims of the Study 3 To identify which elements of dashboards design were most valued by students; To identify students’ learning responses to seeing data presented about themselves via a dashboard; To identify the potential and limitations of using dashboards with undergraduate students; To identify questions raised by their use for future research in the area. This presentation will focus on the first aim and suggest ways we are approaching the second aim
- 5. Sample 5 Invitations to a full cohort of 180 students on one final year module Academic Range: 1st to 116th in a recent assignment and on-track data predicting from 51% (low 2:2) to 74% (first) as final degree outcome Of the 10 volunteers, half had done better in the currently assignment than their average and half had done worse Self-selecting All female
- 9. The Students’ Response - LIKED 9
- 10. The Students’ Response – didn’t like 10
- 11. The Students’ Response - mixed 11
- 12. Motivation 12 I feel like I have more motivation to study because I can see that, you are currently on track for, and I am currently on track for the first class, which motivates me a lot. (Claire) I feel like I’m not so left behind seeing this a 116 out of 170. I think that’s pretty good. (Kirsten) I kind of wish we had seen it from like that presented to us since year one. (Marcia)
- 13. Analysis 13 Sutton (2012) has developed a model of feedback literacy around three interrelated dimension: knowing, being and acting, and suggests that acquiring feedback literacy is mediated by the students’ perceptions of their university teachers’ identities. Feedback Literacy Acting Knowing Being
- 14. Analysis - Knowing 14 Notion of personal best in relation to the data (ipsative feedback) Understanding where their performance sits in the cohort (norm referenced performance) Understanding their performance relative to criteria for good students (criteria referenced performance) Understanding their attendance profile There is no point in seeing an average of everybody’s marks, only because it’s doesn’t really matter what other people get because it’s only your marks that matter (Ingrid) I like the flags because it gives you, it’s another visual aid to see, okay well if I’m green I’m good (Rebecca) It was quite useful for me because I know my position in the class…Knowing your position in a class is always a nice thing because you know where you are, what you need to, do you need to move up or you just need to, are you keep, are you on the right track? Are you following other classmates? (Claire )
- 15. Analysis - Being 15 Investment in their identity in academic work and developing learner’s self-confidence in being a [Law/Chemistry/English/Music] undergraduate, deserving to be there, and feeling they have the ability to achieve the degree. Feedback has the ability to change people’s self confidence in a positive or negative way Recognising feedback as self-development I could’ve done better now, knowing that fourteen people did better than me (Justine) I’ve done better as the years have gone on and it’s really shown how much university has helped me progress with my academic writing skills (Rebecca) If I want to push it so that I get a good degree that’s my responsibility to do so (Jayne)
- 16. Analysis - Doing 16 What sort of behaviours do we want? as soon as I saw it I decided I’m taking a month off work to just get on with my dissertation [Marcia] even if I were on track for a third I think that this probably would motivate me if it had some pointers as to what I could be doing to get a higher grade. [Sarah] I would definitely authorise my absences in future [Jasmin]
- 17. Conclusion 17 We focussed on dashboard design and the way that we are starting to understand the data. Limitations of the sample. Students liked dashboards in general and found it motivating and particularly the on-track scores; Mixed response to the positional (ie norm referenced) information; There is a different sort of feedback literacy that applies to dashboards related to students’ understandings of data and performance; Response to dashboard, like all feedback, is personal and depends on a student’s attitudes, beliefs and dispositions towards learning.
Editor's Notes
- Context and rationale Dashboards are the graphical interface that manipulate and present data about students’ learning behaviours (attendance, visits to the library, which books they take out, their attainment etc). Although only a few UK HEIs have developed a dashboard for students, most other UK HEIs have an aspiration to develop their use (Sclater 2014). NTU is one that has implemented a student facing dashboard – based on student engagement ie using data from attendance, library, vle, but it doesn’t include assessment data. Research into use of dashboards is in its early stages with some evidence of their positive impact on student engagement leading to improvements in student motivation, retention, satisfaction and attainment (Duval, Verbert, Klerkx, Govaerts, & Santos 2013; HEA 2014; Sclater 2014; UCISA 2015). However much of the focus research is on the technical aspects of collecting and analysing data (Papamitsiou & Economides 2014) with little understood about how students respond to seeing data presented in this form (Duval et al 2013) and this is the gap that this research aims to address. The paper presents the emergent findings from a small scale study funded by Society for Research in Higher Education. The paper will focus on the design of dashboards and how students understood and interpreted the data that they presented.
- SRHE scoping award so seen as important and timely; it is about raising questions for future study too.
- Methods The study used a small scale study. Two methods were used to gathering data, focus groups and semi- structured interviews: Pilot focus group with self-selecting sample asked them to comment on the format of the dashboard design elements e.g. pie chart, comparative data, progress, word cloud. We did this in November 16 and presented findings at SRHE. Seven students participated. A second focus group was held with 10 students who were a self-selecting sample, who received £10 gift voucher to thank them for participating. Interviews were held with 10 students who were given their own data in dashboard format and explored their response to seeing their data.
- Sample We drew the sample from a large undergraduate Research Methods module with 180 students in the cohort. We attempted to get a broader cross section of the cohort by targeting invitations across the cohort, but got a self-selecting sample probably representing the most engaged. The academic range of the group ranged from 1st to 116th for the marks for one of the assignments on the module. Of the 10 students interviewed, six had done better than average and four worse than average, so the sample had some academic variation in it. So the group included a range from the across the cohort. The on-track score (showing what class of degree the students were on track to get ranged from 51% (low 2:2) to 74% (1st) which is a range of 23%. Five of the group were doing better in the assignment than their overall average, and five of the group did worse in this assignment than their overall average. This suggests that we had the potential to uncover a range of emotional responses to the assignment data, not just being pleased that this assignment was bringing their average mark up or just disappointment that it was lowering their mark. Hence the sample had significant variation in the academic range of the students and in where this mark sat on their overall profile. However they were self-selecting representing some of the more engaged students. All students were female. Broadly four open interview questions: What was your feelings when you saw the dashboard elements containing your actual data? What action would you take as a result of seeing your data? Whose responsibility do you think it is to act on the results of your data?
- They all reported liking (either like or like very much) the attendance data presented in pie chart format (element figure 1a) They all reported liking (either like or like very much) the summary of the module results presented in tabular form (element figure 1j) They all reported liking (either like or like very much) the simple on track slider showing their current predicted degree classification. (element figure 1e)
- They mostly did not like the VLE data which they did not consider particularly useful. (element figure 1i) The elements that were least liked were the VLE activity, word cloud, emoticons
- There was a range of responses the Personal Tutor Meeting log. (element figure 1c) in particular they were concerned that this section needed to have a clear purpose: There was a variety of responses to the positional data, with some finding it motivating and some (even a high achiever) finding it challenging (element figure 1d – distribution of grades):
- There was evidence that students found receiving data about their progress and motivational some students at both the students at the top and lower end of the spectrum of grades. However in addition there was some evidence that dashboards may well be experienced as demotivating by those at the bottom of the cohort. Ingrid (168th of the 178 students) says “the saddest one [dashboard element] that will make me feel is the core summary overall because looking back on grades that you’ve previously had is, you can’t really change them any more so you can’t really do anything”. (Ingrid) [Liz’s comment emotional response here....potential to feel depressed and saddened by dashboard] Claire 25th = out of 178 Kirsten 116th out of is it 170 or 178??? Marcia 53=
- So how do we understand this data theoretically? Using Sutton’s model of feedback that anayses the way students interpret feedback in 3 dimensions (knowing, being and acting) helps to understand dashboard’s potential: It provides students with ways of understanding their marks in 3 different ways (ipsative, norm and criteria referenced) Processing the individual marks and their significance to overall performance (on track for) (ipsative feedback) Understanding where their performance sits in the cohort (norm referenced performance) Understanding their performance relative to criteria for good students (criteria referenced performance) Understanding their attendance profile (they got very focussed on the accuracy of this)
- Investment of identity in academic work – gives a tool to help in this process Developing learner’s self-confidence in being a xxx undergraduate, deserving to be there, and feeling they have the ability to achieve the degree. Negative feedback can be damaging to some people Feedback has the ability to change people’s self confidence in a positive or negative way Being a student is to be in a state of anxiety (Barnett, 2007) – though this view is contested by others. Learners must be willing to change their identities through feedback. Recognising feedback as self-development For some learners developing their mode of educational being constitutes a challenging and anxiety-provoking experience. Sutton’s notion of feedback as polysemic Middle quote needs explaining Justine feels worse knowing she is 14th out of 180! R Fourteen other people have still done better than me, yeah. I And whereas you thought, seventy-five%, I’ve really, really topped it, I’ve maxed out here. R It’s great, yeah, yeah. I And it’s taken away a bit from that feeling of elation perhaps. Am I putting words in your mouth? R No, no, that’s exactly how I feel about it, that I could’ve done better now, knowing that fourteen people did better than me. Last quote – see how motivating it is for Jayne – hence the power of dashboards to affect students’ emotions
- What sort of behaviours do we want? Attendacne monitoring focus? Works with other forms of feedback
- ALT Conclusion – just focussed on dashboard design and indicated the way that we are starting to understand the data using Sutton. Conclusion is that students liked dashboards in general and found it motivating and particularly the attendance and the on track scores. That Sutton’s feedback literacy can be applied effectively but there is a new type of dashboard literacy related to interpreting the way that the data is presented eg understanding what the flags mean and how they relate to your ambitions (criteria referenced); seeing this about your personal best (ipsative) and understanding that it exposes you to what others in the cohort are doing (norm referencing) how students interpret dashboards data is very personal, so even positive feedback can be interpreted in a negative way (and vice versa). This is in contrast to how a lot of the papers about LA are articulated (see LAK 16) Not covering in this talk: ethics, detailed discussion of norm referenced data, power of words to shape students’ understandings (on track), dispositions/ideographic nature of the data, MacFarlane, critiques of Sclater and Mullan, dashboard literacy. Relationships with academics. Future direction.