Presenters: roughly Nuala Intro/Myths/Backgound/Meet the Data, then Damien Data slice to Students feel the VLE is a genuine learning aid, then Angelica Myths/Summary/Wrap up Claire McAv & Theresa also for questions
VLE's are core infrastructure. Allowing for costs of support, licenses, staff, helpdesk, major investment on VLE's in Ireland. Higher Education and now also used in secondary schools. Educational technologists talk about new technologies – VLE is seen as a common platform/baseline technology
Like a lot of things, it all started in Cork. NAIRTL funding Rob Cosgrave. If conference was in Cork he would be here but passport is out of date!! But not much use having single institution data. Much more useful to be able to compare different institutions, and everyone had the same problem and questions.
People are involved because the information is useful in guiding their own management of the VLE, not because they wrote a grant People contribute data, and work on the analysis according to their level of interest, time, capability Many hands make light work! Sharing in the spirit of collegeality and fair usage. Currently under the umbrella of ILTA Strategy on Research.
Because we had to share data between multiple, competing institutions, there were complexities - data protection and so on, to be adhered to. Row level data had to be shared for the data to be of use, but no student identifiers, emails etc were to be shared. Doug Belsow – social media putting social back
Why? Otherwise, institutions who do not have heavily resourced VLE, who maybe have the most to gain, will be blocked from being involved because it ‘looks bad’ We get into a whole PR / Benchmarking world and lose the benefits. Institutions who put in data can see who is who, so they know who to talk to, but publication and circulation of that information is strictly limited. One could, if industrious, figure out who is who from the authors, data (Cluedo joke – NUIM, in the library, with a webserver…) Also data protection issues – individual identifiers (Students IDs, eMails) are not pooled. Each institution remains legal custodian of it’s own data.
Here's three myths about VLE's which our data will touch on...Keep these in mind as we look at the data [Suggest keep in intro for Nuala before slice slide - Damien]
Fairly large dataset by past standards and growing all the time. Many participating institutions have run the survey 2/3 times, others only once. Image from http://www.c0d3m0nk3y.com/gallery/communication-node.html licenced for reuse. Cool!
Lots of way to cut the data, but today, where data allows, we're focusing on time. We have a number of institutions where we have two surveys, one from an early 08/09 group, and one in a later, 11/12 group. This allows us to compare like for like in two different time groups (with just over 4000 students in each group), so when I put up charts with two time bars, that the subset we're seeing. We're not going to talk, ever, about which institutions are in what data due to the confidentiality issues. [HANDOVER POINT - Nuala to Damien]
So, this year, for the first time, we're starting to get data from enough institutions on a long enough time base to talk about changes over time as well as gross patterns we discussed last time. Getting copies of lecture notes and other materials is the big usage modality. So is this VLE thing just a souped up content management system? Is that bad? Use of a VLE as a simple transmission systems solves a big problem - paper management. If it's transmitting information that can then be discussed in class instead of simply read out, that can still be pedagogically positive. Basic usage can still be a very good thing, let's not knock a system that solves a simple problem well, just because groovy constructivist things like online discussion remain relatively low (and probably much lower in reality, as this is a sample, not a census!) Usage of everything is up, but submit assigments seems to be a notable one upshifting. We think this is driven by integration of systems like turnitin to VLE's, so making it a more fluid process to submit material online.
Some respondents didn't use the VLE, and here's why - lecturers don't use it! Note the numbers here a relatively small, so everything except lecturer non use is trivial. The decrease in that over time is small, but encouraging. Qualitative data supports this - when asked what prevents them from getting the most out of their VLE 42% said it was lecturer non use or poor use that prevented them using VLEs effectively (compared with 49% in 2008) Note the scale, going up to 30% not 100%.
Digitial divide issues that were front of mind in '08 are fading away - see the change in those agreeing with the statement between the early and late groups - especially in the 50+. There will always be some who are not comfortable. Of course, comfortable here is self defined. Some people aren't comfortable using a machine they didn't hand build, others feel 'comfortable' because they can do facebook.
Talk about barriers to access - lecturer non use In context of results the satisfaction question
inconsistent and poor lecturer usage comes up as a theme in comments. Even students who say they find the VLE hard to use, often remark in comments about topics specific to poor or inconsistent lecturer usage
q9 Other Barriers have reduced over time. This shows the 'agree' and 'strongly agree' % for the statements shown. You can see conventional digital literacy issues around computer use are basically gone (note - among respondants - again with the self selecting sample - but it was self selecting in both years) bandwidth and access is muchimproved,usability, reliability - but no big change in the help and support area, where it's only a strong half.... Qualitatively students are saying a major barrier was access e.g. systems down; password problems; broadband issues (36% of qualitative responses related to access issues; compared to 38% in 2008) Navigation was also cited by many as a barrier (up this time at 18%; compared with 10% in 2008) - navigation refers to students difficulty in finding and accessing resources Student quote 'In the second semester i took me ages to find my modules as the first semester ones are still on the home page with a two of my new modules but the rest are under 'my sites' which can get confusing' Ease of use is often around lecturer usage rather than system
The help and support issue is interesting, as increasingly people are accessing the system across a broad range of hours, with big growth in access outside of conventional office hours - a shift towards the 24 hour university. Again, system logs would be great for this kind of data - superior to survey.
The other 'pattern of access' shift in in devices. Access devices and locations have changed. Everything is up, but mobile devices is the big shift since 08. We expect to have to grow further in '13 and on, and will drop some items like internet cafe's. This is an area where we could really use system derived data on accessing devices - but noting that system won't see offline access, like people mailing docs to a kindle to read them offline.
rising...higher agreement on positive lecture great positive improvement...but why? Moving from early majority to late majority Could be just older lecturers retiring off. Could be upskilling over time, helped along by VLE's getting easier to use, and our wonderful professional development programmes.
q9c [Damien - Rob, some postiive student quotes if you want: I think blackboard is a fantastic resource It informs you of what you need to cover in the modules and keeps you on the right track, without it one would be lost. Access to lecturers, classmates and learning materials are the most useful]
Students seem to feel the VLE is a genuine learning aid. This shows the agree+strongly agree % for some questions on that - now the year comparison isn't so important here, you can ignore that as the shifts are quite small. But look, three quarters of students say it helps to clarify what is covered in class. Close to 70% say it helps them to learn. Almost half feel it helps them understand how well they are doing. This is good news - it tells us that students, or at least the ones who care to tell us, like this thing and feel it is of benefit. That said, 40% agreed that it mainly repeats what is in class, but that isn't so awful either.
q10a, 10b, 10f Changes over time not so singificant here, but a substantial group finds the VLE makes it easier to contact their classmates, even in the facebook age. Small groups find it helps them contact their lecturers. We can be happy with that - if these people might otherwise be lurkers or it empowers them to engage with the lecturer, super. A high positive response isn't always the right answer.
mythbuster - excuse number one Lecturers cite declining attendance as the a common reason not to engage with VLE's. Students, at least those who responded, said it didn't matter. And even if it did, so what? If you can cover the transmission stuff in the VLE, maybe you can do something else with the lecture (the flip!) So 62% disagreed or strongly disagreed that getting the notes made them less likely to go to lectures, vs 21% ayes. q10e
Here's three myths about VLE's which our data will touch on...
1.No. You're only using it for content distribution anyway, and there's no evidence of any VLE specific trends in usage. 2.Students disagree. 3.Students disagree. They say it adds to what is covered in class. Image – disney.com
Added value cannot be underestimated Mission critical – there would be uproar if removed Assessment submission – turnitin – promoting academic integrity
Review and keep running the student survey - review every year. Small tweaks needed, but keep to a minimum to ensure continuity. As we move on, we find things that seemed like good questions in 2008 aren't anymore, and assumptions implicit in the questions, which we didn't notice at the time, become obvious and need to be unpacked (for example, the full time, module ased course assumption) National Strategy – evidence of student engagement We designed a staff survey in 08/09 but never got in going. Time to revisit this over the summer and add the staff perspective Survey data is very limited - poor responses rates, self selecting samples and so on. One way to address that is to supplement it with system derived data. We would need to derive common metrics which can be drawn from different VLE's in a consistent and comparable way. That could be a big job.
Finally, our project is open, and we always welcome new participants. Join us, we don't bite. Please feel free to approach any of us about coming into the workgroup. We are very informal, don't bite etc.
Uptake and Usage of Virtual Learning Environments: Findings from a longitudinal multi institutional student usage survey
Uptake and Usage of Virtual Learning Environments: Findings from a longitudinal multi institutional student usage survey Robert Cosgrave, Eamon Costello, Tom Farrelly, Nuala Harding, Theresa Logan- Phelan, Claire McAvinia, Fiona ORiordan, Damien Raftery, Angelica RisquezEdTech 2012, May 31st-June 1st, NUI Maynooth
Virtual Learning Environments are now core infrastructure in most tertiary institutions. But we dont know as much as we should about how they are used, and for what...
Higher Education Times University of Leitrim has best VLE! President Gloats! Researchers sacked! No use of findings for PRAll published results to be anonymous by Institution. Institutions letter coded A,B,C…
Three myths of the VLE:"The choice of VLE will influence how I teach""Using a VLE will reduce attendance""If everything happens online, the face to face teaching will be redundant."
Meet the Data• 15,385 Individual Student Responses• 12 Institutions• 22 Survey Instances• Data collection since 2008• 78 questions (including subquestions)
How to slice it?• Blackboard vs. Moodle• Demographics• Disability• Time
What do you use the VLE for?100%90%80%70%60%50%40%30%20%10% 0% Keep track of Get copies of Get other Online Online quizzes Submit class times lecture notes course discussions assignments material about the course % 2008/09 2011/12
If you do not use [VLE], why not?30%25%20%15%10%5%0% My Lecturers don’t The information on I can’t access it I find it difficult to I don’t have use it it isn’t useful use access to the web % 2008/09 2011/12
I am comfortable with using computers90%80%70%60%50%40%30%20%10%0% 16-23 24-35 36-50 50+ 16-23 24-35 36-50 50+ 2009/09 2011/12 Strongly agree Agree Neutral Disagree Strongly disagree
What is the most important thing you wouldLecturer use to seebig issue like is a improved More use of Audio/Multimedia/ PodcastsBetter Usage by Lecturers More Reliable Easier to Use
When asked what prevents them getting mostof their VLE..."Answer is simple...The system as usual is perfect. What is holding us back is the lecturers...I feel each lecturer should be trained more""I think that if lecturers dont use Moodle effectively (uploading class notes, including research resources, case studies, discussions, etc.), Moodle wouldnt be a useful or helpful resource for students""All lecturers appear to have a different method of using [VLE] and this should be standard"
Digital Literacy, Usability, Support100%90%80%70%60%50%40%30%20%10% 0% I am I have I think systems The VLE is The VLE is I can get comfortable adequate like the VLE easy to use reliable adequate help with using access (ie are helpful and support to computers computers, use the VLE bandwidth) to the web outside college % 2008/09 2011/12
"I like the freedom of having access to info day or night." 60% 50% 40% 30% 20% 10% 0% 07:00 to 09:00 09:00 to 12:00 12:00 to 14:00 14:00 to 18:00 18:00 to 22:00 22:00 to 07:00 % 2008/09 2011/12
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% H om e W or kp la ce O pe n Ac ce ss La b s D ep a rtm en ta lL ab s2008/09 La % pt op s In on te r ne Ca m2011/12 tC pu af s és ou ts i de ca m pu s From where do you access the VLE M ob ile De vi c e
Things are getting better My lecturers make good use of the VLE100%80%60%40%20% 0% 2008/09 2011/12 Disagree/Strongly Disagree Agree/Strongly Agree
Students overwhelmingly like VLEs Systems like VLEs are helpful100%80%60%40%20% 0% 2008/09 2011/12 Disagree/Strongly Disagree Agree/Strongly Agree
90%80%70%60%50%"I find it makes note-taking less hard andmeans I can concentrate more in the40%lecture…"30%20%10% 0% It mainly repeats It adds to what is It helps to clarify Using the VLE Using the VLE what is covered in covered in class what has been makes it easier for helps me class covered in class me to learn understand how well I am doing % 2008/09 2011/12
Getting notes on [VLE] makes me less likely to go to lectures50%40%30%20%10%0% Strongly Agree Agree Neutral Disagree Strongly Disagree
"I believe extra notes online aid the learning of students and not deter them [students] from coming to class""Sometimes lecturers dont use it to its full potential…some are reluctant to put lecture notes up in the flawed attempt to make people come to class but if people dont want to be there they just wont come either way"
Three myths of the VLE?"The choice of VLE will influence how I teach"No. Youre only using it for content distribution anyway, and theres no evidence of any VLE specific trends in usage."Using a VLE will reduce attendance"Students disagree."If everything happens online, the face to face teaching will be redundant."Students disagree. They say it adds to what is covered in class.
Three myths of the VLE?"The choice of VLE willinfluence how I teach""Using a VLE willreduce attendance“"If everything happensonline, the face to faceteaching will beredundant."
So, how are we doing?Our VLEs are doing OK!They solve basic problems wellThey are becoming more widely usedWe need to help lecturers get off the bottom rung where they need to
Next Steps• Keep running the student survey• Complement with a staff survey• Consider system derived metrics and learning analytics type data• Integrate with National Strategy regarding student engagement
Longitudinal Studies never end...New institutions welcome. Contact any of the authors to get involved. Get involved anytime. Plenty of work to go around!
Dissemination Cosgrave, Robert ; McAvinia, Claire ;Risquez, Angelica ;Logan-Phelan, Theresa (2008): Uptake and usage of virtual learning environments: findings from a multi institutional student usage survey. Second Annual Conference of the National Academy for Integration of Research, Teaching and Learning. WIT, 14-15th November Cosgrave, Robert ; McAvinia, Claire ;Risquez, Angelica ;Logan-Phelan, Theresa; Farelly, Tom; Harding, Nuala; Cooper, Rosemary; Vaughan, Noreen; Palmer, Marion (2009): Usage and uptake of virtual learning environments and technology assisted learning tools: Findings from a multi-institutional, multi year comparative study. EdTech 2009, May NCI Cosgrave et al, (2011) Usage and uptake of Virtual Learning Environments and Technology Assisted Learning Tools: Findings from a multi-institutional, multi-year, comparative study, The All Ireland Journal of Teaching and Learning in Higher Education (AISHEJ) http://journals.sfu.ca/aishe/index.php/aishe-j/index Publication pending EDIN publication (June 2013) Emerging Issues III – From Capacity Building to Sustainability