LMS Focus Group Report Background UT Dallas’ current eLearning system, Blackboard Vista 8.2, is an end-of-life system andwill no longer be supported by Blackboard after 2013. With this end in mind, theuniversity is conducting a search for a new learning management system. UTD willpublish a Request for Proposals in early Spring 2011.In October and November 2010, as directed by the LMS Committee, the Office ofEducational Enhancement, assisted by SOM eLearning, conducted a series of focusgroups to solicit feedback from faculty, staff and students about the use of the currenteLearning system and about their needs to determine what features would benecessary and desired in any learning management system. Focus Groups A total of eight (8) focus groups were held - 3 for faculty users, 3 for student users, 1 onaccessibility and 1 for back-end users. See the below tables for attendance anddemographic information of attendees.At the beginning of each focus group (except for the back-end users group), a briefsurvey was administered individually to solicit quick feedback about specific tools used.A comments area was provided.See Appendix A for the survey and survey results.Guiding Questions For the focus group sessions, a set of “guiding questions” was used to direct theconversations. This semi-structured approach allowed the facilitator to guide thediscussion and to prevent the focus groups from becoming “gripe” sessions. SeeAppendix B for the Guiding Questions.Observations Interestingly, many of the features that faculty expressed a desire for in a new systemalready exist in the current eLearning. Faculty were either unaware of them or had notreceived training. Faculty emphasized that migration from eLearning to new systemshould be easy.Students generally wished that all faculty used or were required to use eLearning. “Itwould be convenient.” Not all of the students were taking full online course; thus theirfaculty tended to use eLearning for posting syllabi and lecture notes and homeworkassignments. Some students took online quizzes. Student disappointment witheLearning was more from lack of use by faculty and from stability. Students related thatother students don’t like eLearning because it often didn’t work and they need to feelconfident that the system will work.
Focus Group Results Current usage (from surveys) See Appendix A Usability • “eLearning is powerful but for a power user” (F) • “the learning curve is more work than it’s worth” (F) • navigation is easy (S) • could look more modern, more updated (S) Tools Communication Mail • Tool should be a default in all courses (F) • Need search by name and keyword (F) • If mail forwarded out, need to be able to reply back into (F) • Window doesn’t expand (F) • Connect UTD email and eLearning mail (F) (S) Discussions • Not intuitive to create a new post (F) • In graded discussions, move from student to student (not back to list) (F) • Need to reply privately to a post (F) • grading is clunky (F) • doesn’t that it opens new into separate, smaller windows (F) • needs to be easy to use (like Facebook messaging) (S) • in “show all” should still be threaded (S) • “read” should apply only to ones actually read (not all new) (S) • searchable and tagable threads (S) Calendar • link to Outlook calendar / Google calendar (F) • color code entries by course (user assigned colors) (F) Announcements • need ability to edit, delete announcements by TAs (F) • need ability to hide announcements (F) Chat • would be essential to online course (S) • chat not sufficient (moved to Skype) (S)
Collaboration Groups • useful (F) • manual creation of groups is somewhat hard (F) • nice but not crucial (S) • group contact could be expanded (S) Assessment Assignments • Easy formatting w/o launching html editor (F) • Sequential grading of assignments - student to student (no return to list) (F) • Easy upload of multiple or large files (eg. video) (F) • Somewhat difficult to submit (S) • Some files (MS Access) hard to download (S) Quizzes • Improve graphic ability (embedding, formats) (F) • Change settings while quiz is open (F) • Equation editor isn’t good; integrate “MathType” (F) • System not “dynamic”, though works well for MC tests (S) Grading forms • Ability to add comments with each item (F) Learning Modules • More visual way (F) Course Management • Desired - central location from which to push email, announcements, exams to all sections (F) File manager • Should work more like a drop box (F) • Faster download/uploading of files (F) Syllabus • Integrate with Coursebook (F) • Hypertext to calendar (F) Icons • More choices (F) Edit pages • Make default “See all” (F) MyFiles • Nice, files always there (S) Gradebook Columns • Rearranging is difficult (F) • Need per column settings (F) Formulas • Formula editing is difficult (F) General • Add categories (F) • Have default grading schemes (F) • Make it more like Excel (F)
Integrated Tools • Library, library reserves (F) (S) • Video server, YouTube, iTunesU, etc (F) • Google Docs (F) • Turnitin - use Grademark (online grading) (F) • MuchLearning (calculus) (S) • Skype (S) Additional tools or functionality • Online editing of documents, regardless of format (F) • Better text editor (F) • Notification system (send reminders after missed exam, send new announcement, new discussion, etc, RSS feeds) (F) • Push notification for updates (S) • Ability to embed videos easily (F) • Wikis (F) • LaTex (S) • Smart Tutor (S) • Web conferencing (S) • Resume function on quizzes (S) • Consistent look/structure between courses (S) • Picture roster (S) Accessiblity • Faculty and students confused this with navigation and stability Overall Improvements Login • too many clicks (F) (S) • should be more like shared drives (accessible when log into computer on campus) (F) Mobile • need mobile app (F) • push notifications, RSS feeds (F) (S) Public • need ability to make parts public (resources, etc) (F) (S) Migration • easy porting to new system (F) Listings • make default - latest entry on top (for anything) (F) MyCourses • elearning could be more integrated with other UTD systems • ability to open multiple windows • media library that allows students to post/upload (S) • site map (S)
Results from Back-end Users Focus Group Delivered integration w/ PS • functionality delivered - 50% (MH) • API hooks (SH) • Tight integration (NTH) Structure / hierarchy • Detail level - product / service - how much control do we have (MH) • Conditional course existence (MH) • Roles - admin, support - different levels (MH) Authentication • LDAP (MH) - Shibboleth (SH) • Point of entry control - single sign on (NTH) Hardware • No specific hardware as long as below met • Enterprise level (MH) Database • No flat architecture - Oracle/ MS SQL (MH) • Must be able to query (MH) • Published schemes (SH) External integration control • We know where our data resides (MH) • We know who has access to our data (MH) Archiving • Comprehensive backup strategy (MH) • Capability for restoring deleted content (MH) • Daily backup (SH) Application • Web-based (MH) • Support multiple browsers/OS (MH) • High availability (MH) Migration • Migration support from Vista (SH) Security • SSL - for mobile (grades) (MH) Hosting • Meet UTD/state requirements (MH) Integration Control of content • Course object (MH) • Individual object (MH) • Gradebook (MH) • Member enrollment (MH) • Term-based (MH) Support • What kind (MH) • Multiple levels (MH) • Process (MH) • SLAs (MH) Hosting • Data storage meets FERPA, UTD, state policies (MH) Concern-timeline for vendors to get through the vetting process in Information Security.
Appendix A Student Focus Group Survey What is your status? Freshman Sophomore Junior Senior Graduate StudentWhat is your major / school?How long have you been using eLearning?How comfortable are you using eLearning? Very comfortable Comfortable Neutral Uncomfortable Very uncomfortableWhich course tools in eLearning do your instructors use? • Announcements • Goals • Assessments • Gradebook o Quizzes • Grading forms (rubrics) o Surveys • Groups o Self-tests • Learning Modules • Assignments • Mail o Individual assignments • Posting syllabus o Group assignments • Web links • Calendar • Who’s Online • Chat / Whiteboard • Discussions o Graded discussions o Journals o Blogs o ThreadedWhich student tools do you use? • My Files • My Grades • My Progress • NotesPlease use the back of this survey for any other comments you’d like to add.
Student Focus Group Survey What is your status? Freshman  Sophomore  Junior  Senior  Graduate Student What department are you in? •  Arts and Humanities •  Natural Sciences and Mathematics •  School of Management •  Economic Public Policy Sciences •  Brain and Behavioral Sciences •  Engineering and Computer Sciences •  Undeclared How long have you been using eLearning? •  1 year •  3 years •  5 years How comfortable are you with eLearning (Please mark your response)  Very Comfortable  Comfortable Neutral  Uncomfortable  Very uncomfortable What tools are you currently using in eLearning (Mark all that apply)  Announcements  Assessments  Goals Quizzes  Grade Book  Surveys  Grading Forms (Rubrics)  Self‐Tests  Groups Assignments  Learning Modules Individual Assignments  Mail Group Assignments Calendar  Syllabus Chat/Whiteboard Discussions Graded discussions  Web Links Journals  Who’s Online Blogs Threaded Which student tools do you use?  My Files  My Grades  My Progress  Notes
Please use the back of this survey for any other comments you’d like to add. • I have never had a semester where every professor used eLearning. As of now, I have 2 using eLearning, 2 using their own personal websites, and out one using the library’s eReserves. For this type of software to be effective, use needs to be mandatory by professors. As of right now, it is simply an inconvenience to use the software. Currently, the math and science departments have started using computerized testing. This software, much learning, while primarily a testing/quizzing software, should have the ability to post notes/announcements/grades it would be much better for students if there was one type of software used throughout the university. I imagine it would also save quite a bit of money on licensing fees (the math department has already purchased 6000 copies of this software. It’s already very difficult to get a professor to learn one new piece of software – having one central software would greatly help its adoption. • eLearning is fairly simple to use and I like it but in all honesty eLearning doesn’t look very cool. • Library integration. I am also working at UT Southwestern Medical center (Research) Academic Information Systems. We use UPIC (PeopleSoft now Oracle) to write eLearning and we have ELM (Enterprise Learning Management). I write technical training. We are looking for solutions as well. Suggestions: Saiki, Moodle Rooms • I use eLearning as an instructor (TA). Very little use of eLearning as a student • I am currently doing an independent study, so I am not in a “class” that uses eLearning. However, I am a TA for a professor who uses it only a little (to post assignments and syllabus). I am assigned to teach next semester though, and I will be using it a lot – the class will be a “hybrid” (half on‐line, half in person) class.
Faculty Focus Group Survey What is your role in the university?  Faculty  Staff What department are you in?  Arts and Humanities  Brain and Behavioral Sciences  Natural Sciences and Mathematics  Engineering and Computer Sciences  School of Management  Interdisciplinary Studies  Economic Public Policy Sciences How long have you been using eLearning?  Never  5 years  1 year  8 years  2 years  9 years  4 years How comfortable are you with eLearning (Please mark your response)  Very Comfortable Neutral  Uncomfortable  Comfortable Very uncomfortable What tools are you currently using in eLearning (Mark all that apply)  Announcements  Calendar  File manager  Mail  Assessments   Goals  Syllabus  Quizzes Chat/Whiteboard  Grade Book  SCORM  Surveys  Discussions  Grading Forms  Templates  Self‐Tests  Graded (Rubrics)  Web Links  Assignments discussions  Groups  Who’s Online  Individual  Journals  Learning Assignments  Blogs Modules  Group  Threaded  Selective Assignments Release Please add any other comments below. • I’m interested in open source products like Sakai • It is a great resource. My use depends on the course I am teaching and the level of the course (under graduate or graduate) • I have helped in past webCT / eLearning initiatives and appreciate this chance for us to share our experiences. Please don’t hesitate to ask if I can help in anyway. I feel like the current tool we are using for courses is difficult for students and instructors to use easily and I am excited to explore other opportunities for us to improve our distance learning and enhance learning in all UTD courses. • 1. When grading assignments submitted in eLearning, it should be possible to go straight from one paper to the next without having to go back to the main screen for that
assignment. 2. When Iooking at a list of submitted assignments and sort it by student name, I find two problems a. The list is sorted by first name, not last. B. When I look at a paper and then go back to the list, the sorting reverts back to default. Resorting takes forever. I have 120 students per class submitting at least 10 papers each, in 3 courses. 3. Students simply don’t know how to use eLearning features. They don’t know how to edit a submitted assignment, for example. They don’t understand the difference between a public discussion and email. A more accessible system might cause less confusion in students with more limited computer skills. 4. Links to TurnItIn and CPR would be helpful. Integration, I mean. 5. If a TA posts an announcement, I have no access to edit or delete it. I should be able to change all announcements. Similarly, students can change the calendar for everyone, and I can’t intervene. I had a student post some of her assignments there, and I couldn’t delete them. 6. In assignments, HTML Enabler does not work for most students, which causes widespread panic. 7. The eLearning help number is not helpful. On campus support couldn’t be better, but I hear many bitter students complain about the remote help desk. • I have anywhere from 12 to 17 sections of one course. The most glaring omissions in eLearning is the lack of centralized location where I can store files, quizzes, assessments, etc. that pertain to all the sections. I would also like to send mail and post announcements from one place where I can send it to all the sections simultaneously. Likewise, I would like to post assessments in one place where they can be accessed by all sections. With all of the above, I still would like to retain the individual sections because when it comes to posting grades in Orion, I can just draw them from the individual sections in eLearning. Bunching all the sections into a mega section would not work for me because for grades I still need to retain individual sections. Also, different sections are taught by different TAs, who need to have access only to their sections. • Keep it simple! A greater number of instructors and students will use it. Basic Blackboard worked very well at the University of Arizona – Eller College (Walter is the contact if you would like to call him) • I have not found the current eLearning system to be useful so I do not use it. I would hope we could find a more effective program for the university. • I would love to see a mobile application for the system – particularly discussions and quizzes. • Keep hoping for portfolio developments
Appendix B Focus Group Guiding Questions Let’s talk about your Current usage • What is your level of expertise with eLearning? • What do you use eLearning for? • What do you like about eLearning? Let’s talk about Usability • What do you think about eLearning’s interface? Explain. • What improvements do you need? Let’s talk about Tools in a Learning Management System • What do you like about the tools you are currently using in eLearning? • What do you not like about these tools? • Which tools are essential? (need to be part of whatever system we use?)Let’s talk about Communication • How do you communicate electronically with your students? • How do you use eLearning to communicate with your students? • What is missing from eLearning that would enable you to more effectively communicate with your students?Let’s talk about Collaboration • How do your students collaborate electronically? • How do you use eLearning for collaboration? • What is missing from eLearning that would enable you and your students to more effectively collaborate?Let’s talk about Assessment • How do you assess your students electronically? • How do you use eLearning to assess your students? • Are you able to assess your students effectively using eLearning? • How could you better assess your students in an online format?Let’s talk about Learning Modules • How do you organize your course (in eLearning)? • What content tools in eLearning do you use? • How would you rate each of these content tools and why?Let’s talk about Course Management • What course management tools in eLearning do you use? • How would you rate each of these course management tools and why? • What course management tools would you like to see made available?
Let’s talk about Gradebook • What do you need in an LMS gradebook? • What do you like about the eLearning gradebook? • What do you not like about the eLearning gradebook? • What features would you like to see added (or changes made to) the gradebook?Let’s talk about Integrated Tools • What integrated tools do you use (i.e. Respondus, Turnitin, eInstruction-clickers)? • What external tools would you like to see integrated with an LMS?What additional tools or what functionality would you like to see in an LMS? (egportfolios, wikis, etc)Let’s talk about Accessiblity • Do you feel eLearning is accessible? • Why/why not? Let’s talk about Overall Improvements • How can eLearning (an LMS) be better?