Your SlideShare is downloading. ×
University of Notre Dame Course Management System Replacement ...
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

University of Notre Dame Course Management System Replacement ...

13,802
views

Published on

Published in: Technology, Business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
13,802
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
53
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. University of Notre Dame Course Management System Replacement Project Findings and Recommendations University Council for Academic Technologies Course Management System Subcommittee Public Report May 2008
  • 2. Table of Contents Executive Summary ....................................................................................................................... 3 Overview ............................................................................................................................................ 4 University Council for Academic Technologies ........................................................................................ 4 Office of Information Technologies CMS Replacement Project ......................................................... 4 Course Management System subcommittee .............................................................................................. 5 Activities of the CMSSC ................................................................................................................................................. 6 Vendor Review Process ................................................................................................................................................ 9 Modular and Best of Breed Solutions..................................................................................................................... 9 Course Management Systems at the University of Notre Dame ...................................................... 10 History of CMS at Notre Dame ................................................................................................................................10 Usage of existing CMS .................................................................................................................................................11 Motivations for Change..................................................................................................................................... 12 Instability and lack of vendor support for existing CMS (Blackboard Vista) .....................................13 Uncertainty in the Course Management System Marketplace ..................................................................14 Evalution Process ......................................................................................................................... 14 The Functional Evaluation Matrix ................................................................................................................ 14 First Round Evaluation of CMS options ..................................................................................................... 17 Pilot CMS Choices ................................................................................................................................................ 17 Commercial Systems .........................................................................................................................................................18 Angel Learning Management Suite .......................................................................................................................18 Blackboard Vista ...........................................................................................................................................................18 Open or Community Source System ...........................................................................................................................18 Sakai ....................................................................................................................................................................................18 Modular Solution(s) as an alternative to CMS .......................................................................................................18 The Evaluation Framework ............................................................................................................................ 19 Pilot Courses offered in Angel and Sakai ..................................................................................................................23 Usability Evaluation Sessions ........................................................................................................................................23 Evaluation Results .................................................................................................................................. 24 Student Survey ..................................................................................................................................................... 24 Faculty Feedback ................................................................................................................................................. 27 Technical Integration Evaluation ................................................................................................................. 29 University Libraries Evaluation of Integration and Extensibility ...........................................................29 OIT evaluation of Sakai and Angel performance and stability ..................................................................31 Usability Evaluation ........................................................................................................................................... 33 Faculty Evaluation Sessions .....................................................................................................................................33 Student Evalution Sessions ......................................................................................................................................35 Findings and Conclusions .......................................................................................................... 36 Summary of the Evaluation of CMS Alternatives ................................................................................... 36 Modular Solution ................................................................................................................................................................36 Ranking the three CMS Options ....................................................................................................................................38 Recommendations........................................................................................................................ 40 Future Vision for CMS at Notre Dame .................................................................................... 41
  • 3. Executive Summary The University Council for Academic Technologies (UCAT) requested the Course Management System Subcommittee initiate a thorough review of CMS alternatives for the University of Notre Dame and provide a recommended strategy and course of action for UCAT by the end of the 2007-8 academic year. With the assistance of faculty from a variety of academic disciplines, students, staff from the Office of Information Technologies, and Kaneb Center for Teaching and Learning, the study was conducted from May 2007 – February 2008. The comprehensive CMS evalution process consisted of a series of functional and technical activities including vendor presentations, semester-long pilots of two alternative course management systems, formal faculty and student usability studies, student surveys and faculty focus groups. In addition in-depth technical analyses were conducted to evaluate how each CMS would integrate into the university‟s student information system, how extensible and flexible each system was for future development, and how well each CMS matched the vision for CMS use at the University of Notre Dame. The Course Management System subcommittee worked diligently and collegially to understand the entire scope of possibilities each CMS reviewed offers for supporting teaching and learning for varied University stakeholders. As a result of our work we present the following recommendations for review and approval by the University Council for Academic Technologies. The CMS Subcommittee of the University Council for Academic Technologies recommends:  The University of Notre Dame continue to use Blackboard Vista as the institutionally supported Course Management System at least through Spring Semester 2011.  Review CMS options again in 2010* o Determine if Blackboard has met their contractual obligations o Re-assess course and learning management system marketplace  The University should support a set of usability and functionality enhancements to Blackboard Vista to be phased in over the next 12 months – in place by the end of Spring Semester 2009.  The University strategically engage its faculty in an effort to raise awareness of the benefits of using the Course Management System as a means for enhancing learning in and outside the classroom. *Represents the end of life for Blackboard Vista requiring possible ND move to Blackboard‟s Next Generation System Course Management System Replacement Project – Findings and Recommendations 3
  • 4. Overview University Council for Academic Technologies The Notre Dame University Council for Academic Technologies (UCAT) provides a forum for collaboration between faculty, students and the Office of Information Technologies to manage existing and identify new applications of technology that will enhance teaching, learning, and research. The Council is an advisory body to the University Chief Information Officer (CIO). The Council has twenty members, comprised of seven elected, seven appointed, and six ex-officio members. One member is elected by and from the regular faculties of each of the following: the College of Arts and Letters, the College of Engineering, College of Science, the Law School, the Mendoza College of Business, the School of Architecture, and the University Libraries. Elected members serve staggered three- year terms and may be reelected. UCAT has a number of standing subcommittees including the Software subcommittee, Learning Spaces subcommittee and the Course Management System subcommittee. In the November 6, 2006 UCAT meeting CIO Gordon Wishon acting in his role as UCAT Chairperson requested the Course Management System subcommittee (CMSSC) take the lead on behalf of the full Council in conducting a thorough review of CMS alternatives for Notre Dame and to provide a recommended strategy and course of action by the end of the 2007-8 academic year. Following this call to action, the CMS subcommittee met frequently from January 2007-August 2007 to develop a comprehensive strategy for evaluating CMS alternatives for the University. Office of Information Technologies CMS Replacement Project The Office of Information Technologies created a comprehensive project management process and internal staff working group to assist with activities related to the CMSSC evaluation efforts. The CMS evaluation project is broad in scope and includes: 1) Evaluation of multiple Course Management Systems (CMS) according to a detailed requirements list developed by the CMSSC 2) Implementation of the selected CMS option 3) Defining policy regarding CMS usage and archiving data 4) Conversion of course data to a new CMS solution 5) Enhanced end user support for the chosen CMS The CMS project working group was primarily responsible for performing the necessary research, developing effective processes and providing appropriate documentation to accomplish the above objectives. Its main mission was to assist the CMS subcommittee with its evaluation efforts. This report focuses on the first project goal and reports the findings and recommendations from this evaluation. Course Management System Replacement Project – Findings and Recommendations 4
  • 5. Course Management System subcommittee The Course Management System subcommittee (CMSSC) is a standing committee of UCAT with a stated mission to provide ongoing guidance to the academy about processes, policies, and marketing of the course management system. The subcommittee also includes representatives from campus that have responsibilities for ensuring Notre Dame has a pedagogically sound and technically functional CMS for faculty and student use. The table below lists members of the subcommittee and their academic or administrative department affiliation. Faculty Representing First Last Arts & Letters (Sociology) David Hachen Engineering (Electrical) Mike Schafer Kaneb Center Chris Clark Library Mark Dehmlow MCOB (Accounting) Mike Morris MCOB (Mgmt) Bob Lewandowski Sciences (Biology) Kristin Lewis Sciences (Chem) Sarah West UCAT, MCOB (Mgmt) Robert Easley Staff Representing MCOB (Tech & Courseware) Jon Crutchfield OIT (Functional Lead) Kevin Abbott OIT (Project Sponsor) Craig Brummell OIT (Project Sponsor) Peggy Rowland OIT (Functional Lead) Paul Turner OIT (Project Services) Tracy Weber OIT (CMS Administrator) Laura Gekeler OIT (CMS Administrator) Brandon Rich OIT (InsideND Portal) Patricia Sperry OIT (Project Manager) Scott Kirner Registrar Harold Pace Registrar's Office Don Steinke STEP Tom Cummings STEP Greg Doyle Table 1: Course Management System Subcommittee In addition to the CMSSC the CMS Replacement Project managed by the Office of Information Technologies was led by a steering committee consisting of the CMSSC chair Rob Easley, CIO Gordon Wishon, Deputy CIO Craig Brummell and Vice President and Associate Provost Dennis Jacobs. Course Management System Replacement Project – Findings and Recommendations 5
  • 6. Activities of the CMSSC In a series of meetings held from January – August 2007 the CMSSC considered how best to accomplish the primary objective to: Develop a comprehensive strategy for evaluating appropriate CMS alternatives for the University. A comprehensive vendor review process, and preliminary functional and technical software review process was established in February 2007. Subsequent to the data collection and discussion of CMS alternatives most appropriate to the University of Notre Dame a decision was made by the CMSSC in June 2007 to pilot three systems in the Fall 2007 semester. The committee decided to pursue a pilot process of testing commonly used CMS tools in live courses offered to approximately 600 students in sections taught by six volunteer faculty. The four month long pilot process also included formal usability evaluations of common CMS tasks by twenty three faculty and graduate teaching assistants. Technical evaluations were conducted by OIT CMS integration and administrator staff via vendor meetings, presentations and conference calls. Additional faculty input was solicited via a series of four public presentations on CMS alternatives in September-November 2007. Student feedback was requested from students enrolled in the pilot courses via an end-of- semester survey. Formal usability sessions of five common tasks performed by students in a CMS were also conducted with twenty one randomly chosen students in late November 2007. Total cost of ownership information for the three CMS pilot software options was also collected, analyzed and presented to the CMSSC and the project Steering Committee. A project timeline proposed by the CMMSC is included as Chart 1 below. Course Management System Replacement Project – Findings and Recommendations 6
  • 7. Chart 1: Course Management System Evaluation Project Plan A comprehensive timeline for the entire project is included in Table 2. Course Management System Replacement Project – Findings and Recommendations 7
  • 8. Month Event January 2007 Blackboard support for Vista continues to deteriorate February Notre Dame CIO requests Blackboard issues summary Final Blackboard Issues Summary Delivered to CIO March CIO discusses issues with President of Blackboard, indicates ND desire to explore other CMS options. Blackboard contract renewed until June 2009 to give ND time to move to a new CMS if desired. Notre Dame agrees to include Blackboard Vista as one alternative in the pilot evaluation process. April Notre Dame CMS re-branded “Concourse” CMS Replacement Initial Discussions - RFP vs Pilot approach – Pilot approach chosen and finalized. May CMS vendor and Open Source pre-screening begins – discussions with vendors, universities, Educause. Review of Top 20 Universities CMS Use presented. Results of Interviews with 15 select colleges presented. June Pilot scope is defined: 1 open source, 2 commercial CMS will be formally reviewed. Pilot participants identified. Pilot decision to host both commercial and open source pilot environments off-site. July Angel (hosted by Angel Learning), Sakai (hosted by Optimized Learning). CMS Subcommittee – CIO Gordon Wishon and Assoc. Provost Dennis Jacobs attend August CMS replacement project discussed before official pilots begin. Hosted systems are configured and tested. September Sept. 1 – Pilot courses start. Sept. 19 – Sakai Foundation presentation to Faculty October Oct. 17 – Angel Learning presentation to Faculty Technical conference calls and meetings with Pilot vendors Nov. 13 – Sakai Presentation by IUPUI staff to Faculty Nov. 29 – Blackboard Presentation to Faculty November Technical conference calls and meetings with Pilot Vendors Faculty Usability sessions Conducted (Nov. 1-21) Student Usability sessions Conducted (Nov. 29) Information collected on Modular Solution alternatives December Student pilot survey conducted (December 8-15) Pilot Faculty and Student Focus Groups held CMSSC meetings to review data and findings January 2008 Information compiled from Usability Evaluations and Survey Data Total Cost of Ownership (TCO) data compiled and analyzed Modular Solution information compiled and analyzed CMSSC meetings – Findings and recommendations presented and reviewed February Findings and recommendations presented to CIO and Associate Provost Presentation of Findings and Recommendations to Colleges and Departments for Faculty comment March Findings and Recommendations presented to University Council for Academic Technologies April Final Decision for CMS Replacement or Retention – Move to implementation phase of the project. Table 2: CMS Replacement Project Activities Timeline Course Management System Replacement Project – Findings and Recommendations 8
  • 9. Vendor Review Process In January 2007 the CMSSC developed a list of possible CMS alternatives as a starting point for further research on their suitability for use at Notre Dame. Drawing from a list of both commercial and open source Course/Learning Management Systems in wide use by US institutions of higher education six Course Management Systems were ultimately chosen for further research. Of the six CMS chosen by the committee to investigate four were commercial software programs (Blackboard 7, Blackboard Vista, Angel LMS, Desire2Learn) and two were open or community source software applications (Moodle and Sakai). Course/Learning Management System Commercial/Open Source Angel Learning Management Suite 7.1 Commercial Blackboard Learning System Vista 4.1 Enterprise Commercial Blackboard Learning System Release 7 Enterprise Commercial Desire2Learn 8.1 Commercial Moodle 1.6.1 Open Source Sakai 2.3 Community Source Table 3: Course Management Systems chosen by CMSSC to investigate There are a variety of definitions and related concepts for open source software. Most explanations of open source include the ideas that software source code can be released under an open-source license or to the public domain and freely downloaded and used or modified without cost or license infringement. The Open Source Initiative (http://www.opensource.org) describes open source as “a development method for software that harnesses the power of distributed peer review and transparency of process.” The Sakai Foundation describes Sakai as “a free and open source product that is built and maintained by the Sakai community.” Sakai's development model is called "community source" because many of the developers creating Sakai are drawn from the "community of organizations" that have adopted and are using Sakai. (http://sakaiproject.org/) Modular and Best of Breed Solutions In addition to the six CMS listed above the CMS Replacement Project steering committee also directed OIT staff to investigate the option to assemble a „modular‟ set of web-based course support tools or services. As early as 2003 many educational institutions were hoping to develop a “best of breed” approach to building tailored course management system solutions which would allow them to “plug-in” customized tools and pedagogical practices into a standards-based course management system software “shell” or learning management infrastructure portal. Notre Dame has invested heavily in recent years (2003-2006) to develop an integrated Student Information System (SIS) and portal strategy. The modular
  • 10. solution(s) research undertaken by the CMS Replacement Project Team sought to examine the feasibility of extending the existing SIS and Luminis Portal (InsideND) with a related set of standalone software tools or services that might substitute for the commonly used functions of a monolithic CMS product offering. Course Management Systems at the University of Notre Dame Concourse is the University of Notre Dame‟s current Course Management System (CMS) as of January 2008. Concourse is Notre Dame‟s internal “brand”, or naming convention, for a commercial CMS based on WebCT Vista, which following the merger of WebCT and Blackboard companies in 2005 was renamed the Blackboard Learning System – Vista Enterprise License. Concourse is utilized by approximately 400 faculty and 8,000 student users per semester with the majority being undergraduate students taking traditional classroom-based courses taught on the Notre Dame campus. In addition to Concourse, a smaller group of Notre Dame faculty (approximately 150 per semester) also use a customized version of the Office of Information Technologies (OIT) supported Institutional File Space known as Courseware. Courseware offers faculty the option to use a simple file system which supports uploading electronic course files (syllabi, handouts, presentations) and a drop box feature for students to submit work for assessment in secure folders which can only be viewed by faculty or teaching assistants. Although Courseware is not a full-featured Course Management System the capabilities it offers are related to the review of future CMS replacement options at Notre Dame in this report, particularly what will be labeled the “Modular” option. History of CMS at Notre Dame Notre Dame is primarily a residential campus with approximately 80% of undergraduate students living in campus residence halls. Most faculty-student interaction for course-related learning activities takes place in traditional classroom settings. The average class size is 28 and the faculty-student ratio is 13:1. There are very few off-campus distance learning courses currently supported by the University where use of a CMS or LMS is required by faculty for all students, with the exception of a number of learning at-a-distance programs offered by the College of Business and the Satellite Theological Education Program (STEP). For these courses use of a CMS is essential, especially for providing quizzes, hosting discussions and providing an environment for interaction between instructors and students. The University of Notre Dame first offered an institutionally supported CMS for faculty and students in 1998. Prior to 2005, Notre Dame licensed use of WebCT Campus Edition which was jointly supported by staff from the Office of Information Technologies (OIT) Education Technologies & Services (ETS) group and faculty and staff in the Kaneb Teaching and Learning Center. OIT provided institutionally supported file servers, data center space, and limited system administration support for WebCT CE. A committee of faculty and staff led by Chris Clark and Kevin Barry from the Kaneb Center and Tom Laughner from OIT met on a regular basis in Fall 2003 and early 2004 to conduct a formal evaluation and request for proposal process to procure a new CMS. Ultimately the Course Management System Replacement Project – Findings and Recommendations 10
  • 11. committee recommended the University should continue with WebCT as the commercial CMS software vendor and move to purchase their newer enterprise product (Vista). WebCT Vista was installed at Notre Dame in July 2004 and supported by a new full-time position of CMS Application Administrator within the OIT Education Technologies & Services group. After the merger of Blackboard and WebCT companies in early 2006, WebCT Vista was renamed the Blackboard Learning System Vista Enterprise License (Blackboard Vista). Notre Dame continues to offer Blackboard Vista for faculty & student use and has migrated to new versions soon after they are released. The current version of Blackboard Vista in use at Notre Dame in the Fall 2007/Spring 2008 semesters is 4.2. Notre Dame has „branded‟ Blackboard Vista internally using Concourse as a persistent name for the institutionally supported CMS to avoid confusion by faculty and students given the frequent name changes for WebCT and Blackboard course management system product offerings in the past three years. Usage of existing CMS Adoption by Notre Dame faculty seeking „single sign-on‟ access to a centralized collection of web-based course management tools to augment their traditional classroom-based teaching strategies has grown steadily over the past three years to its present usage outlined in Table 1 below. Approximately one third of teaching faculty (36%) now use Concourse in some manner to support their courses (Fall 2007). Student use of Concourse is much higher, ranging from a low of approximately forty-four (44%) to a high of over eighty percent (83%) in the Fall 2007 semester. Students can be enrolled in as many as six courses per semester increasing the possibility that one or more of their instructors will use Concourse. It appears that faculty who teach larger lecture-style courses with multiple sections utilize Concourse more frequently then faculty who teach smaller seminar-style courses. Concourse is also utilized for non-teaching support by some academic departments and student groups as a secure content repository and for discussions related to maintaining repository documents. Course Management System Replacement Project – Findings and Recommendations 11
  • 12. Semester CMS Version Faculty* Students+ Fall 05 WebCT Vista 3 11.9% 60.7% Spring 2006 WebCT Vista 3 13.5% 44.0% Fall 06 Blackboard Vista 3 23.8% 69.7% Spring 2007 Blackboard Vista 3 19.3% 64.7% Fall 07 BB Vista 4.1 36.0% 83.0% Spring 2008^ BB Vista 4.2 27.0% 68.0% Table 4: Faculty & Student CMS Usage 2005-2008 * Based on Registrar data for number of faculty teaching courses each semester * Does not include faculty teaching seminar, directed reading or dissertation courses + Based on actual CMS usage by undergraduate and graduate students compiled from CMS log data and compared to official Registrar student enrollment data ^ Based on early semester usage data – normally increases towards end of the semester as Faculty sign up to use Concourse for delivering grades to students Motivations for Change There are a number of academic and administrative motivations for the present re- examination of what Course Management System(s), or related web-based software tools, would best serve faculty and students in the near term (2008-11). Despite a steady increase in use of Concourse by faculty in the past three years (2005-7) adoption remains low relative to CMS use by faculty at other institutions of higher education. For example three years ago (2005) a comprehensive survey of faculty teaching in the University System of Georgia indicated that 46% of faculty were already using a CMS to support their teaching activities, with a majority (51%) already using a CMS for three or more years (http://www.alt.usg.edu/research/studies/cms.phtml). Student usage of Concourse at Notre Dame is considerably higher than faculty usage and now approaches the national average of CMS use by undergraduates in U.S. institutions of higher learning reported by the EDUCAUSE Center for Applied Research (ECAR) in their most recent 2007 report (83%). Faculty have frequently complained via annual OIT institutional information technology surveys, e-mail to the Help Desk, or in hands-on training sessions that the current CMS is not “easy to use” for their needs. Course Management System Replacement Project – Findings and Recommendations 12
  • 13. A major faculty criticism is the lack of integration between Concourse and the Banner Student Information System (SIS) for posting official course grades. Faculty often have to maintain or post grades in three different formats and locations:  in a spreadsheet they keep on their local computer  in the Concourse Gradebook tool for posting to students, and  in the Banner Student Information System grading application maintained by the University Registrar. Another general criticism from faculty has been the number of steps required to complete a simple task such as posting a document for students to view, download or print. In addition staff from the OIT and the Kaneb Teaching and Learning Center who support faculty use of the CMS on a daily basis continue to report faculty dissatisfaction and confusion when trying to use Concourse. However following the introduction of version 4 of Blackboard Vista, this has begun to change, with some faculty users reporting increased user-friendliness. Instability and lack of vendor support for existing CMS (Blackboard Vista) Ongoing technical issues with Concourse encountered by faculty, students and OIT staff who manage day-to-day delivery of CMS service to faculty & students include:  Incompatibilities between support for client versus server-side java versions which seriously impacted performance of the Chat and HTML Editor tools in Concourse.  Lack of robust support in Blackboard Vista for Apple‟s Safari Web browser.  Ongoing administrative issues with the lack of ability to synchronize official Registrar data (course sections, teaching assignments, student enrollments) with the Blackboard course database through the real-time interface provided jointly by Blackboard and SCT Sungard. Lack of consistent or functioning integration with Notre Dame‟s Student Information System continues to be a major complaint (Sungard Banner). This issue has not been successfully resolved as repeatedly promised by Blackboard Corporation after they merged with the WebCT company and acquired its WebCT product offerings in early 2006. The 2004 Course Management Systems Selection Process report projected this integration feature would be fully operational in Fall 2005. The ongoing nature of these issues led Notre Dame to limit its commitment to continue to license Blackboard Vista, and acted as a driving force in the University‟s decision to do an in-depth re-evaluation of CMS alternatives. Course Management System Replacement Project – Findings and Recommendations 13
  • 14. Uncertainty in the Course Management System Marketplace Since the Blackboard merger with WebCT in early 2006, there has been an acceleration of competition in the commercial course and learning management system software marketplace. Edutools, a commonly used web service which provides feature-by-feature product comparisons and decision-making supports for reviewing course management systems (CMS), currently lists twenty four commercial and open source learning management systems (LMS) that are in wide use in higher education and K-12 school districts in the United States. (http://www.edutools.org) There is also increasing review and adoption by major institutions of higher education of non-commercial open source or community source CMS or LMS alternatives. Recent statistics gathered by the Campus Computing Project for their annual report indicate that in 2007 approximately eight percent (8%) of US universities have adopted and deployed Open Source Learning Management Systems. According to Kenneth C. Green, founding director of The Campus Computing Project "the recent gains for Moodle and Sakai are very interesting, suggesting that ten years after the deployment of the first commercial LMS applications, campus officials and faculty advisory committees are reviewing seriously the various LMS offerings from both commercial providers and the collaborative Open Source movement." (http://www.campuscomputing.net/survey-summary/2007-campus-computing-survey) Research conducted by OIT staff of course management systems currently used by the US News and World Report Top 20 National Universities revealed that thirteen of these top twenty US universities are currently evaluating the Sakai Community Source Collaborative Learning Environment as a viable alternative to continuing with commercial CMS vendors. Many of these same top twenty universities are prior or existing Blackboard or WebCT customers (12 of 20). Evalution Process The Functional Evaluation Matrix A functional matrix was created to guide information gathering and analysis on how each alternative might meet a wide range of faculty and institutional requirements for CMS use at Notre Dame. Technical considerations included which operating system platform(s), Web browsers and databases each CMS supports. The following table provides an outline view of functional needs and issues developed by the CMSSC to guide research and information gathering on each CMS alternative. Course Management System Replacement Project – Findings and Recommendations 14
  • 15. Functional Needs/Guiding Questions General Platforms and Versions Supported Browsers and Versions Supported Databases and Versions Supported Instructor can View/Perform Tasks Within Course as a Student Course Announcements can be Posted Within a Course Tasks/Assignments can be Selectively Released to Students Based on Criteria On-line, Context-sensitive Help is Available Equation Editor Available Throughout System Breadcrumb Navigation Available Instructor can Customize the Look of the Course Administrative Instructor Can Create/Download Course Backups Individuals can be Manually Entered or Deleted from a Course Instructor can Manually Enter Users into Course Instructors can Add Guest Users Who do not Receive Grades Instructors can Add TAs who Can Enter Grades Instructor can Track all Activities of Students Assessment Integrated Quizzing Tool with Automatic Grading Ability to do Multiple Choice, Short Answer, Essay, and Calculated Questions Quiz Time Limits can be Set Questions can be Used by Multiple Quizzes and Surveys Question Databases can be Shared Across Multiple Courses Questions can be Imported from a Text File Students can be Allowed to Take a Quiz Multiple Times Instructor Controls How and When Quiz Scores are Released Security Measures are Available, Such as Passwords and IP Range Limits Quiz Grades are Automatically Entered in the Grade Book Statistical Analysis of Quiz Question Responses is Available Anonymous Surveys can be Created and Administered Groups Assignment of Users to Groups Selectively or Randomly Multiple Sets of Student Groups Can be Created Discussions Students can Post Anonymous Messages Private or Limited-Group Forums can be Established Discussion Forums can be Locked by Instructor Students can be Given Right to Delete Their Own Messages Discussion Messages can be Archived Instructor can Move Messages Between Forums Course Participants can be Notified via Email When a New Message Has Been Posted Threaded, blog, and journal formats available Table 5: Course Management System Functional Matrix Course Management System Replacement Project – Findings and Recommendations 15
  • 16. Grade Book Grades can be Imported and Exported as a Text File Grades can be Selectively Released to Students Non-grade Information can be Entered (e.g. section number or nickname) Total Points can be Calculated Automatically Instructor can Customize the Way Total Points are Calculated Instructor can Choose to Not View All Grades Letter Grade can be Automatically Determined Based on Total Points Instructor can Change the Grade Display Format, Such as Decimals and Alignment Instructor can Rearrange the Display Order of Grade Columns Instructor can Sort Grades by Name or Score, Ascending or Descending Instructor (and Students with Permissions) can see Basic Statistics on a Grade Assignments Students can Upload Documents to a Drop Box Instructor can Make Comments On and Grade a Document from Within the System Individual Assignment Grades are Automatically Entered into the Grade Book Course Content WYSIWYG HTML Editor Available for Creating and Editing Documents Any Type of Document can be Posted as Content HTML, Word, Powerpoint, Excel, PDF, etc. Files can be Uploaded and Downloaded as Content Content is Searchable Within a Course Table 5: Course Management System Functional Matrix Course Management System Replacement Project – Findings and Recommendations 16
  • 17. First Round Evaluation of CMS options OIT CMS Replacement Project staff spent considerable time on behalf of the CMSSC contacting and interviewing IT staff and select faculty from fifteen universities who had experience using the CMS alternatives under consideration at Notre Dame. University/College CMS Used Licensed Users 1 Elgin Community College Desire2Learn 9,800 2 Ohio State University Desire2Learn 46,064 3 University of Iowa Desire2Learn 24,814 4 Providence College Angel 6.444 5 SUNY Brockport Angel 9,000 6 Lindsey Wilson College Angel 2,000 7 Manatee Community College Angel 6,785 8 Princeton University Blackboard 6,000 9 University of Maryland Blackboard 34,000 10 University of Nebraska-Omaha Blackboard 25,000 11 University of Delaware Sakai Not reported 12 Yale University Sakai Not reported 13 University of California Davis Sakai Not reported 14 Georgia Institute of Technology Sakai Not reported 15 University of North Carolina Sakai Not reported Table 6: Universities and colleges contacted for in-depth interviews Pilot CMS Choices Using the information gained from the interview process the CMSSC thoroughly discussed the pros and cons of each CMS alternative based on the features outlined in the functional matrix in Table 5 above. After careful deliberation the CMSSC narrowed the original CMS review list to three alternatives to pilot with faculty and students in Fall 2007 semester. In addition to two commercial systems and one open/community source option, a „modular‟ option was also considered during the evaluation effort. Course Management System Replacement Project – Findings and Recommendations 17
  • 18. Commercial Systems Angel Learning Management Suite The ANGEL suite of enterprise learning management tools enables efficient and effective development, delivery and management of courses, course content and learning outcomes. Angel Learning is a privately held company based in Indianapolis, Indiana. Notre Dame chose to pilot Angel Learning Management Suite 7.2. Blackboard Vista Blackboard offers two main course management system products: Learning System and Learning System – Vista Enterprise License based on the WebCT Vista product developed by the WebCT corporation prior to its acquistion by Blackboard, Inc. in early 2006. Notre Dame chose to pilot Blackboard Learning System – Vista Enterprise License (Blackboard Vista). This is also the current system in use at Notre Dame (Concourse). Open or Community Source System Sakai Sakai is a free and open source product that is built and maintained by the Sakai community of more than one hundred university and for-profit company partners. Leading university developers of Sakai include the Universities of Michigan, Indiana, Cambridge and Stanford. Notre Dame chose to pilot a freely available version of Sakai developed by rSmart, which is a major commercial Sakai hosting and development vendor (rSmart Sakai 2.4). Modular Solution(s) as an alternative to CMS The CMSSC steering committee wanted to keep open the option of not selecting a particular CMS at all. In this case, the University would wait until a clear CMS leader emerged according to the University‟s requirements, and at that time the University would make a commitment to a specific solution. Additionally, the University would provide some level of basic functionality for course management in the interim until the leading solution was implemented. The basic functionality decided on by the CMSSC includes the following: • Content Management • Assignments • Assessments • Discussions/Collaboration • Grade Book A review of stand-alone web-based tools that accomplished the functions above was conducted. As a result of this review, five „stand alone‟ software options were chosen to evaluate further: insideND (Luminis portal), Xythos file space, SnapGrades, WebAssign and a customized grade upload solution that OIT would design and program internally. Course Management System Replacement Project – Findings and Recommendations 18
  • 19. The Evaluation Framework The CMSSC developed a comprehensive evaluation framework to guide the project. This framework was vetted with industry experts from Educause through a series of conference calls, and reviewed by experienced course management system administrators at other universities who have conducted similar evaluation efforts. Illustrated in Table 7, this framework was used throughout the evaluation process to focus committee discussions and plan evaluation activities. The columns in the matrix represent the four alternatives studied (Angel, Sakai, Blackboard Vista, Modular solution); the rows represent logically separable components of analysis considered in order of importance (Core Functionality/Usability, Integratibility, Extensibility and Future Vision). The most thoroughly applied part of the framework was Functionality and Usability as will be clear from the results reported later in this report. However considerable research effort was also given to studying each system‟s potential for integration, extension and flexibility, primarily by Office of Information Technologies and University library staff members who work with course management and web development systems on a day-to-day basis. Limited pilot and reference checks for integration involved testing various features of each system‟s code and associated technical documentation with follow-up discussions with knowledgeable technical vendor representatives to establish accuracy of understanding. Future vision for each CMS was assessed by market research and through attending presentations, meetings, reading white papers and conducting technical calls with marketing and technical development personnel from each vendor, or in the case of Sakai from Sakai Foundation representatives. This evaluation framework was continually refined by the committee throughout the Fall 2007 and was extremely useful in developing the final evaluation results described later in the report. Course Management System Replacement Project – Findings and Recommendations 19
  • 20. Table 7: Evaluation Criteria Functional Matrix Course Management System Replacement Project – Findings and Recommendations 20
  • 21. After much discussion the subcommittee agreed to assign relative weights, or measures of importance, to each major component in the evaluation criteria matrix. These weights are illustrated by the graphic below. Within the largest evaluation component (Functionality and Usability) five commonly used CMS features were rank-ordered by the committee from a faculty perspective from the most important to least important. Criteria weights were assigned so that Content Management was given approximately five times the weight of the least important functionality (Discussions), Assignments given four times the weight of Discussions and so forth. There was not universal agreement by members of the committee on how to rank or weight various tools. For example instructors from the STEP program who teach 100% online using CMS tools preferred to rank Discussions as their number one requirement. Chart 2: Evalution Criteria and Functionality & Usability Weights Following the guidance offered by the functional matrix the comprehensive CMS evalution process consisted of a series of functional and technical activities:  Pre-pilot testing of Angel and Sakai software with external vendors: o Angel Learning was chosen to host Angel for Notre Dame Pilot courses. o Optimized Learning was chosen to host Sakai for Notre Dame Pilot courses.  Customized development of each externally hosted CMS to allow secure single sign- on for faculty and students in Pilot courses using their Notre Dame netID username and password.  Pilot faculty were assisted by Kaneb/OIT staff in building customized course sites in the respective CMS they were assigned to test.  Five evaluation use cases that covered common tasks faculty were most likely to perform when using a CMS were developed and pre-tested using Morae Usability Testing software. These tasks were attempted by twenty-three faculty members in a controlled usability lab setting. Course Management System Replacement Project – Findings and Recommendations 21
  • 22.  A single evaluation use case of five common tasks which students were likely to perform when using a CMS were developed and pre-tested using Morae Usability Testing software. These tasks were attempted by a random sample of twenty-one students who volunteered to spend twenty minutes performing these tasks in the LaFortune student center.  Technical meetings and conference calls were scheduled with each CMS Pilot vendor to evaluate technical feasibility, stability, database support, security and integration issues with the existing ND Student Information System and Portal applications.  Each CMS Pilot vendor was invited to present a public presentation open to Notre Dame faculty and administrative staff with an interest in CMS alternatives. A representative from the Sakai Foundation, accompanied by a Sakai technical expert from the University of Michigan, presented information on the Sakai Open Source software. A faculty member who currently uses Sakai to teach at Indiana University Purdue University at Indianapolis (IUPUI) was also invited to present examples on how he uses Sakai to support his courses, and to discuss the software development activities that are on going within the Indiana University system to customize and support Sakai as their institutional course management system.  Information was gathered on “best of breed” modular software tools that potentially could be used to build a „modular‟ CMS solution which would integrate with Notre Dame‟s SIS, institutional file space and existing portal software solution(s).  Interviews were conducted with Notre Dame library representatives to determine critical issues and opportunities where students and faculty would benefit from tighter integration of CMS and Library service software.  Wrap-up focus groups were held with pilot faculty and students in November- December 2007.  A survey questionnaire was developed and administered to collect feedback from students participating in the pilot courses offered in either Angel or Sakai during Fall 2007.  Total cost of ownership data was collected from the commercial software vendors including software license fees, hosting and support costs. Cost estimates for Sakai were obtained from a commercial Sakai development and third party hosting vendor. Course Management System Replacement Project – Findings and Recommendations 22
  • 23. Pilot Courses offered in Angel and Sakai Four faculty members and two instructors volunteered to pilot test Angel LMS and Sakai in their courses during Fall 2007. The table below illustrates which course, course sections and CMS tools were evaluated by faculty and the length of time within each course these options were evaluated. Table 8: CMS Pilot Courses and Tool Functionality Testing Timeline Pilot courses in a variety of academic disciplines were offered in Sakai and Angel LMS with a total beginning of semester enrollment of six hundred students. As Blackboard Vista (Concourse) is the current incumbent institutional CMS specific pilot activities were not conducted in this environment. However all faculty chosen to participate in the pilot CMS courses were active users of Concourse and some of the pilot faculty continued to use Concourse tools to support activities in their courses. This required some students to use two different CMS during the semester, sometimes simultaneously. This approach helped inform both pilot faculty and students as to the strengths, weaknesses and differences between different CMS platforms being evaluated. Usability Evaluation Sessions Faculty and student volunteers were also recruited to participate in formal usability analysis sessions. These hands-on sessions were designed using the concept of use cases which is a standard software industry technique for evaluating both prototype and existing software products. A use case is a description of how end-users will use software in certain settings or conditions. A typical use case also describes a scenario that helps frame a task or a series of tasks that users will accomplish using the software. The evaluation sessions conducted using Morae usability testing software from Techsmith Corporation allowed the capture of user actions when attempting to complete the prescribed tasks. (http://www.techsmith.com/morae.asp) The data captured via Morae for each use case Course Management System Replacement Project – Findings and Recommendations 23
  • 24. session included time on task, number of mouse clicks required to complete a task, and the level of ease or difficulty users‟ appeared to have attempting to complete a task. In addition each evaluation session was digitally recorded using Morae‟s built-in audio and video recording tools for later review and analysis. Five use cases were constructed for faculty evaluation based on the functional matrix described above. The use cases examined faculty success or difficulty in working with commonly used CMS tools for creating assignments, assessments, posting grades to the built- in gradebook, content management and discussion board posting, review and grading. Evaluation Results Student Survey At the end of the semester each faculty member who participated in the pilot requested their students fill out an online seventeen question survey to provide feedback on their experiences trying to use one of the pilot CMS environments as part of their course activities. Students participating in the pilot CMS course sites were enrolled in eighteen course sections across five different courses. Four of the faculty members taught the same course in at least two different sections allowing direct comparisons of Sakai versus Angel by faculty. Most students did not use both CMS environments in a single course, with the exception of students enrolled in Pilot Course 3. Two hundred seventy four (274) students completed the survey questionnaire, which represents a survey response rate of approximately 45% from students participating in the pilot courses. One hundred forty six (146) students who used Angel completed surveys and 128 students who used Sakai completed surveys. Survey response information is summarized by course below. Number of Percent of total Course responses survey responses Pilot Course 1 159 58.0 Pilot Course 2 65 23.7 Pilot Course 3 28 10.2 Pilot Course 4 15 5.5 Pilot Course 5 7 2.6 Totals 274 100.0 Table 9: Student Pilot Survey Responses by Course Course Management System Replacement Project – Findings and Recommendations 24
  • 25. The following table summarizes responses from selected results of questions 8 to 10 of the student survey. A copy of the complete student survey instrument is available upon request. CMS ANGEL SAKAI Total n = 146 n = 128 N = 274 Learning to use the Easy 49 43 92 CMS Somewhat easy 84 64 148 (Q8) Somewhat difficult 13 19 32 Difficult 0 2 2 CMS Reliability (Q9) Reliable 58 39 97 Somewhat reliable 61 60 121 Somewhat unreliable 21 21 42 Unreliable 4 7 11 CMS Satisfaction Satisfying 39 40 79 (Q10) Somewhat satisfying 67 53 120 Somewhat frustrating 37 27 64 Frustrating 2 8 10 Table 10: Student Pilot Survey Results, Questions 8-10 A majority of students generally reported that learning either of the pilot CMS environments was “Easy” or “Somewhat easy”. However Sakai was more likely to be rated “Somewhat reliable” versus “Reliable” compared to students‟ ratings of Angel reliability. Users of Angel and Sakai who completed the survey questionnaire were equally likely to report a rating of “Satisfying” or “Somewhat satisfying” (73%) and “Somewhat frustrating” or “Frustrating” (27%) for overall CMS satisfaction in their responses to question 10. Course Management System Replacement Project – Findings and Recommendations 25
  • 26. CMS ANGEL SAKAI n = 146 n = 128 Navigating the CMS Easy 39 39 for my course was: Somewhat easy 67 47 (Q14) Neutral 16 0 Somewhat difficult 19 27 Difficult 5 8 Table 11: Overall student impression of navigating the CMS Question 14 asked students to give an overall impression of ease or difficulty in navigating the CMS environment in their respective pilot course site(s). A majority of Angel users (72%) found navigation to be easy or somewhat easy. Students who participated in courses offered in Sakai found it equally easy or somewhat easy to navigate (71%). A slightly lower percentage of students who used Angel found it difficult or somewhat difficult (16%) to navigate then students using Sakai (29%). Selected comments in answer to Question 17 illustrate students‟ feedback after using Sakai for a semester: “Did not like the user environment. Very confusing. Not easy to navigate to my courses or find the tests.” “I had a terrible time all year trying to upload my assignments. I was never comfortable that they were there. I was constantly asking my T.A. to check for me. Also, navigating through the site was difficult.” “Not difficult to use, but the organization and overall structure/functionality/ease-of-use is not as high as the current concourse system.” “Didn‟t like the red timer w/ a descending red bar which makes the person real anxious while taking the quizzes.” “Sakai was basically an unreliable copy of Concourse. Everything done on Sakai could have been done on Concourse with less hassle.” “Very hard to navigate, would frequently give me error messages. I was never confident that my assignment was turned in correctly.” Course Management System Replacement Project – Findings and Recommendations 26
  • 27. Selected comments to Question 17 from students who used Angel LMS during the semester- long pilot illustrate some of the pros and cons: “Angel feels poorly laid out and somewhat difficult to navigate. There are too many redundancies in the system, especially when trying to upload files.” “Good choice to continue with this program.” “I have a Mac, which made it difficult to upload assignments. In the end, I just emailed all my reports to my TA because Angel was too much of a hassle and I never knew if my reports were successfully uploaded.” “I liked the chat room format [in Angel] much better than Concourse. However, I did not like that I could not cut and paste from MS word to the discussion post; I could not read and print all my posts from its own specific screen.” “I think Angel has a lot of potential, especially as an assignment retrieval and submission system. However, it still has quite a few kinks to be worked out; it was unavailable several times throughout the semester.” “The average lab report grades were not accurate on Angel.” “Too many links to go through to find what you want.” Faculty Feedback Faculty members were asked to utilize the CMS for at least two or more weeks during the semester and to use one or more tools within the pilot CMS for delivering or supporting their course delivery. A focus group was held at the end of the Fall semester to collect faculty feedback. STEP Instructors for the STEP online program reported problems using the Discussions and Chat tools in both Sakai and Angel LMS. As STEP courses are offered 100% online, well designed collaboration tools are important to deliver effective courses which require student participation. STEP instructors felt Sakai lacks useful threaded displays in the discussion tool and presented frequent errors requiring a work-around to restore the correct display of discussion posts. This had a significant negative impact on the class. Angel had better „threaded message display” and search capabilities than Sakai but both still had functional weaknesses compared to Concourse. These included lack of “save as draft” capability and lack of options for sorting messages by author or date. However both the Sakai and Angel discussion tools have an easier to use HTML Creator feature than Concourse. Course Management System Replacement Project – Findings and Recommendations 27
  • 28. The STEP instructors also found the “drag-and-drop” feature in Angel easy to use for uploading content to the course site. Sakai had significant problems allowing PDF and HTML files to display correctly. Sakai also has poor support for printing content directly from the Browser – users are required to download files and print locally from their desktop. Sakai lacked any useful student tracking and reporting features. Angel offers better reporting/tracking features than Concourse. Both Sakai and Angel lacked useful or easy-to- use survey or quiz tools. In Sakai instructors could not upload pre-typed questions (from MS Word or text files) to create surveys, in Angel you could not easily create questions individually you had to import already created surveys. The overall impression of Sakai from STEP faculty was it had “many bugs”. They felt Angel had “less headaches”, but did not out-perform Concourse for their needs. Theology Use of both Sakai and Angel was limited in the pilot Theology courses for use on midterm and final exams that contained True-False questions. Faculty felt Angel was easier to use for setting up multiple questions of the same type. Sakai did not offer features to randomize questions in exams so that every student receives a different set of questions, or the same questions in a different order than the next student. Students taking timed exams using laptops on wireless connections had serious problems with browser “time outs”. In some cases their exams simply disappeared during their exam time, requiring them to start over. Information Technology Management, College of Business Angel had superior calendar features compared to Sakai. The instructor liked the ability to create hyperlinks in Angel calendar tool to link to assignments or required reading files. Quizzes are not automatically linked into the Gradebook in either Sakai or Angel. IT Management students liked the ability to customize the Angel home page screen to their preferences. Biological Sciences The instructor generally preferred the Sakai experience versus Angel. In Angel posting assignments was complex and the status of student submissions was difficult to see at a glance. This made review and tracking more difficult for the graduate teaching assistants. Mac users found the user interface „not usable‟ in Angel. However navigation was confusing in Sakai and bugs repeatedly appeared in the Assignments tool indicating assignment submissions by students were late when they were not. Overall the instructor did not consider either pilot CMS (Angel or Sakai) to be a significant improvement over Concourse. Course Management System Replacement Project – Findings and Recommendations 28
  • 29. Technical Integration Evaluation The following chart shows the project team‟s technical evaluation of several key integration features of the candidate CMS systems. This evaluation was made based on extensive review of the administrative interface of each product and two calls with technical representatives from each vendor in order to get in-depth technical questions answered. Chart 3: Expert evaluation ratings on key technical integration issues with Pilot CMS University Libraries Evaluation of Integration and Extensibility A key strategic goal for University Library staff is to deliver library resources efficiently to patrons' "point of need", so the ability to tightly integrate existing library tools and services into the institution-supported course management system represents a significant strategic opportunity. In the current CMS environment (Concourse), University Libraries is able to offer links out to existing library web tools and the instructions on how to use them. These tools produce URLs which can then be pasted into the current course management system. The library is able to add the links to the tools in CMS course templates, but currently cannot add them retroactively to existing courses in Concourse. The best integration approach would result in a set of library tools that could become part of the standard tool menu in the course management system. The tools would exist in parallel Course Management System Replacement Project – Findings and Recommendations 29
  • 30. with tools like assessments, gradebook, etc. and would appear native to the system, but would interact with existing library servers and services behind the scenes to build and post library content for Concourse from within Concourse itself. This integration could best be supported through an API (Application Programmer Interface), preferably one that uses simple, well-formatted, and open standards and protocols, such as HTTP, XML etc. The most desirable solution would function through a technique called web services, which uses protocols like SOAP (Simple Object Access Protocol), or similar, operates over HTTP connections using simple GET and/or POST requests, and is packaged in XML messages in both directions. The University Library technical staff would prefer a CMS which includes or supports a web Examples With open and robust APIs on both ends, the integration possibilities are endless. The following examples will help illustrate practical examples of tools that could be developed: 1) Electronic Reserves - The tool would check to see if the professor has an electronic reserves course in the system, if so it would be added to their course at the click of a single button. 2) List of Resources Based on Class - Student clicks on a link in the course and they are presented with a list of electronic resources that could help them with their assignments in a particular course. 3) QuickSearch - Course designers could select from a list of databases to search across simultaneously. The selected resources would then be used to create a single search box which would be added to their course at the click of a button. Course Management System Replacement Project – Findings and Recommendations 30
  • 31. The following is a summary of the evaluation of the CMS Pilot products from the University Libraries perspective. Ranked from most preferred to least preferred based on: a. API robustness b. apparent ease of use c. staff time to implement d. apparent ability to accomplish stated goals 1) Angel -- ASP and Web Services based APIs Angel has three APIs and they are fairly robust. Angel API supports programming in the native language, ASP, or ability to utilize their web services based API. After initial review, the path to deep integration looks manageable. 2) Blackboard Vista -- Java and Web Services based APIs Vista has some fairly robust tools, although they appear somewhat complex. You can program in the native language, Java, or utilize their web services based API. After initial review, it is not clear exactly how to proceed, but there are downloadable examples that appear to have accomplished tasks similar to the Libraries' goals, so these samples could be starting points. 3) Sakai -- Java based API After initial review, it looks like modules have to be built for Sakai, which provides for maximal flexibility, but also maximal complexity. The path to integration appears to require a lot of specialized staff expertise and time, but there are also some examples that could be used as starting points. OIT evaluation of Sakai and Angel performance and stability The OIT systems administrator responsible for hands-on maintenance and support of Sakai during the Fall pilot reported the following support issues or lack of functionality requested by pilot faculty. Access Issues  Random server timeouts (temporary, did not recur)  "proxy error" encountered (temporary, did not recur)  Student could not access quiz – required reset of students privileges  Infrequent “404 page not found” errors (required a server restart) Software Bugs  Resources Tool: download all / upload all does not work (a patch didn't fully fix)  Assignment Tool submit button stopped working (required a workaround)  Editing properties of an assignment defaults release mode to "whole site"  Unable to link to a file in "resources" from a content section  Editing a content section results in an error page Course Management System Replacement Project – Findings and Recommendations 31
  • 32.  Clicking "mark as read" in Discussions tool can result in a java error that blocks discussions. (a workaround exists)  Chat room appears unstable (messages come all at once or not at all) and lacking in features Missing Functionality  No ability to set time limit for quizzes  No ability for instructors to easily see information presented from student's perspective  Not able to link to a quiz from a calendar entry The OIT systems administrator responsible for hands-on maintenance and support of Angel LMS during the Fall pilot reported the following support issues or lack of functionality requested by pilot faculty. Access Issues  Assignment release dates buried in advanced settings  Login problems due to students not having accounts in Angel or using the wrong url  Students‟ needed help viewing a previous assessmentSoftware Bugs  VBScript runtime errors after unsuccessful attempts to download assignment submissions  Browser configuration issues which impacted students‟ ability to open assignment files  Creating a link (url) in Lessons tool would not open in a new window  Intermittent problems with Chat tool – while in the Message area typing, the typed text would fade and then disappear  Assessment tool -- no characters are allowed between two $ signs when creating text for a quiz question. Missing Functionality  Angel can‟t import just a survey – forced to import a course copy the survey then delete the course Course Management System Replacement Project – Findings and Recommendations 32
  • 33. Usability Evaluation Highlights from faculty and student usability evaluations is summarized below. Examples of the Use Cases developed for the usability evaluation phase of the project are available upon request. Faculty Evaluation Sessions Use cases were designed based on the five most important functions faculty typically use to support their teaching. The use cases examined faculty relative success or difficulty in working with CMS tools for the following tasks:  creating assignments  designing and evaluating assessments and quizzes  posting grades to a built-in electronic Grade Book, checking and revising grades  content management (uploading, copying, moving and downloading files)  creating, reviewing and grading discussion board posts. Each use case had from four to seven related tasks of varying complexity. Faculty members and graduate teaching assistants were offered a $15 iTunes or Starbucks gift card to participate in a forty five minute hands-on usability evaluation session. The evaluation sessions were conducted using Morae usability testing software which allowed the capture of user actions when attempting to complete the prescribed tasks. A simple „think aloud‟ protocol method was followed in which a facilitator followed a prepared script to introduce each task to the user and asked the user to talk aloud while trying to complete each task. Facilitators were allowed to prompt the user if tasks seemed difficult, or offer a tip to help the user if requested. All evaluation sessions were digitally recorded using Morae‟s video and audio tools. The data captured via Morae for each use case session included time on task, the number of mouse clicks required to complete a task, and a facilitator assigned level of ease or difficulty rating for each task the user attempted to complete in the use case. A faculty member worked in a single CMS during the 45 minute session while attempting 4- 7 related tasks. A total of twenty-three (23) faculty members participated in 40 separate 45 minute sessions with some faculty members working 2 different use cases in 2 different systems. Course Management System Replacement Project – Findings and Recommendations 33
  • 34. The following chart illustrates the overall results of faculty usability findings. Chart 4: Faculty Usability Results The graphic above was developed by transforming the averages compiled for faculty performance based on time on task, number of mouse clicks and success-on-task scores for each use case. Higher numbers are considered better performance than lower numbers. Based on faculty usability performance Angel rated highly in four of five use cases, with the exception being a very low rating on the Assignments Tool tasks. Generally Blackboard Vista rated second best on most tasks, or tied with Angel in most tasks. Although based on a relatively small number of faculty evaluators, the usability data confirms qualitative findings reported by the pilot faculty who used Angel and Sakai for an entire semester. The pilot faculty were not included as testers in the formal usability evaluation exercises. Course Management System Replacement Project – Findings and Recommendations 34
  • 35. Student Evalution Sessions Students completed a single use case which focused on simple activities students‟ would be most likely complete in a CMS during a semester including checking for instructor announcements, their assignment or quiz grades, uploading an assignment, completing a quiz, or participating in a discussion board. Students were selected randomly to participate from students traveling through the LaFortune Student Center lobby on November 21. Students were offered a $10 Starbucks gift card to participate in the evaluation sessions. The following chart illustrates the overall results of student usability findings: Chart 5: Student Usability Results The graphic above was developed by transforming the averages compiled for student performance based on time on task, number of mouse clicks and success-on-task scores for the student use case. Higher numbers are considered better performance than lower numbers. The student use case was designed to be a simple test of five common tasks most students would perform when using a CMS as part of routine course activities. The entire use case took most students less than twenty minutes to complete from start to finish. Based on student usability performance Sakai rated highly in four of five use cases, with the exception being a low rating on the Discussions Tool tasks. Generally Blackboard Vista rated second best on most tasks with Angel coming in third. Course Management System Replacement Project – Findings and Recommendations 35
  • 36. Findings and Conclusions Summary of the Evaluation of CMS Alternatives For both the modular solution and the three CMS Pilot systems a summary matrix was compiled to aid the CMS Subcommittee in assessing which alternative best met the original functional matrix criteria. The matrices use a criteria based approach to help evaluate and prioritize the list of options. Known as the „Pugh Matrix‟ this evaluation instrument uses a prioritization scale with raw scores for each criteria item based on a nine point scale (1-9). Nine (9) is the highest score an item can receive, while the lowest score is one (1). Total weighted scores for each alternative were calculated by multiplying the derived score for each item multiplied by its relative weight (a percentage from 3 to 20) and summing up these results for each system evaluated. A threshold of seventy (70%) was established by the subcommittee as the minimum possible score a system should receive to merit further consideration as a finalist. Results from this approach are summarized below. Modular Solution The relative weights for individual products in the Modular solution are qualitative in nature, based on a best estimate by the OIT working group to evaluate each product‟s capabilities. There was no hands-on pilot testing by faculty or students of the modular solution products, nor was any attempt made to test how difficult it would be to integrate each product into a coherent and easy-to-use product offerings. Two hypothetical „composite‟ modular offerings were created to illustrate the problems inherent in trying to integrate disparate products from different vendors into a system that would meet the subcommittee‟s functional matrix goals of providing a full-featured course management system solution. [Luminis/Xythos/WebAssign | Luminis/Xythos/Custom Grade Upload] Luminis is the underlying portal system currently in use to support InsideND. Xythos is one example product that might be used as an institutional file space which has rudimentary course management features like private drop boxes or discussion capabilities. SnapGrades and WebAssign are externally hosted product offerings in use by K-12 and higher education for providing specific course management support features. The Custom Grade Upload solution would be a customized grading software program created in-house by OIT at Notre Dame to allow faculty an easier-to-use method of uploading mid-term and final semester grades into the Banner system. The resulting analysis shows the difficulties any of the stand-alone or composite „modular solutions‟ have to reach an acceptable threshold where faculty or students might find them as suitable alternatives to a full-featured single or „monolithic‟ course management system product. The OIT technical working group was particularly concerned with how extensible, scalable or flexible these stand-alone solutions might be in practice. Once the individual and collective costs of each stand alone option was considered, it quickly became obvious the modular solution approach would not meet the original objectives of the CMSSC to find an Course Management System Replacement Project – Findings and Recommendations 36
  • 37. improved alternative to the existing CMS product already in use at Notre Dame (Blackboard Vista). Chart 6: Modular CMS Solution Requirement Scores The stand-alone products reviewed as part of a „modular solution‟ do not come close to meeting the 70% threshold established by the Committee when judged independently using the requirements matrix. The two composite alternatives which sought to build a modular solution comprised of individual stand-alone products also do not meet the minimum threshold for further consideration. Course Management System Replacement Project – Findings and Recommendations 37
  • 38. Ranking the three CMS Options The table below illustrates the final weighted rankings using the functional requirements matrix of each of the three Course Management Systems evaluated. Angel LMS and Sakai ranked even and just below the cut-off threshold of seventy percent (70%) established by the Subcommittee. Blackboard had the highest final rating of the three systems evaluated (83%). The numbers below are not truly scientific but represent a best effort approach to compile and rank numerous disparate types of data and findings (both qualitative and quantitative) in a manner which helped the committee discuss the overall findings of the evaluation process to reach a final conclusion as to which Course Management System best meets the current and near-term needs of faculty and students at Notre Dame. Table 12: CMS Functional Requirements -- Final Rankings Blackboard scored highest on Integratibility with the existing Student Information System (Banner) and for out-of-the-box integration with the existing authentication system. Angel LMS scored slightly higher on extensibility, scalability, flexibility and stability factors. However Notre Dame faculty participating in the pilot courses felt Angel and Sakai displayed significant shortcomings in key tools or functions they felt essential to their use of Course Management System Replacement Project – Findings and Recommendations 38
  • 39. a CMS to support their classes and students. Angel received low ratings by faculty on the Assignments and Gradebook tools; Sakai received low ratings on its discussion tools. However Sakai was given the highest rating based on the future vision of the Sakai development community to develop an open source collaborative learning environment for higher education. The Committee felt that Notre Dame should continue to monitor the development of the Sakai platform over the next three years and its adoption and use by Notre Dame‟s peer institutions of higher education. The illustration below was developed for use in a series of public presentations to faculty across campus to report the CMS subcommittee findings and conclusions. Compared to the existing CMS used at the University of Notre Dame, Sakai offers some improvements in flexibility and high usability ratings by students. Angel when compared to Blackboard Vista offers slighter better flexibility and extensibility. Blackboard offers higher functionality, usability (as rated by faculty), and clearly better integratibility with existing campus systems. Continuing to use Blackboard Vista also offered the option of not making a change to a new system at this time, thus avoiding considerable time and expense to make a full-scale transition with the subsequent impact on the existing faculty and student CMS user community. The CMS subcommittee felt strongly that unless Sakai or Angel offered clear improvements over the existing CMS that the “no change needed” factor was an important consideration to faculty who have invested considerable energy in learning and using Blackboard Vista to support their courses. Course Management System Replacement Project – Findings and Recommendations 39
  • 40. Recommendations The CMS Subcommittee of the University Council for Academic Technologies reached the following recommendations as a result of the comprehensive evaluation process described in this report. The University of Notre Dame continue to use Blackboard Vista as the institutionally supported Course Management System at least through Spring Semester 2011. The University plan to review CMS options again in three years (beginning in 2010). The review process illustrated that each CMS evaluated had strengths and weaknesses and that the course management system “space” is rapidly changing along with the overall online learning system landscape. The committee felt that staying with the current system another three years offered a commitment to faculty that their investment in learning the CMS will not be short-changed by switching in the near future to a new system. The three year period was also influenced by information presented by Blackboard of its intent to develop a “next generation” learning management system which will replace its existing CMS offerings, with a projected rollout of 2010. The Blackboard Vista product line will cease to be upgraded and enhanced once the next generation product is released, thus requiring Notre Dame to switch from Vista to a new product as early as 2011. The University should support a set of usability and functionality enhancements to Blackboard Vista. These enhancements should address as many of the perceived shortcomings of Vista as possible and should be phased in over the next 12 months – ideally in place by the end of Spring Semester 2009 in time for use by faculty and students in the 2009-2010 academic year. The University strategically engage its faculty in an effort to raise awareness of the benefits of using the Course Management System as a means for enhancing learning in and outside the classroom. Course Management System Replacement Project – Findings and Recommendations 40
  • 41. Future Vision for CMS at Notre Dame The vision for CMS development encompasses activities from the present (2008) through the end of academic year 2010-11. The main goal is to improve basic usability and functionality for the five critical CMS functions that were identified by faculty as most often used to support courses. However over the next academic year (2008-9) the subcommittee identified a range of additional functions or tools that could enhance the faculty/student experience of using Concourse. These are identified in the chart above and are not yet prioritized. Longer term goals include creating a more „personalized‟ CMS experience for students and adding structured learning outcomes tools or features that mesh with broader University objectives for certain academic disciplines. In addition to the goals outlined above there are expressed needs by some faculty, departments and colleges to gain better usage data gathering and reporting capabilities, peer learning and self-evaluation support, e-Portfolio options and perhaps more support for alumni lifelong learning. Course Management System Replacement Project – Findings and Recommendations 41