Presentation on the Usability Study performed on a Learning Management System used at Rutgers University.

1. Usability Study Results
Evaluation of Sakai

2. Goals and Methods
 Goals
 Primary: Test ease of using Sakai
 Secondary: Feedback of standard site setup versus
previous experiences with other classes
 Methods
 Create a customized dummy Sakai course/site for user to
navigate and complete a set of defined tasks
 Use SilverbackApp to record video of user and interface

3. Study Design
 Informed Consent
 Demographic and Experience Data
 Execution of Tasks
 Video Recording and Screen Capture via Silverback
 Questions

4. Sakai Test Site

5. Task List
Course-Specific
1. Find a Reading for a class
2. Submitting an assignment
3. Upload a document
4. Looking for recent announcements
5. Reply to Threads
6. Take test
7. Check Grades
System Preferences/Settings
8. Change Tab order
9. Hide a Tab
10. Update user profile
11. Change notification preferences

6. Participant Demographics:
Experienced User
Experienced
Student (Undergraduate; Upperclassman)
Male
Engineering Major
Wants to use Sakai on a regular basis
Advanced computer literacy
More prone to providing suggestions

7. Participant Demographics:
New User
New User
Non-Student
Female
Working Professional
Exploring Sakai for the first time
Average computer literacy
Focusing on completing the
individual tasks

8. Task Ratings
 Each task scored as follows:
 1 = completion, independently
 0.5 = completion, with assistance
 0 = failure

9. Task Ratings:
Individual Participants
Participant # 1 2 3 4 Mean STDEV
1 Find a Reading for a class 1 1 1 1 1.0 0.0
2 Submitting an assignment 1 1 1 1 1.0 0.0
3 Upload a document 0 0.0 0.0
4 Looking for recent
announcements
1 1 1 1.0 0.0
5 Reply to Threads 1 0.5 1 0.8 0.3
6 Take test 1 1 1.0 0.0
7 Check Grades 1 1 1.0 0.0
8 Change Tab order 1 1 1 0.5 0.9 0.3
9 Hide a Tab 1 1 1 1 1.0 0.0
10 Update user profile 0 1 1 0.7 0.6
11 Change notification preferences 1 1 1.0 0.0
Mean 1.0 0.8 0.9 0.9
STDEV 0.0 0.4 0.3 0.2

10. Task Ratings:
By Persona
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Experienced
New User

11. Efficiency
 Time to Complete Tasks
 mm:ss (minutes:seconds)
 Actions Per Task
 number of mouse clicks

12. Time to Complete Tasks:
Individual Participants
Participant # 1 2 3 4 Mean
1 Find a Reading for a class 00:07 00:05 00:26 00:15 00:13
2 Submitting an assignment 00:30 00:40 00:05 02:30 00:56
3 Upload a document 06:00 06:00
4 Looking for recent announcements 00:03 00:15 00:30 00:16
5 Reply to Threads 00:10 01:30 00:50 00:50
6 Take test 01:50 00:45 01:18
7 Check Grades 00:02 00:04 00:03
8 Change Tab order 00:04 00:04 01:22 03:00 01:07
9 Hide a Tab 00:15 00:07 00:21 00:10 00:13
10 Update user profile 00:06 00:45 00:30 00:27
11 Change notification preferences 00:05 01:10 00:38
Mean 00:10 00:11 01:15 01:04

13. Time To Complete Tasks:
By Persona
00:00
00:43
01:26
02:10
02:53
03:36
04:19
05:02
05:46
06:29
Experienced
New User

14. Number of Actions (Clicks):
Individual Participants
Participant # 1 2 3 4 Mean STDEV
1 Find a Reading for a class 4 3 3 3 3.3 0.5
2 Submitting an assignment 9 10 2 3 6.0 4.1
3 Upload a document 17 17.0 0.0
4 Looking for recent
announcements
1 1 1 1.0 0.0
5 Reply to Threads 4 8 3 5.0 2.6
6 Take test 7 3 5.0 2.8
7 Check Grades 1 1 1.0 0.0
8 Change Tab order 4 3 5 15 6.8 5.6
9 Hide a Tab 9 4 3 2 4.5 3.1
10 Update user profile 4 2 3.0 1.4
11 Change notification preferences 4 3 2 3.0 1.0
Mean 4.6 4.8 4.9 3.8
STDEV 3.3 2.9 4.6 4.3

15. Number of Actions (Clicks):
By Persona
0
2
4
6
8
10
12
14
16
18
Experienced
New User

16. Positives
 One-stop shopping for communication on all classes
 Generally self-explanatory functionality
 Customizable content per class/subject

17. Errors

18. Types of Errors
 Making the wrong menu choice
 Taking an incorrect sequence of actions
 Unaware of functionality

19. Errors:
Individual Participants
Participant # 1 2 3 4 Mean STDEV
1 Find a Reading for a class 0 0 0 0 0.0 0.0
2 Submitting an assignment 0 0 0 0 0.0 0.0
3 Upload a document 1 5 3.0 2.8
4 Looking for recent
announcements
0 0 0 0.0 0.0
5 Reply to Threads 0 0 0 0.0 0.0
6 Take test 0 0 0.0 0.0
7 Check Grades 0 0 0.0 0.0
8 Change Tab order 1 0 1 7 2.3 3.2
9 Hide a Tab 0 1 0 1 0.5 0.6
10 Update user profile 1 0 0 0.3 0.6
11 Change notification preferences 0 0 0.0 0.0
Mean 0.1 0.3 0.2 1.4
STDEV 0.4 0.5 0.4 2.7

20. Errors:
By Persona
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Experienced
New User

21. Issues
Impact Frequency Overall
1 Course name conventions on global navigation lead
with course numbers instead of names
Low Med Low
2 “Meetings” functionality was not intuitive to user Med Med Med
3 Could not upload document in Resources using built-
in Sakai function per instructions (Site Resources >
Add > Upload Files)
High Low Med
4 Change order of tabs – user clicked up/down arrows
before selecting a course/tab to change
Low Med Low
5 Change order of tabs – location of Preferences Med Low Low

22. Recommendations:
Sakai Interface
 Rename “Meeting” navigation to “Virtual Classroom” or
other name to more intuitively convey functionality
 Private Message functionality (Group Chat is publicly
viewable by all class/project participants)

23. Recommendations:
Managing Sakai

24. Recommendations:
Managing Sakai
 Increased Faculty engagement / knowledge
 Environmental considerations of online use
 Copyright capabilities
 Increased promotion of existing Sakai training classes
and materials available to Faculty

25. Recommendations:
Further Testing
 Additional Study Participants
 Faculty
 Freshmen (i.e., Sakai-naïve Rutgers students)
 Improve structure / design of task list based on data
analysis and metrics requirements
 Customized task list for each persona

26. Study Team Self-Evaluation
 Level of familiarity impacted ability to give tasks to
participants
 More experienced users would discuss a functionality in
detail before study team administered the task for that
functionality
 Task detail and order impacted quantitative analysis
 E.g., number of clicks could vary between subjects varies
since there was no navigation reset step to “Home” or “My
Workspace” before starting the next task
 Faculty as test participants

27. Questions?