This document discusses methods for evaluating eLearning programs, including formative evaluation during development and usability testing with end users. It describes the Kirkpatrick model for evaluating learning programs at different levels. Formative evaluation seeks to improve a program and ensure it is effective, efficient and meets user needs. Methods include expert reviews, user reviews, usability testing, and alpha/beta testing. Usability testing involves observing representative users performing tasks to evaluate ease of use, speed, errors and satisfaction. Multiple evaluators are better than a single evaluator to find most usability problems.

1. Notes on the Evaluation of eLearning<br />Michael M. Grant, PhD<br />Kirkpatrick (& Phillips) Model<br />Level 1: Reaction — participants’ reactions to the instruction

2. Level 2: Learning — the degree to which learning occurs as a result of the instruction

3. Level 3: Transfers — the transfer of learning to impact on job behavior

4. Level 4: Organizational performance — the impact the learning has on the organization

5. Level 5: ROI — the investment of the training compared to its relative benefits to the organization and/or productivity/revenueFormative Evaluation<br />Overview

6. What’s the purpose?

7. A focus on improvement during development.

8. Level 2 Evaluations

9. Appeal (Level 1: Reaction)

10. Effectiveness

11. Efficiency

12. Data Collection MatrixMethods1. What are the logistical requirements?2. What are user reactions?3. What are trainer reactions?4. What are expert reactions?5. What corrections must be made?6. What enhancements can be made?Anecdotal recordsXXXXXUser questionnairesXXXXUser interviewsXXXXUser focus groupsXXXUsability observationsXXXXOnline data collectionXXExpert reviewsXXX<br />“Vote early and often.”

13. The sooner formative evaluation is conducted during development, the more likely that substantive improvements will be made and costly errors avoided (Reeves & Hedberg, 2003, p. 142).

14. Formative Evaluation Methods (Reeves & Hedberg, 2003)

15. Expert Review

16. User Review

17. Usability Testing

18. Alpha and beta prototypes and field trials

19. Expert Review

20. “Experts are anyone with specialized knowledge that is relevant to the design of your ILE” (Reeves & Hedberg, 2003, p. 145).

21. SMEs, instructional experts, graphic designers, teachers, and trainers

22. Interface Review Guidelines from http://it.coe.uga.edu/~treeves/edit8350/UIRF.html

23. User Review

24. Observations from one-on-ones and small groups

25. We need additional information about how the elearning product is being used and how it will perform.What Is Usability?<br />“The most common user action on a Web site is to flee.” —Edward Tufte

26. “at least 90% of all commercial Web sites are overly difficult to use….the average outcome of Web usability studies is that test users fail when they try to perform a test task on the Web. Thus, when you try something new on the Web, the expected outcome is failure.” —Jakob Nielsen

27. Nielsen’s Web Usability Rules

28. Visibility of system status

29. Match between system and real world

30. User control and freedom

31. Consistency and standards

32. Error prevention

33. Recognition rather than recall

34. Flexibility and efficiency of use

35. Help users recognize, diagnose, and recover from errors

36. Help and documentation

37. Aesthetic and minimalist design

38. Usability.gov

39. Ease of learning - How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?

40. Efficiency of use - Once an experienced user has learned to use the system, how fast can he or she accomplish tasks?

41. Memorability - If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?

42. Error frequency and severity - How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors?

43. Subjective satisfaction - How much does the user like using the system?

44. Two Major Methods to Evaluate Usability

45. Heuristic Evaluation Process

46. Several experts individually compare a product to a set of usability heuristics

47. Violations of the heuristics are evaluated for their severity and extent suggested solutions

48. At a group meeting, violation reports are categorized and assigned

49. average severity ratings, extents, heuristics violated, description of opportunity for improvement

50. Advantages of Heuristic Evaluation

51. Quick: Do not need to find or schedule users

52. Easy to review problem areas many times

53. Inexpensive: No fancy equipment

54. Disadvantages of Heuristic Evaluation

55. Validity: No users involved

56. Finds fewer problems (40-60% less??)

57. Getting good experts

58. Building consensus with experts

59. User Testing

60. People whose characteristics (or profiles) match those of the Web site’s target audience perform a sequence of typical tasks using the site.

61. Examines:

62. Ease of learning

63. Speed of task performance

64. Error rates

65. User satisfaction

66. User retention over time

67. Elements of User Testing

68. Define target users

69. Have users perform representative tasks

70. Observe users

71. Report results

72. Why Multiple Evaluators?

73. Single evaluator achieves poor results

74. Only finds about 35% of usability problems

75. 5 evaluators find more than 75%

76. Reporting User Testing

77. Overall goals/objectives

78. Methodology

79. Target profile

80. Testing outline with test script

81. Specific task list to perform

82. Data analysis & results

83. Recommendations

84. Recent Methods for User Testing

85. CrazyEgg

86. Userfly

87. fivesecondtest

88. 10 Second Usability Test

89. Disable stylesheets

90. Check for the following: Semantic markup, logical organization & only images related to content appearAlpha, Beta & Field Testing<br />Akin to prototypingReferences & Acknolwedgements<br />American Society for Training & Development. (2009). The value of evaluation: Making training evaluations more effective. Author. <br />Nielsen, J. (2000, March 19). Why you only need to test with 5 users. Jakob Nielsen’s Alertbox. Retrieved from http://www.useit.com/alertbox/20000319.html<br />Reeves, T.C. (2004, December 9). Design research for advancing the integration of digital technologies into teaching and learning: Developing and evaluating educational interventions. Paper presented to the Columbia Center for New Media Teaching and Learning, New York, NY. Available at http://ccnmtl.columbia.edu/seminars/reeves/CCNMTLFormative.ppt<br />Reeves, T.C. & Hedberg, J.C. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.<br />