More Related Content
Similar to ICBL Evaluation
ICBL Evaluation
- 1. Course evaluation inblended courses Matt Russell, Ph.D. & Gerald Bergtrom, Ph.D. Learning Technology Consultants Learning Technology Center University of Wisconsin-Milwaukee
- 2. Assessing Student LearningDetermining the value or quality of a student’s work Evaluating Your Course Determining the worth or effectiveness of a course design or teaching What is the difference between assessment and evaluation?
- 3. Why is evaluation particularly important for hybrid and online courses? The experimental mood: hybrid and online courses are different than face-to-face Ideally, “experiment” ensures that we come as novices to the hybrid or online course structure Pedagogical experiment argues that we demonstrate academic rigor in the online environment Progressive evaluation permits making changes throughout course – before, during and after the course is offered
- 4. What tools canbe used to evaluate a hybrid or online course? Evaluation checklists For an online course For a hybrid course Evaluation can involve yourself, colleagues, or students
- 5. What do wewant to evaluate? Learner Support Course Organization and Design Instructional Design and Delivery Integration of Face-to-Face and Online Activities (hybrid only) Student Assessment Student Feedback
- 6. Learner support Nota significant issue in traditional face-to-face courses Student self-assessment: is s/he likely to succeed as an online or hybrid learner? Acquiring the technical skills and requisites Knowing what to do when troubles arise
- 7. Course organization anddesign A basic syllabus affords a contract between instructor and students The use of modules to organize course activity is more pronounced in online and hybrid courses The course Web site is a visual representation of the learning goals and activities
- 8. Instructional design anddelivery A relationship between learning objectives and learning activities A progression towards critical thinking Ongoing efforts to develop an online learning community of peers
- 9. Integration of face-to-faceand online work (hybrid only) If course redesign is not completely thought through, there is a tendency to favor the face-to-face over the online. Running two modes of instruction parallel and independently is a sure recipe for the course-and-a-half syndrome Each form of learning must affect -- extend, elaborate, intensify – the other
- 10. Student assessment Theonline environment lends itself to frequent, low-stakes assessment with ample feedback Traditional forms of assessment offer little information about the learning taking (or not taking) place Rubrics help both instructor and student apply abstract knowledge to disciplinary practice
- 11. Student feedback Likestudent assessment: frequent, low-stakes, and information-rich The simple “reality check” is an extremely valuable tool The students find their voices within the course The community of learners benefits from a give and take between instructor and students
- 12. Conclusion Why isevaluation integral to hybrid and online courses? A variety of evaluation tools: http://LTC.uwm.edu/resources.html
Editor's Notes
- #2 In a traditional face-to-face course, evaluations occur at exactly the point when they are guaranteed to be almost entirely useless – at the end of the semester, when it’s too late for the instructor to respond to them until s/he teaches the course again! The perversity of this timing is underscored when instructors have incorporated an extensive use of online learning into the design of their hybrid or fully online courses. I say this because experience tells us that it is precisely this environment that lends itself to much more extensive documentation of learning and more rapid paced feedback to students, so the instructor knows far earlier whether students are “getting it.” When you take this environment into account together with the likelihood that hybrid and online courses enjoy greater design flexibility because of the expanded repertoire of possible modes of content delivery and types of learning activities, it’s easy to understand why course evaluation should not wait until the class is over. There is a very real advantage to adjusting the course along the way in a well-informed and timely fashion.
- #3 I want to take a moment to make an important distinction: although the terms “assessment” and “evaluation” are often taken as synonymous in everyday speaking, the scholarly literature of course design uses the term “assessment” to refer specifically to grading students, while the term “evaluation” is used to refer to the systematic critique of a course or a program. In this presentation, we follow this convention; hence, we are speaking of course evaluation here.
- #4 I think it’s important to understand why evaluation is especially important when you teach hybrid or online courses. My sense of one of the key values of teaching hybrid and online courses is that I do it because it allows me to teach in a mood of experimentation. I don’t mean by this notion of “experimentation” that the value of hybrid and online courses has not been established by scholarly inquiry – quite the contrary – nor do I mean that I am employing methods of teaching that are untried and therefore somewhat suspect. What I do mean is that by challenging the boundaries and limitations of the traditional face-to-face classroom, I have made it possible to think how teaching might be “otherwise” – other than merely the dominant voice of the instructor or the canonical text. Having thought beyond the walls of the classroom is an invitation to try things that you couldn’t do before, to see whether they fit well with your teaching and whether they actively engage the students in your discipline. This notion of experimentation partially accounts, in my view, for the highly positive feelings that both instructors and students take away from online or hybrid courses. There is a sense of freedom fostered by the new learning environment that invites a new appreciation of teaching and learning. All that said, experiment has its costs, and should not be undertaken lightly or introduced unsystematically. But since you are doing something new and different, and your students are likewise doing something new and different, evaluation matters in a much more urgent way than in a traditional face-to-face course. Moreover, as would any experiment, it’s critical to be able to demonstrate both to yourself and to your colleagues the rigor of the work accomplished. Since the documentation of learning in a hybrid or online course is more comprehensive, the data are already at hand. What’s important is to focus attention on those data through evaluation, and place them in context. All of this argues, finally, that evaluation can and should occur at every point in the course cycle, not just at the end of a course offering. Finding ways to test a course design before the course is offered allows you to fix things that seem likely to become problematic. Even more to the point, seeking and taking seriously the feedback that you can obtain while the course is being taught encourages the sort of fine-tuning that can address unforeseen issues by allowing you to make course corrections along the way.
- #5 More to the point, perhaps, we have used existing evaluation tools (with appropriate attribution) to assemble two evaluation checklists, one for online courses and one for hybrid courses. The two checklists are very similar to one another, with one significant difference that I mention later on. The checklist is more substantial than an ordinary course evaluation, but that is meant to ensure both comprehensiveness and flexibility. The comprehensiveness of the checklist means that you will never find yourself at a loss to identify any missing component of your course design – it’s all right here on the checklist. The flexibility means that you can select particular questions for evaluation at various stages of the course offering where they seem most pertinent. You can also choose specific questions or sections that invite collegial review rather than student evaluation (or vice versa), or even that lend themselves to personal reflection. One typical use of portions of the checklist is to produce a “reality check” about a third of the way through the semester: ask your students to evaluate their course experience to date, then respond by noting which areas indicated in the checklist require further development. Another simple suggestion that is often overlooked is for instructors to keep a self-reflection diary – sort of a course design blog – during the course. If you focus your reflections on one or more features of the checklist each time, you will soon compile a valuable record that you can consult to guide your “experiment” in its subsequent stages, as you redesign your course a little each time it is offered.
- #6 I’m not going to present in detail each item in the checklist. Most of the items are self-explanatory, particularly if you have encountered our other modules in this series. I do wish to speak briefly about each of the main categories, however, since each illustrates in its own way the special character of online and hybrid instruction. You see here the list of items that I address below.
- #7 It’s entirely characteristic of online and hybrid courses that “learner support” takes pride of place in an instructor’s task of course redesign. No one ever says – though perhaps they should! – what kinds of special learner support do you provide your students in a traditional face-to-face course? The traditional course format is the privileged mode of instruction to which we are all socialized from a very early age, so it’s simply assumed that “good” students will know what to do to get by. But the online or hybrid course requires a different approach. First, there are a number of self-assessment tools online that students can complete to find out whether they are likely to succeed in such courses. Readiness for online work is by no means assured, since it involves a willingness on the part of the student to be more active and accountable. Second, students must either possess the necessary technical skills and equipment or be willing to acquire them to handle the course work. Again, the initial responsibility for stating plainly what is required lies with the instructor; follow-up, naturally, falls to the student. For instance, orientation to the local course management system is always a sine qua non of this support; so are simple one-pagers written by the instructor or by the IT support unit, that describe how to perform a particular task online. Students must be aware of their need to download and install plugins for specific programs that will be used during the course, or even to purchase a specific course-related program such as SPSS or Photoshop. Finally, making certain that students know what help resources are available extends their social network when difficulties arise. None of these is a desideratum of an ordinary face-to-face course, so the very first category of the evaluation checklist reminds us that we are experimenting with a new dimension of instruction where different needs are made visible, and resolution of those needs is available if the instructor is alerted to them well in advance.
- #8 In one sense, course organization is not a “new” category for experienced teachers – everyone knows that you have to have a course syllabus to act as a contract between yourself and your students, and these days most academic units provide templates for what must be included in a syllabus. So this is no mystery, though I confess that I’m sometimes surprised at the extent to which many instructors, both traditional and otherwise, find themselves in the early weeks of a semester with a syllabus still in flux! I’m not quite so much of an experimenter in that respect. There are two respects in which online and hybrid courses differ from more traditional teaching, however. First is the explicit use of the module concept, which aligns particular course objectives with “chunks” of content, learning activities, and rapid assessment and feedback. In my experience, it’s not typical for instructors to approach their courses in quite so systematic a fashion; most tend to begin with an intuition of what content would be appropriate, then insert assessments at appropriate moments. Our suggested “backwards design” approach really argues that instructors adopt an assessment-centered design that will then be thoroughly elaborated in their syllabus. This results in its turn to an attention to syllabus construction that is more a literary genre than a contract! The other difference between traditional course design and that of the hybrid or online method of delivery is the development of an artifact, the course Web site. This artifact serves multiple purposes, and responds to a number of evaluative criteria. For instance, it documents many of the learning activities of the course and their corresponding assessments of student work. It incorporates a range of content, from PowerPoint to video to Web links, and as such, may present the “main business” of the course in a variety of ways. And finally, the course Web site is sensitive to design criteria that make it accessible and usable to students. In this last regard, the Web site needs to be consistent and easily navigated, relying upon just enough redundancy to ensure that students understand what they are to do now, and then what comes next.
- #9 At first glance, the idea of instructional design and delivery is scarcely a sea change from the traditional face-to-face course. Even if we add “backwards design” to the mix, we are merely emphasizing what has always been latent in standard real-time courses: there ought to be learning objectives associated with the course, and the learning activities that are affiliated with those objectives are to be made utterly transparent to even the most dilatory and inattentive student. But there is more here, as a component of online or hybrid courses, than at first seems apparent. For instance, the progression of learning tasks from most ordinary to most sophisticated – typically identified by a phrase such as “critical thinking” – arises naturally from the manner in which active learning proceeds in the online environment. You begin with description and straightforward application, to produce a critical mass of understanding that has both pedagogical and technical implications. You gradually raise the bar as rapidly as your students can accommodate, teaching them, through feedback, to become more analytic. For instance, I begin each course with a requirement that students introduce themselves online, partly as an inducement to self-disclosure and in part to examine the limit of their technical skills. They must routinely produce examples from their own cultural experience that match the theoretical perspectives at hand. By the end of the course the students should be capable of sustaining an argument of high quality that encompasses both theoretical and ethnographic goals. And perhaps most demanding of all, online and hybrid courses require the fostering of an online community of active peer learning that is every bit the counterpart – preferably, more fully so – of the face-to-face community that forms itself in the face-to-face classroom. The online peer community is not a simple cumulative by-product of individual learning activities. Instead, an instructor must take positive steps to nurture it and to ensure its growth over the semester. Although the learning community may take different forms – discussion forums and small group projects are prime examples – they anticipate an instructor’s ability to proceed from the individualistic and privatized tenor of much traditional classroom instruction to the more challenging public space of online discussion and peer learning activities. In this sense, a hybrid or online course runs counter to normal expectations of how students should learn in the traditional classroom, and this dimension of the checklist reminds the instructor of the “art that conceals art” which renders a community of learners both visible and sustained.
- #10 In the hybrid mode of instruction, there are two modes of learning operating simultaneously: the standard face-to-face and the online. If these two run more or less independently in parallel to one another, the course is not an integral whole, but is merely the sum of its parts. The consequences of this are well understood, I believe. First, both students and instructor are likely to treat the online work as “busy-work” or as “icing on the cake,” to be completed but not to be taken seriously as part of the scholarly enterprise. Second, because the face-to-face portion of the course tends to be treated as of greater significance, the instructor is likely to retain most of the learning activities and assessments characteristic of the traditional course, while adding several online activities on top. The result, as we have mentioned many times, is the course-and-a-half syndrome, in which both instructor and students suffer burnout because they are trying to do more than a single course’ worth of work. A thorough hybrid course redesign involves setting aside the traditional modes of instruction and assessment native to a fully face-to-face course to use online work with far greater effectiveness. This is why, in large part, we argue that a modular redesign is desirable: it removes instructors from their familiar classroom habits sufficiently that they can incorporate online work more fully. In the hybrid, each course finds its own rhythm between face-to-face and online modes of instruction, but the goal is always the same. What is done face-to-face affects what happens online, and in its turn what is done online returns to the face-to-face classroom. I think one of the most surprising discoveries for a first-time hybrid instructor is that the face-to-face and online work function differently, and thereby accomplish quite different – though complementary – course goals. The trick is to make this work for you rather than against you! Again, I see the process of integration as experimental. You have to find a way to create a learning community both online and face-to-face, then make them work together. That is something that only happens through trial and error. Finally, even though I’ve said that this portion of the checklist applies only to the hybrid – and technically, that must be correct – in fact there is an important lesson here for those who are teaching fully online. Online work can be valuable or ineffective in precisely the same way that face-to-face work can meet the learning goals of the course, or fail to do so. Neither mode is transparent in itself, since any form of instruction requires a meticulous alignment of learning goals, content, and assessment that has been assembled and closely examined by the instructor to ensure quality. The online world makes available many opportunities for learning, but there is nothing magical there. The online work of the course, whether fully online or hybrid, has to be carefully wrought in order to realize its full value.
- #11 Assessment in online and hybrid courses follows well-known rules of thumb that are implicit in the checklist. Low-stakes, frequent assessment with ample feedback is essential to student learning in the online environment, and gives the instructor a clear warning if specific students need additional help, or if the course material needs further elaboration. By contrast, the notion of three tests and a term paper is characteristic of the more static, regimented face-to-face classroom, where instructors know relatively little about the quality and extent of the actual learning that is taking place other than at specific “catastrophic” moments when students are subjected to sudden, intense scrutiny with limited opportunities for amelioration. I’m often surprised by the confidence with which instructors insist that their lectures are well-wrought and thoroughly understood by “the better students” in the class. I think that the ability to distinguish “the better students” from everyone else, or even the desire to identify an elite of learners who are privileged by aheavy doses of lecture and examination, is simply an instance of the fallacy of misplaced concreteness. The checklist emphasizes the use of rubrics to lay out the instructor’s expectations and guide the students’ understanding of what counts as excellent work (or for that matter, work of any level of quality). The importance of rubrics in online or hybrid instruction brings to the fore the idea that it is insufficient merely to convey information to students for them to learn in a course. That information must be placed in a context where it becomes meaningful. In this sense, a rubric becomes an elaborated statement of appropriate disciplinary context. From an instructor’s point of view, the rubric is a way to locate expectations in scholarship pertinent to a field of study; from a student’s point of view, the rubric is a way to learn how to apply accrued information most effectively in a particular course. Remember that “backwards design” tells us that we are better off framing course objectives in terms of what a student can do , not what a student knows. Rubrics allow both instructor and student to cut to the bone of disciplinary practice.
- #12 Student feedback, like student assessment, should be frequent, low-stakes, and information-rich. I could simply repeat the remarks made at the start of this presentation – or better yet, compare the traditional “three exams and a term paper” assessment to the “summative” evaluation. Summative course evaluations, like final exams, are catastrophic moments in which little of value is accomplished, though precise statistics may be generated in the process (another instance of the fallacy of misplaced concreteness). I think that the main point here is that the entire purpose of a hybrid or an online course is to move from the single voice of the instructor and the text to allow students to develop their own voices in the classroom, whether face-to-face or online. Until students find their voices, they cannot effectively take responsibility for their learning. One way in which students do speak is through the kinds of student feedback that have been suggested, such as the early semester “reality check” and the regular use of Classroom Assessment Techniques (CATs) at the end of a class or a module. One way in which I have interpreted the role of student feedback is in terms of becoming a good citizen within the course framework (and after all, isn’t citizenship typically identified as a goal of a liberal education?). As citizens, students participate, they “vote” by making valid claims on the attention of the instructor, they labor in the service of the course when they complete assigned learning activities. While I don’t want to press too hard this notion of the community of learners as commonwealth, I do think it is worth considering as one of many pedagogical impulses underlying the operation of a hybrid or an online course. Instructors should ask for feedback throughout the course, and when given feedback that deserves to be taken seriously, acknowledge as much to the students. I typically will at least ask ‘what is one thing you like about the course?’ ‘what is one thing you’d like to fix?’ and ‘what is something that I need to know about this course?’ These simple prompts produce a great deal of information for me to consider. Naturally, not all of it is useful; students are apprentices for good reason. But there are always things that give me direction, that suggest a need for change, that help me appreciate what is going on in the learning experience that I need to know about. I summarize my students’ comments and post them on the course Web site, together with an indication of what I will fix, or if necessary, what must remain unaltered. I think, in short, that student feedback is a significant part of the normal give and take that should be part of the dialog in an online or a hybrid course.
- #13 My discussion of course evaluation has focused mainly on the role that students can play in providing useful information to the instructor of an online or hybrid course. I think that this emphasis is entirely correct, since evaluation is a key aspect of active learning. If I may return to the image of hybrid or online course as an experiment in learning, I would like to note, in conclusion, that pedagogical experiments are most successful when evaluated by the very persons whose learning they are meant to improve. Evaluation is integral, not incidental to the nature of hybrid and online learning. These few remarks have stressed the conditions that are necessary for a consistent, helpful flow of information between instructor and students: the availability of support resources, the design of a syllabus and Web site, the gradual engagement of students with critical thinking, the use of feedback as a tool to empower students, not disenfranchise them, the development of mutual respect for a common goal through the provision of regular and systematic opportunities for progressive evaluation. All of these are reflected in our checklist because we think that they are fundamental to good course redesign. Our Web site provides a range of evaluation tools so that people can see what’s “out there”.