2. 12/23/10
2012 ECP Submission Timeline
Submission window: December 1, 2011 – March 2, 2012
Directors
Reviews Meet to Anonymous
Conducted Discuss Reviews
March19 – Reviews: Sent: Mid
April 6 April 10-11 May
Review Winners
Deadline: Notified:
April 6th April 23
2
3. 12/23/10
Overview of the ECP Rubric
• The ECP Rubric addresses four areas that are
considered by this program to be essential in
effective, high-quality online instruction:
– Course Design
– Interaction & Collaboration
– Assessment
– Learner Support
3
4. 12/23/10
1. Course Design
Course design addresses
instructional design, and the
guidelines
for this program reference the
elements of:
• Goals and objectives
• Content presentation
• Learner engagement
• Technology use
4
5. 12/23/10
2. Interaction and Collaboration
Interaction denotes communication between and among
learners and instructors, synchronously or asynchronously.
Collaboration is a subset of interaction and refers
specifically to those activities in which groups of students
are working toward a shared result.
Special Note to Reviewers
Please be sure to look for evidence of interaction and
collaboration in special files or supplemental resources.
Submitters may have provided special files and resources in
order to hide the identity of students for privacy purposes. 5
6. 12/23/10
3. Assessment
Assessment focuses on
instructional activities designed
to measure progress
towards learning outcomes,
provide feedback to students
and instructor, and/or
enable the assigning of a
grade.
6
7. 12/23/10
4. Learner Support
Learner Support addresses the support resources
made available to students taking the course. Such
resources may be accessible within or external to the
course environment.
Learner support resources should address the variety
of topics that are appropriate for learner success in
the course. The topics will vary depending on
resources that are available and the uniqueness of
each course.
7
9. What’s the difference between 12/23/10
Exemplary and Accomplished?
Exemplary Accomplished
Goals and objectives are easily Goals and objectives are
Goals and located within the course at all located within the course
Objectives appropriate places such as syllabus or the individual
syllabus and Individual learning learning units;
units;
are clearly written at the objectives are written to reflect
appropriate level and reflect desired learning outcomes,
desired outcomes;
are written in measureable although not all are written as
outcomes (students know what measureable outcomes;
they are expected to be able to
do),
9
10. 12/23/10
A Note on Visual Appeal of Courses
• There is no area of the
rubric focused on course
look-and-feel or on how
“pretty” the course
design is
• Visual appeal is very
personal
• Provide constructive
feedback on the course
look-and-feel as you
wish
• Do not let visual appeal
cloud your judgment or
interfere with your review
10
11. 12/23/10
What Makes a Review Successful?
Completeness:
– Thoroughly complete Parts 1 and 2 of the Reviewer Form.
Validation:
– Carefully review the comments and examples of exemplary practices
presented in the narrative components on the submitter’s form to
determine whether or not you agree.
– Base your review on the detailed criteria of the ECP Rubric.
Constructive Feedback:
– The ECP program is based on a premise that participation in the
program as a reviewer or a submitter will enable individuals in
both groups to get better at creating and delivering effective online
instruction.
– This positive theme should be foremost in the minds of reviewers
and should be evident in the feedback you prepare for course 11
submitters.
12. 12/23/10
Examples of Feedback
Feedback that isn’t very constructive:
Too much text on the page. I didn’t like to scroll.
12
13. 12/23/10
Examples of Feedback
Positive, constructive feedback:
It is great that you provide supplementary reading materials within the
course itself! This is a real time-saver for students.
The materials may be easier for students to read and recall if the items
were organized by theme or in chronological order and placed into
dedicated folders or subfolders rather than listed on one page. While
having all of the information on one page makes it easy for people to print
out, it makes for an awfully long page to scroll through. In addition,
because of some of the large images, the page takes a long time to load.
This might be a problem for students with slow internet/dial-up
connections.
13
14. 12/23/10
Examples of Feedback
Feedback that isn’t very constructive:
Learning objectives are not placed in appropriate places.
Your learning objectives are not good.
14
15. 12/23/10
Examples of Feedback
Positive, constructive feedback:
Although your learning objectives (LO’s) are included in the course
syllabus, it would be helpful to students to relate specific LO’s to
course activities and assignments.
The learning objectives (LO’s) could be improved if written in
measureable terms. For example, in the LO’s below, the word
“know” is vague and isn’t measureable. However, the action verb
“write” is measureable.
Not measureable: The student will know the Spanish alphabet.
Measureable: The student will be able to write the Spanish alphabet.
15
16. 12/23/10
The Reviewer’s Responsibilities – Part 1
On the Reviewer Form:
Rate each of the rubric components by selecting from
the drop-down
Provide feedback on each of the following four sections:
• Course Design
• Interaction and Collaboration
• Assessment
• Learner Support
16
17. 12/23/10
The Reviewer’s Responsibilities – Part 2
• On the Reviewer Form enter:
– Your overall rating for the
course (pull-down)
– Your summary to describe
your overall impressions of
the course.
– Your suggestions for course
improvements.
– A description of the top three
“stand-out practices” in the
course.
• These practices are
those that you think are
worthy of being shared
with the broader
educational community.
17
18. 12/23/10
Reviewer Checklist
Skim the entire submission.
Note initial “gut” reactions
Read the submission.
Review your initial reactions and prepare a response
Include the following in your response:
Complete sentences
Begin and end with a positive statement
Include suggestions for improving or enhancing the course
18
20. 12/23/10
Agenda
• Steps for reviewing and
rating courses
20
21. 12/23/10
Overview of the ECP Submission Form
• There is an online ECP
Submission form for each
course
• This form is a key resource
Reviewers use in their
evaluation of the course.
• The Submission form:
– Sets context for the course
– Provides evidence as to how
the course meets each
requirement in the ECP
Rubric
– Contains course login
information
21
22. 12/23/10
New ECP Reviewer Form
• Each course reviewed must have a completed
reviewer form
– If you are reviewing as a team please submit one review
with all your notes consolidated in it
• The new online form contains:
– The ECP Submission form
– Reviewer Information
– Part 1: Feedback using the Rubric:
• Course Design
• Interaction & Collaboration
• Assessment
• Learner Support
– Part 2: Overall Feedback
22
23. 12/23/10
Ready to Review?
1. On March 19th you will receive your course assignments from
Sheryn Anthes. Be on the lookout and check your spam filter
2. Create a login and find your assigned courses. Access the
online Reviewer Form and review the instructions
3. Evaluate your assigned course (if you have trouble logging in,
please email us ASAP)
4. Save your feedback, constructive comments, and
evaluation information in the Reviewer Form
5. Deadline to complete reviews: April 6th
23
30. 12/23/10
Help – I’ve got questions!
• I can’t login to the course! Email us at
community@blackboard.com
immediately.
• Any other questions - Email us at
community@blackboard.com
30
Here is the high-level timeline of the Review Process.You will be sent an email on March 19th with your course assignments. You have 3 weeks to review the courses that have been assigned to you.The deadline is April 6th. You will be sent weekly reminders.In Mid April, the ECP Directors will meet to finalize the list of ECP winners.Winners will be notified later in April. Late May, reviews are sent back to the Course Owners.You will be sent a Certificate of Appreciation for your efforts as a course reviewer. Please be sure to fill out your address in the Reviewer details of the Reviewer form.
Let’s talk about the Exemplary Course Program Rubric. The rubric addresses four areas of a course that are essential for effective, high-quality online instruction. We’ll cover each of these in more detail in the following slides. Course design Interaction and Collaboration Assessment Learner Support
Course Design covers the goals and objectives of the course – are they clearly stated for the students? Content presentation – is it well organized and straightforward to find and progress through the materials?Learner engagement – is the course designed to be engaging and appealing?Technology use – online learning can take advantage of a variety of technologies, does the course use these technologies appropriately and effectively?
Interaction and collaboration can be a significant portion of as well as a critical driver of the learning experience whether it happens between and among learners or between students and their instructor. Interaction can occur synchronously or asynchronously. The rubric asks you as reviewers to determine whether there is sufficient and effective interaction within the online course.Collaboration is a subset of interaction and refers specifically to those activities in which groups of students are working towards a shared result. This may occur as a group project, team wiki page, video presentation, to name just a few examples. Because there are so many different ways interaction and collaboration especially can be incorporated into a course it is very important for you to look for evidence of it. Oftentimes this evidence is submitted as special files or supplemental resources. In addition, student privacy concerns often mean that evidence of student collaboration has been anonymized and provided in a location separate from the course or included within the course in a specially marked folder. This information will be provided to you within your reviewer assignment email.
The Assessment portion of the ECP Rubric focuses on examining instructional activities designed to measure progress towards the learning outcomes. Assessments also give instructors the opportunity to provide feedback to the student and enable the assigning of a grade. Assessments can be in many forms so please refer to the course submission form for evidence that the course submitter specifically called out as examples of Assessment.
The Learner Support portion of the rubric addresses the support resources made available to students taking the course. These resources may be included within the online course itself, or linked to from the course, for example a link to the student writing center or library. The types of resources available to the student will vary depending upon the topic taught and the technologies incorporated into the course.
Each area of the Rubric has a set of sub-areas. In our example the Course Design area has four sub-areas. Each of those sub-areas has a detailed definition of the level of achievement - Exemplary, Accomplished, Promising or Incomplete. Download the ECP Rubric from the website to see the detailed definitions for each area and sub-area. You will use these detailed definitions as you review the course. Let’s go through a quick example.
Here is the actual text from the ECP rubric for Course Design: Goals and Objectives. We’re just looking at the definitions for Exemplary and Accomplished to give you an idea of how the definitions of achievement become more detailed and demonstrate greater mastery as one moves from Incomplete to Promising to Accomplished to Exemplary.Let’s start with the first sentence in the Exemplary column. Now compare it to the text in the Accomplished column. Do you see the differences between the two? To achieve an exemplary rating the bar is set higher for the course designer. An exemplary course will have goals and objectives that are easily located within the course at all appropriate places, whereas an Accomplished course will just have the goals and objectives somewhere within the course syllabus or individual learning units.If we walk through the sentences below you will see similar differences between the definitions in the two columns. These differences may be subtle. You will need to familiarize yourself with the rubric in advance of your review and keep it close to you as you are reviewing the course so that you can effectively determine how to rate each aspect of the course using the rubric.
This is a screenshot of a 2009 Exemplary Course Award Winner. Not all ECP Exemplary Courses look like they were designed by experts in Photoshop. Do not let your reaction to the colors selected or icons used sway your review of the course based on the elements of the ECP Rubric. There is no area of the rubric focused on how “pretty” the course design is or how cool the images are. Visual appeal and design sense is very personal. The ECP Directors deliberately left out of the ECP Rubric an evaluation of the prettiness of the course. Instead, we want you to focus on the design of the course from the ability to navigate through the course content, how the course design aids in content organization, and so forth. That said, if you want to provide constructive feedback to the course creator as part of your overall feedback please do so.
What makes a review a successful one? Three things:Completeness: There are two parts to the Reviewer form – your feedback on each section of the rubric and then your overall feedback on the course Validation: The course submitters will have provided their evidence or examples of what makes their course exemplary as part of their course submission. It is important for you to review these items and decide whether you or not you agree with them. You should base your review of their course using the detailed criteria of the ECP rubric. We’ll cover this in more detail shortly. Constructive Feedback: This is one of the most important aspects of the Exemplary Course Program. The Exemplary Course Program is based on the premise that participation in the program – whether as a reviewer or course submitter, will enable participants to get better at creating and delivering effective online instruction. Constructive feedback is at the heart of this. Please be positive and constructive as you provide feedback to the course submitters.
If there is one thing to stress here is that your feedback is the most critical element of the review. It provides the justification for your ratings, but most importantly it provides the submitters with necessary information on the types of improvements they need to consider to move their courses to an exemplary standard. Submitters will know where to focus their efforts. The above comment really doesn’t tell us anything. It is framed from a negative viewpoint. The same comments can be made much more positively – keeping comments collegial and tactful – by clarifying with suggestions and examples such as chunking the text into sections, having separators using color coded headings, even the use of page templates to provide consistency. It is easy to be critical – it takes more thought to be constructive.
Notice the difference in this feedback – starting with a positive comment and then offering suggestions on how to improve. Having it presented in such a manner provided a benefit to the learners – but here are some other suggestions that would provide even more benefits…. so it’s constructive, it’s positive, and here’s why.
Again, these statements are quite vague – what do you mean by “objectives are not good” or “appropriate places”. These statements lead us only to assumptions – more definitive comments need to be included. In a case such as this, you might want to start out by saying, “while the learning objectives were readily available in the course syllabus…then lead into the suggestions. You also need to keep in mind that not all faculty are instructional designers – and not all of the courses submitted were created by instructional design teams – there are still some lone rangers out there. That’s why sometimes having a team consisting of an instructional designer, faculty expert, etc. is a good approach to the review process. Let’s see what the next slide reads….
One of the best practices we have noted over the years is the placement of learning objectives in areas other than just the syllabus – allowing for Making them available on content module pages – within assessment activities, etc. as noted above. These types of strategies help students relate the objectives to the activities required of them. And measurable objectives also helps align student expectations – helping them understand what they are going to be required to do. The next comment clarifies the improvements needed in the learning objectives. And an example provided. In this case the reviewer re-wrote an objective as an example.
You will be required to download the reviewer’s form from the website Jan will provide more instruction on that later. The next couple of slides relate what your responsibilities are with respect to completing that form. One form for each course you review. If you do a team review – then all comments from the individual members of the team need to be compiled into one review form. Select the rating for each component in the rubric. In the example here you are rating the Goals and Objectives section based on your observation and notes you’ve made. At the end of each section, you will be required to enter overall comments on the section summarizing your observation, positively making comments and offering further suggestions where necessary.
Review and reflect on your ratings and comments for each section – this will help you determine your overall rating. If the majority of your ratings are less than exemplary, then it is unlikely you would give an overall rating of exemplary to the course. Provide your overall concluding comments about the course. Reiterate what the submitter needs to take away – the good and the weak areas noted. Remember – positive and constructive. Then add your overall suggestions for how the course (or various areas) can be improved. Look at the areas you’ve noted as exemplary – they will help inform your best practices for the course. Most courses will have at least three areas that could be considered as being well done. This is the area where you have a chance to praise what is done well that might be worth sharing with the online community. So, consider what was done really well. It does not necessarily mean it will make the final list of best practices once compared to others.
Carefully review the submitters form what have they said about their course highlight areas you want to validate especially those noted as exemplary by the submitter. But also highlight anything that stands out – whether it’s a design element or a certain adaption of a tool, or certain piece of technology, etc. Note the course level teaching methods varies based on level of course blended delivery may affect level of communication/collaboration the choice of tools and how used will be impacted depending on totally online vs blended knowing this information will influence your choices. Remember: you are essentially a student in the course how oriented do you feel so taking some time to become acquainted with the course prior to start reviewing is important. Can make some initial jot notes during this process. Some courses will take less time – most will take 3 to 4 hours to complete to do a good constructive job. You may need less time as you become more familiar with the rubric and you’ve developed a system for yourself. Sometimes you need to leave a course and come back to it – especially if you are struggling with some of the ratings. You will likely find you need to look at a course a couple of times.
Each course submitted to the ECP program has a completed nomination form. This form contains their contact information, the ECP Rubric with their notes as to how their course meets each part of the rubric. We’ll take a moment to walk through this rubric together.[switch to live demo of rubric]
For each course you review you must complete the reviewer form. If you are reviewing courses as a team, please compile your individual reviews into one review form. We want only one review form per course.[switch to live demo of reviewer form]
Enter your login information.Select the submission you would like to review by pressing "Review" and press "Add Review" to complete the review form.Any attachments will appear next to the "Review" tab. They appear in order of the upload field on the submission form.Press "Save" to submit the review. You can not save drafts.You can then move to the next submission and repeat the process. "Done" will appear next to the submissions you have already reviewed.