This document discusses measuring program objectives for a new Master's in Marketing degree via online assessment. It outlines creating program objectives and mapping them to two direct and two indirect measures. Direct measures include entrance and exit exams, with a goal of 80% of students scoring over 73%. Indirect measures include mid-term and end-of-program surveys, with a goal of 80% of students agreeing or strongly agreeing that they can apply program learning outcomes. The document then discusses using case analysis as one direct measure, outlining four cases and central questions. It provides results from the first exam that were skewed and adjustments made going forward. Finally, it covers setting up the case analysis in an LMS like Blackboard, including why cases
1. Measuring Program Objectives
Via Online Assessment
Julia Cronin-Gilmore, Ph.D
Tracy Gies
Bellevue University
International Assembly for Collegiate Business Education
9. Create Program Objectives
• Demonstrate problem solving and critical
thinking skills in order to be successful in
the business world
• effective communication skills and
professionalism when working with others
in a business setting.
• Demonstrate effective interpersonal skills
and an appreciation of business ethics.
• Demonstrate the ability to apply content
knowledge to create value as business
leaders in an organization.
10. Two Direct & Two Indirect
Measures
Direct measures:
Entrance & Exit Exams:
80% of students will
score at least 73% or
greater, which is at a
level of proficiency.
11. Two Direct & Two Indirect
Measures
Indirect Measures:
Mid-term and End of
Program Surveys: At least
80% of the students' will
note the ability to apply
program learning outcomes
as noted by their responses
of "Agree" or "Strongly
Agree".
12. Measure Program Objectives
Two Direct Measures
• In other programs, we used a
business plan and a group of
professors would evaluate their
specialty area (marketing, finance,…).
• Found to be a time consuming
process.
• Needed something manageable for
one person
• Decided to try analyzing a case study
13. Case Analysis
Four Cases Total
• All four are equal in length – about ¾
page.
• Each has multiple things going on
Three Central Questions
• What are the relevant facts of the
case?
• What are the ethical issues?
• What are the ethical alternatives?
15. Test Results
Test Results were Skewed…
• Students are used to answering once
on a multiple choice exam.
• There were several answers that were
correct.
• If students answered correctly, they
received a point
• If students missed a correct answer, it
deducted points from ones they
earned.
16. Going Forward
• Questions reflect how many correct
answers are possible.
• Students will earn one point per
correct answer
• Students will still loose points earned
for incorrect answers (couldn’t fix in
the system).
17. Test Results
Test Results were Skewed…
• Exam #1 was extra credit
• Eliminated the issue with it being a
major portion of their grade or
• Something I had to prepare them for.
• Most were encouraged to take
• Not every student completed the
exam.
19. Agenda
• Why use cases online?
• Mapping
• Creating the tests in the LMS
20. Why Use Cases
• Authenticity (Suskie, 2009)
• Effectiveness and student
acceptance of CBL (Barnes et al, 1994;
Srinivasan et al, 2007)
• Program curriculum favors CBL
learning
• CBL in assessment good tie-in
21. Issues
• Grading responses to cases can be
very time consuming
• There can be a wide range of correct
answers
• Setting up auto graded tests that
capture a wide range of correct
answers can be difficult
22. Our Approach
• Use Multiple Answer questions
• Try to capture as broad of a range as
possible of correct answers in terms
of ethical considerations, and
resolutions
23. What We’ve Learned So Far
• Generally well received by students
• Refine instructions
24. Mapping
• Provides a common document to prompt
discussion for all parties involved in course
design, development and review
• Keeps the two main components of learning in a
program (program outcomes and learning
objectives) a the forefront of design and
development discussions
• Helps ensure alignment of program outcomes
and course objectives
25. How to use the Excel Mapping Tool
• Some cells are linked so you can enter info on
the Data Entry worksheet, and it will fill the
appropriate cells on the other worksheets.
• Demo
26. Setting up the Test
• Blackboard 9.1.201404.160205
• Fairly standard Multiple Answer set
up
• Partial Credit
• Group cases and MC questions
Authenticity: Authenticity in educational assessment relates to assessing students in a manner that more closely resembles the way in which students will employ the skills. CBL learning encourages students to think like a marketing professional operating under conditions that require action while working with problems that are similar to problems they will face on the job.
Effectiveness and acceptance: CBL learning turns the classroon into learning laboratory, in which students are active participants rather than passive observers. Studies have shown that students prefer the more active experience
Program Curriculum: The Marketing program curriculum uses cases heavily throughout as learning activities
Tie in: Therefore, it makes sense to use cases in assessment of program outcomes
Responses to cases typically take the form of essays in which there are a wide range of possible correct responses. Because of this wide range of possible correct responses it is difficult to set up tests that capture this range
