Describes methods and tools for maintaining educational assessment information - primarily for program outcomes assessment. This is describing outcomes assessment processes and tools for an ABET-Accredited university undergraduate programs. Discusses direct and indirect evidence use, and how to use these to efficiently validate the program.

1. ICEE 2006F. Mak & S. Frezza © 2006
Collection,
Maintenance, and
Validation (CMV)
of a Set of
Effective Objective Evidence
Fong Mak, Ph.D., P.E.
Electrical and Computer Engineering
Steve Frezza, Ph.D., C.S.D.P.
Computer and Information Science
Gannon University

2. F. Mak & S. Frezza © 2006 ICEE 2006
Why CMV Objective Evidence ?
• Foundation:
Outcomes-Based Continuous Quality
Improvement
• Objective Evidence …
– Shows we’ve engaged in…
– Reflects the quality of…
…the educational process
Documents…
Supports…

3. F. Mak & S. Frezza © 2006 ICEE 2006
Why Objective Evidence ?
• Process:
– Ensure the quality of the educational process
• Product:
– Support and ensure the quality of educational
delivery
– Multiple constituents, measures

4. F. Mak & S. Frezza © 2006 ICEE 2006
What is Effective Evidence ?
Objective Evidence that
• Maximizes Information Content
– Process documents should provide decision-
makers information to support their decisions
• Minimizes Item Count
– Collect as few items as needed
Golden Rule:
Documents Add Value

5. F. Mak & S. Frezza © 2006 ICEE 2006
Effective Objective Evidence
Does:
• Inform the process
• Support decisions
• Flow from the process
Key:
Process Integration
Typical Educational Assessment Processes: dd
Untracked, Indirect, Direct
Does Not:
Just document it
Record decisions
Add steps

6. F. Mak & S. Frezza © 2006 ICEE 2006
DirectAssessments
Disconnected Assessment Process
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Implementation
Corrective
Actions
Process
Objective
Evidence
Course
Portfolios
Course
Portfolios
Course
Portfolios
No feedback
Minimal documentation
Minimal additional effort
Discontinuous improvement

7. F. Mak & S. Frezza © 2006 ICEE 2006
Indirect Assessment
• Respond to perceived effectiveness by:
– Students
– Faculty
– Others

8. F. Mak & S. Frezza © 2006 ICEE 2006
Indirect Assessment Process
Indirect
Assessments
DirectAssessments
Process
Objective
Evidence
Meeting
Minutes
Action
Item
Matrix
Course Exit
Survey Results
Senior Exit
Survey Results
Employer
Survey Results
Alumni
Survey Results
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
Course
Portfolios
Course
Implementation
Faculty
Review
Corrective
Actions
Continuous feedback
Structured documentation
Some additional effort
Tracked improvement
Only indirect assessment

9. F. Mak & S. Frezza © 2006 ICEE 2006
Direct Assessment
• Direct measures of effectiveness:
– Student performance in meeting/exceeding
• Course objectives
• Program objectives (more difficult)
• Our Innovation:
Link grading & course outcomes
– Key Assignments: P/F to course
– Faculty-assigned, linked to course outcomes

10. F. Mak & S. Frezza © 2006 ICEE 2006
Minimized Direct Assessment
DirectAssessments
Process
Objective
Evidence Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
Course
Portfolios
Course
Implementation
Faculty
Review
Corrective
Actions
Course
Grade
Reports
Course
Grade
Reports
Summary
Grade
Reports
Meeting
Minutes
Action
Item
Matrix
FCARs
FCARs
FCARs
Continuous feedback
Tracked improvement
Effort focused on
student performance
of outcomes

11. F. Mak & S. Frezza © 2006 ICEE 2006
Complete Assessment Process
Indirect
Assessments
DirectAssessments
Process
Objective
Evidence
Meeting
Minutes
Action
Item
Matrix
Course Exit
Survey Results
Senior Exit
Survey Results
Employer
Survey Results
Alumni
Survey Results
Course
Grade
Reports
Course
Grade
Reports
Summary
Grade
Reports
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
Course
Portfolios
FCARs
FCARs
FCARs
Course
Implementation
Faculty
Review
Corrective
Actions

12. F. Mak & S. Frezza © 2006 ICEE 2006
Effective Collection
• Direct & indirect measures
– Improvements based on collected evidence
– Evidence, decisions and actions documented
– Continuous quality improvement tracked
Normal part of improvement process

13. F. Mak & S. Frezza © 2006 ICEE 2006
Effective Maintenance
• Automation
– Maximize measures collected/maintained online
• Survey creation, management: EvalTools
• Simple paper trails, used for review
– Standardized templates
• simplify use and review
– E.g., Summary Grade Reports, FCARs, Meeting
Minutes

14. F. Mak & S. Frezza © 2006 ICEE 2006
Effective Validation
• Review evidence as part of process
• Summarize decisions/planned actions
• Simple paper trails, used for validation
Analogy:
Bug Tracking
Describe (open) issues
Assign responsibility
Track changes
Track effectiveness (close issues)

15. F. Mak & S. Frezza © 2006 ICEE 2006
Summary
CMV Effective Objective Evidence
• Requires:
– Process design & commitment
• Benefits:
– Students
– Program Communicators
– Program Stakeholders (even faculty)
– Program Evaluators