Evaluation finds out “what works” and “what does not work.” A process or outcome evaluation enables to answer basic questions about a program’s effectiveness, including: Are participants benefiting from program services? Do instructors have the necessary skills and training to deliver services? Are participants satisfied with the program? Are some sub-groups benefiting, but not others?Knowing “what works” helps to focus resources on the essential components of the program that benefit participants.Knowing “what does not work” allows to improve and strengthen the service delivery models. Not knowing what is working may waste valuable time and resources.
Instructional Design and Content Development Workshop
TopicsDefine a program of evaluationEvaluation in learningOutline major reasons to conduct a programevaluationWhy conduct an evaluation?General evaluation models.Who is Kirkpatrick?Kirkpatrick’s four level learning evaluationmodelWrite questions for each level to assess and
What is an evaluation?Evaluation is a systematic method for collecting, analyzing, andusing information to answer basic questions about a program.Evaluation is valuable for strengthen the quality of theprogram and improve outcomes for the learners.Evaluation answers basic questions about a program’seffectiveness, and evaluation data can be used to improveprogram services.
Evaluation In Learning- Evaluation means assessing the effectiveness and possible improvement of a course.- Evaluation is part of any instructional design model.- Evaluation provides review checkpoints for each phase of ADDIE that allow the instructor to evaluate- Without completing this produced. the work that has been portion of ADDIE model, the e-course is incomplete because the course cannot be redesigned and improved without
MAJOR REASONS TO CONDUCT AN EVALUATIONEvaluation finds out “what works” and “what does not work.”Evaluation showcases the effectiveness of a program to the community.Evaluation improves staff’s practice with participants.Evaluation increases transfer of learning to behavior/performance inorder to maximize program results.Evaluation increases a program’s capacity to conduct a critical selfassessment and plan for the future.Evaluation builds knowledge for the out-of-campus time field.
Why conduct an evaluation?The ADDIE model stresses the concept thatgood training programs requireplanning, review, and revision.The evaluation phase focuses on gabs to mendthem.Increase transfer of learning in order to maximizelearning / e-content results.The evaluation phase measures the courses
Benefits of EvaluationEffectiveness: It shows the progress made toward program goals and objectives.Best Practices: It provides the ability to determine what program approaches are mosteffective.Improvement: It provides ongoing assessment of program design and implementationto identify areas of improvement.Impact: It demonstrates economic or human impact.Accountability: It provides the base for interpreting an organization or programs’ worthto its stakeholders.Promotion & Advocacy: It informs policymakers about programs successes.Appraisal & Coordination: It gives managers the performance information to makebetter operational decisions.
General Evaluation ModelsThere are a lot of models clarifying the evaluation process and determining theinformation needs of the intended audiences of the evaluation, these modelssuch as: Tyler’s early conception of assessing attainment of program objectives. Decision-Making Evaluation Approaches. Naturalistic Evaluation Approaches Kirkpatrick’s Four Levels for evaluating program effectiveness; which suggests the most appropriate evaluation methodology to be used.
Who is Kirkpatrick?Donald Kirkpartick is Professor of theUniversity of Wisconsin in the USA and apast president of the American Society forTraining and Development.He has developed a very popular evaluationmodel that has been used by thetraining/learning community.He focused on measuring four kinds of
Kirkpatrick’s four levels of evaluating learningKirkpatrick’s model includes four levels or steps ofoutcome evaluation:
Level one - Evaluate Learner ReactionLevel One: Evaluate Learner Reaction How well did the learners like the learning process? The goal is find out the reaction of the learners to the instructor, course and learning environment. The purpose is not to measure what the learners has learned, but whether the delivery method was effective and appreciated. This type of evaluation can be incorporated at the end of the instruction and can be delivered online.
Level one - Evaluate Learner ReactionGeneral questions can include the following: • Did the instructor attend the sessions on time? • Did the instructor respond to learners’ comments and questions? • Did the instructor deliver the information clearly and smoothly? • Were there distraction? • Did the learners feel comfortable in the surroundings?E-content questions can include the following: • Level of appeal of the instruction. • The ease of navigation and use of tools. • The ability of the course to motivate and retain interest. • The amount of interactive exercises. • The relevance of the objectives. • Quality and relevance of multimedia. • The ease of navigation and use of tools.
Level two - Evaluate LearningLevel two: Evaluate Learning What did learners learn? (the extent to which they gain knowledge, skills and attitude) The goal is measuring learning results. In other words: did the learners learn what intended to be taught?
Level two - Evaluate LearningMeasurement methods of level two include: Formal and informal testing. Self assessments at the beginning (pre-test) and end (post-test) for learners. http://www.reap.ac.uk/reap/public/papers//DN_SHE_Final.pdf Interviews, observation and feedback. Product Assessment. Creation of a project as an authentic assessment to showcase the knowledge and talents. This form of assessment evaluates whether the learner can apply the learned skills or concepts in a concrete fashion.
Level three - Evaluate PerformanceLevel three: Evaluate Performance What changes in performance resulted from the learning process? To what degree participants apply what they learned during training when they are back on the job. The goal is measuring “what happens when learners leave the classroom and return to their daily lives/job. Did the learners put their learning into effect when back on the job/live? Would the learners be able to transfer their learning to another person? How much transfer of learning occurs? It is the most obvious sign to evaluate the training program’s effectiveness.
Level three - Evaluate PerformanceMeasurement methods of level two include:Allow time for a change in behavior to take place Testing Observation, interview and survey. The learner interactions on the job. Evaluate both before and after the program if that is practical
Level four - Evaluate ResultsLevel four: Evaluate Results What are the tangible results of the learning process in terms improved quality, increased production, efficiency, etc.? To what degree targeted outcomes occur as a result of learning event(s) and subsequent reinforcement The goal is to find out if the training program led to final results, especially business or environment results Level four outcomes include the major results that contribute to the well functioning of an organization. Level four outcomes Improve quality of work, increase productivity and profits. Across an entire organization it becomes more challenging and affect organizational and business performance, which cloud the true cause of good or poor results.
Level four - Evaluate ResultsMeasurement methods of level four include: Methods of Measuring Learning Outcomes Grid How colleges and universities can measure and report on the knowledge and abilities their students have acquired during their college years is an issue of growing interest. http://web.mit.edu/tll/assessment-evaluation/methods-of-measuring-learning-outcomes- grid.doc http://www.elcamino.edu/academics/slo/docs/SLOFocusOnResults.doc Course-embedded assessment/Assurance of Learning Standards. http://www2.cortland.edu/dotAsset/f2e9f1ee-0100-40ab-b78d-48ead9c047d3.pdf
Checklist for effective questionsDo Do NotGive clear instructions Use complex phrasesKeep questions structure simple Frame questions in the negativeAsk one question at a time Use abbreviations, contractions or symbolsMaintain a parallel structure for all questions Mix different words for the same conceptDefine terms before asking the question Use “loaded” words or phrasesBe explicit about the period of time being referenced by the Combine multiple response dimensions in the same questionquestionProvide a list of acceptable responses to closed questions Give the impression that you are expecting a certain responseEnsure that response categories are both exhaustive and mutually Bounce around between topics or time periodsexclusiveLabel response categories with words rather than numbers Insert unnecessary graphics or mix many font styles and sizesAsk for number of occurrences, rather than providing response Forget to provide instructions for returning the completed surveySave personal and demographic questions for the end of thesurvey
Activity Based on your study ofKirkpatricks model of learning/ training programsevaluation, apply this model toevaluate your course using the4 levels of Kirkpatricks model.o Write questions that explores the items within each level