2. FINAL EXAM BREAKDOWN
• As a reminder the Final Exam will be scored as follows:
• 25 Multiple Choice Questions @ 4 points each = 100 points total
• 25 Matching Questions @ 2 points each = 50 points total
• 50 True or False Questions @ 2 points each = 100 points total
• 15 Short Answer Questions @ 10 points each = 150 points total
• For a total of 400 points
• You will be required to complete the Final Exam during the Week 7 module
• When you complete the exam on Canvas don’t worry about the score that comes up; as the 100 MC, Matching
& T/F Questions will be auto graded correctly, but the short answer responses will require me to grade them
• You will be given two opportunities to take the exam; your highest score will be the grade you receive [ie a 75
on the first attempt and a 95 on the second attempt means you’ll receive a 95]
3. FOLLOWING THIS PPT
• This PPT will be broken down into the four sections you are tested on the Final Exam: Multiple
Choice, Matching, T/F, and Short Answer questions
• For the Multiple Choice section the question you will be asked will be presented as the title and the
correct answer with be in the body of the slide
• For the Matching section you will be provided with the concepts requiring you to match them – I
will provide the concept and what Week’s PPT it is in; you will be able to match the picture and the
concept on the exam
• For the T/F questions the concepts you will be asked whether they are True or False will be
presented; not the specific T/F question on the exam
• For the Short Answer questions each will have its own slide and a reference to how I would like it
answered
4. MULTIPLE CHOICE QUESTIONS
• Slides 5 through 29 feature the 25 multiple choice questions on
the exam
• As a reminder, the title is the question on the exam and the
content is the answer on the exam
5. HOW WAS EVALUATION DEFINED WITHIN
THIS COURSE?
• For our purposes within this course consider evaluation to be
an assessment or judgment on what the best approach is for
addressing an issue.
• [Week 1, Slide #2]
6. WHAT IS ONE REASON EVALUATION FAILS?
• Generally, people don’t ask for evaluations or improvements
• [Week 1, Slide #5]
7. WHAT IS THE FOCUS OF AN INTERNAL
EVALUATION?
• Specific focus
• Largely focused on improving one specific aspect
• [Week 1, Slide #10]
8. WHAT IS THE FOCUS OF AN EXTERNAL
EVALUATION?
• Vague focus
• Largely focused on improving all aspects
• [Week 1, Slide #10]
9. WHAT IS ONE-SHOT DESIGN?
• One-Shot Design – evaluation takes place at one time and one
time only
• [Week 2, Slide #17]
10. WHAT IS RETROSPECTIVE PRETEST DESIGN?
• Retrospective Pretest Design – info is collected prior to design,
but also at the end as well to show if participant data has
changed
• [Week 2, Slide #17]
11. WHAT IS ONE-GROUP PRETEST-POSTTEST
DESIGN?
• One-Group Pretest-Posttest Design – pretest data is compared
to posttest data for one group
• [Week 2, Slide #17]
12. WHAT IS TIME SERIES DESIGN?
• Time Series Design – graphing results over a period of time to
see if anything is changing through the evaluation process
• [Week 2, Slide #17]
13. WHAT IS PRETEST-POSTTEST CONTROL-
GROUP DESIGN?
• Pretest-Posttest Control-Group Design – similar to One-Group
Pretest-Posttest Design, but there are two groups [control and
non-control]
• [Week 2, Slide #17]
14. WHAT IS SOLOMON FOUR-GROUP DESIGN?
• Solomon Four-Group Design – two control groups and two
non-control groups; often provides stronger data than just two
groups
• [Week 2, Slide #17]
15. WHAT IS CASE STUDY DESIGN?
• Case Study Design – examines one specific situation
• [Week 2, Slide #17]
16. WHAT ARE IMPORTANT CONSIDERATIONS
TO EVALUATION?
• Availability of data
• Timeliness
• Objectivity
• Structure
• [Week 3, Slide #11]
17. WHAT ARE THE THREE MOST COMMON
TYPES OF ARCHIVAL DATA?
• Records
• Documents
• Databases
• [Week 3, Slide #12]
18. WHAT IS THE PRIMARY ADVANTAGE OF
USING ARCHIVAL DATA?
• Data can be easily accessed if you know where to look
• [Week 3, Slide #14]
19. WHAT IS THE PRIMARY DISADVANTAGE OF
USING ARCHIVAL DATA?
• It may be difficult to adequately interpret the data
• [Week 3, Slide #14]
20. WHEN REVISITING PURPOSE AND
RATIONALE WHAT IS MOST IMPORTANT?
• Was the evaluation easy to follow? Ties to the planning involved
• Week 6, Slide #2
21. POSING QUESTIONS ABOUT EVALUATION'S
POSSIBILITY AND UNCOVERING OF DATA IS?
• Revisiting purpose and rationale
• Week 6, Slide #2
22. WHAT CONSIDERATION IS GIVEN TO
STAKEHOLDERS AFTER THE EVALUATION?
• Their ability to have followed along with the purpose and
rationale of the evaluation
• Week 6, Slide #2
23. WHEN REVISITING KEY QUESTIONS, WHAT
NEEDS TO BE CONSIDERED?
• Were all, some, or none of the key questions answered
• Week 6, Slide #3
24. HOW DO YOU DETERMINE IF YOUR
EVALUATION IS COST EFFICIENT?
• The pre-planned budget was followed
• Week 6, Slide #4
25. WHAT CAN INFLUENCE THE DATA
COLLECTED?
• sample, population, and response level
• Week 6, Slide #5
26. WHAT CAN IMPROVE FUTURE
EVALUATIONS?
• feedback, suggestions, or recommendations
• Week 6, Slide #5
27. WHAT CAN HELP MOVING FORWARD
RELATED TO DATA?
• keeping a record
• Week 6, Slide #6
28. WHO SHOULD BE ABLE TO PROVIDE INPUT
ON AN EVALUATION?
• Co-workers [not just stakeholders], those knowledgeable, and
those with experience
• Week 6, Slide #7
29. WHAT NEEDS TO BE CONSIDERED THAT
GOES BEYOND THE INTERNAL EVALUATION?
• outside influences
• Week 6, Slide #7
30. MATCHING QUESTIONS
• Two questions will be associated with Slide #3 from Week 1
associated with Evaluation and Research
• Three questions will be associated with Slide #6 from Week 1
associated with Types of Evaluation
• Three questions will be associated with Slide #7 from Week 1
associated with Types of Evaluation
• For these questions you will see the same definitions and need to match them
to the appropriate concept
31. MATCHING QUESTIONS
• Two questions will be associated with Slide #10 from Week 1
associated with Internal and External Evaluation
• Again you will match the definition to the concept
• Kirkpatrick’s Four Level Evaluation Model [Slide #2 Week 2]
• Hamblin’s Five Level Model [Slide #3 Week 2]
• Training Effectiveness Evaluation (TEE) System [Slide #4 Week 2]
• Brinkerhoff’s Six Stage Model [Slide #5 Week 2]
32. MATCHING QUESTIONS
• Input, Process, Output Model [Slide #6 Week 2]
• Systematic Model of Factors [Slide #7 Week 2]
• Kaufman, Keller, and Watkins Five Level Model [Slide #8 Week 2]
• Training Efficiency and Effectiveness Model (TEEM) [Slide #9 Week 2]
• Swanson and Holton Results Focus [Slide #10 Week 2]
• Holton’s HRD Evaluation Research and Measurement Model [Slide #11
Week 2]
33. MATCHING QUESTIONS
• From the entirety of the course textbook; be able to match
• Evaluation Purpose
• Evaluation Rationale
• Data Collection Instrument – surveys, observations, focus groups, etc.
• Evaluation Budget
• Evaluation Stakeholders
34. T/F QUESTIONS
• The following are all True statements from the course content
• On the Final Exam these statements will be modified [or not] in some
way to test your understanding of them
• Slides 35-44 will each have 5 True statements from the course
content [to total the 50 T/F Q’s]
35. FIRST 5
• The main primary reason individuals or companies or universities evaluate
something is to see if there is a better option out there [Week 1]
• Reminder, the next three concepts are associated with What is Needed in Evaluation
• Planning: setting criteria and standards [Week 1]
• Implementing and Completing: showing potential evidence for each case and
testing those that are most appropriate [Week 1]
• Analyzing: discussing or presenting recommendations based on the evaluation
results [Week 1]
• Archival data has been previously collected, but there is no guarantee it has
been analyzed [Week 3]
36. NEXT 5
• Produces time sensitive qualitative data [meaning data is only
collected during the observation period ie 60 minutes] [Week 3]
• Limited response rate (associated with surveys) [Week 3]
• Allows for follow up [like an interview] which can produce substantial
qualitative data (associated with focus groups) [Week 3]
• Pilot testing methodology is a good way to check for instrument
validity [Week 3]
• Can someone else do what you’ve done and produce results? They
may not be similar, but the instrument’s reliability depends on how
consistent it is [Week 3]
37. NEXT 5
• Observation requires permission/approval [Week 4]
• Observation purpose and key questions need to be determined
beforehand [Week 4]
• Without clear information there may be a refusal to participate
in an observation [Week 4]
• Surveys make it easier to reach a larger population and sample
[Week 4]
• Surveys often have limited response rates [Week 4]
38. NEXT 5
• Type of survey and questions influence response rates [Week 4]
• Surveys require informed consent prior to agreeing to survey and debriefing
info after completion of survey [Week 4]
• Data collectors need to express in the informed consent how they will
protect participants identity [Week 4]
• Open-ended questions allow participants to provide a rationale behind their
answer [Week 4]
• Fill-in-the-blank questions are not often used in research, but can be used
in education and management [Week 4]
39. NEXT 5
• Lists or multiple answers allow participants to identify importance
[Week 4]
• Likert scale – strongly disagree, disagree, neutral, agree, strongly
agree [limited in what it can measure] [Week 4]
• Focus groups can occur individually or in a group [Week 4]
• Focus groups feature largely opinion based responses which limit the
strength of the data [Week 4]
• For focus groups there is a need to actively recruit people to
participate; rather than having a potentially set population [Week 4]
40. NEXT 5
• With focus groups it is easier to schedule individuals; rather
than groups with different schedules [Week 4]
• Sampling size should produce a 95% confidence level [Week 4]
• Sampling errors often occur due to nonresponses [Week 4]
• Stakeholders may not understand how to analyze data [Week 5]
• Basic methods of data analysis may make it easier to
communicate results [Week 5]
41. NEXT 5
• Nominal data relates to numbers [Week 5]
• Ordinal data relates to rankings [Week 5]
• Interval data covers an interval scale [Week 5]
• Ratio data is often tied to trends [Week 5]
• Most qualitative methods can be modified to secure
quantitative data [Week 5]
42. NEXT 5
• Items such as observations and surveys often seek both
quantitative and qualitative data [Week 5]
• Archival data can also feature both qualitative and quantitative
data [Week 5]
• Analyzing data is difficult [Week 5]
• Interpreting data is easier with experience [Week 5]
• The evaluation being easy to follow ties to the planning
involved [Week 6]
43. NEXT 5
• All parties impacted by the evaluation should receive a copy of
results [Week 6]
• Evaluations should answer the key questions [Week 6]
• If the appropriate evaluation method and data collection instrument
are chosen the data should be large [Week 6]
• You should have extensive data if your evaluation method and data
collection instrument were appropriate [Week 6]
• Some evaluations deviate from the plan [Week 6]
44. FINAL 5
• Sample size and population influence data results [Week 6]
• Results of an evaluation can lead to recommendations [Week 6]
• Seeking feedback from respondents of the data collection instrument
can be helpful for future evaluations [Week 6]
• You can determine after an evaluation if your data collection
instrument was appropriate [Week 6]
• Budgets need to be considered prior to evaluation, during evaluation,
and after evaluation [Week 6]
45. SHORT ANSWER QUESTIONS
• The following 15 slides are the short answer questions as the title and
the acceptable answers [put them in your own words on the exam]
within the body
• I will point you to certain aspects here, but feel free to add more to
these answers if you deem it appropriate
46. DESCRIBE THE THREE COMMON
OBSERVATION TYPES
• Type of observation needs to be clear
• Full participant observer – interacting with what you’re observing
• Partial observer – interacting with observation only if asked/suggested
• Nonparticipant observer – only observe, no interacting
• Week 4, Slide #2
47. DISCUSS OBSERVATION OBSTACLES
• Length/Amount – how long will you observe? How many times will you observe?
This determines if you can get the appropriate amount of data collected
• Location – can you easily access the site?
• Timeliness – are you observing on a bad day? This could prevent you from
obtaining useful data
• Ethics – do you have the appropriate permission to observe people and to use
the site?
• Training – can you train others to conduct observations to help collect more data
at one time?
• Week 4, Slide #5
48. DISCUSS THE PROS AND CONS OF
OBSERVATIONS
• Advantages
• Can produce a lot of data if planned correctly
• Considers emotion
• Helps define key themes
• Disadvantages
• Bias [you’re interpreting data as it happens]
• Cost [you may have to visit site multiple times]
• Missing data [you may miss something while writing about something else]
• Week 4, Slide #6
49. DESCRIBE THREE COMMON SAMPLING
METHODS
• Convenience – you solely sample from a population those directly available to you [can
introduce bias to the study/evaluation]
• Purposive – you have a direct purpose in mind, so you only use certain people that can
provide effective data. For example, if you’re studying school administrators you would
sample a principal, not a gym teacher.
• Snowball – referral based sampling; where one lead suggests another
• Week 4, Slide #13
50. DESCRIBE THREE LESS COMMON SAMPLING
METHODS
• Random Sampling – rarely provides effective data, but no bias exists
• Stratified Random Sampling – random sampling where participants are
broken into groups
• Cluster Sampling – selecting a large population, say 100 individuals, then
collected data from subgroups [in essence two types of data collection] –
generally based on responses
• Week 4, Slide #14
51. COMPARE AND CONTRAST QUANTITATIVE
AND QUALITATIVE DATA
• Quantitative
• Easy to understand
• Nearly all numbers based [Likert scales too]
• Sometimes viewed as more credible than qualitative data
• Often used to prove something
• Qualitative
• Can be difficult to explain
• Nearly all verbal
• Sometimes analyzed incorrectly
• Often used to find the underlying cause behind something
• Week 5, Slide #3
52. WHY IS PRODUCING A LOT OF DATA A PRO
• Producing a lot of data can be a huge pro
• There may be validity and reliability
• Data eliminates assumptions; may prove or disprove hypothesis
• Robust data can eliminate bias [depending on population and sample]
• Possible to replicate
• Possible to generalize
• Allows for quantitative, qualitative, or mixed methods approach
• Easier to analyze with experience
• Week 5, Slide #5
53. WHAT ARE THE MAIN TENETS BEHIND
QUALITATIVE STUDIES
• Qualitative studies are often by two main tenets:
• Literature Review – you find a pattern in scholarly journals which suggest there is a problem in need
of studying
• Theoretical Framework – you find a theory, perhaps UTAUT, leading you to consider changing
something
• In research both of used; in evaluation only one may be used
• Week 5, Slide #7
54. WHAT GUIDELINES SHOULD BE FOLLOWED
WHEN ANALYZING DATA
• Read, reread, and reread again – qualitative data. In research multiple
people may take notes on qualitative data
• Develop codes/themes – can use software to do this if not
comfortable
• Take notes
• Find patterns
• Code/theme in time intervals; difficult to deal with a lot of data at
one time
• Week 5, Slide #8
55. WHAT DO YOU NEED TO CONSIDER WHEN
DESCRIBING DATA
• Difficult to do for the novice
• Consider,
• Level of data – nominal, ordinal, interval, ratio, etc.
• # of Variables studied – will drive how you analyze the data
• Relationships – is there a correlation between two variables [or more]
• Predictions – does one variable have an influence on another; common example in the work force is
“will experience influence work performance”
• Significant Difference – helps shows a positive, neutral, or negative relationship
• Week 5, Slide #9
56. WHY DO WE NEED TO COMMUNICATE DATA
• Often you’re trying to solve a problem or modify what you’re doing
• If only you know the results the data collected and analyzed serves
no purpose
• Presenting the results allows for others to interpret the results and
present solutions
• Think of journal articles – they often have a results and discussion
section, but they also have a “future considerations” section where
they discuss how to use the results to drive future actions
• Week 5, Slide #12
57. WHO SHOULD WE COMMUNICATE DATA TO
• Often in evaluation, data results are merely reported to stakeholders or those involved/impacted
in/by the evaluation process
• This may not be appropriate
• Consider everyone impacted by the evaluation [can go beyond just those involved in the process]
• Consider new employees; you may be doing something as a result of your evaluation and need to
explain it
• Consider outside relationships; you may need to explain to a vendor
• Consider those within the organization with expertise in communicating data results; they may not
have been involved in this process, but you can seek advice from them prior to disseminating your
results
• Week 5, Slide #13
58. WHAT NEEDS TO BE CONSIDERED WHEN
COMMUNICATING DATA
• Timing – need to provide/report results so those reading reports have time to
understand; perhaps provide them with the report on a Friday and have them look it over
and bring in any questions on Monday
• Structure – if a reporting of data is disorganized and hard to read it won’t be read
• Presentation – use words everyone can understand, use tables/graphs if needed;
understand qualitative and quantitative data will be presented differently
• Communicate negative data results in an effective manner; explaining why, to avoid bias
against results due to personal intentions
• Week 5, Slide #14
59. IN WHAT WAYS CAN DATA BE
COMMUNICATED
• Work Sessions/Meetings – hard to organize and time consuming
• Emails/Online Chats – easier to organize, but can be unstructured
• Presentations/Posters/Video – visual interpretation of facts
• Cartoon/Poetry/Drama – mentioned in text, but not legitimately used
• Memos/Postcards – outdated
• Written Reports/Posted on Website – more common; allows individuals to read findings on
their own time
• Week 5, Slide #15
60. DISCUSS WHAT NEEDS TO BE CONSIDERED
AFTER COMPLETING THE EVALUATION
PROCESS
• Utilize all info from Week 6, Slides #2-6 to construct your
response
![FINAL EXAM BREAKDOWN
• As a reminder the Final Exam will be scored as follows:
• 25 Multiple Choice Questions @ 4 points each = 100 points total
• 25 Matching Questions @ 2 points each = 50 points total
• 50 True or False Questions @ 2 points each = 100 points total
• 15 Short Answer Questions @ 10 points each = 150 points total
• For a total of 400 points
• You will be required to complete the Final Exam during the Week 7 module
• When you complete the exam on Canvas don’t worry about the score that comes up; as the 100 MC, Matching
& T/F Questions will be auto graded correctly, but the short answer responses will require me to grade them
• You will be given two opportunities to take the exam; your highest score will be the grade you receive [ie a 75
on the first attempt and a 95 on the second attempt means you’ll receive a 95]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)