Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. PRR 475 Review - Tips• Powerpoint Presentations – widgets too• Topic Handouts• Sample Questions• Other Handouts e.g. Statistics• Exercises from Lab• Trochim chapters• Other texts, websites
  2. 2. Research/Evaluation Concepts – evaluation.pptProposals & Research Design –propdesb.pptMeasurement –measure.pptSampling –sample.pptSurvey methods –survey99.pptBasic Statistics/Data analysis - handoutsExperiments –expmt.pptObserv., qualitative, other methods – QLQN.ppt, econevaluation.pptCommunication & Ethics researchwriting.ppt, ethics.ppt
  3. 3. Outline• Definitions of Research/Evaluation• Purposes of Evaluation / Research• Science and Scientific Management• Research/Evaluation as Process• Types of Research / Evaluation
  4. 4. Definitions• Evaluation = Process of judging the merit or worth of something• Research – application of scientific methods to answer questions – controlled inquiry directed at increasing knowledge/establishing truth• Evaluation Research - combine the two
  5. 5. Science• Body of Knowledge • Method of Inquiry• systematic • logical• abstract – induction• – deduction general• parsimonious • self-corrective • empirical
  6. 6. Scientific Management• Application of scientific principles to management and decision making – systematic information gathering – empirical, objective, self-corrective
  7. 7. Process -- Steps Research Evaluation• define problem • describe program• objectives/hypotheses • evaluation criteria• literature review • program scoping• research methods • evaluation methods• gather data/analysis • gather data/analysis• conclusions • conclusions
  8. 8. Types of Evaluation by Program Stage• formative (conceptualization/design)]• process (implementation)• summative (outcomes, impacts, efficiency)
  9. 9. Types - By Approach• Standards – norm-based – criterion-referenced• Goals and objectives• Impacts or effects
  10. 10. Evaluation Criteria• Effort - qnty and qlty of inputs• Performance - qnty and qlty of outputs• Adequacy - meet needs?• Efficiency - benefits/costs• Equity - distributional issues, fairness
  11. 11. Process Evaluation• Identifies how and why program works – attributes – recipients – conditions – effects • single or multiple • intended or side effects • timing & duration, long/short term • cognitive, affective or behavioral
  12. 12. Research Process Define Problem, Research Objectives HOW? What? Overall Method Who?•Concepts •Survey •Experiment •Population•Variables •Case Study •Sampling •Secondary Data•Measures Data Gathering Analysis Application
  13. 13. Proposal Format 1. Problem Statement - define program to be evaluated/problem to be studied, users & uses of results. Justify importance of the problem/study. 2. Objectives : Concise listing . In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives. 3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods. 4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why? 5. Attachments - budget, timeline, measurement instruments, etc.NOTE: Most “programs” must be narrowed to specific components to be evaluated.Think of a “Program of studies” rather than a single evaluation study. The proposalshould define this specific study & how it fits into a broader program of studies.
  14. 14. Purposes of Proposal• Communicate with Client• Demonstrate your grasp of problem• Plan the study in advance, so others can evaluate the study approach – will it work? – have you overlooked something? – will results be useful to client? – Can we afford it?
  15. 15. Sample Objectives1. Estimate benefits and costs of program2. Estimate economic impacts of program on local community (social, environmental, fiscal).3. Determine effects of program on target population.4. Describe users and non-users of program5. Assess community recreation needs, preferences6. Determine market/financial feasibility of program7. Evaluate adequacy or performance of program
  16. 16. Methods Choices• Overall Approach/Design – Qualitative or Quantitative – Primary or secondary data – Survey, experiment, case study, etc.• Who to study - population, sample – individuals, market segments, populations• What to study - concepts, measures – behavior, knowledge, attitudes• Cost vs Benefit of Study
  17. 17. Definition & Measurement“measurement is the beginning of science, … until youcan measure something, your knowledge is meager andunsatisfactory” Lord Kelvin Nominal/Conceptual Definition - define concept in terms of other concepts, links concepts without tying them to real world Operational definition - equates definition with measurement, specify procedures/operations to generate the concept.
  18. 18. Levels of MeasurementLevel Characteristic ExampleNominal Unordered Race, gender categoriesOrdinal Ordered categories Sm, med.lg Hardness scaleInterval Consistent distance Temp in fahrenheit between categories or CelsiusRatio Natural zero Temp in Kelvin
  19. 19. Validity vs Reliability
  20. 20. Questionnaire Design1. Preliminary Info Information needed Who are subjects Method of communication2. Question Content3. Question Wording4. Response Format5. Question Sequencing/Layout
  21. 21. What Info?Demographic, Socioeconomic, PhysicalCognitive - Knowledge & beliefsAffective - attitudes, feelings, preferencesBehavioral - actions
  22. 22. Sampling• Always define study population first• Use element/unit/extent/time for complete definition • element - who is interviewed • sampling unit - basic unit containing elements • extent - limit population (often spatially) • time - fix population in time
  23. 23. Types of Sampling Approaches• Probability vs non-Probability• Judgment, Simple Random, Systematic• Stratify or Cluster (Area Sample)• Time Sampling
  24. 24. Sample size• Based on four factors • Cost/budget • Accuracy desired • variance in popln on variable of interest • subgroup analysis planned• Formula: n= Z2 σ2 / e 2 • n= sample size • Z indicates confidence level (95% = 1.96) ∀ σ = standard deviation of variable in population • e = sampling error
  25. 25. Sampling errors for binomial (95% confidence interval) percent distribution in populationSample 50/50 60/40 70/30 80/20 90/10 size 100 10.0% 9.8% 9.2% 8.0% 6.0% 200 7.1% 6.9% 6.5% 5.7% 4.2% 400 5.0% 4.9% 4.6% 4.0% 3.0% 1000 3.2% 3.1% 2.9% 2.5% 1.9% 1500 2.6% 2.5% 2.4% 2.1% 1.5% 2000 2.2% 2.2% 2.0% 1.8% 1.3%
  26. 26. Computing 95% confidence interval• N= 100 , sample mean = 46%, use p= 50/50,• sampling error from table = 10%• 95% CI is 46% + or - 10% = (36, 56)• N=1,000 sample mean =22%• sampling error from table = 2.5%• 95% CI is 22% + or - 2.5% = (19.5, 24.5)
  27. 27. Research Designs/Data Collection ApproachesHow ....Where Household On-Site LaboratoryGatheredPersonal Surveys Surveys, Focus GroupsInterview Field ExpmtsTelephone/ Surveys Computer ComputerComputer Interviews InterviewsSelf-Admin. Surveys, ExperimentsQuest. Field ExpmtsObservation NA Observable Observable& Traces Characteristics CharacteristicsSecondary NA Internal NASources Records
  28. 28. Major Design Types• Surveys• Experiments• Observation• Secondary Data• Qualitative Approaches – Focus Group – Case Study
  29. 29. General Guidelines on when to use different approaches1. Describing a population - surveys2. Describing users/visitors - on-site survey3. Describing non-users, potential users or general population - household survey4. Describing observable characteristics of visitors - on-site observation5. Measuring impacts, cause-effect relationships - experiments
  30. 30. Guidelines (cont)6. Anytime suitable secondary data exists - secondary data7. Short, simple household studies - phone8. Captive audience or very interested population - self-administered survey9. Testing new ideas - experimentation or focus groups10. In-depth study - in-depth personal interviews, focus groups, case studies
  31. 31. Primary or Secondary Data• Secondary data are data that were collected for some purpose other than your study, e.g. government records, internal documents, previous surveys• Choice between Primary /Secondary Data – Costs (time, money, personnel) – Relevance, accuracy, adequacy of data
  32. 32. Qualitative vs Quantitative Quantitative QualitativePurpose Gen’l Laws Unique/Individual case Test Hypotheses Understanding Predict behavior Meanings/IntentionsPerspective Outsider-Objective Insider-SubjectiveProcedures Structured Unstructured formal measures open ended measures probability samples judgement samples statistical analysis interpretation of data
  33. 33. Qualitative vs Quantitative Approaches Qualitative Focus Group In-Depth Interview Case Study Participant observation Secondary data analysis Quantitative Surveys Experiments Structured observation Secondary data analysis
  34. 34. Survey vs ExperimentSurvey - measure things as they are, snapshot of population at one point in time, generally refers to questionnaires (telephone, self-administered, personal interview)Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships (field and lab experiments)
  35. 35. STEPS IN A SURVEY1. Define problem and study objectives2. Identify information needs & study population(s)3. Determine basic design/approach - cross sectional vs longitudinal - on-site vs household vs other - self-admin. vs personal interview vs phone - structured or unstructured questions4. Questionnaire design5. Choose sample (frame, size, sampling design) 6. Estimate time, costs, manpower needs, etc.
  36. 36. Survey Implementation7. Proposal & “Human subjects” review8. Line up necessary resources9. Pre-test instruments and field procedures10. Data gathering and follow-up procedures11. Coding, cleaning and data processing12. Analysis: preliminary, then final.13. Communication and presentation of results.
  37. 37. Characteristics of a true Experiment 1. Sample equivalent experimental and control groups 2. Isolate and control the treatment 3. Measure the effect
  38. 38. Pre-test/Post-test with ControlR MB1 X MA1 Experimental groupR MB2 MA2 Control group R denotes random assignment to groups X denotes the treatmentMeasure of effect = ∆ Expmt gp - ∆ Control gp = (MA1-MB1) - (MA2-MB2) = with vs without
  39. 39. Threats to Internal validity• * Pre-measurement (Testing) : effect of pre- measurement on dependent variable (post-test)• * Selection: nonequivalent experimental & control groups, (statistical regression a special case)• * History: impact of any other events between pre- and post measures on dependent variable• * Interaction: alteration of the “effect” due to interaction between treatment & pre-test.• Maturation: aging of subjects or measurement procedures• Instrumentation: changes in instruments between pre and post.• Mortality: loss of some subjects
  40. 40. Economic Evaluation Tools• Benefit-Cost Analysis• Cost Effectiveness Analysis• Financial Analysis/Feasibility• Fiscal Impact Analysis• Economic Impact Analysis• Social/Environmental Impact Analysis
  41. 41. Ethical Issues• Human Subjects• Honest and Open Reporting• Client confidentiality• No selling
  42. 42. Human Subjects• Voluntary Participation• Informed Consent• Risk-benefit ratio• Anonymity vs Confidentiality
  43. 43. Research Writing• Like any writing, – Know audience – Know Story/Message/Subject – Follow Formats for Outlet• Research Style – Objective, Factual – Impersonal – Dense, concise, logical
  44. 44. Sections• Abstract – Executive Summary• Introd/Problem statement• Objectives• Lit Review/Background• Methods• Results• Discussion, Implications• References
  45. 45. Three Audiences/styles• Researchers – research journal style – Technical, methods, statistical tests• Managers – business style – Results and implications• Public – newspaper style – Interesting, no jargon, highlights
  46. 46. Research vs Business Reports• Written/Research • Oral/Business – Problem – Objectives – Objectives – Key Results & – Methods Recommendations – Results – Justify from study – Discussion – Brief methods – Discussion
  47. 47. Reminders• Final Exam is Friday Dec 15, 7:45-9:45 am, this room• Final Papers due by Wednesday Dec 13• See YaYen Sun to finish lab work by end of week.