Your SlideShare is downloading. ×
0
Measurement 201:
Collecting quantitative information
Institute for Public Relations Summit on Measurement

               ...
Today’s objectives

• Smart consumer
• Quantitative survey methods
  1. Telephone
  2. Web
  3. Mail and multi-modal
  4. ...
Great resources




       Copyright © 2010 by The Institute for Public Relations   3
First steps


What are your objectives?

   What do you want to explore,
   discover, test, or document?

      Who are th...
Telephone surveys

• Types
  – Random digit dial (RDD)
  – List from sample provider or panel
  – Company or client list (...
Telephone surveys: Current issues

Cell phone only households
  – Same or different?
  – What to do?
  – Ethical and legis...
Surveys in general: Current issues

                 • Non-response                                                       ...
Are more intensive methods helpful?
          The Keeter et al study

• Standard Survey: 36% response rate
    – Calling d...
Other design issues

• Oversampling
  – Example: Attitudes to location-based apps
  – Example: National survey on water co...
Telephone omnibus polls

• National random sample of 1,000 households
• Offered by all major research firms
• Fast
• Low c...
Internet surveys

• Advantages and uses




• Limitations
  – http://www.aapor.org/Content/NavigationMenu/Home/Left/A
    ...
Mail surveys and multi-modal surveys

• Don Dillman et al, Internet, Mail, and Mixed-Mode
  Surveys: The Tailored Design M...
Sampling


– Examine different sampling techniques
– Strengths and shortcomings
– Cases and examples
– Tools to make decis...
Why care about sampling?


• Formal definition of our targets
   – Example: Caregivers of Type II diabetes patients
• Gene...
What is sampling?


• Probability sampling
  – “Sampling is the science of systematically
    drawing a valid group of obj...
Some definitions


• Universe
  – General concept of who or what will be sampled
• Population
  – People or units to be sa...
Some definitions


• Sample
  – Actual people chosen for inclusion in the research
  – Example: Selection of 10,000 veteri...
Types of error


• Sampling error
  – Issue: Potential error or uncertainty as a result of not
    sampling from all membe...
Some definitions


• Measurement error
  – Error when respondents misunderstand or
    incorrectly respond to questions
• ...
Problems and errors in sampling


• Understanding and reducing coverage
  error
  – Does the sampling frame (list) contain...
Three approaches


1. Census
2. Probability sample
3. Nonprobability sample




               Copyright © 2010 by The Ins...
Census sampling


• Interview or measure all members of a
  population
   – Example: Wal-Mart annual employee survey

• No...
Probability sampling


• Every individual in a population has an
  equal chance of being chosen
  – In theory
  – In pract...
Probability sampling


Key types of probability sampling
• Simple random sampling
• Systematic sampling
• Stratified rando...
Nonprobability sampling


• Interview or measure without access to every
  individual in a population
 – Examples
• Situat...
Nonprobability sampling


• Convenience sampling
  – Selecting based on availability
  – Example: Hospital survey of nurse...
Nonprobability sampling


• Purposive sampling
  – Selecting participants based on knowledge of the
    population and foc...
Sample size in probability sampling


• Key questions:
 – How much might our results differ had we
   interviewed another ...
Sample size


• “Normal” curve
  – Mean, standard deviation,                                                 68%
    we ca...
Sample size


– Sample size of 385 is
  necessary for a
  confidence level of plus
  or minus 5 percentage
  points at the...
Case: National omnibus poll

• National random digit dialing completing surveys
  with 1,000 adults
• Conducted Friday thr...
Case: Online survey


– National online panel survey with 1,000 adults
– Balanced post-survey to census figures for age,
 ...
Case: Employee survey at a multi-
      division corporation


–   Four divisions
–   Management vs. non-management
–   Re...
Case: Veterinarian survey

– American Veterinary Medicine Association
   • 90,000 veterinarians under age 65
   • 50,000 v...
Case: Journalist survey

– Client: Financial services company
– Respondents: List of 1,000 journalists who cover
  persona...
Some resources

Public relations research
• Don W. Stacks and David Michaelson. 2010. A Practitioner's
  Guide to Public R...
Upcoming SlideShare
Loading in...5
×

Institute for public relations summit on measurement class measurement 201

975

Published on

Published in: Marketing, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
975
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Institute for public relations summit on measurement class measurement 201"

  1. 1. Measurement 201: Collecting quantitative information Institute for Public Relations Summit on Measurement David Geddes, Ph.D. evolve24, a Maritz Research company Saint Louis, Missouri david.geddes@evolve24.com October 6, 2010 Copyright © 2010 by The Institute for Public Relations 1
  2. 2. Today’s objectives • Smart consumer • Quantitative survey methods 1. Telephone 2. Web 3. Mail and multi-modal 4. Face-to-face • Sampling • Case studies • Questions … as they come Copyright © 2010 by The Institute for Public Relations 2 2
  3. 3. Great resources Copyright © 2010 by The Institute for Public Relations 3
  4. 4. First steps What are your objectives? What do you want to explore, discover, test, or document? Who are the right people to talk with? What are appropriate data collection methods? What is the value of the information? Copyright © 2010 by The Institute for Public Relations 4
  5. 5. Telephone surveys • Types – Random digit dial (RDD) – List from sample provider or panel – Company or client list (customers, employees, industry analysts, donors, partners, etc.) • Uses and advantages • Limitations and weaknesses Copyright © 2010 by The Institute for Public Relations 5 5
  6. 6. Telephone surveys: Current issues Cell phone only households – Same or different? – What to do? – Ethical and legislative issues – A trend to follow Copyright © 2010 by The Institute for Public Relations 6
  7. 7. Surveys in general: Current issues • Non-response -.74% bias Behavior Risk Factors Survey Response Rate -1.5% 100 90 Maximum Median All 80 States Response Rate 70 60 50 Pennsylvania 40 Minimum 30 20 7 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 Year Copyright © 2010 by The Institute for Public Relations
  8. 8. Are more intensive methods helpful? The Keeter et al study • Standard Survey: 36% response rate – Calling done over five days – Selected respondent from people at home at time of call (no random selection) – Five call-backs, one call-back to refusals • Rigorous Survey: 60.6% response rate – Eight-week calling period – Random selection of respondent from list – Pre-notification letters with $2 incentive – Multiple attempts (including letters to refusals) – Multiple call-backs Source: Scott Keeter et al, Consequences of Reducing Nonresponse in a National Telephone Survey, Public Opinion Quarterly 64:125-148 (2000) Copyright © 2010 by The Institute for Public Relations 8
  9. 9. Other design issues • Oversampling – Example: Attitudes to location-based apps – Example: National survey on water conservation • Weighting – Example: Ethnicity Copyright © 2010 by The Institute for Public Relations 9
  10. 10. Telephone omnibus polls • National random sample of 1,000 households • Offered by all major research firms • Fast • Low cost – Cost per question – Demographics included – Costing parameters • Deliverables • Applications • Limitations Copyright © 2010 by The Institute for Public Relations 10
  11. 11. Internet surveys • Advantages and uses • Limitations – http://www.aapor.org/Content/NavigationMenu/Home/Left/A APOROnlinePanelsTFReportFinalRevised.pdf Copyright © 2010 by The Institute for Public Relations 11
  12. 12. Mail surveys and multi-modal surveys • Don Dillman et al, Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, (Wiley, 2008) • TDM method: 1. Respondent-friendly questionnaire 2. Personalized correspondence 3. Token financial incentive ($1 or $2 prepaid) 4. Up to five phone contacts 5. Mail survey with stamped return envelopes 6. Phone again • Other ways to improve response rates Copyright © 2010 by The Institute for Public Relations 12
  13. 13. Sampling – Examine different sampling techniques – Strengths and shortcomings – Cases and examples – Tools to make decisions in practice – Not a comprehensive textbook treatment Copyright © 2010 by The Institute for Public Relations 13
  14. 14. Why care about sampling? • Formal definition of our targets – Example: Caregivers of Type II diabetes patients • Generalize or project? – Example: Poll of Kansas City voters about rental car tax • Understand and minimize sampling error – How far off might our result be if we interviewed another group of individuals? • Make tradeoffs – Budget, time, other factors given objectives Copyright © 2010 by The Institute for Public Relations 14
  15. 15. What is sampling? • Probability sampling – “Sampling is the science of systematically drawing a valid group of objects from a population reliably.” (Stacks, p. 150) • Non-probability sampling (informal definition) – Process of systematically drawing a group of objects from a population sufficient to meet information needs. (Adapted from Stacks) Copyright © 2010 by The Institute for Public Relations 15
  16. 16. Some definitions • Universe – General concept of who or what will be sampled • Population – People or units to be sampled, formally defined and described • Sampling frame – List of all people to be surveyed – Example: List of all 90,000 registered veterinarians under age 65 Copyright © 2010 by The Institute for Public Relations 16
  17. 17. Some definitions • Sample – Actual people chosen for inclusion in the research – Example: Selection of 10,000 veterinarians from the list • Completed sample – People who actually responded to the survey – Example: 3,000 veterinarians completed the survey Copyright © 2010 by The Institute for Public Relations 17
  18. 18. Types of error • Sampling error – Issue: Potential error or uncertainty as a result of not sampling from all members of sampling frame – How far off would we be if we interviewed a different 500 people? • Coverage error – Issue: The sampling frame does not contain all members of a population or contains a biased list – Example: People without landlines in a telephone poll – Example: People with invalid e-mail addresses in membership Copyright © 2010 by The Institute for Public Relations 18
  19. 19. Some definitions • Measurement error – Error when respondents misunderstand or incorrectly respond to questions • Nonresponse error – Respondents unlike nonrespondents Copyright © 2010 by The Institute for Public Relations 19
  20. 20. Problems and errors in sampling • Understanding and reducing coverage error – Does the sampling frame (list) contain everyone in the population? – Does the list contain people who are not in the sampling frame? – How is the list maintained and updated? – Does the list contain other information that can be used to improve sampling? Copyright © 2010 by The Institute for Public Relations 20
  21. 21. Three approaches 1. Census 2. Probability sample 3. Nonprobability sample Copyright © 2010 by The Institute for Public Relations 21
  22. 22. Census sampling • Interview or measure all members of a population – Example: Wal-Mart annual employee survey • No error due to sampling – Other types of error • Rare in practice • Is it worth the effort? Copyright © 2010 by The Institute for Public Relations 22
  23. 23. Probability sampling • Every individual in a population has an equal chance of being chosen – In theory – In practice • Allows generalization or projection to the population • Known sampling error parameters • What other sources of error? • How much to invest, given objectives? Copyright © 2010 by The Institute for Public Relations 23
  24. 24. Probability sampling Key types of probability sampling • Simple random sampling • Systematic sampling • Stratified random sampling • Cluster sampling Copyright © 2010 by The Institute for Public Relations 24
  25. 25. Nonprobability sampling • Interview or measure without access to every individual in a population – Examples • Situations where it is difficult to fully specify the population or sampling frame – Examples • Cannot generalize – How far off might our result be if we interviewed another group of individuals? • Key: Understand limitations … justify choice Copyright © 2010 by The Institute for Public Relations 25
  26. 26. Nonprobability sampling • Convenience sampling – Selecting based on availability – Example: Hospital survey of nurses leaving a shift • Quota sampling – Selecting based on availability but weight based on predetermined characteristics – Example: Mall intercept sampling Copyright © 2010 by The Institute for Public Relations 26
  27. 27. Nonprobability sampling • Purposive sampling – Selecting participants based on knowledge of the population and focus or objectives of the research – Example: Survey of most influential journalists covering the air transport industry • Volunteer sampling – Select based on agreement to participate • Snowball sampling – Selecting participants based on recommendations of other participants Copyright © 2010 by The Institute for Public Relations 27
  28. 28. Sample size in probability sampling • Key questions: – How much might our results differ had we interviewed another 100 American voters? – How much more would we learn, given our objectives, had we interviewed another 100 customers? – More technically, how much sampling and measurement error can we tolerate? • To reduce sampling error and measurement error, you must increase sample size Copyright © 2010 by The Institute for Public Relations 28
  29. 29. Sample size • “Normal” curve – Mean, standard deviation, 68% we can calculate confidence 95% intervals 99% – See an interactive demo at http://geographyfieldwork.com/StandardDeviation1.htm – Sample size calculators on Web • Maritz Stats (download) • National Statistical Service http://www.nss.gov.au/nss/home.NSF/pages/ Sample+Size+Calculator+Description?OpenD ocument Copyright © 2010 by The Institute for Public Relations 29
  30. 30. Sample size – Sample size of 385 is necessary for a confidence level of plus or minus 5 percentage points at the 95% confidence level. – Is this the biggest source of error? Copyright © 2010 by The Institute for Public Relations 30
  31. 31. Case: National omnibus poll • National random digit dialing completing surveys with 1,000 adults • Conducted Friday through Sunday • Balanced post-survey to census figures for age, gender, HHI, ethnicity (results only differ slightly) • Evaluation • Universe and population • Sampling frame • Sample and completed sample • Sources of error or bias • Final assessment – when is this appropriate? 31 Copyright © 2010 by The Institute for Public Relations
  32. 32. Case: Online survey – National online panel survey with 1,000 adults – Balanced post-survey to census figures for age, gender, HHI, ethnicity (results only differ slightly) – Evaluation • Universe and population • Sampling frame • Sample and completed sample • Sources of error or bias • Final assessment Copyright © 2010 by The Institute for Public Relations 32
  33. 33. Case: Employee survey at a multi- division corporation – Four divisions – Management vs. non-management – Results by age, gender, tenure at company – Which survey methods? – Develop a sampling plan: • Universe and population • Sampling frame • Sample and completed sample • Sources of error or bias • Final assessment Copyright © 2010 by The Institute for Public Relations 33
  34. 34. Case: Veterinarian survey – American Veterinary Medicine Association • 90,000 veterinarians under age 65 • 50,000 valid email addresses • Goal: Low-cost survey – Evaluation • Which methods? • Universe and population • Sampling frame • Sample and completed sample • Sources of error or bias • Can we work around the limits? • Final assessment and recommendation Copyright © 2010 by The Institute for Public Relations 34
  35. 35. Case: Journalist survey – Client: Financial services company – Respondents: List of 1,000 journalists who cover personal finance, the economy, and lifestyle. – Which survey methods? – Evaluation • Universe and population • Sampling frame • Sample and completed sample • Sources of error or bias • Final assessment and recommendation Copyright © 2010 by The Institute for Public Relations 35
  36. 36. Some resources Public relations research • Don W. Stacks and David Michaelson. 2010. A Practitioner's Guide to Public Relations Research, Measurement and Evaluation. Businessexpert Press. • Don W. Stacks. 2002. Primer of Public Relations Research. New York: Guilford Press. Market research (leading business school texts) • Gilbert A. Churchill and Dawn Iacobucci. 2004. Marketing Research: Methodological Foundations. Mason, OH: South- Western Cengage Learning. • Naresh K. Malhotra. 2007. Marketing Research: An Applied Orientation. Upper Saddle River, NJ: Prentice Hall. Copyright © 2010 by The Institute for Public Relations 36
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×