Online Research Arf Quality Enhancement Process Esomar Presentation

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    2 Favorites

    Online Research Arf Quality Enhancement Process Esomar Presentation - Presentation Transcript

    1. The ARF Quality Enhancement Process
      Joel Rubinson: Chief Research Officer, The ARF
      ESOMAR’s Online Research Conference
      October 28, 2009
      Contact Joel at:joel@thearf.org
      Follow him on Twitter @joelrubinson
    2. Stormy weather.
      A few years ago, Procter & Gamble publicly stated that they experienced online research results from successive projects that appeared to be inconsistent. Others shared similar experiences. Questions were raised publicly about “professional respondents”. The trustworthiness of online research was in question. Two years ago, in a spirit of collaboration and transparency, leading buyers and sellers came together at The ARF for a path forward.
    3. ORQC formed; working committees assembled
      “Foundations of Quality” $1MM piece of R on R was planned and executed in Oct/Nov ‘08
      • 17 panel companies, phone, mail
      • 700,000 panelists, over 100,000 online interviews in 3 waves
      An “industry solutions” committee was formed to translate insights into an action plan.
      …always buyers and sellers in partnership - always open and frank discussions.
    4. Participants in ORQC, Foundations of Quality, and Industry Solutions
      4
      4
    5. ARF Quality Enhancement Process V1.0
      Sept 29, 2009
      Confidential Review Document
      5
    6. ARF Quality Enhancement Process V1.0
      • A factually informed process with templates, definitions, metrics and declarations
      • Intended to bring structure to the conversations that buyers and sellers are having or need to have about data quality…control the root causes of problems
      • Will help buyers and sellers achieve their shared goal of providing valid and consistent data in a way that is effective, efficient, and does not violate privacy guidelines
      • A process, not a service or single solution
      • The approach is similar to Category Management
      Confidential Review Document
      6
    7. Common Goal
      Category Profit Growth
      Seller
      Buyer
      e.g. Unilever
      e.g. Kroger
      Research Company
      Trustworthy research to inform marketing decision making
      Building Collaboration
    8. Online research ecosystem requires industry alignment
      8
    9. QeP an industry approach that is constructive and encourages innovation
      9
    10. FoQFacts:
      Panels replicate their own answers but different online panels produce different answers so major changes to a panel, switching panels or changing the mix of panels from which respondents are sourced can lead to results that are not comparable across studies.
      Purchase intent for a concept is related to panelist longevity (newer panelists are more favorable).  Online panels vary in their longevity profiles.
      Even after considering many factors, study results still vary across panels – due to multiple panel characteristics working together to produce variation. No weighting scheme was able to completely smooth out this variation, which underscores the need for sample sourcing control.
      Taking multiple surveys per month (up to 10) is good in terms of respondent engagement
      Being on multiple panels less of an issue than presumed
      Fact-Based Process
      10
    11. Quality Enhancement Process 1.0 (QeP)
      1
      3
      2
      11
    12. Panel Profile Snapshot Report
      12
    13. Panel Consistency/Change Report
      13
    14. Step 2: Sample/Study Level Processes
      14
    15. Sample Level QeP Template
      15
      Details tomorrow at SMG in Chicago. Free to attend (physically or virtually), go to www.thearf.org to register
    16. Next Step: Pilot Testing QeP
      16
      Pilot testing scheduled for November; numerous concept testing, ad testing, and tracking programs, many suppliers already chosen
    17. Pilot Test Objectives
      Ensure that all templates, calculations, and definitions are crystal clear (e.g., same interpretation across the 8 tests)
      Assess workability of the QeP (did participants see this as facilitating or weighing down their discussions?)
      Were the reports useful and comprehensive in assessing suppliers and the comparability of waves of research?
      Were any issues flagged (and addressed) that led to quality improvement?
      Results will be shared within the QeP pilot program user group first, then more broadly to inform:
      QeP revisions and upgrades
      17
    18. ARF Online Data Quality Path forward
      Make learning available:
      3 Knowledge Briefs: Executive summaries of key facets of the analysis
      Draft QeP templates and guide
      Full FoQ 200+ page analytic report
      Encourage debate on contemporary science of online data quality
      Test and evolve industry approach via QeP
      Conduct regular industry briefings
      Ensure scale
      18
      More info and downloads at www.thearf.org

    + joel rubinsonjoel rubinson, 1 month ago

    custom

    734 views, 2 favs, 1 embeds more stats

    The ARF Quality Enhancement Process. Description o more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 734
      • 726 on SlideShare
      • 8 from embeds
    • Comments 0
    • Favorites 2
    • Downloads 22
    Most viewed embeds
    • 8 views on http://blog.vovici.com

    more

    All embeds
    • 8 views on http://blog.vovici.com

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories