Online Research Arf Quality Enhancement Process Esomar Presentation

The ARF Quality Enhancement Process
Joel Rubinson: Chief Research Officer, The ARF
ESOMAR’s Online Research Conference
October 28, 2009
Contact Joel at:joel@thearf.org
Follow him on Twitter @joelrubinson

Stormy weather.
A few years ago, Procter & Gamble publicly stated that they experienced online research results from successive projects that appeared to be inconsistent. Others shared similar experiences. Questions were raised publicly about “professional respondents”. The trustworthiness of online research was in question. Two years ago, in a spirit of collaboration and transparency, leading buyers and sellers came together at The ARF for a path forward.

ORQC formed; working committees assembled
“Foundations of Quality” $1MM piece of R on R was planned and executed in Oct/Nov ‘08
17 panel companies, phone, mail
700,000 panelists, over 100,000 online interviews in 3 waves
An “industry solutions” committee was formed to translate insights into an action plan.
…always buyers and sellers in partnership - always open and frank discussions.

Participants in ORQC, Foundations of Quality, and Industry Solutions
4
4

ARF Quality Enhancement Process V1.0
Sept 29, 2009
Confidential Review Document
5

ARF Quality Enhancement Process V1.0
A factually informed process with templates, definitions, metrics and declarations
Intended to bring structure to the conversations that buyers and sellers are having or need to have about data quality…control the root causes of problems
Will help buyers and sellers achieve their shared goal of providing valid and consistent data in a way that is effective, efficient, and does not violate privacy guidelines
A process, not a service or single solution
The approach is similar to Category Management
Confidential Review Document
6

Common Goal
Category Profit Growth
Seller
Buyer
e.g. Unilever
e.g. Kroger
Research Company
Trustworthy research to inform marketing decision making
Building Collaboration

Online research ecosystem requires industry alignment
8

QeP an industry approach that is constructive and encourages innovation
9

FoQFacts:
Panels replicate their own answers but different online panels produce different answers so major changes to a panel, switching panels or changing the mix of panels from which respondents are sourced can lead to results that are not comparable across studies.
Purchase intent for a concept is related to panelist longevity (newer panelists are more favorable). Online panels vary in their longevity profiles.
Even after considering many factors, study results still vary across panels – due to multiple panel characteristics working together to produce variation. No weighting scheme was able to completely smooth out this variation, which underscores the need for sample sourcing control.
Taking multiple surveys per month (up to 10) is good in terms of respondent engagement
Being on multiple panels less of an issue than presumed
Fact-Based Process
10

Quality Enhancement Process 1.0 (QeP)
1
3
2
11

Panel Profile Snapshot Report
12

Panel Consistency/Change Report
13

Step 2: Sample/Study Level Processes
14

Sample Level QeP Template
15
Details tomorrow at SMG in Chicago. Free to attend (physically or virtually), go to www.thearf.org to register

Next Step: Pilot Testing QeP
16
Pilot testing scheduled for November; numerous concept testing, ad testing, and tracking programs, many suppliers already chosen

Pilot Test Objectives
Ensure that all templates, calculations, and definitions are crystal clear (e.g., same interpretation across the 8 tests)
Assess workability of the QeP (did participants see this as facilitating or weighing down their discussions?)
Were the reports useful and comprehensive in assessing suppliers and the comparability of waves of research?
Were any issues flagged (and addressed) that led to quality improvement?
Results will be shared within the QeP pilot program user group first, then more broadly to inform:
QeP revisions and upgrades
17

ARF Online Data Quality Path forward
Make learning available:
3 Knowledge Briefs: Executive summaries of key facets of the analysis
Draft QeP templates and guide
Full FoQ 200+ page analytic report
Encourage debate on contemporary science of online data quality
Test and evolve industry approach via QeP
Conduct regular industry briefings
Ensure scale
18
More info and downloads at www.thearf.org

http://blog.vovici.com	80
http://www.slideshare.net	11
http://translate.googleusercontent.com	3
http://74.6.238.254	1

Online Research Arf Quality Enhancement Process Esomar Presentation

by joel rubinson on Oct 28, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 95

Statistics

Online Research Arf Quality Enhancement Process Esomar Presentation — Presentation Transcript