The ARF Quality Enhancement ProcessJoel Rubinson: Chief Research Officer, The ARFESOMAR’s Online Research ConferenceOctober 28, 2009Contact Joel at:joel@thearf.orgFollow him on Twitter @joelrubinson

Stormy weather.A few years ago, Procter & Gamble publicly stated that they experienced online research results from successive projects that appeared to be inconsistent. Others shared similar experiences. Questions were raised publicly about “professional respondents”. The trustworthiness of online research was in question. Two years ago, in a spirit of collaboration and transparency, leading buyers and sellers came together at The ARF for a path forward.

ORQC formed; working committees assembled“Foundations of Quality” $1MM piece of R on R was planned and executed in Oct/Nov ‘0817 panel companies, phone, mail700,000 panelists, over 100,000 online interviews in 3 wavesAn “industry solutions” committee was formed to translate insights into an action plan. …always buyers and sellers in partnership - always open and frank discussions.

17 panel companies, phone, mail

700,000 panelists, over 100,000 online interviews in 3 waves

Participants in ORQC, Foundations of Quality, and Industry Solutions44

ARF Quality Enhancement Process V1.0Sept 29, 2009Confidential Review Document5

ARF Quality Enhancement Process V1.0 A factually informed process with templates, definitions, metrics and declarations Intended to bring structure to the conversations that buyers and sellers are having or need to have about data quality…control the root causes of problems Will help buyers and sellers achieve their shared goal of providing valid and consistent data in a way that is effective, efficient, and does not violate privacy guidelines A process, not a service or single solution The approach is similar to Category ManagementConfidential Review Document6

A factually informed process with templates, definitions, metrics and declarations

Intended to bring structure to the conversations that buyers and sellers are having or need to have about data quality…control the root causes of problems

Will help buyers and sellers achieve their shared goal of providing valid and consistent data in a way that is effective, efficient, and does not violate privacy guidelines

A process, not a service or single solution

The approach is similar to Category Management

Common GoalCategory Profit GrowthSellerBuyere.g. Unilevere.g. KrogerResearch CompanyTrustworthy research to inform marketing decision makingBuilding Collaboration

Online research ecosystem requires industry alignment8

QeP an industry approach that is constructive and encourages innovation9

FoQFacts:Panels replicate their own answers but different online panels produce different answers so major changes to a panel, switching panels or changing the mix of panels from which respondents are sourced can lead to results that are not comparable across studies. Purchase intent for a concept is related to panelist longevity (newer panelists are more favorable). Online panels vary in their longevity profiles. Even after considering many factors, study results still vary across panels – due to multiple panel characteristics working together to produce variation. No weighting scheme was able to completely smooth out this variation, which underscores the need for sample sourcing control.Taking multiple surveys per month (up to 10) is good in terms of respondent engagementBeing on multiple panels less of an issue than presumedFact-Based Process10

Quality Enhancement Process 1.0 (QeP)13211

Panel Profile Snapshot Report12

Panel Consistency/Change Report13

Step 2: Sample/Study Level Processes14

Sample Level QeP Template15Details tomorrow at SMG in Chicago. Free to attend (physically or virtually), go to www.thearf.org to register

Next Step: Pilot Testing QeP16Pilot testing scheduled for November; numerous concept testing, ad testing, and tracking programs, many suppliers already chosen

Pilot Test ObjectivesEnsure that all templates, calculations, and definitions are crystal clear (e.g., same interpretation across the 8 tests)Assess workability of the QeP (did participants see this as facilitating or weighing down their discussions?)Were the reports useful and comprehensive in assessing suppliers and the comparability of waves of research?Were any issues flagged (and addressed) that led to quality improvement?Results will be shared within the QeP pilot program user group first, then more broadly to inform:QeP revisions and upgrades17

ARF Online Data Quality Path forwardMake learning available:3 Knowledge Briefs: Executive summaries of key facets of the analysisDraft QeP templates and guideFull FoQ 200+ page analytic reportEncourage debate on contemporary science of online data qualityTest and evolve industry approach via QePConduct regular industry briefingsEnsure scale18More info and downloads at www.thearf.org