Recent Findings on Using Rich Media in Online Surveys Bernie Malinoff element54 Jeffrey Henning, PRC Vovici This presentation will explore some hotly debated questions about data consistency, and what respondents are actually doing with our online surveys — spanning the most widely used 'radio buttons', to the more recently introduced 'rich media' types. Researchers concerned about declining online survey response rates have rushed to embrace "rich media" (aka Web 2.0) as a way to enliven the survey taking process and reignite respondent engagement. Yet engagement is not the same as entertainment. When can rich media foster engagement, reliability and validity and when can rich media damage it? Bernie Malinoff, CEO of element54, and Jeffrey Henning, founder of Vovici, have both independently conducted research-on-research studies comparing "traditional" online questionnaires to "rich media" online survey formats. Both have consistently seen dramatic variations in results.

1. Recent Findings on Using Rich
Media in Online Surveys
Bernie Malinoff
Jeffrey Henning, PRC

2. Panning for Respondents

3. Engaged Respondents are Golden

4. Losing Respondents Along the Way
Invited
ParRcipated
Completed
Engaged
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Source: Vovici

5. But is a Ghost Town our Future?

6. Paper Surveys Inspired the Web

7. Maybe Technology is the Answer?

8. Web Surveys as Entertainment

9. What Can We Take to the Bank?

10. ELEMENT54 R&D

11. Drivers of Survey CooperaRon
Topic Interest
QuesRonnaire
Panel/Client
Length/
ReputaRon
IncenRve
Respondent
Survey Invite CooperaRon
Visual Appeal

12. Typical respondent experience
Respondent
Onscreen
ExpectaRons
Layout
"Engagement Response
& Pa^erns/
Enjoyment" Flatlining

13. Divergence/Variety of New QuesRon Types
Sources: Spyglass Solutions/2x4 Research & Consulting

14. ‘Black Box’ Phase of Online Design
Project Kick‐off Interviewing
Briefing Coding
Planning TabulaRon
Sample Analysis
Draj(s) ... Report
Programming
Final Delivers a
QuesRonnaire “Link”
Researcher Researcher/
Gives to Client Review
Programming

15. ‘Respondent Engagement’
Converging Issues
Data Respondent
Consistency Usability

16. Major R&D IniRaRves
Data Consistency Respondent Usability

17. Balanced 5 Cell Design:
2000 Total Interviews
Radio Bu^on
Anchor @ ‘1’
Anchor @ ‘5’
Anchor @ ’10’
Experimental

18. QuesRonnaire Coverage

19. Study Highlights
+8% overstated
past 7 day consumpRon ‐ slider vs. radio bu^on
‐8% understated
issue importance ‐ slider vs. radio bu^on
36% variance
how ojen brush teeth ‐ slider vs. radio bu^on

20. eg ‐ NetPromoter Score (NPS)

21. Given Differences, What is the Benchmark?
Radio Bu^on Sliders Drag n Drop
Default, Easier Scales/Anchors CauRon:
Standards Ma^er Contents Hot

22. Survey CompleRon Time
& Respondent Enjoyment
Rich Media vs. Radio Bu^on QuesRons

23. Eyes Don’t Lie ‐ Research Protocol
Eye Tracking
Recording
Interviewing

24. Eye Tracking

25. "1.0 vs. 2.0"
Using Eye Tracking for Survey Usability
... to better understand the destination, by examining the journey.

26. Eye Tracking ‐ 'Areas of Interest'

27. Inference vs. InstrucRon

28. Inference vs. InstrucRon
42% did not read
the quesRon
33% did not read
the "tradiRonal"
quesRon

29. Don’t Interrupt Me When I’m Thinking

30. Error PrevenRon & Recovery
This respondent needed 5
minutes to complete this
quesRon and encountered
3 error messages.
The quesRon is ‐ how do
we treat the data which
emerges from someone
who tried hisng "next",
and then added or changed
data to escape?

31. Error PrevenRon & Recovery
Respondent interpreted the error message
locaRon as a guide to where to put the answer.
RecommendaRon ‐ have some way to highlight
the enRre row, avoiding any visual cues or
biases.

32. Minimum Standards ‐ eg: Horizontal Layout
Horizontal layout ensures consistent viewing, regardless of screen size

33. Equal RaRngs for ‘Engagement’ Metrics
1.0 2.0
Survey
Enjoyment
8.20 8.25
Ease of Use 8.86 8.85
Desire to
Repeat 7.02 6.56

34. Sexy QuesRons, Dangerous Results?
ConsideraRons
1. There is no Be^er, Faster, Cheaper ‐ you can only pick 2.
2. We must return to the usability principles which guided
paper survey design, not CATI design, to maximize
confidence in our data, and business decisions.
3. There are ways to make all quesRons/surveys more
appealing, intuiRve and future‐relevant.
4. 'Responsible InnovaRon' is key.
5. As an industry, we must devise and reinforce best
pracRces for online research interfaces, like we have for
so many elements of the research process.

35. Tips for MigraRng to Rich Media
• DiscreRon is the be^er part of valor
• Split ballot test rich media quesRons against
tradiRonal quesRons to idenRfy differences
• Visual choices introduce bias, especially for
more abstract choices

36. Technology is Not the Only Answer

37. Tips for Engagement
• Keep the survey short
• Make the quesRons interesRng
• Minimize the use of matrixes
• Use rich media in a considered manner

38. CreaRng Engaging Surveys is Hard

39. We Welcome Your QuesRons
Bernie Malinoff NPR Listens:
How Panel Management helps NPR
President
‘Dial In’ on Unmet Needs
element54
Lessons from a loyalty Leader:
A conversa>on with Navy Federal Credit Union
h^p://twi^er.com/BernieMalinoff
Bernie.malinoff@element‐54.com
(514) 833‐2268 Jeanne Bliss:
Earn your Customers Rave
Jeffrey Henning
Founder & VP, Strategy
Vovici Blog
Vovici
Vovici ’s Jeffrey Henning blogs about survey research,
h^p://twi^er.com/jhenning EFM, voice of the customer and online feedback.
Jhenning@vovici.com blog.vovici.com
(800) 787‐8755

40. Contact InformaRon
Bernie Malinoff Jeffrey Henning, PRC
• bernie.malinoff@element‐54.com • jhenning@vovici.com
• 514‐833‐2268 • 781‐848‐8100 x 550
• @berniemalinoff • @jhenning
• h^p://www.element‐54.com • h^p://www.vovici.com