Research on research: an examination of the quality-cost-speed factors in online qualitative research
1. Research on Research:
An Examination of the Quality, Cost, & Speed of
Online Qualitative Research
May, 2010
By
2. Introduction
For decades, face‐to‐face (F2F) interviewing, as the method for conducting qualitative research, has
been viewed as the optimal condition for collecting data. By being present at the time data are
collected, a skilled qualitative research consultant (QRC) can listen to and understand the direct
answers that participants give to the QRC’s questions and combine those data with visual cues from
non‐verbal behavior such as body language and facial expressions, as well as with word emphasis and
voice inflection. In so doing, that QRC may get a deeper understanding and a clearer view of
participants’ attitudes, self‐reported behavior, and the emotional basis for their actions, over and
above what is afforded by verbal data alone.
Yet, when we review the history of quantitative market research, we see that what began as F2F
interviewing (whether done by door‐to‐door or mall intercepts), eventually shifted to different
methodologies which capitalized on advances in technology. These became opportunities for our
industry to drive down study costs and time requirements. Over time, study methods migrated to
computer assisted telephone interviews (CATI) which, in turn, was followed by the Internet. Each shift
was caused by the cost and time benefits to organizations that commissioned research, as well as the
proven viability of the method to the market research industry. Nowadays, it is only under special
circumstances that a quantitative study will deploy a F2F method, having to do with certain
characteristics of the target population or the test stimuli or some other unique set of conditions.
Granted, quantitative research does not have at its core the quality standard of yielding data that are
in‐depth and emotionally‐based as does its qualitative counterpart. Rather, quantitative data that are
high quality are closely representative of a population so that statistical tests of significance can be
performed and inferences can be drawn. In contrast, qualitative research must yield data that reflect
the dynamic interrelationships between an individual’s thoughts, feelings, and actions. These studies
are designed to pose short, open‐ended questions, receive detailed answers and, based on those
answers, probe with other questions that may not necessarily have been planned before the outset of
the study. The QRC must use special interviewing skills that enable the discussion to veer into
unplanned directions and otherwise seize opportunities for a deeper glimpse into the inner workings of
study participants’ hearts and minds.
Perhaps now we have reached a point during which new technologies may be used in qualitative
research to provide similar reductions in cost and time and provide more options to QRCs in offering
new, alternative solutions to clients’ needs that may be more appropriate than traditional ones alone.
For that matter, these same technologies could also be used to augment qualitative research by
combining them with F2F participant interaction. Study costs and cycle times may be sharply reduced,
surely. But if the quality of the data was comparable or represented a worthwhile trade‐off, the entire
market research industry and all of its participating members would benefit. QRCs, in particular would
be able to provide solutions to clients that may be more appropriate to the study objectives using
entirely non‐F2F or hybrid methods, and this would enable them to provide better service. Also, there
2
3. would be increased capacity by freeing up time that would be otherwise spent traveling. Less
considered but of distinct importance is that the carbon footprint associated with a given study can be
reduced or virtually eliminated, too. So, the question may be asked “are there Internet‐based
technologies that can be leveraged to conduct in‐depth qualitative research in a way that affords the
professional QRC the opportunity to exercise all the alacrity and prowess that is brought to bear in a
typical F2F study?”
As it is, social media usage has developed and become widespread very quickly. Social networks,
blogsites, and other online “beehives” appear to be all around us. Millions of people continue to adopt
social media sites as their primary source of all kinds of information. Over the past few years, more
and more of the general population has embraced the social networking phenomenon. Blogging, for
example, has become widespread and continues to grow. The term “blog” (a portmanteau of the term
"web log") is a type of website usually maintained by an individual with regular entries of commentary,
descriptions of events, or other material such as graphics or video. This term only recently came into
existence, but now is a household word. For all intents and purposes, blogging has become a
comfortable, almost natural form of interpersonal communication in today’s society.
So, if blogging is as common a behavior as we assume, we can combine the ease with which general
consumers blog with current mainstream technologies, such as broadband connectivity and digital
technology, and use it to facilitate qualitative studies in market research. The potential benefits of
doing so include those for organizations, QRCs, and the market research industry as a whole.
Based on these observed conditions, we set out to test the viability of an online platform to facilitate a
truly in‐depth study in qualitative research, an ethnography, which we reasoned would be the optimal
litmus test of these assumptions. In essence, if we were successful in conducting a “BlogNography,”
we would have demonstrated that we can use technology to improve conditions for qualitative
research in general. It was our main goal in this research to test and then tell, what we think, is a very
compelling story to research professionals. That is, conditions have emerged in society and in
technology that, when aligned, can alter the way in which qualitative research is done. The benefits to
be gained are as great and similar in nature to those enjoyed by changes in methodology in
quantitative research. As such, our mission now is to begin the process of raising industry‐wide
consciousness to the benefits of replacing in‐person qualitative research with online platforms that are
designed to capitalize on society’s comfort level with blogging and enable consumers to share their
lives, stories, aspirations, fears, and hopes using textual and visual data.
Background
What is more basic and fundamental to our lives than the American family? How better can we
understand what is important to the family than by studying them while under challenging conditions,
such as dinner time in the home? We chose this topic because we sensed that it would be rich and
fertile ground for us to explore. We assumed that if we were successful in achieving what qualitative
research is designed for (to develop an in‐depth understanding of human behavior and the reasons
3
4. that govern such behavior) using the Internet rather than face‐to‐face methods, it would be a clear
demonstration of the viability of the online medium for qualitative research.
Life today typically contains several competing objectives and goals. Raising children, jobs, school,
friends, extra‐curricular activities, volunteerism, homework, relaxation, and sleep all have the potential
to impinge upon the day and its schedule and demand attention at the expense of the other pressing
needs. Since we are all part of a family, we know that there are many factors that shape, mold,
facilitate, inhibit and impinge upon how the “typical dinner” is executed in a family. Consider social
needs, historical traditions, biological and nutritional requirements, financial and budgetary limitations,
religious practices, physical space, gender roles, political divisions of labor, time pressures,
occupational characteristics, and not least of all, culinary skills. The constellation of all these factors
contributes to the unique manner in which dinner is done for each individual family, and their blogging
provided a looking‐glass into their experiences and associated emotional content.
The dinner process provides a direct view of the real value placed by the family on getting together,
conversing, exchanging experiences, discussing plans, developing strategies, and surviving as a family
unit. There is only so much time and, more and more these days, there is less free time. Arguably,
dinner is the event during the day when the distinguishing characteristics of any family unit is most
starkly evident. Indeed, the way a family processes dinner manifests and reveals what is most
important to the group and to the individuals that are members. Combining their blogs with our
ethnographic research approach yielded a “BlogNography” on the American family as we immersed
ourselves in their lives through cyberspace and witnessed their account of whether and to what extent
their lives have changed and on what path they are heading in the future.
To facilitate this endeavor, we used BlogNogTM (see http://blognogresearch.com), an online platform
specifically designed to capitalize on today’s blogging phenomenon and able to leverage the
widespread access in homes to broadband connectivity and digital technology to test the viability of
conducting an online ethnography.
4
5. Methodology
We recruited 250 respondents to this study vis‐à‐vis the following screening criteria:
• Aged 18 or older
• Not employed in food service industry
• Not employed in sensitive industry (advertising/public relations, market research)
• Presence of at least one child aged 17 years old or less in the household
• Access to the Internet and to a digital camera or mobile phone camera, knowledge of how to
upload photos onto a computer, and agreement to do so
The data collection process lasted one week beginning January 14, 2010 and ending January 21, 2010.
Each respondent was given three (3) distinct assignments in which to participate, and we inserted
specific probing questions to individual respondents based on their blogged responses to our standard
questions over the course of this time period. On average, each participant spent a total of 60 to 90
minutes in fulfilling their requirements for this study. Special services were provided, with our
appreciation, by Decipher who programmed the screener and managed the recruitment process for us.
Likewise, Survey Sampling International provided the list of its panel members from which we
randomly sampled to enlist our study respondents. The following is an outline of the Discussion Guide.
ASSIGNMENT 1: Present
Take photos of the inside of the refrigerator, freezer, and pantry (or other places in which you store dry
goods) and upload them to your computer, then to your account in BlogNog.
Describe your typical dinner process beginning with grocery shopping, then meal preparation, serving,
eating and cleaning up. Include answers to who, what, where, when, and how.
ASSIGNMENT 2: Past and Future
Describe the typical dinner process of your family when you were a child along the same lines as above
and compare that to the current dinner process. Then predict the future dinner process for your
child(ren) and comment on the influences of current practices on the future.
ASSIGNMENT 3: Projectives using Archetypal Symbols
Choose separate images that best represent:
1. your typical dinner process
2. you in relation to your typical dinner process
3. your IDEAL dinner process
4. how you would be transformed by the IDEAL dinner process if it existed nowadays
5
6. Detailed Findings: Current Dinner Practices
Nowadays, getting dinner on the table is not a simple matter; perhaps it never was. Several important
factors play a significant role in how the dinner process for any given family unfolds. For some, it is
easier than for others, but most balance their job schedule, budget, living condition, cooking space,
and culinary skill against their needs for tradition, customs, food preference, nutrition and family
togetherness. Also, they do this while abiding by the laws of physics, in terms of time and space, to
prepare, serve, eat, and clean up after the meal is consumed.
Some general information was found, such as that “Moms” are still the household member that is the
primary dinner chef. It is mainly her that cooks the bacon regardless of whether she brings it home or
not. However, more frequently men and children are responsible for supporting roles in the process,
nowadays, whether it is in preparation, serving and/or clean up. According to these consumers’
recollections, their mother was usually the one responsible for dinner when they were children.
Another piece of general information found is that people’s schedules have more influence on dinner
than it did when these consumers were children growing up in their homes. As such, there is a greater
societal reliance on prepared foods from various providers such as grocery stores’ prepared or frozen
meals, restaurants’ eat‐in or take‐out, fast food, drive‐thru, etc.
As much as I wish we can eat like a typical family around the dinner table, we usually eat working on the
computer or watching TV. Maybe I’m a bit lazy. I truly dislike the fact that we are not like the traditional
family that eats at the dinner table, as I did as a child. I feel like a failure both as a woman and as a wife and
mother because of my inabilities. This is not how I want my daughter to think "family life" is supposed to be.
Because I am not creative enough nor a well enough cook, our meals usual consist of things like Hamburger
Helper, Betty Crocker prepared meal kits or kits I can throw in the crock pot. On occasion I will throw
something together from a recipe book but it never comes out right. My ideal dinner would be the whole
family gathered around the table to a "nice" well‐balance home cooked meal. Talking about our days as we
enjoy each other's company. We hardly ever go out to eat simply because I cannot afford it. I just recently lost
my job and money is very tight. Clean up usually falls upon me too. Since we do not have a dish washer, I have
a whole day (sometimes 2 or 3 days) of dishes that have to be washed by hand.
As family members describe their dinner process, they reveal aspects of their family life that are deeply
felt, and it is these aspects that most poignantly describe their plight in life. As they do so, what
emerges are a set of factors of varying levels of impact on each family’s unique dinner process. These
factors can be dichotomized as external – somewhat beyond the family’s control, such as their budget
and job schedule and internal – those factors that families impose upon themselves and somewhat
within their control, such as nutrition requirements, family traditions and culinary skill. Below is a
description of how each of these factors affects the dinner process of the American Family as well as
how they interact with each other in some cases.
6
11. We did not eat dinner together. Most often ate when we wanted to go make something for ourselves. If
dinner was prepared for the family it was prepared and cleaned by my mother. During dinner we would eat
alone or in front of the television. I personally wanted to have family meals. At least one meal a day that
everyone came together to recap on what happened. I would have liked to have more balanced meals and not
something a child could think of and prepare themselves. So that’s how I run things in my home – the right
way.
Some even have profoundly negative memories associated with dinner in the distant past and have
taken great effort to positively change things and erase those memories.
We ate together a couple times a week. Often it was fend for yourself. I believe this is because both my
parents worked and were overwhelmed. Dinner was unpleasant b/c of the high level of stress and an
unhealthy view of food. When a full meal was cooked it was usually baked ham, chicken and broccoli, broiled
fish and other yummy well‐balanced meals. When it was fend for yourself, it was microwavable meals or
snacks, at least once or twice a week we ate fast food meals. (McDonald's, Hardees, or Burger Kind) My
stepmother usually prepared the meals and the kids took turns cleaning up as part of our chores. We usually
either all ate in the living room watching T.V. or in separate rooms watching T.V. or doing homework. My
sister and I found dinner unpleasant b/c we were often criticized for our portions and commented about our
weight, even though in hindsight we were within a normal weight range for our age and height. My sister and
I are still rather bitter about dinner time in our family so we both try very hard to change things for our own
family and children. I’d rather not even think about those days b/c I get angry.
Conclusions and Implications
As mentioned, the entire study was conducted online as respondents, representing their own
respective family, blogged responses to our questions about the entire process of dinner in their
homes: from grocery shopping to food preparation, serving, eating, and cleaning up. We studied their
memories of their families when they were children growing up, and finally, posed several questions to
have them predict what it will be like for their children when they have families of their own.
We set out to test whether in‐depth qualitative research can be conducted online in a manner that
produces consumer insights at least as well as traditional in‐person methods while capitalizing on
current technologies that can improve other research study factors such as cost, speed, effort, and
data quality. Since blogging is a mode of communication that has been so widely adopted and
comfortable for many consumers, we set our sights on performing the optimal litmus test – an online
ethnography; one in which participants were required to tell us about some of the most poignant
experiences from their past, confess to aspects of their present lives that they feel are less than
perfect, and share their deepest desires for the future, not only for themselves, but more importantly,
for their children. We collected hundreds of artifacts in the form of photos of participants’ homes,
where they store, prepare, and eat their dinners. Indeed, we received more than we bargained for.
The consumer insights unearthed in this study are foundational in nature. That is to say, they are the
kind of data that are used to provide a framework for the development of organizational initiatives
11
12. such as a creative brief for advertising strategies or the grist for ideation sessions to seed product
development pipelines. Quite easily, other objectives could have easily been brought to bear in this
study, such as customer experience, naming/positioning, copy testing, and concept/product testing.
But doing so was out of scope for this study whose sole purpose was to execute and witness whether
online qualitative research, even ethnography, was reachable.
Our conclusion is that it is. Moreover, consumers seem to enjoy the assignments and their
participation was worthwhile.
I did enjoy this survey!!! You really made me look hard at my family's dinner habits and made me think about
what I could do to possibly change our family meal time.
This was a wonderful study, I thoroughly enjoyed it. Very creative!
Thank you for the opportunity to participate. This was fun.
I did enjoy this survey, I realized that I need to make some changes with my family meal time.
Key Takeaways
• Ethnography conducted successfully online
• Rich, symbolic, emotional‐based data were collected
• Data were gathered in the context of the subject of study where artifacts were gathered
• 250 participants involved and all data were collected in one week
• Online platform served as blogspace (http://www.blognogresearch.com/) for participants to
tell their stories
• Hundreds of photos, videos and mountains of text‐based data gathered
• Projective technique was utilized
• New solutions and enhanced service to clients’ needs may be provided
• Could be used to inform the design of quant survey for subsequent phase of research
• Could be used to provide the basis for creative brief development
• Could be used to provide grist for product development ideation
• Could be used for competitive intelligence gathering
• No traveling, no scheduling, no logistical constraints
• Fun and engaging for respondents despite the amount of work they were required to do
• Entirely “green” – virtually no carbon footprint
If we assume that the quality of the data in this study are comparable to that which could have been
collected using a traditional in‐person process of conducting these ethnographic interviews, then a
review of cost and time requirements will provide a more thorough evaluation of the viability of this
methodology. The table below is an attempt to directly compare cost and time factors among
standard qualitative research methods by holding constant the number of respondents involved in a
given study.
12
13. Study Focus In-Depth Online
Characteristics Groups Interviews Ethnography Ethnography
# Respondents 60 60 60 60
# Minutes input per 10 50 120 120
respondent
Cycle Time to 5 weeks 2 months 3 - 4 months 3 - 4 weeks
deliver final report
Total Cost of Study $50,000 $60,000 $250,000 $50,000
Cost per minute $83.33 $20.00 $34.72 $6.95
per respondent
While there may be variations on both cost and time from one study to another or one research
provider or another, the cell entries in the above table are only estimates based on general
experiences and conventional wisdom in the market research industry.
In essence, for approximately the same cost of a round of six focus groups, an online ethnography can
be conducted yielding over ten times the amount of respondent input. Generating that much more
data drives down the cost per minute per respondent by over 90%. In addition, the total amount of
time required to complete the study is reduced by one to two weeks. Likewise, in comparison to F2F
ethnography, conducting an online ethnography can cut the cost of the study by about 80% and deliver
a final report four times as fast.
This is not to say that all qualitative studies would be better served by an online method. Rather, the
online technology that exists today may serve to provide new options, alternatives, and enhanced
solutions, whether done in substitution for F2F or in combination with that method. As such, QRCs
who embrace the new technologies may be in a position to better serve their clients. Client
organizations may benefit through more in‐depth consumer insights and more bang for the research
buck, speedier‐to‐market solutions, enhanced geographic reach, and less wear and tear and downtime
on its members.
13