Rethink your Research Design
•Download as PPTX, PDF•
1 like•241 views
Are you still separating sample and surveys for your multi-mode market research projects?
Recommended
Recommended
More Related Content
Viewers also liked
Similar to Rethink your Research Design
Similar to Rethink your Research Design (20)
Recently uploaded
Recently uploaded (20)
Rethink your Research Design
- 1. NorthWest MRA Bay Area Research Forum May 8, 2015 Rethink Your Research Design Brian Fowler COO | CFMC Survox Solutions
- 2. Good data begets great insight. © Copyright 2015 CFMC | All rights reserved. 27/24/2015
- 3. Are you reaching the right audience?
- 4. Rethink Your Research Design 2015 Bay Area Research Forum • Representative • Insightful • Inclusive • Respondent-focused 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 4
- 5. Data Collection Options CFMC | Copyright 2014 5
- 6. Research Decision | Which Mode(s) to Use? 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 6 COSTS RESULTS - Time - Quantity - Demographics
- 7. Rethink Your Survey Design 2015 Bay Area Research Forum Silo’d Surveying SOP 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 7 Web Phone IVR Client DIY / D.C. Partner Phone Specialists IVR Specialists Often at odds with today’s communication preferences In Perso n Qual/D.C. Partner
- 8. Rethink Your Survey Design 2015 Bay Area Research Forum Consider Your Audience Most of us: • Have preferred communication vehicles • Prefer to give feedback when convenient Why limit the opportunity to provide feedback & input to one channel? 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 8
- 9. Strategic value Multi-mode approach Research design impact Technology enablement Multi-Mode data collection 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 9
- 10. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 10 Web interviewing
- 11. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 11 Phone interviewing
- 12. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 12 IVR* answering, screening, interviewing * Interactive Voice Response
- 13. + Multi-Mode Data Analysis 13© Copyright 2015 CFMC | All rights reserved.7/24/2015 O N L I N E P H O N E + I V R
- 14. Why Web?
- 15. Why Web? Speed Cost Convenience Researchers Respondents
- 16. Why Phone?
- 17. Who are you missing? Who are they? Non-white Over 65 Under $30,000 No high school degree Rural © Copyright 2015 CFMC | All rights reserved. 17 13% US adults 43% Age 65+ 23% Income < $30,000 24% High School or less
- 18. Reach the right people Fill quota & minimize weighting Complete projects faster Why Phone? Reach demographics not accessible online Validate identities better Target under-quota segments to complete study faster Prompt and Probe participants’ responses Record the true Voice of the Consumer CFMC | Copyright 2014 18
- 19. SU RVEYURLPUSH PULL CALL CENTER Why Phone 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 19
- 20. Why IVR?
- 21. Why IVR? Speed Cost Convenience for Respondents Flexibility Push and pull Unattended survey or Agent Partial or complete interview
- 22. SURVEY COLLECTION METHODS DATA SET FOR ANALYSIS MULTI-MODE RESEARCH 22© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 23. TALK ISN’T CHEAP
- 24. COST COMPONENTS SAMPLESAMPLESAMPLE TELCO LABOR TELCO PHONE INTERVIEW IVR INTERVIEW ONLINE SURVEY $ $$ © Copyright 2015 CFMC | All rights reserved. 24
- 25. CUSTOMERS New Product Development Feature Testing Pricing Analysis Customer Feedback Choice of Mode for Loyalty Programs Customer Experience Outbound Follow-up Transfer to Survey 25© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 26. Differentiate by Value SURVEY DESIGN One survey → different groups Short online survey for majority Additional questions for select segments COLLECTION MODE Mode → reflects respondent status Online survey the majority Phone interview select segment Visit in-person the elite 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 26
- 27. Saves Time & Labor While Eliminating Inconsistencies SURVEY CREATION 1 vs. 3 surveys SAVINGS: LABOR & TIME in preparation. Survey to survey differences eliminated. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 27 SURVEY COLLECTION METHODS DATA SET FOR ANALYSIS DATA ANALYSIS 1 data set vs. 3 data sets in various formats SAVINGS: LABOR & TIME to export & import across solutions
- 28. NATIONAL RETAILER | A multi-mode story…. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 28
- 29. Increase Response Rate | Expand Reach 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 29 URL + 800# 15% higher overall 70% more Seniors 25% more customers who are… • Rural resident • Lower income
- 30. Offer Engagement Choices 30 Invitation URL WEB Survey PHONE # In-coming Call PHONE Survey IVR Survey IVR Screen © Copyright 2015 CFMC | All rights reserved.7/24/2015
- 31. EMPLOYEES Workforce Development Key Employees In-Depth Drill Downs Employee Feedback Choice of Modes Proactive Sampling Exit Interviews Key Employees Strategic Functions 31© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 32. Differentiate by Value SURVEY DESIGN One survey → different groups Short online survey for majority Additional questions for select segments COLLECTION MODE Mode → reflects respondent status Online survey the majority Phone interview select segment Visit in-person the elite 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 32
- 33. CITIZENS Public Opinion Policy Input New Policy Testing Political Strategies Campaign Strategy Message Testing Citizen Feedback Choice of Modes e.g. Inbound IVR 33© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 34. Local Government | A multi-mode story…. 7/24/2015 © Copyright 2015 CFMC | All rights reserved. 34
- 35. QUOTA SAMPLE SURVEY METHOD $ ONLINE $ IVR or PHONE ONLINE + IVR ONLINE + PHONE Complete Quota Within Budget 35© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 36. SURVEY COLLECTION METHODS DATA SET FOR ANALYSIS MULTI-MODE RESEARCH 36© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 37. TECHNOLOGY IS NO LONGER A BARRIER
- 38. MULTI MODE RESEARCH DESIGN web & phone
- 39. YOUR CALL CENTER….OR “THEIRS” SINGLE DATA SET SPECIALIZED PARTNER CALL CENTER ONLINE SURVEY USER ONLINE RESEARCHER ONLINE RESEARCHER CALL CENTER 39© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 40. 40 CREATE A SURVEY © Copyright 2015 CFMC | All rights reserved.7/24/2015
- 41. Web Instructions Phone Instructions SET-UP MODE-BASED INSTRUCTIONS 41© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 42. Mode-based Response Data 42© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 43. MULTI-MODE DATA COLLECTION Mode is tracked by “Survey Type” Start on Web, finish with Phone Analysis should consider mode and note any suspected mode affects 43© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 44. Reach all demographics URL PHONE PHONE ONLINE Improve customer experience Use Offer a choice #800 Match mode to value Finish on Phone Gain better insights Fill quota at lowest cost ONLINE PHONE
- 45. SURVEY COLLECTION METHODS DATA SET FOR ANALYSIS MULTI-MODE RESEARCH 45© Copyright 2015 CFMC | All rights reserved.7/24/2015
- 46. Once in a while, remember to dial!
- 47. B r i a n F o w l e r C O O , C F M C b f o w l e r @ c f m c . c o m 4 1 5 - 3 6 9 - 3 0 6 2 W W W . C F M C . C O M
- 48. THE END
Editor's Notes
- Challenge you to consider how we collect survey data today
- For quick turnaround
- Also balancing : Effects of self selection vs proactive outreach
- How did we get here? Organizational and technical barriers, cost, time to combine, skill sets / vendor capabilities, not necessarily effectiveness of the work produced. But what are we giving up by limiting the vehicles for response?
- How did we get here? Silos were built based on specialties, cost and technological limitations, not necessarily effectiveness or accuracy of the work produced. But what are we giving up by limiting the vehicles for response?
- Underutilized mode: Advantages of phone but without labor
- As mentioned, this integration allows you to marry multiple data collection modes maintaining 1 data set. You have the choice of starting in one mode and adding another later or beginning multiple modes from the start. The streamlined workflow will hopefully let you worry more about what is right for your study and less about the logistics to achieve your goals.
- Convenience = DIY, “launch anytime”
- Large segments of the US population may not be reached online. And of those who ARE Online: How different are they from younger, more affluent who are online more regularly and more consistently. - More data: The CDC report classified another 16.1% of U.S. households as being “wireless-mostly.” That means that although they have a land line, “all or almost all” of their calls are received on a wireless phone. The researchers identified five groups of Americans in which the majority had dropped their land lines. Topping that list were adults who lived with unrelated adult roommates — 76.1% of them relied solely on a wireless phone. Next up were adults between 25 and 29, a group in which 65.7% were wireless-only. They were followed closely by adults who live in rental homes — 61.7% had only a wireless phone, according to the report. FROM A SEPARATE CDC STUDY IN 2H 2013 Among adults living in poverty, 56.2% had a wireless phone but no land line, the researchers found. That compares with 46.1% of adults who were “near poverty” and 36.6% of adults who had a “higher income.” Finally, 53.1% of Latino adults were wireless-only, the only racial or ethnic group to cross the 50% threshold. In comparison, 42.7% of blacks, 38.1% of Asians and 35.1% of whites had also ditched their land lines, according to the report. Midwesterners were more likely than Americans from any other part of the country to be classified as wireless-only, with 43.7% earning that designation. Americans in the South and West were close behind, with 41.9% and 41.2%, respectively, relying solely on their wireless phones. By comparison, only 24.9% of Northeasterners had cut the cord to their land line, the researchers reported. Men held the edge over women when it came to living in wireless households, by a slim margin of 40.4% to 37.9%.
- 98% of the US population has phone service. 20% of the US population does not have Internet access. Start in online survey Show existing survey; show existing panel Switch to Survox Console Register the Online Project (mention that it only displays surveys that are active and that the user has access to) Import the sample from a online panel Activate the survey Take the survey as an interviewer (talk through the dialer interaction but we won't use an actual dialer) After running through a few respondents - go back into the online solution and show a report (report should include responses from both the web and phone interviews)
- Least Cost Method first . Web is only a push. Who’s going to show up when? Manage your Sample to fill quota Unlike online surveys where you need to push and prod people to take the survey… With good sample management, call centers can reach exactly who you need to fill quotas and nurture them through the whole survey to get the results you want.
- Convenience = DIY, “launch anytime” Examples: inbound/outbound.
- But those barriers are coming down today we have the ability to create a single survey, employ multiple data collection strategies, and have the data all available for analysis in a single data set. Now researchers can choose the right methods that fit EACH job and not be constrained by the technology.
- RECORD TRUE VOICE of the customer Voice of the Customer Examples Retailers – provide incentive to collect feedback and now on receipt offer choice – URL or Phone # (which usually goes to an IVR system) Service Quality checks Major national home improvement chain that uses 3rd party installers uses phone interviewers to check on customer experience. Sample is provided by a feed from other systems such as an ERP or POS Too important to them to let respondent self-select who provides feedback. Customer Support departments transfer caller to IVR survey post transaction One large consumer products company uses national sampling and phone recruitment to get the right mix of consumers to take mail and web surveys for new product input. Example: Automaker feeding recorded VOC directly back to automaker.
- One survey, one data set produced….different modes of data collection employed for different audience segments Online Phone In person using a tablet device For example, if you’re a casino you’re going to treat your high rollers differently from your tourists Or if you’re an airline you’ll treat your Platinum and 1K members differently than your Bronze level members.
- Systematic and Coordinated approach
- Once upon a time there was a Franchise Manager named Molly who had the tricky challenge of maintaining a consistent, high quality customer experience from services delivered by organizations that only indirectly reported to her. Every day Molly made decisions about which locations to renew and which ones to drop, based solely on sales data. This approach created pretty high turnover and meant that at any given time there were a fair number of new franchises coming online, needing training, and depressing sales. One day, Molly realized that her pipeline of new applicants was running low and she’d have to find a different way to deal with the problem locations. Because of that, Molly worked with her team to come up with the key metrics that may yield an early warning of problems to come. She also developed a set of questions to ask customers to provide feedback on various aspects of the service. Now she needed to collect the data which is a problem since the franchisees ‘owned’ the customer relationship and customer contact information was not available. Molly was blocked. Until finally, she realized that her organization supplied the POS system used to print receipts. Now she could appeal directly to the customers themselves with an incentive to tell her what she needed to know….and to maximize the feedback she decided to give her customers a choice of ways to respond. Now, each receipt is now printed with both an 800# and a URL to a quick survey…giving Molly and her team the insight needed to identify potential problems and how to help them. Franchise turnover has been dramatically reduced.
- ~100% have phone 14% all US adults don’t have internet access so 14/86 = 16% increase in potential respondents or ‘reach’ 41% seniors don’t have access so 41/59=70% increase in reach to this segment ~20% of minorities, rural, etc don’t have access so 20/80 = 25% increase in reach
- Now you can give people a choice of ways to engage to maximize your results. Invite them to a web survey or provide a phone number. When they call in you can have a live person greet them Or you can use an automated IVR system to screen and forward or simply conduct the survey
- And if you’re a large company… Differentiated data collection strategy for 360 reviews or new policy/product roll-outs. Executives – In-person (web survey on tablet) High Potential & Management – recorded Phone interviews Staff – Web survey Or a Hybrid: web for basics, phone for indepth questions/clarification One survey – possibly different lengths Central analysis
- One survey, one data set produced….different modes of data collection employed for different audience segments Online Phone In person using a tablet device For example, if you’re a casino you’re going to treat your high rollers differently from your tourists Or if you’re an airline you’ll treat your Platinum and 1K members differently than your Bronze level members.
- When quota and demographics matter most. Polling used to be phone only, due to time requirements to ‘fill quota’. Now, IVR and Web are being used to Daily public opinion poll – EVERY day 1000 responses are collected using RDD sampling; Not 999 or 1001 but exactly 1000 or not a valid sample. - Phone provides reliable response rates so that they get all the responses needed - Quota management ensures that they don’t oversample Campaign strategists Call with the candidate every night at 10 pm At 9 pm the strategist calls the polling center to get the days results Accurate insights depend on tight sample control that produces responses from a replicate of the target population. i.e. The polling center had to have collected input from exactly the demographic mix that is expected to vote
- Once upon a time there was a city manager named Harold who prided himself on always engaging in public discourse before making a policy change or offering a new service. Every day Harold would check the polls on the city’s web page to see what his constituents were saying. He would use the data he gathered from these surveys to inform the various decisions he and his staff made on behalf of its taxpayers. One day, the City Council (his boss) decided to do some Town Hall meetings (sort of like focus groups) in neighborhoods across the city to make sure there were no surprises in the upcoming election. You see, City Council members, unlike the City Manager, are each elected by the voters. So they held the Town Hall meetings and boy, did they get an earful! Council Members came back to Harold with a long list of issues and general unhappiness in the way the town was managed. Quite a gap emerged in what Harold was seeing online and what the Council members were hearing in person. Because of that, Harold knew he was in danger of losing his job and had to rethink his citizen outreach strategy. Whereas he felt pretty good about the profile of respondents online, he realized that those who showed up to the Town Halls may have been a completely different segment of voters. In order to avoid future surprises like this, the City needed a broader polling strategy to ensure that all citizen voices were being heard. Harold was in trouble. Until finally, after researching the best practices of other city leaders, Harold decided to invest in an approach that used a variety of methods to hear from voters. He kept his online surveys going but also ran ads in the local paper with two ways to take the survey – via the online survey or by calling a local number for a phone-based survey. Harold also kept track of who was responding. He wanted to make sure he had a good cross section from each neighborhood, age group, and ethnic backgrounds. Now, when he needs input from a given group, Harold arranges for outbound calls to be made to citizens in those segments. Never again will he make the mistake of assuming that a passive approach to collecting feedback is sufficient.
- Use multiple modes to stay within budget AND still reach quota. Maximize responses from most cost effective method first – online. Finish with outbound IVR or phone interviewing to complete quota. - IVR has the Telco expense - Phone interviewing has Telco and labor expense
- Open up your thinking: Don’t be constrained in your practice by organizational structure, technology or vendor. But those barriers are coming down and what we are bringing to the market is the ability to create a single survey, employ multiple data collection strategies, and have the data all available for analysis in a single data set. Now researchers can choose the right methods that fit EACH job and not be constrained by the technology.
- Can have mode based differences. Embedded data field call SurveyType is passed (mobile or web) then you can have skip logic based on that value Optional: Screens customized by mode Introduction to survey– web vs. Interviewer script Additional Interviewer instructions – “read the list” Interviewer questions / selections
- Regardless of the point at which phone is brought in to the flow, all survey response data resides within the Qualtrics database. No importing, no merging Existing reports and integrations continue to work Because the modality of data collection is stored in the data, you can create mode specific reports and analysis.
- If multimode you might start with web and phone with fixed daily targets. Or You might start on web then bring on phone to round out your data collection and fill in the quotas after giving respondents some time to complete the survey via web and a subsequent reminder blast as illustrated in the Production report on the right.
- Open up your thinking: Don’t be constrained in your practice by organizational structure, technology or vendor. But those barriers are coming down and what we are bringing to the market is the ability to create a single survey, employ multiple data collection strategies, and have the data all available for analysis in a single data set. Now researchers can choose the right methods that fit EACH job and not be constrained by the technology.