Improve Your Customers' Experience By Listening to Unstructured Feedback


Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Improve Your Customers' Experience By Listening to Unstructured Feedback

  1. 1. Improve Your Customers’Experience By Listening toUnstructured Feedback
  2. 2. Good experiences. On brand. On budget.Table of ContentsExecutive Summary .................................................................................................................................................................................3Finding the Golden Nuggets ...................................................................................................................................................................4Capitalizing on Fresh Insight ...................................................................................................................................................................5Emerging Market Appraoches Tipping Point ..........................................................................................................................................8Making the Business Case .......................................................................................................................................................................9Smart Practices for Success ..................................................................................................................................................................10About the Author and Sponsor .............................................................................................................................................................11Appendix: Text Mining Fundamentals ..................................................................................................................................................12AcknowledgmentsThis paper was made possible with the generous sponsorship of Island Data Corp. and in-depth interviewswith its customers, including Meredith Sime, associate director, Customer Experience, for AT&T’s U-verse;Jill Trecker, guest loyalty manager for Garden Fresh Restaurant Corp.; and Devon Child, senior productmanager for Yahoo!.We also sincerely thank CustomerThink community members for their input in an online survey aboutmanaging unstructured customer feedback, which provided most of the statistical information included inthis paper. Those seeking more details on text mining are encouraged to read our key backgroundsources: The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge University Press,2007) and Machine Learning in Automated Text Classification by Fabrizio Sebastiani (ACM ComputingSurveys, Vol. 34, No. 1, March 2002).©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 2
  3. 3. Good experiences. On brand. On budget.Executive Summary Customer-centric businesses must, above all else, be committed to listening to the voice of customers and acting on that feedback. But, these days, that “voice” includes unstructured text that can’t be handled with conventional data analysis. If your business is like most, you hear from your customers all the time. They send you email. They unload earfuls to your customer service reps. They write in comments on your product surveys. They leave feedback on your web site. You probably would like to find out what customers are saying, but in practice, you don’t cull through the massive amount of feedback for a variety of reasons: You’re understaffed, there’s too much of it and you have other fires to put out. “This kind of feedback is giving us what the Forward-looking business leaders are now recognizing that this unstructured feedback customer wants to tell contains the key to amplify customer listening programs. And they recognize that the us, not what we want to effort to mine their unstructured customer feedback is worth the rewards: early warnings ask them.” about calamities, insights about pain-points that could cause customer defection and a view into future customer trends. - Survey respondent In this paper, I’ll look at the business case for mining your unstructured customer feedback, and Ill discuss the basics of text mining, looking in simple terms at how it works. I’ll examine the key findings from a 2007 CustomerThink survey on how businesses are managing unstructured customer feedback, and I’ll discuss what executives from some successful businesses say about what text mining has meant for their strategy. Finally, I’ll discuss the practices that will work best when you do take the plunge and adopt a text-mining solution for your organization. A 2007 CustomerThink survey of members found that business leaders are certainly aware of the importance of mining unstructured customer feedback, but they also recognize they are, by and large, not doing enough about it. Here are some key findings from that survey: The majority of businesses surveyed received unstructured customer feedback in the forms of comments from surveys, unsolicited email and online feedback forms. An overwhelming majority (80 percent) of businesses surveyed said managing unstructured customer feedback would help improve their business performance. More than half of the businesses surveyed said that managing unstructured customer feedback is important to top executives, and that importance will only increase in the future.©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 3
  4. 4. Good experiences. On brand. On budget. Yet few (less than 25 percent) of the businesses we surveyed felt that their companies were doing an “excellent” or “very good” job of capturing, analyzing, reporting or acting on unstructured customer feedback. Only 12 percent said they used a software tool of any kind. Managers surveyed felt that the two most important potential benefits to effectively managing unstructured feedback were improving products and services and improving customer loyalty. One emerging best practice stands out, from both executive interviews and survey respondents. For success in managing unstructured customer feedback, the top priority is to act on the feedback received. Otherwise, why should customers invest their time in telling you what they think?Finding the Golden Nuggets Text mining, like its geological counterpart, is sifting through vast amounts of debris to find the gold. As Ronen Feldman and James Sanger, the authors of The Text-Mining Handbook (Cambridge University Press, 2007) note, what we call “unstructured data” isn’t completely unstructured. Text follows some basic tenets of natural language, and text mining involves analyzing text to 1) determine what the original author was trying to say or 2) learn something completely new. Like data mining, the idea is simple, but what’s “under the hood” in text mining applications is complex. One common technique is called “categorization,” which simply means classifying text into categories. An example would be deciding whether customer emails represented “happy,” “unhappy” or “neutral” customers, based on the types of words used in those emails. You can imagine that each category should be handled differently, and it might be useful to track the percentages in each category over time. For more details on text mining methods, please see the Appendix or read our reference sources. Categorization has been used since the late 1960s in areas such as medicine and news services, where there were large stores of documents, a strong desire or need to make sense of it and the financial wherewithal to handle the expense of the computing power necessary to crunch through the data. Text Mining Comes of Age In recent years, text mining algorithms have been more widely adopted by businesses, thanks to “Moore’s Law,” which drives continued computer performance improvements. But equally importantly, as you’ll learn in this paper, the setup and usage of text mining systems have become much easier with the adoption of more “turnkey” implementation approaches, including hosted or “on-demand” applications. Executives at companies that are using text mining to analyze their unstructured customer feedback are enthusiastic about the benefits. Early adopters usually are. For©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 4
  5. 5. Good experiences. On brand. On budget. them, not only did text mining help them meet the challenge they knew they had, but also they found numerous other uses and benefits. Consider these illustrations: In one case, a floral company was getting a high volume of returns of one particular floral arrangement. The company thought that it was because customers were disappointed in the look of the flowers. But after mining the unstructured customer comments, they discovered that wasn’t true after all. In another example, a telecommunications company introducing a new service was able to analyze unstructured feedback over a period of time. This enabled executives to see the nature of complaints progress from predominantly service-related to encompassing more aspects of the customer experience as the product evolved in the marketplace. And in yet another case, restaurant chain executives were able to see that the new buildings they were designing for additional restaurants had some critical issues. Executives were able to make design changes before replicating the faulty design to multiple locations. In every one of those cases, executives are thankful for what they see as a robust tool for looking into the minds and experiences of the customer. In general, they see text analytics for customer feedback as a necessary part of a customer-centric enterprise. It’s not about just automating a process to gain efficiency, although technology does enable text analysis that is impractical to do manually. The real key is understanding what customers are saying in their own words and then acting on that insight as quickly as possible.Capitalizing on Fresh Insight If you’ve eaten at a restaurant or shopped at a store lately, you’ve no doubt seen an invitation on your receipt to call a toll-free number and respond to a survey about your recent experience. Surveying customers is a popular way to quickly gauge the thoughts of motivated customers, whether they’re interested in venting about a problem or praising your business for a great product or outstanding service. Surveys are now commonplace tools for companies to solicit customer feedback. In a 2007 CustomerThink survey, nearly 80 percent of respondents said they were conducting customer surveys at least annually, up from 70 percent in 2004. It’s a safe bet that nearly all of those surveys include some questions for customers to write in their comments.©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 5
  6. 6. Good experiences. On brand. On budget. In another survey in late 2007, we learned that, sure enough, customer surveys are a popular source of unstructured feedback. But unsolicited emails and web site forms are also popular sources of customer feedback text for the majority of respondents. Other sources cited included call center agent logs, transcripts from recorded phone calls, text messages, chat sessions and posts on discussion forums or blogs. Now, the fact that this feedback is available does not mean that it is analyzed and acted upon! It turns out that not enough are taking advantage of it, according to our survey. When we asked people how effective their companies were managing unstructured customer feedback, more said they were best at capturing customer comments. But the average ranking they gave to that activity was only 2.66, when a rating of 3.0 is “good.” And 46.2 percent rated their efforts as less effective than that. Its clear that businesses have a long way to go. What about taking advantage of the feedback? Well, the story there is somewhat worse, with fewer than 25 percent of those surveyed rating their effectiveness as “excellent” or “good” and more than 50 percent giving their organizations a “fair” or “poor” rating. These findings are not at all surprising, given the emerging market for text analytics solutions. Judging from the comments from early-adopter executives, however, there’s a tremendous opportunity for a wide variety of companies to listen better to customers and act on the insights gained. Listening to Vocal Customers If you give customers a chance, they’ll communicate with you in many ways. “It’s worth noting our guests are very vocal,” says Jill Trecker, guest loyalty manager for Garden Fresh Restaurant Corp., a chain of 104 buffet-style restaurants known as Souplantation in Southern California and Sweet Tomatoes in other parts of the United States’ Sun Belt. The©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 6
  7. 7. Good experiences. On brand. On budget. restaurants boast 55-foot salad bars and a selection of eight soups, baked goods and frozen yogurts—on an “all you care to eat” basis for one price. In addition to inviting customers on restaurant receipts to take surveys, Garden Fresh posts a toll-free phone number in its restaurants, inviting customer feedback. They listen to the voice prompts, and they rate the restaurant experience. Guests are also given the opportunity to record a 60-second message. A service management company handles the “You’re one of our responses to the structured questions and transcribes the unstructured recordings to be favorite places because of processed by Island Data. the healthy benefits of your food. BUT, we have a The company also has an online “eat club.” Six weeks after customers have joined the few suggestions: I would club, they receive a short survey with structured questions and a place for free-form like to see raspberry responses. vinaigrette as a regular The philosophy at Garden Fresh is that “no news is bad news,” but with so many vocal dressing and see you customers, the company was overwhelmed by the volume of customer feedback. The improve the blue cheese company receives about 10,000 pieces of unstructured feedback a month from the dressing, which is much different channels. Going through them manually would be, in Trecker’s words, too ‘mayonnaisey.’” “prohibitive.” “There were things we knew that we were probably missing that we didn’t have a structured question about,” Trecker says. Additionally, Trecker and company President Kenneth J. Keane worried that if they were soliciting feedback and not responding to important information in that feedback, they really were wasting people’s time—or worse—risking angering customers. Sifting Through Mountains Garden Fresh contracted with Island Data in 2005 to find meaning in all that feedback. It took the company about three weeks to set up shop, beginning by taking about 1,200 records of the 60-second transcribed comments and using them to train the system to identify different categories of praise, criticism or requests. Now the company has dashboards and receives monthly “praise” and “complaint” reports, both showing trends and drilling down to individual customer comments. The data is used mainly by the marketing team, but Trecker regularly shares reports with Keane and the rest of the executive team. They now can look at trends “month to month to month,” says Trecker, and drill down to investigate trends. One “aha!” moment early on was in responding to customers’ demands for more soup varieties, including vegetarian soups. The company expanded the number of soup varieties offered, and complaints about variety lessened.©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 7
  8. 8. Good experiences. On brand. On budget. The company also was finally able to confirm why it typically received more complaints in February and March, traditionally the months when Garden Fresh restaurants have a higher volume of activity. The hunch was that when volumes increased, attention to cleanliness dropped, as employees struggled to keep up with the load. Sure enough, the Island Data reports showed people were complaining about cleanliness. In another example, as mentioned earlier, Garden Fresh found that customers were complaining that the new restaurant design seemed too crowded and noisy. Because of its window into customer feedback, executives had time to make changes before it was too late.Emerging Market ApproachesTipping Point It’s early yet in the market for unstructured customer feedback solutions, according to Lane Michel, executive vice president and managing director of the Marketing Performance Management unit at Quaero Corp. The most active applications are in customer experience assessment and monitoring; new or upgraded product feedback; call center performance measurement and efficiency; and competitive intelligence gathering. “Marketing executives cannot continue to wait Call center executives, for example, are looking to a new generation of more effective for the cumbersome tools to address customer experience and operational performance issues. “Marketing 360-degree customer executives cannot continue to wait for the cumbersome 360-degree customer databases databases and slow and slow analytics to generate intelligence, early warnings and pragmatic segmentation analytics to generate schemes,” Michel says. “My guess is that we should see a steep climb in text mining tool intelligence, early sales in the next 18 months, with a shift out of early adopters predominately purchasing warnings and pragmatic today.” segmentation schemes.” In most cases, companies were already gathering feedback. The decision to actually - Lane Michel, Quaero implement a system came when the volume of that feedback reached a tipping point. One retailer was receiving from 4,000 responses in a slow week to 150,000 to 200,000 responses a week during peak holiday seasons. When creating customer feedback tools, “we took a pretty decent stab at anticipating what our customers might want to tell us," said Meredith Sime, associate director, Customer Experience, with AT&T’s U-verse, a new platform for services provided over©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 8
  9. 9. Good experiences. On brand. On budget. Internet Protocol. "But we did not initially invest enough resources in mining the wealth of information from an unstructured format.” And, she said, those instincts were borne out once they started analyzing the data. “We learned a tremendous amount of information that helped us drive improvement plans, and it taught us to ask better questions.” Customers, Sime said, “raised issues we had not previously considered because the products were new to us.” Like others I spoke with, Sime would love to get near a real- time response in the future. “We would have a first-response mechanism if any quality of service issues were to arise,” she said. In fact, according to Scott Austin, Island Data’s chief technology officer, the company found immediate reception to its “early-warning” alerts, notifications that there was a spike in complaints about a product or service. When Island Data first rolled out the service, it was temporarily offered to customers on a complimentary basis. “Within a week and a half, [customers] said, ‘We absolutely have to have it. We will pay you money to make it a permanent part of what we have.’”Making the Business Case We asked survey respondents to rate the potential benefits of effectively managing unstructured customer feedback. “Improve products and services” and “improve customer loyalty” tied for top ranking, followed closely by “gain insight hidden in customer comments” and “listen to customers in a more natural way.” However, when we asked people how they would justify an investment, the top three factors in their rationale were: 1) improving the customer experience, 2) improving customer loyalty and 3) increasing revenue. Collectively, these factors accounted for 53 percent of the weighting. Other factors included: improving product/service quality, supporting a customer-centric strategy, improving decision-making and reducing operational costs. Considering the survey results along with executive interviews, it seems clear that the ROI from managing unstructured customer feedback is geared toward improving customer©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 9
  10. 10. Good experiences. On brand. On budget. retention by focusing on trends, “listening” to customers and improving the customer experience. One of those early-warning alerts came in handy at Yahoo!, according to Devon Child, senior product manager for the portal’s internal feedback platform. It highlighted a problem that product managers knew was festering but didn’t consider a priority. “They had it on their radars.” The alert came back, and they realized that it was more serious than they previously thought. That’s a great “aha!” moment, but does it justify implementing a text-mining solution? “It’s hard to argue with How do you convince the rest of your company that managing unstructured feedback is wanting to listen to the important? I asked executives this question, and they were almost puzzled by it. Perhaps customers – our these are stellar firms when it comes to customer-centricity, but these executives did not customers. Their opinions encounter internal resistance to text mining. matter.” Their companies wanted to know what customers were thinking, which, again, is why they - Devon Child, Yahoo! were collecting the feedback in the first place. The question for each was: Do we do it in- house or do we outsource? Executives I interviewed said money wasnt the issue. Their aim was to do what was best for their companies—and their customers. At U-verse, which performed an IT assessment of the costs, comparing outsourcing with in-house text mining, Sime saw no resistance to the question of whether to analyze the text. “We had all this data coming in, but we had no idea what to do with it.” The ability to cull through all the unstructured feedback has “been a godsend to us,” Sime said. “Before, the best we could do was paste the information in quotes.” What gems of information are hidden within the customer feedback your organization collects? “It’s hard to argue with wanting to listen to the customers—our customers,” Child said. “Their opinions matter.”Smart Practices for Success It may be a bit premature to declare the definitive list of “best practices” for success with projects to manage unstructured customer feedback. For now, let’s call them “smart practices.” The chart below shows the opinions of CustomerThink members, based on our 2007 online survey. Given this input, along with my interviews with early adopters, here are five practices that will help you succeed. 1. First and foremost, it’s critical that you are truly committed to act on the feedback you receive, whether it’s unstructured. Nothing is more disillusioning to a customer than taking the time to provide constructive praise or criticism and then have it be ignored.©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 10
  11. 11. Good experiences. On brand. On budget. 2. Make sure the insights are correct. “Best algorithm” was not rated highly in our survey, but its very important to the results you get. Garbage in means garbage out. The analysis-and- categorization algorithm must be reasonably effective; otherwise, “The value of the you’re acting on bad information. That said, unstructured feedback is there is probably a point phenomenal.” of diminishing returns in striving for the “perfect” - Meredith Sime, AT&T’s algorithm. U-verse 3. You must get the “golden nuggets” of insights to the right people, at the right time, in a format that is easy for business people to use. You can see these priorities clearly in the survey, and several interviewees commented that ease-of-use was extremely important. 4. Make sure the project has senior level sponsorship. Odds are you won’t be able to create a business case that “proves” in a spreadsheet how your company will increase revenue or cut costs by investing in text mining. You need an executive’s vision of a customer-centric organization. 5. Finally, pick a solution that is cost effective and fits your business and IT environment. Consider the total cost of ownership when comparing an installed to on-demand solution. Keep in mind, however, that cost is probably not going to be the central issue if your business strategy dictates that better listening to customers is a critical success factor. Follow those recommendations, and you, too, can reap the benefits of text mining. As Sime says, “The value of the unstructured feedback is phenomenal.” You already have the content. It’s time to put it to work to improve your customers’ experience—and your business performance.About the Author and Sponsor About the Author – Bob Thompson, CustomerThink Corp. Bob Thompson is CEO of CustomerThink Corp., an independent customer management research and publishing firm. He is also founder of, the world’s largest online community dedicated to helping business leaders improve customer-centric business strategies. Since 1998, Thompson has researched the leading industry trends, including partner relationship management, customer value networks and customer experience management. In January 2000, he launched (renamed in 2007) which now serves 300,000 business leaders monthly through its web site and email newsletters.©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 11
  12. 12. Good experiences. On brand. On budget. Thompson is a popular keynote speaker at conferences worldwide and has written numerous articles and papers, including his most recent report, Customer Experience Management: A Winning Business Strategy for a Flat World. Before starting CustomerThink, he had 15 years of experience in the IT industry, including positions as business unit executive and IT strategy consultant at IBM. For more information, visit or contact Thompson at About the Sponsor – Island Data Corp. Island Data Corp., the innovator in Customer Analytics solutions, delivers real-time market and business intelligence for global online enterprises. Island Data’s software-as-a-service, Insight RT™, allows customer-centric organizations to listen to the voice of their customers by managing the unstructured customer feedback. Insight RT enables executives to track key performance indicators, collect actionable intelligence and optimize the customer experience at every touch-point. Companies employ Insight RT for timely discovery of critical facts and issues that support continuous improvement of product management, product quality, market and online presence, brand loyalty, customer retention, customer service and customer satisfaction. Headquartered in Carlsbad, California, Island Data is funded by top-tier venture capital firms Dolphin Equity Partners and ABS Ventures. For more information, visit Text Mining Fundamentals Text mining is the discovery of information by analyzing text. Sources could include any written form of communication captured electronically or verbal communication transcribed into text. Insights gleaned from this text could reflect the meaning of what the author intended or provide entirely new information. Whereas data mining extracts information from structured databases, text mining extracts information from natural language text. There are two approaches: linguistic and statistical. The linguistic approach involves identifying linguistic elements of language and structures that relate them to each other. Those elements are the keys to meaning. This is much like parsing or diagramming a sentence to identify parts of speech. If you have an adjective, for example, what noun does it modify? You don’t need human intervention to train a classifier, but you do want to identify parts of speech (which is done easily with a conveniently packaged list known as a “dictionary”); words that are commonly used by people who are angry (which can come from a©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 12
  13. 13. Good experiences. On brand. On budget. dictionary or thesaurus); and information about sentence structure (if you have two nouns, which is the subject and which is the object? Did John throw the ball or did the ball throw John?). In the statistical approach, words and phrases are treated as abstract objects. You use purely their mathematical relationship to each other. This approach most often involves machine-learning. Is your customer angry? Is he pleased? Is another customer talking about your latest product? Using a sample of the text and assignments in a number of categories, the computer scans them for common elements. The machine learns by example based on training data assigned by human beings. If a business receives a million emails a day, you would take a small sample—say, 500 to 2,000 emails—and manually classify them. Then the computer would scan the sample to identify relationships in the text that hold clues for whether a particular email may be from a happy customer. People using obscene language tend to be unhappy, so simply scanning for profanity in your sample can distinguish email from irate customers. There are many ways to train a classifier without human intervention. One important method is clustering, in which you look at what words naturally go together, automatically identifying common themes just by looking at one or two emails, rather than 200,000, for instance. For more information, please read: The Text-Mining Handbook by Ronen Feldman and James Sanger (Cambridge University Press, 2007). Machine Learning in Automated Text Classification, by Fabrizio Sebastiani (ACM Computing Surveys, Vol. 34, No. 1, March 2002)©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 13
  14. 14. Good experiences. On brand. On budget. Copyright 2011 KANA Software, Inc. KANA and the KANA logo are registered trademarks of KANA. Other company, product and service names may be service marks of their respective owners. 840 W California Avenue, Suite 100 Sunnyvale, CA 94086 T 650.614.8300 | F 408.736.7613 Contact us at Twitter @KANASoftware Linkedin: Facebook: All Rights Reserved. 0110-01©2011 KANA Software, Inc. • 840 W California Ave, Ste 100, Sunnyvale CA 94086 • 1.800.737.8738 • • PAGE 14