What Can Be Learned From A One-Minute Online Survey


Published on

What do people who use the Amazon website think of their experience?

Results of this Customer Centric Index survey reveal the strengths of the website and changes that might make it even better.

Published in: Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

What Can Be Learned From A One-Minute Online Survey

  1. 1. What Can Be Learned From a One-Minute Online Survey How people really feel about Amazon.com (and what does this mean)? An Illustrative Case Study Using
  2. 2. Foreword: Web teams need constant feedback By Gerry McGovern Living systems get constant feedback from their external environment. To truly succeed, web teams need constant feedback from their customers. You're a manager in a restaurant. It's raining. A customer walks in and almost slips on the mat in front of the door. You're very busy at this stage, but you make a mental note: "I must change that mat." About 15 minutes later another customer comes in. She, too, almost slips on the mat. You rush up to her, apologize profusely and then change the mat. People are slipping on our websites right now but, because we don't see them slip, we don't change the mat. I'm one of the biggest offenders. Over the years I have left content and applications on my websites that had problems that I was vaguely aware of, but they just didn't seem important enough to warrant any action. Even when I became clearly aware of the issue I didn't react with enough urgency. Why was that? Why was I so complacent? I would like to think that if I was running a restaurant I would have apologized to the customer and changed the mat. Why don't I do that when it comes down to managing a website? I think a core part of the problem is the lack of real feedback. I'm not actually seeing the customer slip. I don't actually see real people use my websites. Customers are hugely impatient on the Web. When they slip, their first impulse is to hit the Back button. Jared Spool wrote an excellent article in 2009 called the "The $300 Million Button." In it he explained how the removal of a registration button from a particular step in a purchase process resulted in a dramatic improvement in sales. The Web team had created the registration button so as to make it easier and faster for regular customers to buy. But people absolutely hate registration. New customers felt they would be spammed if they registered. One potential customer summed up their feelings as follows: "I'm not here to enter into a relationship. I just want to buy something."
  3. 3. The regular customers didn't feel much happier. "45% of all customers had multiple registrations in the system, some as many as 10," Jared wrote. "We also analyzed how many people requested passwords, to find out it reached about 160,000 per day." The Web is so important today. And yet many of the web teams I deal with are way down the management hierarchy. Intranet teams, in particular, tend to get negligible resources. That needs to change because the reality is that the Web is central to the present and future success of most organizations. One of the ways we make that change happen is that we start developing much better feedback mechanisms for our websites. At a most basic level, we must find ways to regularly (weekly at minimum) observe our customers carry out top tasks on our websites. That's how Jared Spool discovered there was a problem: by watching customers trying to buy. According to Wikipedia, "Living things are systems that tend to respond to changes in their environment." Let us embrace our customer environment. Let us observe and evolve. The rewards are very substantial. Gerry is the founder and CEO of Customer Carewords. He is widely regarded as the worldwide authority on increasing web satisfaction by managing customer tasks. Gerry has spoken, written and consulted extensively on web content management issues since 1994. His new book, The Stranger's Long Neck: How to Deliver What Your Customers Really Want Online, was published in July 2010. Read the first chapter of the book at http://www.gerrymcgovern.com/sln-ch1.htm
  4. 4. What Can Be Learned From a One-Minute Online Survey Executive summary In order to demonstrate what can be learned from a carefully-designed survey that is very quick to complete, we asked a combination of consumers and Web professionals to rate Amazon.com (chosen as it is very familiar to many people). As expected, the site was rated very positively yet there was still much to learn. The most compelling characteristic of the site for most visitors was the ratings and reviews feature. But while ratings and reviews are a strength now, they might also be under threat. The growth of social media has diminished the advantage of ratings and reviews by strangers and Amazon is arguably no longer at the leading edge in this respect. Other findings demonstrate that female visitors like to ‘get things done quickly’ and are more troubled by the difficulty of ‘contacting a person’ while men value ‘complete information’ a little more. Perhaps surprisingly, Web professionals complained more than consumers that Amazon.com had a cluttered layout, hinting that expert opinions cannot be relied on exclusively without the input of regular Web users. 4
  5. 5. What Can Be Learned From a One-Minute Online Survey Table of contents Introduction..........................................................................................6 Key findings..........................................................................................7 Results.................................................................................................8 ............................................................................................................8 Analysis................................................................................................9 Ratings, reviews and recommendations are popular but under threat. . .9 ......................................................................................................11 People like Amazon’s search but there is no room for complacency ....11 Clutter and layout were problems, but not for all...............................12 Some people find it hard to contact a person.....................................13 Visual appearance not of overriding importance.................................15 Methodology.......................................................................................18 5
  6. 6. What Can Be Learned From a One-Minute Online Survey Introduction Consumers have opinions on websites; they express them to friends and family, not in the technical and analytical terms that professionals might use, but in quick summary statements that capture the essence of the sites. Typically, Web surveys ask consumers to rate sites in technical terms, but this is expecting consumers to think the way we experts do, to tell us what and where site issues exist. . We need to simulate being present at hundreds of social gatherings and eavesdropping on conversations about websites. The approach used in this We instead need to collect Amazon review, tried and trusted over many years, puts the onus on the analyst what people feel strongly to interpret consumer sentiment, and makes it easier for them to express their about; what they would feelings. It was our goal to run through a illustrative example from start to finish say over the barbecue or to show you the level of actionable information can be gleamed from this simple at the water cooler and comparatively inexpensive method. You can be the judge of the success of that objective. This technique does not ask questions in company-centric language (e.g. ‘Please rate the options that are available for you to navigate this website?’), but instead simply asks people to express what they felt strongly about. This ensures that their instinctive reactions are captured. It also helps that the survey was very quick to complete — it genuinely took less than a minute and nobody abandoned it. The success of the survey depends of course on the results. You have your opinion of Amazon, so how well do the findings match your expectations, but also surprise and provide ‘food for thought’? Ultimately, what value does this survey bring to website analysis? This study was conducted by The Customer Respect Group using the Customer Carewords methodology. It is far from perfect. We would have liked larger participant numbers and we did invite web professionals with their different preconceptions to get involved, which may have skewed overall results a little. But a lot of data and analysis emerged from what was, superficially, a very simple survey. 6
  7. 7. What Can Be Learned From a One-Minute Online Survey Key findings A very clear trend emerged from the results – the top five attributes accounted for over half of the overall impressions. The strong trend showed that the survey successfully identified the aspects of Amazon.com that are most compelling. The positive sentiments were dominant and the site recorded an extremely high rating of 86 out of 100. The general rating for websites tested with this methodology is typically in the 50-60 range. The top five attributes were: 1. Site has ratings, reviews and recommendations 2. Helpful search results 3. Fast to do things 4. Simple layout/easy to read 5. Accurate information An important consequence of the survey is that it highlights site strengths just as much as weaknesses. So while ratings, reviews and recommendations are a much-loved feature this strength can become a threat should competitors neutralize their value. Although the negative sentiments were in the minority, a significant minority is not happy with some aspects of Amazon.com. Even a successful site like Amazon can be improved. Given the volume of traffic, a small improvement could have a significant effect. The three main areas of concern were: 1. Cluttered layout 2. Hard to contact a person 3. The site was visually unattractive   7
  8. 8. What Can Be Learned From a One-Minute Online Survey Results Participants came from a pool of demographically evenly-spread consumers (approx 60% of the total) and web professionals. The graph below reflects the importance of each of the 26 attributes in the survey. A red bar indicates a negative sentiment. 8
  9. 9. What Can Be Learned From a One-Minute Online Survey Analysis The data is a signpost and tells us where to look; the interpretation brings the data to life and leads to actionable tasks and strategies to pursue. With websites so complex, it is critical to garner consumer sentiment continuously to reflect changing expectations as much as changes in functionality. When so many companies focus on customer retention and growth, sentiment towards websites is a critical metric that up to now has been difficult to harness. Ratings, reviews and recommendations are popular but under threat 84% of US customers prefer The presence of ratings, reviews and recommendations was the top-rated the opinion of other customers attribute. It was once what differentiated Amazon.com but the landscape has versus experts. changed. Most retailers have adopted, and even improved, reviews and for very good commercial reasons: —Marketing Sherpa 78% of Internet users rate recommendations as most credible form of advertising (Neilsen) 84% of US customers prefer the opinion of other customers versus experts (Marketing Sherpa) 96% of online retailers rank customer ratings and reviews as an effective or very effective tactic for driving conversion (Forrester) 65% trust ‘friends’ recommendations while 33% trust company-supplied recommendations (emarketer) 69% of consumers who read reviews then go on to share them with family and friends (Deloitte) Products that have reviews show a 35% increase in conversion rates (Bazaarvoice) One weakness of the Amazon implementation is that it does not allow for recommendations by friends. We can read that hundreds of people rated this product as 5-star, but it’s not immediately clear if they share our tastes or preferences. A clear finding of various studies is that consumers want to hear from “someone like me”. 9
  10. 10. What Can Be Learned From a One-Minute Online Survey Best Buy’s Facebook page, which has almost 1.2 million fans (compared to Amazon’s 90,000) includes a ‘Shop + Share’ tab on which consumers can browse products and traditional reviews. A key enhancement, however, is the ‘Ask Friends’ button which allows fans to collect input on their decision from friends’. This makes it very easy to get the opinions of trusted peers. Now, friends can offer an opinion before we buy, in a manner similar to bringing a friend to a bricks-and-mortar store. There are many other examples of how ratings and reviews have evolved and developed. Once considered a unique feature of Amazon, this could be a game- changing feature and Amazon can’t afford to fall behind the innovation curve in an area so ingrained in its users’ experience. If a key strength is perceived as being inferior to systems in use elsewhere, there is a danger that Amazon might lose some of its popularity. 10
  11. 11. What Can Be Learned From a One-Minute Online Survey Best Buy’s Facebook page brings reviews and ratings to a new level by allowing people to ask their friends for advice on selected products Best Buy has recruited 15 times more ‘fans’ on Facebook than Amazon and more likely to benefit from recommendations from friends People like Amazon’s search but there is no room for complacency Good search results was considered to be the next most important attribute. Search is sometimes considered to be ‘Plan B’ when navigating through menus and links has not resulted in success, but in this case it seems unlikely. Very few people chose ‘confusing menus and links’, so it seems that Amazon’s search is actually particularly good – good enough to be a Plan A when arriving on the site. Customers know that searching for that book or DVD will produce really accurate and fast results. Amazon search has customer-oriented features that can help them find what they Amazon cannot afford need very quickly. Sub-categories on the left menu allow customers to filter to neglect improving results if the default results are not quite right, ratings are visible, and related searches are suggested. its search facility... 34.5% of traffic to top retail sites in July 2010 came directly from [external] search 11
  12. 12. What Can Be Learned From a One-Minute Online Survey Amazon cannot afford to neglect improving its search facility. There are many web shoppers that are not wedded to an Amazon-like one-stop shopping experience. According to Hitwise, 34.5% of traffic to top retail sites in July 2010 came directly from search engines. Google’s new instant search introduces the concept of consumers fine-tuning search terms dynamically. If this proves popular, Amazon will need to respond. There are also examples of reviews being included in search results (Bazaarvoice is supplying reviews to Google that includes content from sites like BestBuy and Macys). As Facebook becomes more and more a destination site for consumers, its search feature might drive traffic to rival sites, particularly if their Facebook presences are more advanced or feature-rich. Negative sentiments were in the minority but can’t be ignored The important thing about negative sentiments in a one-minute survey is that they help companies isolate general issues and provide the roadmap of where to start ‘digging’. Furthermore, we have found that Web teams welcome some negative findings, which are often seen as a validation of their own opinions. A more typical ratio of positive to negative is 65/35 or 70/30, and in these cases the discussion of how to rectify problems would dominate the analysis. Clutter and layout were problems, but not for all The most significant negative aspect of Amazon.com was ‘cluttered layout, hard to read’. Clearly, page layout matters - it has direct impact on the ability to complete tasks. Too many long blocks of text, too many links, clutter, not enough white space, lack of a coherent structure; these can all slow people down. ... ‘you guys are so According to a consumer for whom this is the most important facet of primitive, you are Amazon.com: like cavemen. Don’t I’ve used the site many times and have found it confusing to use. It has taken you have any sense me too long in the past to get to the information I need or to find out that I of aesthetic?’... had to do something else to get where I wanted to be. Just a hassle! In all the complaints - Female consumer, 25-34 years old, quite familiar with the site and requests we get The bigger takeaway story on this factor was that this was much more important from users, this is for professionals than consumers. It was the only factor where professionals never one of them. differed from consumers to any degree of significance. It may indicate that web professionals tend to judge layout more than consumers do, years of experience — Jim Buckmaster, having trained them to look at websites differently. Very few Web professionals CEO of Craigslist would not criticize Craigslist yet it remains a very popular site. According to Jim Buckmaster, CEO of Craigslist: 12
  13. 13. What Can Be Learned From a One-Minute Online Survey “I hear this all the time, ‘you guys are so primitive, you are like cavemen. Don’t you have any sense of aesthetic? ‘ But the people I hear it from are invariably working for firms that want the job of redoing the site. In all the complaints and requests we get from users, this is never one of them. Time spent on the site, the number of people who post—we’re the leader. It could be we’re doing one or two things right.” To reinforce this view, far more people chose ‘simple layout, easy to read’, with consumers more satisfied with the layout than the professionals. Although subjectivity is also a factor in this attribute, the difference of opinion shows the importance of getting the opinions of non-expert consumers, who vastly outnumber the Web professionals in the online space. Without intending to, we were able to show that web professionals may not always be the best predictors of consumer sentiment. Some people find it hard to contact a person Sears offers chat, call At one time, it was extremely difficult to get in touch with Amazon support staff. back and email on Things have improved, but it still is not that easy to engage with a every page to assist the representative. consumer complete the The issue is the lack of real-time escalation – being able to contact an agent by buying process — this phone or chat without leaving the page. Many competitive sites offer telephone is a standard of numbers, live chat, instant callback and click-to-call to ensure that consumers availability of help that don’t lose momentum by having to navigate to another page to resolve an issue. sets expectations for According to Art Technology Group, 58% of consumers now look for live help if missing information while online, while 53% will do so if there are problems visitors to sites like checking out. It was interesting that men looked for help less than women, Amazon’s. reinforcing the general perception of men having an aversion to asking for directions. Sears, a retailer that has announced major plans to expand its online offering with the adoption of an ‘Amazon-like’ marketplace that offers products from third Chatters that engage parties, provides prominent options for consumers to reach out. Service agents via proactive invitation can view the contents of the consumer’s shopping cart and answer any questions. are 6.3X This helps to reduce cart abandonment, a major issue for online retailers. Sears more likely to convert offers chat, call-back and email on every page to assist consumers complete the than visitors who don’t buying process — this is a standard of availability of help that sets expectations for visitors to sites like Amazon’s. chat. — Bold Software Live Chat Performance Benchmarks October 2009 13
  14. 14. What Can Be Learned From a One-Minute Online Survey Live chat with an agent from product pages allows Sears customers to escalate issues or get answers without interrupting their tasks It would be very revealing to drill down with the participants that expressed concern about contact options to find out, for example: Why is this an issue – what was the reason they needed to contact Amazon when they found it difficult? Was the issue they tried to escalate to do with a site issue, or perhaps an offline issue like late delivery or a billing discrepancy? Did the issue cause them to abandon a purchase and go elsewhere? What were their expectations of Amazon – a telephone number on every page, or just an easier path to an email contact form? 14
  15. 15. What Can Be Learned From a One-Minute Online Survey Visual appearance not of overriding importance Visual appearance is of course another subjective characteristic, but what is striking is that the ‘look’ of the site was not important to most people, whether they liked it or not. (There was only a slight difference here between consumers … the ‘look’ of the site was and professionals.) not important to most people, whether they liked it or not Functionality seems to be what matters — with the appearance of Amazon not getting in the way of people browsing products and buying. This indicates that once the appearance of a website is not strikingly bad (and Amazon’s certainly isn’t), it does not really register prominently with visitors. Once customers can do what they need to do, they’ll not be too concerned about colors, fonts, imagery or other design features. That’s not to say that any design will do – it still needs a professional to design an interface that is visually coherent and serves the brand well. But once people can get where they need to go quickly, and complete their tasks, the visual appearance is very much secondary to functionality. Consumers are probably too short on time to notice the visual design of site that is designed to get things done. This finding also raises further questions: What type of site visitor finds Amazon.com unattractive? Are new visitors more critical of the appearance, perhaps it becomes less important as they find out that the site is functionally excellent? What exactly about the appearance is causing an issue – is it related to clutter, or is it a purely subjective attribute like the choice of colors and fonts? Why do they feel strongly enough about appearance to pick this attribute rather than one of the other 24 available?   15
  16. 16. What Can Be Learned From a One-Minute Online Survey Report Summary Consumers want choice and they want low prices. That’s a big reason for the success of Amazon.com. Alongside its own merchandise, Amazon also offers goods from other retailers and the revenue from this ‘marketplace’ in the first half of the year was roughly $4.2 billion. Major online competitors including Sears and Wal-Mart are aggressively building their own marketplaces and industry experts say several more large retailers will launch similar efforts in the coming months. So, does our study help understand Amazon’s strengths, weaknesses, opportunities and threats? Clearly, ratings and reviews are a critical attraction for Amazon but they need a significant upgrade to exploit new channels of social media, especially Facebook. Sears has already developed a sophisticated online community which looks to be more advanced than Amazon’s. This is critical to consumers and in a world of diminishing loyalty may serve to weaken Amazon’s dominant position Customers increasingly see online retail as a multi-channel experience and it is likely that Amazon will need to provide better options for speaking to a service person in real time. The study indicated a small (but what is likely to be a growing) dissatisfaction in this area. A continuous review process would be able to monitor that. Both Wal- Mart and Sears already offer better capabilities as well as store pick- up options and are experimenting with same-day delivery in major cities Search is highlighted as critical and while Amazon’s is good, there are sites which already surpass its capabilities. There is the added threat of Facebook and Google search as the most popular starting points for consumers. In summary, Amazon received a very positive rating so on the surface it has little to put right. But as we have shown significant challenges are emerging.   16
  17. 17. What Can Be Learned From a One-Minute Online Survey Conclusions We produced this report to step through a complete process; we used Amazon simply because many people are familiar with the site and have impressions. Many readers of the report will have provided input so you know that it did take the 1 minute we say. You all have regular visitors to your sites; what are their impressions, what effect does that have on viral recommendations or on repetitive use. The benefit of the methodology is the combination of a unique survey technique in combination with the expert interpretation from a range of partners that have extensive online experience. Already the methodology has been applied to sites across multiple industries and countries, it provides great value for a comparative inexpensive commitment. 17
  18. 18. What Can Be Learned From a One-Minute Online Survey Methodology We created an online poll and asked participants to tell us which three of 26 statements most closely described how they felt about Amazon.com, and allocate ‘3’ to the strongest sentiment, ‘2’ to the next, and ‘1’ to the third (mirroring Gerry McGovern’s widely-used Customer Centric Index). The statements were displayed in a randomly-ordered list. Although there were 26 statements, they were in fact positive and negative sentiments of 13 core attributes. Visual/Architectural factors Clear menus and links Confusing menus and links Helpful search results Poor search results Simple layout / easy to read Cluttered layout / hard to read Looks attractive / appealing Looks unattractive / unappealing Fast to do things Slow to do things Content factors Accurate information Inaccurate information Complete information Incomplete information Up-to-date information Out-of-date information Plain language Full of jargon / corporate speak Social factors Gives me the facts / transparent Misleading / not transparent Easy to contact a person Hard to contact a person Easy to participate / give feedback Hard to participate / give feedback Has ratings, reviews, Has no ratings, reviews, recommendations recommendations 18
  19. 19. What Can Be Learned From a One-Minute Online Survey The statements as they were presented to participants (randomized for each) 19
  20. 20. What Can Be Learned From a One-Minute Online Survey USA The Customer Respect Group 978.412.4047 http://www.customerrespect.com Net Reflector 206.462.4240 http://www.netreflector.com/ Bob Johnson Consulting 248.766.6425 http://www.bobjohnsonconsulting.com Canada Customer Carewords is a methodology Neo Insight, Inc that helps you to truly understand what 613.271.3001 the top tasks of your customers are when www.neoinsight.com they come to your website and the critical impressions they take away. Europe By continuously improving the perform- Ireland ance of your customers' top tasks, you will Customer Carewords, LLC maximize the performance of your web- www.customercarewords.com site. +353 87 238.6136 United Kingdom We have been developing customer cent- ric strategies for websites since 1994, Barry Hagan identifying the top tasks of thousands of +44 77 1470.0066 customers and employees for organiza- www.customercarewords.com tions such as Microsoft, Cisco, Tetra Pak, Thrivent Financial, US Internal Revenue Brian Lamb Service, NHS Choices, Rolls-Royce, BBC, +44 79 8070.0075 www.customercarewords.com Innovation Norway, etc. We have partners in the UK, Holland, Sweden, Norway, Sweden Canada and United States. Webbrådgivaren Sverige AB +46 040.134.200 http://www.wpr.se Netherlands Sabel Communicatie +31 088.227.2240 20
  21. 21. What Can Be Learned From a One-Minute Online Survey http://www.sabelcommunicatie.nl Norway NetLife Research +47 9240.3165 http://www.netliferesearch.no 21