Listening to the Voice of the Customer: Top 6 Findings from the Revulytics Product Management Survey
•Download as PPTX, PDF•
3 likes•157 views
WATCH the full webinar here: https://www.revulytics.com/webinar-pm-survey-results-2018 Gartner has predicted that by 2021, 75% of software providers will rely on insights from embedded software usage analytics to inform product management decisions and measure customer health. Based on Gartner’s prediction, Revulytics was interested to see where things stand today and wanted to get a better sense of how Product Managers are currently using customer data. This recorded webinar reveals and analyzes the findings of our Q3 Product Management survey including: Effectiveness of tools for collecting customer data Usefulness of customer data for specific product management functions How often customer data is actually being used to make decisions What’s the one thing Product Managers would like customer data to tell them We explore the survey results through the lenses of direct vs. indirect sources of customer data, qualitative vs. quantitative data, and manual vs. automatic collection methods.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Listening to the Voice of the Customer: Top 6 Findings from the Revulytics Product Management Survey
Similar to Listening to the Voice of the Customer: Top 6 Findings from the Revulytics Product Management Survey (20)
Recently uploaded
Recently uploaded (20)
Listening to the Voice of the Customer: Top 6 Findings from the Revulytics Product Management Survey
- 1. Listening to the Voice of the Customer Top 6 Findings from the Revulytics Product Management Survey Keith Fenech VP, Analytics Michael Goff Marketing Director
- 2. Agenda • About the Survey • Survey Results and Reactions • Case Studies – Customer Data Applications • Q&A 2
- 3. About the Survey “By 2021, 75% of software providers will rely on insights from embedded software usage analytics to inform product management decisions and measure customer health.” (Gartner, Predicts 2018: Technology Go-to-Market) 3
- 4. Who Responded? 4 Product Manager 70% Other 30% Job Title 319 Respondents ISV 67% Consulting 17% HW/Equip Mfr 11% SW for Internal Use 5% Company Type
- 5. Who Responded? 5 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 1-24 25-49 50-99 100-499 500-999 1,000-4,999 5,000 or more Company Size Desktop/S erver 40% Cloud/Saa S 30% Embedded/IoT 7% Mobile 23% Application Types
- 6. One thing you would like customer data to tell you? 6
- 7. 7 One thing you would like customer data to tell you? “I would like more customers to respond and be willing to provide feedback. Then, I would like customer data [to] tell us which features we should prioritize working on first. I’d also like better insights into whether a release was successful.” “If they’re telling the truth” “I would want to know each customer more individually so that I can treat them that way as much as possible”
- 8. How effective are the following for collecting customer data in your organization? 8 0% 10% 20% 30% 40% 50% 60% 70% 80% Customer Phone/In-person Interviews Advanced Product Usage Analytics Sales Feedback Support Calls Basic Telemetry from the Product Email Surveys
- 9. Who Uses Customer Data In Your Organization? 9 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Product Management Marketing Software Engineering Senior Management Sales Customer Success Not Used Other
- 10. Customer Data Is Useful For… 10 “What the customer actually values and how the product use has helped them realize their business results” 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Feature Prioritization for Roadmap Development UI/UX Design Beta Testing Deprecating Features Licensing/Pricing Decisions Tracking Software Versions Tracking Piracy or Overuse “Are users able to complete the jobs they want to complete in our software” “What feature would my users miss the most if I [have it] removed? What feature would my users love the most if I added it in?”
- 11. But… 58% use customer data less than half the time to make product roadmap decisions 11 42% 58% Don’t Use Use
- 12. What tool do you use to collect and analyze customers’ product usage data today? 12 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% We use usage analytics software We built our own solution We do not have a tool, but are planning to implement in next 2 years We built a solution using a packaged third party We do not have a tool, and have no plans to implement We collect telemetry data, but do not analyze it “By 2021, 75% of software providers will rely on insights from embedded software usage analytics to inform product management decisions and measure customer health.” (Gartner, Predicts 2018: Technology Go-to-Market)
- 13. What percentage of time do you use customer data to make product roadmap decisions (by tool)? 13 0% 10% 20% 30% 40% 50% 60% 70% We built a solution using a packaged third party We use usage analytics software We built our own solution We do not have a tool, but are planning to implement in next 2 years We do not have a tool, and have no plans to implement We collect telemetry data, but do not analyze it
- 14. Leveraging Customer Data Real World Examples 14
- 15. Accelerating UI/UX Redesign Product Redesign in Half the Time • Mature CAD/CAM software, thousands of customers – Deep and complex product with 1,200+ functions – Needed a UI overhaul: Menu vs Ribbon? • Homebuilt software usage analytics package – Too much time and effort to visualize, analyze, and report on the terabytes of data it was generating • Redesign executed in 18 months at significantly reduced cost by using data from Revulytics Usage Intelligence – which features are used most often – which functions could be grouped together – Result: more intuitive UI with overwhelmingly positive feedback 15
- 16. Mastercam Mill X7 – Before Redesign 16
- 17. Mastercam Mill 2017– After Redesign 17
- 18. Delivering Innovative Features Continuous Customer Feedback Loop, More Data-Driven Org • Building Information Modeling (BIM) software with customers in over 70 countries • Previously relied on customer interviews, user interactions with Sales and Support • Wanted a better understanding of user engagement by geography • Discovered that customer base is using more powerful computers than previously recognized • Developers can now implement features requiring more robust hardware, confident that their innovations won’t leave customers behind 18
- 19. Beta Testing Focusing Engineering and QA Resources • More than 5,000 creative organizations and 100,000 professionals • No data to support intuitive sense of how customers used applications, many for decades • Shift towards agile methodologies required a more quantitative way to capture customers’ needs • Can now direct QA resources more effectively and decrease waste on edge cases and configurations that are rarely used – Savings of $10-20K per release in QA time alone • Extended the functionality of existing products and shaped entirely new offerings based on Revulytics usage data 19 “We are absolutely hooked on usage data. Its one of the first dashboards I fire up every morning. It helps us all know exactly what’s going on, so we’re aware of problems sooner – and opportunities, too.” (Toby Martin, VP, Development & Strategy)
- 20. About Revulytics • Revulytics offers cloud-based software usage analytics that give software producers deep visibility into how their products are being used and misused 20 www.revulytics.com/pmsurvey
- 21. Q&A 21 • Thank you for joining us today - you will receive a link to the recording • Please enter your questions in the GoToWebinar control panel’s Questions box • Download the Report and Case Studies at www.revulytics.com/pmsurvey • Start tracking your product usage in just 30 minutes with a free trial at www.revulytics.com/freetrial • Learn more at devzone.revulytics.com Keith Fenech VP, Analytics Michael Goff Marketing Director
Editor's Notes
- Hello everyone and thank you for joining us. I’m Michael Goff from Revulytics and I’d like to welcome you to today’s webinar, “Listening to the Voice of the Customer: Top 6 Findings from Revulytics’ Product Management Survey.” I am joined today by Revulytics VP of Software Analytics Keith Fenech. Keith and I are excited to share the responses to our recent survey and Keith will share some insights and opinions based on his years as a Product Manager and his experiences working with PMs here at Revulytics. Before we get started with today’s discussion, though, I’d like to take a moment to go over some housekeeping details. First, we’ll address all questions at the end of the webinar. At any time during the webinar please feel free to type your questions into the Questions Box on your GoToWebinar control panel. We’ll address as many of your questions as time allows, and if we don’t get to some questions, we’ll address them individually after the event. We’d also love to get your comments and feedback on the findings as well, so be sure to use the Questions Box for those, too! Second, the entire webinar is being recorded and will be archived for future viewing. You’ll receive an email with the link to the recording in the next day or so.
- Today, we’ll talk about the survey itself and who responded and the results and some thoughts on what we think they mean. We’ll briefly share some case studies about how companies are using customer data to make decisions in the real world, and then we’ll wrap up with Q&A So let’s get started…
- In “Gartner Predicts 2018: Technology Go-To-Market,” Gartner predicted that by 2021, 75% of software providers will rely on insights from embedded software usage analytics to inform product management decisions and measure customer health. Based on this prediction, we were interested to see where things stand today and wanted to get a better sense of how Product Managers are currently using customer data.
- So in Q3 of this year, we sent a survey to the members of the very active Product Management Networking Group on LinkedIn and we received 319 responses. The vast majority of the respondents were Product Managers, but we also saw responses from the C-suite, Marketing, Sales, and Engineering. More than two-thirds of respondents worked for independent software vendors.
- We got responses from a good mix of companies by size, and 70% of respondents are developing desktop/server and cloud/SaaS applications. So most likely you’re well-represented in the responses, but again, be sure to send your comments and questions throughout the webinar!
- So let’s start by looking at what people told us in response to the Open Ended Question: “What is the one thing you would like customer data to tell you?” As you can see, the words that appeared most often are “customer” (obviously!), “features,” “product,” and “users/used/usage,” with significant emphasis on “need,” “value,” and “problem.” This isn’t too surprising as customer needs are – or should be – at the core of PM decisions. Getting insight into the features and uses that customers value can drive more informed (and data-driven) prioritization, roadmap decisions, and resource allocation, which in turn helps PMs address customers’ needs and build applications that delight them. So let’s dig in to a few of the specific comments from the PMs themselves….
- Here are a few of the responses we received – we’ll share some others later in the presentation, but we thought these were both representative and interesting. The first response demonstrates a real thirst for customer data: [CLICK] “I would like more customers to respond and be willing to provide feedback.” The second part speaks more to the ability to analyze that data: “Then, I would like customer data [to] tell us which features we should prioritize working on first. I’d also like better insights into whether a release was successful.” The second comment underscores the sentiment that while aggregate customer data is important, it becomes even more valuable when it allows PMs to understand specific customer needs: : [CLICK] “I would want to know each customer more individually so that I can treat them that way as much as possible.” The key phrase here is, “as much as possible.” While it’s not possible to develop applications for each individual user, it is possible to develop applications – and for that matter features and workflows – that are more attuned to specific types of users or personas. Finally, the most interesting response – in my mind anyway – was, : [CLICK] “If they’re telling the truth.” Without reading too much intent into what prompted this response, I think it’s fair to say that it suggests the reality that much of the customer feedback PMs receive is biased, skewed, or subjective. We’ll touch on this later, but the source and the context of customer data and feedback is relevant. Is it direct or indirect? Is it subjective or objective? Are customers voicing their true needs, or are they trying to negotiate a lower price because of features that are allegedly missing? Do you know if you’re getting feedback on functionality from the users with the most experience with that feature? ************************************************ Other responses: What is never or rarely used My pain point is actually getting the data - I need a service that gives me access to great B2B qualitative data Developing multiple Buyer personas and integrating the User journey in the product itself. Simply, Customer data should be able to tell the next step a user might take and we can enforce that step in the communication and user engagement plan. How our products are used. We offer several features but we know the vast majority of those features are never or rarely used. So what does the customer really want/care about? Feature use - we don't have tooling today to gather this info. How closely do users follow the workflow we designed versus using the product in unanticipated ways Who is using what features when What’s the percentage of people that switched from PC to Mac, or do they use both? Occurrence of a given event as a percentage of total. E.g.: number of users that clicked button A vs other buttons in the same menu Value of enhancements. We get suggestions all the time, but how much value is it really bringing to them? What edge cases are we ignoring that could create new growth opportunities?
- With that in mind, we asked PMs to rate the effectiveness of various methods of collecting customer data, and customer interviews (whether over the phone or in-person) were rated the highest at 69%. But, perhaps in recognition that interviews are subjective and typically represent a small sample of users, it was interesting to see that Advanced Product Usage Analytics (a more automated way of collecting objective data) was the second highest rated method at 63%. As we noted earlier, Gartner predicts 75% of companies will rely on it by 2021, and PMs are already recognizing its effectiveness. It’s also interesting to note that while the top two methods represent qualitative and quantitative data, they are both direct sources of customer data, compared to the next two methods which are more indirect. With Sales Feedback and Support Calls, you’re getting customer data through the lens of someone other than the customer. Of course, these all represent complementary ways of collecting customer data. It’s useful to pause for a moment and look at how Gartner defines different forms of customer feedback in the context of “voice of the customer:” Direct feedback is feedback that the customer intends to provide to the organization – examples include surveys, complaints, market research, or panels. Indirect feedback is derived from instances when the customer is speaking about the organization without specifically intending to provide feedback to the organization – examples include insight from review sites, social networks, and customer care interactions via phone, email and chat sessions. Inferred feedback is operational and transactional data associated with a customer experience or journey such as website clickstream data, purchase history or contact center operational data – and software usage analytics falls under this as well. The customer’s behavior is the source of the feedback here. And before we move on – one of the most interesting findings is how ineffective Email Surveys are viewed to be – especially since we seem to do them all the time! We suspect that this could be recognition that you’re not getting the response rates you need, you’re fighting for attention in the respondents’ in-box, or that you’re getting feedback from users who haven’t actually used the product enough to provide informed responses. *********************************************************** Michael asks: So Keith were you surprised by these responses? Keith Replies: Well initially I had expected Sales & Support feedback to be closer in terms of usefulness, since they are both down there in the arena with customers, but as you pointed out the bias in Support is usually leaning towards unhappy customers having issues with the product, so not the ideal representation of your user base. On the other hand the bias from the Sales channel is usually leaning towards bigger promises and brighter future – the kind of if we develop this we will sell X% more or win these deals. As Michael pointed out, these feedback channels are not mutually exclusive but can very effectively complement each other. One part of the result I would like to point out is how the top 2 sources came so interestingly close in terms of usefulness. The difference between these two is that in-person customer conversations (which ranks as the most effective) is a very manual and resource intensive channel, that typically can only reach a limited sample set of users… On the other hand Product Usage Analytics (which ranks as the 2nd most effective) can be a fully automated data collection mechanism, that fills the rest of the gap by reaching all your customers (or even trial users that never turn into customers). So companies who make use of both of these channels are getting the added value of having 1-on-1’s with a select set of customers, whilst backing up that sample data with automated usage analytics. So my observation here is that usage analytics seems to be one of the most effective means to get as close as possible to in-person conversations in a more scalable manner. PLUS, companies who use this type of inferred feedback do not have to deal with the bias that Michael was mentioning earlier...since they are not pre-selecting the sample of users on whom to conduct in-person interviews, such as: Customers who are Paying the most? Or those calling in when they have a problem (support) Or the most ‘happy’ customers that accept to talk to us to give us feedback Again, as we think about these complementary sources of customer data, it’s important to consider the different spectrums like: direct vs. indirect and objective vs. subjective feedback.
- So who are the consumers of customer data in your organizations? It shouldn’t surprise you that PMs recognize that leveraging a wide range of customer data is essential to meeting the needs of their users, and that PMs are the largest consumers of customer data. Some of you may recall a webinar we did with Steve Johnson – now VP of Product at Pragmatic Marketing – where we discussed using data to respond to HiPPOs – the Highest Paid Person’s Opinions. PMs rely on data to add context to these discussions and prioritize their roadmaps in the face of more subjective demands. Because it’s the PM’s role to consider ideas and feedback from a wide range of sources (no matter how crazy they may sound) they need to be able to filter what is most relevant and important. It makes sense that PMs rely on customer data and insight to help determine what Engineering will be building to meet and exceed customers’ needs. Customer data is also being leveraged to varying degrees across the whole organization. Marketing benefits from customer data along with data from web analytics, market research, etc. as it works on go-to-market plans and specific marketing campaigns. The interest from Senior Management also aligns with the 2016 PwC Global Data and Analytics Survey findings that executives want decision making to be faster and more data-driven. But with so much focus and discussion on Customer Experience and the Voice of the Customer, it is surprising that Customer Success is not using customer data more often. We suspect that usage will increase as there is more focus on tools to enable customer experience leaders, but given the cost of customer acquisition, this response really is a bit of a shock. ************************************************ https://www.pwc.com/us/en/services/consulting/analytics/big-decision-survey.html
- We also wanted to understand specifically what PMs think customer data is useful for. Looking at respondents who replied that Customer Data was Useful, Very Useful, or Highly Useful, 97 percent of the respondents said feature prioritization for roadmap development. As they could choose more than one response for this question, 90 percent also said UI/UX design, 86 percent beta testing, and 78 percent deprecating features. You may have seen that SD Times conducted a similar survey of software developers in Q2 2018. They also responded with the same top 3 uses of customer data: Feature Prioritization for Roadmap Development (89%) UI/UX Design (79%) Beta Testing (78%) But do these responses align with our open-ended question about “the one thing PMs would like customer data to tell them?” They do – and there’s significant alignment here. [CLICK] They responded that they wanted to know “What the customer actually values and how the product use has helped them realize their business results” – aligning with Feature Prioritization and Roadmap Development. [CLICK] They want to know “Are users able to complete the jobs they want to complete in our software” – aligning with UI/UX Design. [CLICK] They also want to know “What feature would my users miss the most if I [have it] removed? What feature would my users love the most if I added it in?” – aligning with Deprecating Features. [CLICK] All of that alignment sounds great, but surprisingly as the next slide shows…
- …58 percent said their companies use customer data less than half the time to make product roadmap decisions. Michael asks: Keith, what do you make of this? Why are the same people who say customer data is useful in several areas not using that data to drive roadmap decisions? KEITH: Well that’s a good point. It seems people do realize that the data is valuable, yet it could be that the data they have is somehow “not usable”. From experience over the years I would say there are 3 major reasons companies end up NOT using data, even if they believe in the usefulness of this data: 1 - They do not have easy access to the data Not the first time I hear PMs saying, oh we implemented a call home system inside the product but all this data is hidden in some database and I need to ask Jack the engineering guy to run SQL queries every time I need to pull up some stats. This takes time, I can’t afford to keep going to Jack to drill down into different areas so it is way faster to just use my gut feeling when working on the roadmap! 2 – There is just too much data and they don’t know where to start. This is usually the result of collecting way too much raw data from different sources. Data is usually not correlated so you have a lot of pieces of the puzzle but lack an easy way to join the data into something that is actionable. 3 – They cannot trust the source or the data itself. This usually happens either because the data is incomplete or unreliable or biased because the sample set is too small or not a true representation of the whole customer base. Of course there could also be some edge cases like the HIPPO situation where the boss’s opinion always takes over.... But outside of that, if you are considering a data-based PM strategy, always ask yourself: 1. Do you have direct & real-time access to the data? 2. Do you have the right tools to help you analyze the data? 3. Is the data easily digestible so you can filter noise from actionable answers? Michael: Great points Keith – and it’s worth reiterating the benefit of having a mix of Qualitative AND Quantitative means of data collection that are representative of the whole customer base. But let’s hear what our audience today thinks about this.
- We asked respondents to select one tool that they use to analyze customers’ product usage today, and the responses seem to align with Gartner’s prediction: 43% use usage analytics software 29% built their own solution 9% don’t have a tool but are planning to implement in the next 2 years And only 2% are collecting telemetry data, but not analyzing it. Interestingly, 8% do not have a tool and have no plans to implement one. Why not? Keith, how does this map to what you’re seeing when you talk to PMs? KEITH: A good chunk of Companies we’ve worked with in the past have gone through several cycles of collecting and analyzing data. - For those having engineering resources, it is very common for them to take a shot at building some form of data collector inhouse - Usually this ends up as part of their check for updates system, where raw data gets appended to a log or database. - A common mistake is they start collecting too much raw data without thinking about how they can analyze it once it accumulates - This difficulty to analyze the data often causes them to shelve the project as it becomes impractical to extract real insight from the noise. - Companies who have already been through this learning cycle would typically revisit the project with greater awareness of how the data needs to be consumed. This is the point where they would invest in tools/solutions to make their data life easier… So my suggestion is when you are building a data strategy, you focus most of your effort on how you will be doing the analysis of the data, and not just the collection… Make sure you have a fast and reliable visualization tool to help you digest data in bites that you can chew, because as a PM you need actionable insight, not just raw data.
- Finally, it’s interesting to return to our question about the percentage of time respondents used customer data to make product roadmap decisions, but look at it through the filter of the product usage tools they are using today. Clearly, Product Managers that have the ability to collect and analyze customer’s product usage data are more comfortable using that insight to make roadmap decisions. Those without a tool and no plans to implement one, are only using customer data to make roadmap decisions 35% of the time. Keith – what do you make of these results? KEITH: Well the results speak for themselves here… People who have the proper tools in place tend to rely more often on customer data than those without a means to easily analyze the data. It’s rather concerning to see the people who took the time and effort to build their own solutions are only using it 38% of the time. This ties in with what we discussed in the previous slide, where some companies who go down the route of building a data collection system, tend to fall short when it comes to building a full analysis and visualization tool to derive actionable insight from their raw data. So the reason for seeing a low 38% usage could be that whatever they built does not satisfy the practical needs of the PMs. Let’s actually see some success cases of how companies are using customer data to make decisions….
- When you make improvements or changes to a product, you can usually validate them by monitoring how user behavior changes after you deploy such changes. However mistakes can be expensive to fix after the product is deployed. So taking the right decisions beforehand is crucial to guarantee success of your project. And this is exactly what Mastercam did… So what were the Challenges for this company? CNC Software had a CAD/CAM product that had been around for a number of years. The UI on this product had become rather complex so the Mastercam team decided it was time for a major UI overhaul. They turned to Revulytics usage intelligence and started tracking how their users were interacting with the old UI…
- This what their software looked like…. After analysing what function-groups their customers were using and in what order, they decided to go for a UI based on a Microsoft-style ribbon interface. The daunting task was to determine which of the 1200 functionalities could be grouped up and which could be totally eliminated from the new UI. So over the course of a year, they used usage analytics to study detailed usage behaviour from various user segments and deployed a single version of their software with a totally new UI.
- And this is what their new version looks like today… You can see the new ribbon interface and a much cleaner UI... Mastercam estimated that without usage analytics it would have taken them at least 36 months to survey users and stagger the changes over several releases. With usage analytics they did this in half the time because they could move way faster and without making mistakes since all the changes were based on accurate usage data from all customers, not a small sample of interviewed customers.
- PUT MINOR POINTS IN THE NOTES, NOT ON THE SLIDE – add more about the customer feedback Another case was Solibri. These guys have a Building Information Modeling (BIM) software with customers in over 70 countries. Given buildings vary drastically between different countries, Solibri knew there were significant variances in how professionals in different countries use its products. By using Revulytics, they segmented the usage patterns by geography to understand how the software was being used in each region. Their product teams used this regional data to better reflect customers’ localization requirements more closely. Previously they already ran customer interviews, and relied on user interactions with Sales and Support, but the automated analytics data could give them faster and more complete insight. For example something else that they discovered was that they were always afraid of adding too many resource intensive simulation functions, so for example reducing the amount of datapoints in a model simulation so that it can be computed faster. However within 6 weeks of using Revulytics they found out that in reality their typical customers were running way more powerful machines than they originally had imagined. So that allowed them to tweak their software to take full advantage of the machines they were running on. This increased speed, agility, and data-driven insight enabled Solibri to build on its strengths and extend its competitive advantage.
- Extensis is another case. This is a company developing professional font management software. In this case they had no data to support their intuition of how customers were using their applications. And they wanted to bring more "voice of the customer" approach into their agile development process. So Extensis uses Usage Intelligence to capture data on: - Feature Usage, How customers used specific file types, manipulations and other user behaviour. Using this data, Extensis was able to extend the functionality of existing products They created entirely new offerings based on Revulytics usage data. And more interestingly, they estimated that they are now saving $10,000-$20,000 per release in QA costs alone, with future anticipated savings on feature development.
- Thanks Keith – those are really good examples of ways software producers are using customer data to inform decisions on roadmaps, prioritizing development, and beta testing. It looks like Gartner’s prediction is on target. Before we jump into the Q&A, I wanted to tell you a little bit about Revulytics. We offer cloud-based software usage analytics that give software producers deep visibility into how their products are being used and misused, with solutions for product management, software development, marketing, sales, and license compliance. Our customers gain actionable intelligence to generate revenue, optimize product development, and make data-driven decisions across their business. Please contact us if you’d like to learn more. As an attendee, we’ll be sending you a link to the webinar recording and a copy of our report on the product management survey, but they will also be available along with the case studies at www.revulytics.com/pmsurvey
- Q&A Our time is just about up, and I want to thank Keith for his time and insights today. As I mentioned earlier, we will send all attendees an email with a link to the recording and the product management survey report, but please feel free to contact us if you have any questions in the meantime. Finally, we’d like to thank each of you for attending today’s webinar and hope that you enjoy the rest of your day.