Katy Wilburn lead the third session of the day at the 6th annual Voluntas conference. Her presentation focused on hot to reach customer goals using research design/methods and analysis.
Discover Katy's top seven tips now.
This document outlines seven questions that can help consultants rapidly assess problems and get results for clients with limited time. The questions are: 1) What does success look like? 2) What is the problem? 3) Which metrics are you trying to positively affect? 4) What are employees doing/not doing that they shouldn't/should be doing? 5) When did the problem first begin and when does it occur? 6) Where does the problem occur? 7) How big is the problem in measurable terms? Asking these questions helps ensure consultants solve the right problem and deliver value to their clients.
Nilfisk – Net Promoter Score® ExperienceCustomerGauge
1) The document discusses Nilfisk's experience with Net Promoter Score (NPS) as a metric to measure customer satisfaction and loyalty over time.
2) Initial challenges included low response rates to surveys and a lack of engagement across the organization. Actions taken focused on improving communication to customers and internal training.
3) By integrating NPS into key processes and communicating results regularly, it became a core business metric that influenced strategy and identified areas for structural change based on customer feedback.
Realizing Immediate Value from Customer SuccessGainsight
The document discusses a presentation about realizing immediate value from customer success. It describes a methodology called the Cadence methodology for transforming customer success processes. The Cadence methodology is an 11-step approach involving touching customers at scale, tracking customer health consistently, and transforming customer success processes. It also describes how the company Gainsight can help organizations implement the Cadence methodology through tools and services like Experience Success Express.
4 User Experience Questions to Include in Your Next UX SurveySogolytics
Make the most of every outreach by using key questions to target opportunities for improvement in your users' experience. If you don't know what's wrong (or right!), how can you decide where to spend your time and energy? Make the most of your resources by using smart surveys to uncover insights.
An introduction to NPS - Net Promoter ScoreStartquestion
Learn how to gather valuable data with the use of a client loyalty assessment tool.
Startquestion - Online surveys, forms, and tests.
Net Promoter, Net Promoter Score, and NPS are registered trademarks of Satmetrix Inc., Bain & Company Inc., and Fred Reicheld.
Learn more about how to avoid mistakes in NPS surveys:
https://blog.startquestion.com/7-mistakes-in-nps-questions-find-out-how-not-to-make-them-in-your-online-survey/
The document outlines the agenda for a quarterly business review meeting. It includes sections to discuss wins and positive metrics, areas still needing improvement, progress on joint goals, short-term goals for the next 1-3 months, long-term goals for the next 3-6 months, new products and features in development, and getting feedback from the customer on where the company's product stands currently and for renewal. The meeting aims to review performance, identify opportunities, and make plans to further goals.
Beyond Stakeholder Analysis And ManagementJoe Newbert
This document summarizes a presentation titled "Beyond Stakeholder Analysis And Management" given by Joe Newbert on November 8, 2017 at the PMI Business Analysis Virtual Conference. The presentation provides tactics for building relationships that satisfy stakeholders, organizations, and oneself. It discusses performing practical stakeholder analysis and management through activities. It also provides context for business analysis, project delivery, and change management.
This document outlines seven questions that can help consultants rapidly assess problems and get results for clients with limited time. The questions are: 1) What does success look like? 2) What is the problem? 3) Which metrics are you trying to positively affect? 4) What are employees doing/not doing that they shouldn't/should be doing? 5) When did the problem first begin and when does it occur? 6) Where does the problem occur? 7) How big is the problem in measurable terms? Asking these questions helps ensure consultants solve the right problem and deliver value to their clients.
Nilfisk – Net Promoter Score® ExperienceCustomerGauge
1) The document discusses Nilfisk's experience with Net Promoter Score (NPS) as a metric to measure customer satisfaction and loyalty over time.
2) Initial challenges included low response rates to surveys and a lack of engagement across the organization. Actions taken focused on improving communication to customers and internal training.
3) By integrating NPS into key processes and communicating results regularly, it became a core business metric that influenced strategy and identified areas for structural change based on customer feedback.
Realizing Immediate Value from Customer SuccessGainsight
The document discusses a presentation about realizing immediate value from customer success. It describes a methodology called the Cadence methodology for transforming customer success processes. The Cadence methodology is an 11-step approach involving touching customers at scale, tracking customer health consistently, and transforming customer success processes. It also describes how the company Gainsight can help organizations implement the Cadence methodology through tools and services like Experience Success Express.
4 User Experience Questions to Include in Your Next UX SurveySogolytics
Make the most of every outreach by using key questions to target opportunities for improvement in your users' experience. If you don't know what's wrong (or right!), how can you decide where to spend your time and energy? Make the most of your resources by using smart surveys to uncover insights.
An introduction to NPS - Net Promoter ScoreStartquestion
Learn how to gather valuable data with the use of a client loyalty assessment tool.
Startquestion - Online surveys, forms, and tests.
Net Promoter, Net Promoter Score, and NPS are registered trademarks of Satmetrix Inc., Bain & Company Inc., and Fred Reicheld.
Learn more about how to avoid mistakes in NPS surveys:
https://blog.startquestion.com/7-mistakes-in-nps-questions-find-out-how-not-to-make-them-in-your-online-survey/
The document outlines the agenda for a quarterly business review meeting. It includes sections to discuss wins and positive metrics, areas still needing improvement, progress on joint goals, short-term goals for the next 1-3 months, long-term goals for the next 3-6 months, new products and features in development, and getting feedback from the customer on where the company's product stands currently and for renewal. The meeting aims to review performance, identify opportunities, and make plans to further goals.
Beyond Stakeholder Analysis And ManagementJoe Newbert
This document summarizes a presentation titled "Beyond Stakeholder Analysis And Management" given by Joe Newbert on November 8, 2017 at the PMI Business Analysis Virtual Conference. The presentation provides tactics for building relationships that satisfy stakeholders, organizations, and oneself. It discusses performing practical stakeholder analysis and management through activities. It also provides context for business analysis, project delivery, and change management.
Coaching And Development Presentation And Leaders Guide 2008winpuc
Coaching is a development tool used by managers to provide feedback to employees and improve performance. It involves having face-to-face, casual conversations where the manager prompts self-discovery and focuses on business and process-related topics. Effective coaching is timely, consistent, results-oriented, direct but gentle, and honest. When coaching, managers should congratulate positive behavior, outline expectations, analyze opportunities for improvement, communicate benefits, and have the employee commit to action.
This is a Quarterly Business Review Template to be used by Customer Success Management organizations.
One of the most important activities your Customer Success Managers (CSMs) will perform is the Quarterly Business Review (QBR).
QBRs are sometimes known by different names – Business Reviews or Executive Business Reviews – but no matter what they’re called, they’re incredibly important and the agenda and flow are largely going to fall on the CSM, so it’s critical to help them prepare for, and perform QBRs, the right way.
The most successful Enterprise SaaS companies know that growing revenue only through new customer acquisition is the less efficient way to scale. Rather, they understand that growing revenue within your existing customer base - through up-sells, cross-sells, and expanded use - is the most profitable way to scale.
In fact, Enterprise SaaS companies that grow revenue - and company valuation - by expanding revenue within their existing customer base also know the key to making this work is to focus on - and operationalize - Customer Success.
10 Essential Tips For An Effective Business ReviewWarwick Brown
A business review is the cornerstone of any client/supplier partnership and one every Account Manager must master. What follows are 10 essential tactics to get your reviews back on track and turn them into one of the most anticipated events on your clients’ calendar. I hope you find them useful
Improve Your Law Firm Performance With These Smart TipsBenny Henson
This document provides seven tips to improve law firm performance and efficiency without affecting quality work. The tips are to 1) map out work processes and information flow, 2) eliminate unnecessary steps by outsourcing tasks, 3) balance client handling with other work responsibilities, 4) implement robust knowledge management practices to share information and make work easier, 5) focus by only doing core tasks and hiring others for other categories of work, 6) outsource legal work to reduce costs and ease scheduling, and 7) define service levels for clients. Adopting these tactics can help firms finish tasks faster with less effort and improve overall efficiency.
Customer Satisfaction Research has two primary purposes:
1) Measure and track your effectiveness in managing customer satisfaction.
2) Provide insights for improving customer satifaction.
Presentation by Paul van Veen, Customer ServiceBenchmarking Australia (CSBA) at the Serve You Right Conference, Melbourne, September 15-16, 2011. Provides an overview of What to look for in customer service measures; The overall service improvement model; Customer satisfaction measures; Understand current performance; Key points for successful customer satisfaction measurement
This document provides guidance on creating an effective proposal. It emphasizes including an executive summary to clearly state your understanding of the client's needs. Key components discussed are the budget, timeline, exclusions/limitations, and evaluations. The document stresses customizing the proposal to each client rather than copying others, and guarding your intellectual property while providing initial recommendations. It also recommends including your expertise, experience, capabilities, case studies as evidence of past success, and references to help sell your abilities.
How to Build an Effective Customer Health ModelTotango
Your customers’ success is critical to your business. But how do you define and measure their success? Should you look at the number of support tickets, rate of adoption, revenue per customer, churn rate, net promoter score, or customer lifetime value?
Instead of looking at specific metrics, you need to take a holistic view of the customer using a Customer Health value. Health definitions should incorporate several KPIs and variables, such as business outcomes, lifecycle stage, product usage, license utilization, support tickets and other relevant information.
In these slides presented during our webinar, you'll learn how to build an effective health model to measure real business outcomes, drive retention and increase revenue. Presented by Jill Rubin, CMO of Totango, and Isabelle Rochon, Director of Customer Success from iPerceptions, a Totango customer.
Ken Sandy - Ten tips to lead as a PM through influenceProduct Anonymous
Product managers are under pressure to drive results, but cannot wield direct power or authority to achieve their objectives. If you don't know how to influence people at all levels of the organization, how will you create the best possible product? In this talk, Ken Sandy shares ten techniques from The Influential Product Manager that product managers can immediately apply at each stage of the product life cycle to achieve the best outcome for the customer and their organization.
A CEO once made the case that business success is built on the millions of brief interactions between a business and its customers.
Those interactions were referred to as "Moments of Truth".
Since then, the importance of flawless customer service has become accepted across most industries.
The essential role of measurement in customer satisfaction improvement is well recognized.
However, measurement must be tied to management action to create a truly world-class customer satisfaction
Explore the bottom-line payoffs for building customer satisfaction and you'll discover elements of an effective customer satisfaction system.
Measuring and Managing Customer Satisfaction ProgramsMARY MALASZEK
The document discusses various methods for measuring and managing customer satisfaction, including the customer decision journey, post-purchase behaviors, and metrics like NPS and Kano. It recommends collecting both quantitative data through surveys, ratings, and feedback as well as qualitative data from social media, reviews, and user groups. Measuring customer satisfaction provides strategic value when the results are used to address weaknesses, when shared company-wide, and when tied to employee performance and compensation.
The document outlines the steps for outsourcing a division, including gathering facts, brainstorming solutions, identifying advantages and disadvantages of solutions, deciding on the most appropriate solution, implementing the solution, following through on results, defining issues that need changed, and repeating the process. It discusses using prescriptive and descriptive decision making and defending the reasoning for outsourcing through a transparent problem-solving approach. The resources cited provide additional information on factors that affect decisions, improving decision making, ethical considerations, and board room dynamics.
Your data is only as good as the survey behind it – here, Qualtrics’ Dave Vanette talks best practice for designing surveys and how to manage them to get the best possible data back.
This document describes a consulting firm that uses diagnostic software to help companies improve business performance and strategy execution. The software conducts an in-depth survey that provides a comprehensive analysis of an organization, identifying opportunities for improvement. Managers receive individual reports analyzing their workgroups and recommending actions to boost efficiency, effectiveness and alignment with company goals. The consultants claim their approach produces measurable gains in workforce performance through ongoing support, accountability systems and targeted action plans.
Class 7 adlt 610 fall 2011 diagnosis to discovery and data collection rev tjctjcarter
The document discusses the discovery phase of a consulting engagement. It emphasizes maintaining an open dialogue with the client to help them identify underlying problems and be open to the discovery process. Data should be collected jointly with the client to understand how the problem is currently being managed. The consultant should distinguish between the presenting problem and underlying issues and involve the client in interpreting findings to develop recommendations for long-term solutions.
The document discusses customer satisfaction and outlines a process for organizations to focus on customers. It recommends determining who customers are and what attributes of products/services are most important. Organizations should arrange attributes by customer importance, measure satisfaction levels, and tie feedback to processes. The process involves implementing metrics, working on low satisfaction/high importance areas, continually updating customer input, and maintaining communication with stakeholders. The goal is to increase customer satisfaction through continuous process improvements.
Leadership Series #2 - Structured Approach to Problem SolvingZana Gawan-Taylor
It is easy to quickly jump into a solution when we are busy, to fight the fire. Take time to breathe, and have the space to think it through and be rational. It is worth the investment.
Learn how to solve problems effectively using Root Cause Analysis - Five Whys and Fishbone Diagram.
Share other tools that you have found to be effective.
Measuring Customer Satisfaction: trends and challenges for Customer ServiceEstelle Wienk
This trend report looks at developments customer feedback programs, the importance of measuring your customers' journey and how you can measure it: NPS, CES or CSAT?
The document provides examples of objectives and key results (OKRs) for different roles within a customer success organization. It includes OKRs for roles like Head of Customer Success, Customer Success Lead, Customer Success Manager, and Customer Success Executive. The OKRs follow the five pillars of Goal Science - connected, supported, adaptable, progress-based, and aspirational. They also adhere to the format of stating the goal followed by "as measured by" and key results. The examples are meant to guide organizations in setting their own OKRs.
The Value of Developing Relationships in SellingJames Muir
Presentation given in 2012 to the NextGen Healthcare national sales force. On the value of developing relationships and genuinely providing value for clients.
You want to manage an important Change culture in your firm or team? First, Initiate your Change Story with your Customer needs. Learn to better know him, to better communicate with him to build a relation of Trust & partnership.
Coaching And Development Presentation And Leaders Guide 2008winpuc
Coaching is a development tool used by managers to provide feedback to employees and improve performance. It involves having face-to-face, casual conversations where the manager prompts self-discovery and focuses on business and process-related topics. Effective coaching is timely, consistent, results-oriented, direct but gentle, and honest. When coaching, managers should congratulate positive behavior, outline expectations, analyze opportunities for improvement, communicate benefits, and have the employee commit to action.
This is a Quarterly Business Review Template to be used by Customer Success Management organizations.
One of the most important activities your Customer Success Managers (CSMs) will perform is the Quarterly Business Review (QBR).
QBRs are sometimes known by different names – Business Reviews or Executive Business Reviews – but no matter what they’re called, they’re incredibly important and the agenda and flow are largely going to fall on the CSM, so it’s critical to help them prepare for, and perform QBRs, the right way.
The most successful Enterprise SaaS companies know that growing revenue only through new customer acquisition is the less efficient way to scale. Rather, they understand that growing revenue within your existing customer base - through up-sells, cross-sells, and expanded use - is the most profitable way to scale.
In fact, Enterprise SaaS companies that grow revenue - and company valuation - by expanding revenue within their existing customer base also know the key to making this work is to focus on - and operationalize - Customer Success.
10 Essential Tips For An Effective Business ReviewWarwick Brown
A business review is the cornerstone of any client/supplier partnership and one every Account Manager must master. What follows are 10 essential tactics to get your reviews back on track and turn them into one of the most anticipated events on your clients’ calendar. I hope you find them useful
Improve Your Law Firm Performance With These Smart TipsBenny Henson
This document provides seven tips to improve law firm performance and efficiency without affecting quality work. The tips are to 1) map out work processes and information flow, 2) eliminate unnecessary steps by outsourcing tasks, 3) balance client handling with other work responsibilities, 4) implement robust knowledge management practices to share information and make work easier, 5) focus by only doing core tasks and hiring others for other categories of work, 6) outsource legal work to reduce costs and ease scheduling, and 7) define service levels for clients. Adopting these tactics can help firms finish tasks faster with less effort and improve overall efficiency.
Customer Satisfaction Research has two primary purposes:
1) Measure and track your effectiveness in managing customer satisfaction.
2) Provide insights for improving customer satifaction.
Presentation by Paul van Veen, Customer ServiceBenchmarking Australia (CSBA) at the Serve You Right Conference, Melbourne, September 15-16, 2011. Provides an overview of What to look for in customer service measures; The overall service improvement model; Customer satisfaction measures; Understand current performance; Key points for successful customer satisfaction measurement
This document provides guidance on creating an effective proposal. It emphasizes including an executive summary to clearly state your understanding of the client's needs. Key components discussed are the budget, timeline, exclusions/limitations, and evaluations. The document stresses customizing the proposal to each client rather than copying others, and guarding your intellectual property while providing initial recommendations. It also recommends including your expertise, experience, capabilities, case studies as evidence of past success, and references to help sell your abilities.
How to Build an Effective Customer Health ModelTotango
Your customers’ success is critical to your business. But how do you define and measure their success? Should you look at the number of support tickets, rate of adoption, revenue per customer, churn rate, net promoter score, or customer lifetime value?
Instead of looking at specific metrics, you need to take a holistic view of the customer using a Customer Health value. Health definitions should incorporate several KPIs and variables, such as business outcomes, lifecycle stage, product usage, license utilization, support tickets and other relevant information.
In these slides presented during our webinar, you'll learn how to build an effective health model to measure real business outcomes, drive retention and increase revenue. Presented by Jill Rubin, CMO of Totango, and Isabelle Rochon, Director of Customer Success from iPerceptions, a Totango customer.
Ken Sandy - Ten tips to lead as a PM through influenceProduct Anonymous
Product managers are under pressure to drive results, but cannot wield direct power or authority to achieve their objectives. If you don't know how to influence people at all levels of the organization, how will you create the best possible product? In this talk, Ken Sandy shares ten techniques from The Influential Product Manager that product managers can immediately apply at each stage of the product life cycle to achieve the best outcome for the customer and their organization.
A CEO once made the case that business success is built on the millions of brief interactions between a business and its customers.
Those interactions were referred to as "Moments of Truth".
Since then, the importance of flawless customer service has become accepted across most industries.
The essential role of measurement in customer satisfaction improvement is well recognized.
However, measurement must be tied to management action to create a truly world-class customer satisfaction
Explore the bottom-line payoffs for building customer satisfaction and you'll discover elements of an effective customer satisfaction system.
Measuring and Managing Customer Satisfaction ProgramsMARY MALASZEK
The document discusses various methods for measuring and managing customer satisfaction, including the customer decision journey, post-purchase behaviors, and metrics like NPS and Kano. It recommends collecting both quantitative data through surveys, ratings, and feedback as well as qualitative data from social media, reviews, and user groups. Measuring customer satisfaction provides strategic value when the results are used to address weaknesses, when shared company-wide, and when tied to employee performance and compensation.
The document outlines the steps for outsourcing a division, including gathering facts, brainstorming solutions, identifying advantages and disadvantages of solutions, deciding on the most appropriate solution, implementing the solution, following through on results, defining issues that need changed, and repeating the process. It discusses using prescriptive and descriptive decision making and defending the reasoning for outsourcing through a transparent problem-solving approach. The resources cited provide additional information on factors that affect decisions, improving decision making, ethical considerations, and board room dynamics.
Your data is only as good as the survey behind it – here, Qualtrics’ Dave Vanette talks best practice for designing surveys and how to manage them to get the best possible data back.
This document describes a consulting firm that uses diagnostic software to help companies improve business performance and strategy execution. The software conducts an in-depth survey that provides a comprehensive analysis of an organization, identifying opportunities for improvement. Managers receive individual reports analyzing their workgroups and recommending actions to boost efficiency, effectiveness and alignment with company goals. The consultants claim their approach produces measurable gains in workforce performance through ongoing support, accountability systems and targeted action plans.
Class 7 adlt 610 fall 2011 diagnosis to discovery and data collection rev tjctjcarter
The document discusses the discovery phase of a consulting engagement. It emphasizes maintaining an open dialogue with the client to help them identify underlying problems and be open to the discovery process. Data should be collected jointly with the client to understand how the problem is currently being managed. The consultant should distinguish between the presenting problem and underlying issues and involve the client in interpreting findings to develop recommendations for long-term solutions.
The document discusses customer satisfaction and outlines a process for organizations to focus on customers. It recommends determining who customers are and what attributes of products/services are most important. Organizations should arrange attributes by customer importance, measure satisfaction levels, and tie feedback to processes. The process involves implementing metrics, working on low satisfaction/high importance areas, continually updating customer input, and maintaining communication with stakeholders. The goal is to increase customer satisfaction through continuous process improvements.
Leadership Series #2 - Structured Approach to Problem SolvingZana Gawan-Taylor
It is easy to quickly jump into a solution when we are busy, to fight the fire. Take time to breathe, and have the space to think it through and be rational. It is worth the investment.
Learn how to solve problems effectively using Root Cause Analysis - Five Whys and Fishbone Diagram.
Share other tools that you have found to be effective.
Measuring Customer Satisfaction: trends and challenges for Customer ServiceEstelle Wienk
This trend report looks at developments customer feedback programs, the importance of measuring your customers' journey and how you can measure it: NPS, CES or CSAT?
The document provides examples of objectives and key results (OKRs) for different roles within a customer success organization. It includes OKRs for roles like Head of Customer Success, Customer Success Lead, Customer Success Manager, and Customer Success Executive. The OKRs follow the five pillars of Goal Science - connected, supported, adaptable, progress-based, and aspirational. They also adhere to the format of stating the goal followed by "as measured by" and key results. The examples are meant to guide organizations in setting their own OKRs.
The Value of Developing Relationships in SellingJames Muir
Presentation given in 2012 to the NextGen Healthcare national sales force. On the value of developing relationships and genuinely providing value for clients.
You want to manage an important Change culture in your firm or team? First, Initiate your Change Story with your Customer needs. Learn to better know him, to better communicate with him to build a relation of Trust & partnership.
Your Value Proposition is the reason that customers choose to buy from you, it needs to be a core business skill. This slideshare shows you how to develop compelling value propositions with a focus on B2B companies
The 9 secrets of successful customer feedback and action programsGenroe
Ever wondered how successful customer feedback and action programs differ from unsuccessful programs? We did. So we asked 80 organizations, operating at various levels of success, what they were doing.
Are your Product Managers using an appropriate framework? What do Sales, Implementations and your customers say about your products? Is too much time spend on process, and not enough on value and outcomes?
These are some ideas on a simple framework for Product Management that might work for you.
The document discusses creating a compelling client proposition. It suggests that clients will expect clear answers about what they are paying for in a more transparent pricing environment post-RDR. A client proposition should explain the services being provided in a way clients understand and demonstrate value. Many advisers currently provide services clients don't need or want. Research shows clients want responsive, personalized service and are willing to pay more for better service. High net worth clients want specialized, independent advice and transparency. An effective proposition must meet rising client expectations.
This document provides an overview of the consulting framework and process. It defines a consultant as someone who can influence an organization without direct power, while a manager has direct responsibility. The core transaction is transferring expertise from consultant to client. Key skills include technical knowledge, interpersonal skills, and consulting skills like contracting, discovery, analysis, engagement, and termination. The consulting process involves defining problems, selecting dimensions to study, gathering data, providing recommendations, and iterating if needed. Building solutions requires a fact-based, hypothesis-driven approach addressing all key drivers. Selling solutions requires building trust and gaining internal commitment from clients.
This document outlines a client feedback program to help businesses improve through gathering client opinions. It discusses the benefits of client research, how the program works through surveys and analysis, and provides examples of sample reports including key metrics on brand image, satisfaction, and opportunities. The goal is to help businesses better understand clients, drive growth, and enhance loyalty through a continuous feedback process.
eSavvy webinar: Top 5+1 Tips of How to Maximize the ROI of a CRM InvestmenteSavvy
Do you want to know how to maximize the return of investment in a CRM system? Are you interested to find out how to truly use a CRM system to its full potential and how you can benefit from doing so? Then watch our next webinar, during which our Managing Director David Goad will share with you top tips on how to maximize the ROI of a CRM investment. David has been involved in more than 200 Business Application Implementations and was one of the contributing authors of the proven project management methodology recommended by Microsoft – Dynamics Sure Step. He will use his vast experience and project management knowledge to provide you with valuable insights about how to select your new CRM, how to prepare for CRM deployment projects and what to do during and after the project to maximize the results from your investment.
How to run a perfect Net Promoter Score campaign [Webinar]Retently
We explain the basics of Net Promoter Score and how to choose a good service to get started with. We’ll show you how to personalize your survey, what are the advantages and disadvantages of various survey channels. You will learn the main differences between on demand and transactional NPS, how to avoid the most common mistakes, how to efficiently close the feedback loop and finally - what is a good NPS score.
You can watch a recording of this webinar here: https://retently.wistia.com/medias/jrorlz4ie2#
In this webinar, our in-house expert, Nancy Thakur, VWO’s Sr. Optimization Consultant, walks you through the key elements that can help you run your CRO program the right way.
Maximizing the Value of Customer Feedback Surveys Design and Benefits.pdfCViewSurvey
Maximizing the Value of Customer Feedback Surveys Design and Benefits" is a PDF document that discusses the importance of designing effective customer feedback surveys and the benefits they provide to businesses. The document begins by highlighting the importance of customer feedback in driving business growth and improving customer satisfaction.
The PDF then goes on to discuss the key factors to consider when designing a customer feedback survey, including the types of questions to ask, the format of the survey, and the timing and frequency of the survey. The importance of ensuring the survey is easy to complete and understand is also emphasized.
The document then outlines the benefits of customer feedback surveys, including improved customer loyalty, increased revenue, and enhanced product development. The PDF also discusses how customer feedback surveys can help identify areas of improvement and provide insights into customer preferences and behaviors.
To ensure maximum value from customer feedback surveys, the document recommends that businesses take a strategic approach to survey design and analysis, and use the insights gained from the surveys to inform their decision-making processes. The document concludes by emphasizing the importance of continuous improvement in survey design and implementation to ensure ongoing success.
Here's my collection of Marketing & Product Management best practices, appropriate to B2B tech companies. Would love to hear from comments and suggested improvements and additions!
5 controversial secrets of customer successVenk Chandran
A brief overview of 5 controversial 'secrets' in Customer Success for Startups and the Enterprise. Distilled from some of my experiences delivering Customer Success in startups, Salesforce, and our latest acquisitions.
1. Four Product Management mindsets Deploy and balance the Explorer, Analyst, Challenger and Evangelist mindset throughout the product life cycle to avoid common pitfalls and deliver a superior solution.
2. Create context to motivate a high-performing team Practical tips and real-world examples to drive innovation, shared understanding, mitigate risks, and create energy and focus.
3.Understand your profile Evaluate your "go-to" strengths versus where you need to consciously practice, and how to recognize and balance stakeholders’ own.
4. Tools to help you Navigate challenging stakeholder relationships. Emerge with a stronger reputation as a leader when faced with conflicting business priorities, changes in direction, misaligned incentives, resource constraints, unexpected disruptions, and aggressive deadlines.
5. And many more strategies Techniques to say “no” given common stakeholder archetypes, how to diplomatically, authentically yet firmly approach keeping your priorities on track.
Harvesting the value from Advanced AnalyticsJaap Vink
In general, Analytics help you leverage investments that you have done already in your IT investments, on ERP, on CRM systems, on sales
force automation systems, and on all
the data collection that you put in
place.
Unfortunately, reality isn’t that
straightforward. It’s still a struggle
for most companies to drive valuable
insight into the data they have.
The document discusses measuring and managing customer satisfaction. It provides an overview of key metrics to measure like complaints, word of mouth, loyalty, and repurchase rates. It explains that totally satisfied customers contribute significantly more revenue. The document also discusses using customer satisfaction surveys to measure individual touchpoints and a company overall. It provides examples of survey questions and using metrics like Net Promoter Score. The conclusion emphasizes implementing a formal customer satisfaction process to improve business performance.
How To Market a Professional Services FirmPaul Banks
This document provides an overview of marketing strategies for professional services firms. It discusses six core marketing strategies: marketing to new clients, marketing to existing clients, superpleasing, nurturing, courting, and branding. For each strategy, it provides examples of specific tactics and ideas for implementation. It emphasizes focusing efforts on the most effective strategies and consistently implementing tactics over time. The document concludes by recommending professionals commit to their top three strategies from the six discussed to focus their marketing efforts.
Similar to Getting at the goals of the customer (20)
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Aggregage
This webinar will explore cutting-edge, less familiar but powerful experimentation methodologies which address well-known limitations of standard A/B Testing. Designed for data and product leaders, this session aims to inspire the embrace of innovative approaches and provide insights into the frontiers of experimentation!
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
End-to-end pipeline agility - Berlin Buzzwords 2024Lars Albertsson
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
4. • Measuring contractor/ internal KPI’s
• Benchmarking against peers
• Benchmarking against rest of the business
• Monitoring end to end service delivery
• Highlighting failures in process
• Improving value for money
• Identifying cost savings
• Understanding what we do well/
not so well
• Understanding what’s important
to tenants
What’s the point?: Objectives-led design
Objectives:
We need to ask ‘what is the point’ to get everybody thinking from the
start about where they want to get to
6. Measure what’s important: Don’t try to make
important what you
measure
• Don’t just ask whether you delivered your service standards
• Shine a light on your known areas of weakness
• Ask about things outside your control
• Check the open comments for gaps
Benchmarking has it’s place….. But remember it’s not
important to your customers
7. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Use open
questions
wisely
Measure
what’s
important...
3. Use open questions wisely
8. Use open questions wisely
Better communication
when appointments
are booked as to who
and when they are
coming to do what
repairs if more than
one job is booked
Communicate -
give a choice on
appointment
times/days
The fan is working
but I have been
told not to have my
window open when
it is in use which is
not what I
expected
Because it
was
terrible
Why were you
dissatisfied?
What would you
most like to see
improve?
9. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Use open
questions
wisely
Focus on
the fence
sitters
Measure
what’s
important...
4. Use open questions wisely
10. The ‘neither’: “Embracing ambivalence”
Think more about the middle ground neither responders - they don’t hate
you, they just don’t feel that strongly about your service on way or another.
It is a shorter distance to travel from that attitude up into satisfied.
11. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Results are
just the start
Use open
questions
wisely
Focus on the
fence sitters
Measure
what’s
important...
5. Results are just the start.
12. Results are just the start: Difference between
results and insight
“80% of customers were satisfied
with appointments”
“20% dissatisfied are all young
working families”
“95% of the time the operative
showed their ID….”
“the other 5% is always Nigel”
Don’t just break data down by all the usual suspects (single
demographics, age, property type) think about what might really have an
affect on satisfaction or expectations of service delivery…
13. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Results are
just the start
Everyone
loves a
good story
Use open
questions
wisely
Focus on the
fence sitters
Measure
what’s
important...
6. Everyone loves a good story.
‘people don’t get data, but they love a good story’
14. A lunchtime story….
This confirms the desk research we did on the main causes of responsive repairs complaints
before we started the survey, which also highlighted appointments not being kept causing issues.
23% of customers said they were dissatisfied with the system for allocating repairs appointments.
This would benefit all customers, but is particularly important for customers who work.
Our sub group analysis tells us that dissatisfaction is highest for households who work part
time, but we’ve looked at our in-house data and know that this group have no more
appointments missed than any other.
We’ve looked at the suggestions for improvement in this area, and customers would like 2 hr
appointment slots, or a text message when the operative is on their way, so they don’t have to
take a whole day off work, or wait in all day for the operative to arrive.
This is a high level of dissatisfaction compared to the other measures in the survey so we have
identified appointments as a key area for improvement.
15. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Results are
just the start
Everyone
loves a good
story
Use open
questions
wisely
Focus on the
fence sitters
Measure
what’s
important...
‘people don’t get data, but they love a good story’
16. Objectives
led
Engage the
Business
Theme and Question
Design
Analyse
Data
Reporting
Insight
What’s the
point?
Review, Tweak, Change, Improve….
Results are
just the start
Everyone
loves a good
story
Use open
questions
wisely
Focus on the
fence sitters
Measure
what’s
important...
7. Review, Tweak, Change, Improve…
17. Want a free conference debrief?
Contact us HERE
Editor's Notes
The seven tips I want to give you today are spread across the whole design and reporting process, starting with the first day we meet a new client.
So starting at the beginning, at Voluntas we talk a lot about objectives led design when we set up a new project, and this runs right the way through how we design the survey, the samples we choose and how we report the data back to you.
When we turn up for first survey set up meeting or first talk on the phone or by email, first thing we ask is ‘in 6/12/18 month’s time what do you expect this research to give you?’ ‘what do you expect to get out of it’? which is a bit of an odd question because generally at this point you’ve just commissioned us to deliver a survey and we arrive and say ‘what’s this for exactly?’ ‘Do you really need it?’
What’s the point of doing customer satisfaction research?
What we’re really trying to do by starting the meeting in that way is step back from our clients coming to us with a list of questions and saying ‘we need to ask this’ and get everybody thinking from the start about where you’re looking to get to, what do you want from the survey and also where your service is heading, starting at the end if you like and working backwards.
Why do we do that/ how does that help? – we find if we verbalise the aims of the research and how that links to the direction the service is travelling in, the research has a much better chance of delivering something actionable at the end, because we set a fixed structure for the whole of the research project, which you can refer back to; this question links to this objective, this headline result tells us how close we are to achieving this, these suggestions for improvement should help us get here.
Designing research by objective also tends to make it more fluid – if you know what you’re looking to achieve from the research it makes it obvious much quicker when the current questions, design or reporting isn’t working and they get changed and improved quickly rather than ending up with a survey which hasn’t changed or delivered anything in 12 months.
Satisfaction research can have very different objectives in different organisations, (click list) which also means that starting your survey design this way makes your research more personalised to your business and less abstract – so you’ve already starting to knock down that data silo that can sometimes build up around research in Housing Associations which are a predominantly operational business.
So, we discussed the objectives and set a structure for what the research is trying to achieve. Now we need to start designing a survey which unpacks how to get to those objectives in sufficient detail for you to be able to take action to deliver them.
Starting on the quantitative side first….
We need to measure what’s important.
This might sound like the most obvious piece of advice you’ve ever heard in your life but it’s almost always where customer research fails – straight forward asking the wrong questions – collecting loads of data and absolutely no insight, and no amount of clever analysis is going to salvage that situation once it’s done.
As an example - We do a lot of key driver analysis at the moment looking at which other questions in a survey have the greatest pull on overall satisfaction. Improving ‘overall satisfaction’ is an incredibly difficult objective to achieve because ‘overall satisfaction’ is abstract, it’s basically a feeling, you’re trying to shift how your customers feel about you and that could be impacted by all sort of things both inside and some outside of your control – so you have to work out what concrete things are causing that feeling (and even that can be fluid day to day), before you can work on doing those things more or less or better or differently, to have a knock on effect on customer’s overall satisfaction.
KDA helps because it starts to do that, it highlights the other questions in the survey which are having the greatest pull on whether a residents is ‘satisfied with you overall or not’, BUT it’s an analysis tool not a crystal ball it can only tell you about the questions you had in the questionnaire in the first place – if the real issue was never asked about, if all your residents are dissatisfied with waiting time but you only ask about reporting and then service delivery, we’re never going to be able to feed that bit of insight back to you – the KDA will tell you which is more important ‘reporting’ or ‘service delivery’ but it wont tell you the most important thing because it was never there to find – applying advanced analysis to poor data is a bit like employing a crowd of blokes with spades to try to help you find enlightenment.
So how do you make sure you’re asking about what’s important? – the first stage is to recognise that what's important to customers does fit in a neat little box, sometimes just delivering your service standards to the letter wont improve satisfaction, so you need to leave space for customers to give you that feedback – in ‘getting at the goals of the customer’ you’re not the customer – we’re trying to get at what your customers want, the households you house and support.
Measuring what’s important is going to involve asking difficult questions and putting your service out their for criticism – asking about things you already know you do badly, at least then you can measure and publicise the improvement in that sore spot over time.
Ask about things outside your control – this comes up a lot in repairs and typically around the relationship between contact centres taking reports and repairs team delivering the service.
I’ve been told by a lot of repairs managers that one of their main issues is the reporting of repairs – inaccuracies in recording or contact centre staff increasing customers expectations, potentially beyond service standards and that’s impacting our satisfaction and it’s outside of our control, but it comes out in the open comments because we don’t measure it explicitly.
I don’t want to start a discussion about whether that’s true or not, but my argument always is it might be outside of your control, but include it in your survey anyway, from the tenants point of view that’s all one service and from your point of view, that way you can measure the number of times that’s the case, the impact that’s having on your satisfaction score and potentially how that needs to be improved, then you’ll either be proven right or wrong in your assumption that this is one of your major issues, and if you’re right when you’re feeding it back you’ll have facts not anecdotes.
With the best will in the world, we’re all still going to miss something when we design our surveys, so after 3 or 6 months of surveying you need to check the open comments - because that’s where those missed bits will appear and if it’s mentioned enough you might need to add it as a long term measure to start addressing it.
Finally, think carefully about benchmarking - some organisations swear by benchmarking others don’t. Don’t worry this is not my well-wore rant about STAR questions (anyone who wants to hear that can find me at the BBQ later) but what I will say is some high level benchmarking fits squarely in the ‘trying to make important what you measure’ camp rather than actually measuring what's important in terms of improving services because to make them generic they are so far abstracted from the service they are measuring – if you run a key driver analysis on the 7 core star questions 9 times out of 10 ‘repairs and maintenance’ and ‘listening to your views and acting upon them’ come out as the key areas to work on which gives you some headlines but falls a very long way short of helping you know exactly what to do to improve.
So, we nailed our quantitative questions, what about the qualitative?
Open questions are really important in terms of understanding how to improve service – quantitative scores tell you where the service is failing or could be improved, qualitative tells you how to improve it.
BUT they have to be used sparingly in a survey – there nothing more fatiguing in a survey than being asked how do you rate x, and why do you say that? How do you rate y, and why do you say that, and z, how do you rate z….. If you’re anything like me and you’re confronted with that in an online survey you either give up or start going back to change your answers to try and avoid the follow up. Position them where they will have most benefit, after questions which need some elaboration to help you action the feedback and make them varied and specific – follow up where customers said they weren't happy with the waiting time with ‘how long do you feel would have been OK to wait for this repair?’ isn’t of ‘why do you say this?’
Open questions should be about finding ways to improve rather than going over previous issues, so word them that way – don’t ask why were you dissatisfied? Ask what one thing can we do to improve? You’ll be surprised the difference it makes in terms of the directness of the responses you get, how actionable they are and the tone of the interview itself, switches from evaluate to service recovery.
Analysing data…..
I’ve lost count of the number of times I’ve had the ‘neither’ debate this year!
And I know there are people in the room with a range of views on this subject and I don’t want to upset anyone, so putting aside the methodological and academic arguments for today I want to make a case for why I think the ‘neither’ should stay in relation to how it helps identify improvements in service and satisfaction.
I think a lot of this centres on how you think about customers who say they are neither satisfied or dissatisfied so my aim is to maybe make you think about them differently – as ‘potentially satisfied’ rather than ‘without a view’.
In our web portal, in Housemark and pretty much everywhere else satisfaction scales are broken down into satisfied, dissatisfied and neither in the middle – and when looking to ‘improve satisfaction’ our instinct seems to go directly to this group at the end here the ‘dissatisfied’s and try and work out why that is, we regularly have tailored follows ups which ask why are you dissatisfied etc.
However, the sad truth is, some of this percentage are going to be customers who will simply never like you whatever you do, the hard core detractors who will never in a million years be shifted in their negative opinion to the extent that they will give you a satisfied score.
Think about the last time you said you were ‘dissatisfied’ with a service you received, the strength of negative feeling you have to have about a service to give that sort of score – it’s a long way from there back up to satisfied – now I’m not saying we should therefore ignore that group and certainly from a service recovery point of view you wouldn’t want to, what I’m trying to say is if you’re looking for quick wins to improve satisfaction they’re probably not the place to look.
Now think again about our middle ground neither responders - they don’t hate you, they just don’t feel strongly about your service one way or another, or potentially they had a mixed experience, some of it was good but something grated on them enough to not give you a satisfied score. It’s a lot shorter distance to travel from that attitude up into satisfied, so maybe focusing on the reasons for that ambivalence and addressing that will identify a few quick wins and easy changes to make while you’re addressing that hard core of dissatisfaction in the longer term?
I’ve had clients say to me over the past year, the neither's are unfairly pulling our satisfaction score down or they’re basically counted as dissatisfied because the benchmarking scores are positive responses only which is actually a helpful way to think about them – a group that pulls your satisfaction score down but who might be more easily swayed than the real dissatisfieds.
So, we’ve designed the survey, we have data and the last stage is to report that data back to the business and start improving.
I’ve linked reporting and engaging the business together here because as a bonus tip I think it’s important to link think about these two as one and the same thing – what you report should always be about engaging the business and moving towards taking action it shouldn’t be report and then work out what to feedback.
Reporting Insight. The important word here is ‘insight’.
Insight to me has to be something more than a headline – 80% of customers satisfied with appointments and 95% of operatives showing their ID is not insight it’s a finding.
Identifying who was dissatisfied and potentially why or where the service failure is is insight – and the difference is that insight is actionable, it moves you closer to doing something to improve, it directs your efforts.
To do that you need to break the data down further, you need to look at wider data, you need to look at your sub-groups. And think outside the box a little bit – don’t just break your data down by all the usual suspects (single demographics, age, property type) think about what might really have an effect on satisfaction or expectations of service delivery – not just HB or not, level of contribution towards your rent has a massive impact – partial HB are always the least satisfied with vfm because they struggle to find that money.
We’re asked a lot by performance teams about feeding results back to service leads and operational staff in a way that makes them take action – My last tip is really to share with you some great advice I was given by boss in my first ever research job.
I’d just written what I thought was a brilliant report – for a start it was about 60 pages long and who doesn’t love 60 pages of stats right? It had graphs for every point, it have every question broken down by at least three different sub groups, it was ridiculously detailed, but what it didn’t have was a ‘so what?’
So what should we do? So what should we change? So what will make our service better?
The advice my boss gave me was ‘people don’t get data, but they love a good story’ take it away, and tell a continuous story with the data, write it like a murder mystery, and at the end tell them who done it.
So here’s a good example of a data story, which uses some of the tips I’ve highlighted today and draws in wider business intelligence to make it more relevant.
23% of customers said they were dissatisfied with the system for allocating repairs appointments.
This is a high level of dissatisfaction compared to the other measures in the survey so we have identified appointments as a key area for improvement.
This confirms the desk research we did on the main causes of responsive repairs complaints before we started the survey, which also highlighted appointments not being kept causing issues.
Our sub group analysis tells us that dissatisfaction is highest for households who work part time, but we’ve looked at our in-house data and know that this group have no more appointments missed than any other.
We’ve looked at the suggestions for improvement in this area, and customers would like 2 hr appointment slots, or a text message when the operative is on their way, so they don’t have to take a whole day off work, or wait in all day for the operative to arrive.
This would benefit all customers, but is particularly important for customers who work.
I did say at the start 7 hints and tips and the more observant amongst you will of noticed that’s just 6 – so the 7th and final tip is about remembering to review – your business and the way you delivery your services evolves so so should your research, every 6 months look at what your research has given you in that time and whether it’s really helping you move towards your objective and if it’s not it’s time to look at these points again and think about what needs to change.