Ceci N'est Pas Une KPI (Interaction 10) with notes!

I have been interested in how to measuring success in the context of the user experience practice for a long time. Unfortunately I never had had the opportunity to really dig into this topic until recently. Last September my boss came to me and asked me to figure out what where the right KPIs by which we could measure user experience success and manage the efforts of our organization in that regard. Note: That’s where I work! 2

I have only been investigating this in more depth for the past 4 months and in this time I have been able to formulate better questions than when I started, but it is fairly early to make any conclusions that could provide prescriptive approaches to how user experience practitioners should be measuring success. Bonus: http://www.youtube.com/watch?v=j_gpe2‐oaB0 3

In exploring the topic and attempting to deconstruct the question of what performance indicators we should be utilizing to measure user experience, I interviewed designers across my organization and reached out to peers in the larger community. Three main themes emerged, including a desire by designers themselves to be able to tell if their hard work was paying off. The questioned of how to assess that exposed the need to measure it in the context of user experience and once the problem was framed that way, the fear of numbers became an immediate concern as designers generally see themselves of consumers of information not analysts of numbers. consumers of information not analysts of numbers 4

So let’s talk about numbers! Designers are not generally attracted to numbers. This is a clear barrier in empowering designers to make informed and confident decisions with supporting data. 5

What meaning does this hold for you? 6

Some numbers may hold meaning to you and some may not. When they do, they really are expressing a prior understanding, a relationship to an existing context. More importantly, they answer a question (regardless of the utility of the answer, that’s a whole different topic!). 11

In my research I came across a book by Michael Lewis which describes how transformative numbers have been to the world of baseball (in a nutshell, allowing teams to acquire talent relying less on the gut feeling of scouts and the concrete predictive qualities of players’ scores). The key insight for me, however, was about the lack of meaning in numbers ‐‐ which opposes the richness of language ‐‐ Numbers are only a representation, and do not possess inherent value. This gave me some insight as to why designers are often disinterested in numbers, given our interest in people and behaviors. 12

That’s what made me think Magritte’s 1928 The Treachery of Images. It reads “this is not a pipe” under the image of a pipe. The point is that in fact, this is not a pipe, but rather a representation of that object. If you have seen a pipe before it is hard to disassociate the actual object and the representation because it holds so much meaning. (Likewise, if you had never seen a pipe, it would be difficult to understand the point because you might just believe that this was the wrong representation rather than not the right representation and not the actual object). 13

The same is true for numbers. When we see a number we forget that it is just a representation of something else and we can become really attached to the representation and detached from the real meaning of what it represents. Numbers are an abstraction, but when we give them context, they have the strength, significance and nuance of language. Designers need to find meaning in numbers by seeing them as a design challenge and figuring out the context and questions they relate to. 14

Now that you have opened your heart and mind to meaningful numbers, let’s discuss a specific kind: metrics. 15

This old management adage is true in the sense that it is not possible to understand your circumstances if you have no way to assess them. 16

But the relevance of this for us as designers is not in managing in the sense of monitoring, but managing as a way to understand so we can improve the quality of the user’s experience with what we create. 17

As designers we get insights from many sources by applying a multitude of different approaches (these are just some examples). Regardless of where they come from, our real interest is in how they help inform design. This is why design research is irreplaceable and absolutely needed – I don’t expect anyone would contend this point – but metrics provide an additional layer of information, one that expresses what events and trends we should pay attention to and investigate further to understand motivations, expectations and needs. 18

A Key Performance Indicator is nothing if a select set of metrics that express where we are, giving us a sense of place relative to our intentions. When trying to figure out what KPIs we should pay attention to we should think of what are the critical few metrics we will want to look to know if things are going well or badly. If you want to fill up your car with $20 worth of gas, you start pumping and look at pump indicator that shows how many gallons are exiting the pump and how much cost is adding up. That tells you what you need to know and answer the “when do I need to stop filling” question. “Filling the tank performance” is your KPI for this particular objective. 19

But how do you figure out which metrics make sense and which do not? Experimentation. Here’s an example: I work for a cable company that has conducted satisfaction surveys for many years. It has found a correlation between people’s satisfaction scores and their likelihood to continue giving us money for services. Then came along the Net Promoter Score, which promised to be a better predictor of this behavior. So we started tracking it and after a few months observed that there was no relation between the Net Promoter Score and our customers’ behaviors. The good old Satisfaction Score was much more reliable for our context and objectives, even if it s all the rage in other industries and even reliable for our context and objectives even if it’s all the rage in other industries and even with our competitors. The lesson is that no two products have the same objectives and users and context, so the key performance indicators need to reflect its unique context. Note: Net promoter score is calculated by asking users about their likelihood to recommend service X to a friend, rated 1‐10, then 8, 9, 10 ratings are aggregated to make up the score. Satisfaction asks how satisfied one is with service X, rated 1‐5, then 4 up the score Satisfaction asks how satisfied one is with service X rated 1‐5 then 4 (satisfied) and 5 (very satisfied) are added to make up the score. 20

As designers we have an edge because we know how to experiment with different possibilities. We just need to embrace KPIs as another design challenge that we need to explore. More importantly, he who knows the KPIs, directs the conversation. John Kolko in his Interaction 10 session was talking about bridging the gap between data and information, when he asked two excellent questions: 1. When do designers have the time to look into this? 2. How will designers acquire the language needed to discuss this? This is precisely what we need to tackle; we need to acquire the language to understand what these metrics mean so we can play with them and test to see what can help measure the outcomes of our efforts in creating great experiences for users. As David Gray put it during his Knowledge Games presentation, “If you don’t know the meaning of the pieced you can’t play the game”. Note: See KPIlibrary.com, SmartKPIs.com and The Big Book of Performance Indicators for thousands of KPIs to help you get started and familiarized with them and see what has been employed in services like yours so you can select a few to start with. 21

If we don’t need to define user experience in explicit ways to be able to design for it, we do not need to define it in order to measure it. The real relevance is the outcome of the experience, what the user is actually experiencing. In other words, how well the goals were fulfilled. Measure the outcomes. Apply some deductive thinking: 1. We design with a goal in mind, and 2. We can measure against goals using metrics. 3. Logically, we can we measure success of users’ experiences. (Provided you don’t design with the spaghetti method, where you throw things up the wall to see if they stick and do not have clear goals). The most critical piece to manage the success of users’ experiences is clarity of intent. Let’s look at how we’ve been trying to do that in our community 23

Usability is an established practice and has very good established metrics of success. However, user experience is far more than usability. If you apply usability metrics to represent success in user experience as a whole, we all lose. Use usability metrics to measure usability. We must move beyond usability. 24

Peter Morville created this model to help clients understand why they must move beyond usability. The problem is that for any given context (your service, with your users and your objectives), you will find that some of these attributes are more relevant than others. And you might even add other aspects that are not found here (as many have added in response to Peter’s first version and have continued to extend it). Measuring improvement in these areas, does not report on the progress in achieving objectives. You may correlate these actions to outcomes, but you still have to articulate the KPIs. Intangibles, are not concrete and quantifiable and therefore are not KPIs. Moving beyond usability is the right goal, but where is beyond? That horizon is determined by your unique balance of context, content and users. 25

Adaptive Path introduced the Linking Elephant model (found in their excellent report Leveraging Business Value: How ROI Changes User Experience) in order to use explicit business value metrics to brainstorm design ideas. The two challenges with this model are, it assumes business know what their problems are (which is sometimes true) but have articulated them (which is often not true or done poorly). As we discussed clarity of intent is key so this is an area of risk. Second, it assumes that there is known value for all behaviors we identify. I am seeing many people struggle with that. Specially when goals are “fuzzy”. fuzzy But this is a brilliant model because it demonstrates that businesses can achieve that value by encouraging desired behaviors. We like behaviors, that is what we design for! So even though this is not a KPI on its own right, it introduces a really helpful language and framework for us to articulate what objectives we are interested in and what metrics we may apply to assess them. may apply to assess them It’s the most helpful linkage between design interests (behaviors) and designers fears (numbers) that I have found. 26

The three contributions we have seen are helpful and bring us closer to embracing performance indicators, but they are not quite there yet. KPIs as an additional layer of information to help us make decision need to be done properly and not used as a buzzword. Metrics, specifically KPIs will tell us “whats” and provide a starting place for us to ask “why” questions, which can only be answered by the qualitative insight we know are irreplaceable. The fuzzier the goal, harder to articulate how well you re getting there. irreplaceable The fuzzier the goal harder to articulate how well you’re getting there But even a stake in the ground is better than not articulating and not being able to measure success. KPIs can also help us because of how contextual they are. They force us to stop comparing ourselves to others (the plague of corporate environments). With concrete metrics we avoid distancing ourselves from users. Lastly, KPIs are about behaviors. They are not far removed from the action, but they need to become connected. Find the meaning, acquire the language and relate them to your design decisions. 27

These were not included in the presentation during Interactions 10 but I pulled up after we were done during Q&A because the audience wanted to know how to get started and if they should start by selecting technology and other questions that were about data sourcing. My big message is to not get tactical before you clarify objectives (even if they are fuzzy definitions) and identify how you want to measure them. 29

This is just a list of questions I would like to answer next. 31

Somebody wanted to know about how to measure project success (as opposed to the whole experience) and I reinforced that you should have your KPIs relating to the overall experience regardless of projects, but then apply any relevant metrics you might find useful to assess how well the objectives of a project were met. I additionally walked through the User Experience Health Check (find out more at uxhealthcheck.com). It’s a method to quantify aspects of the user experience through quantitative measures. It is not a KPI on its own, but can be used to show directionality of UX effort and outcomes (comparing metrics). It basically is a made up a metric and like any other, it tells you what, not why. But you can color that what with all your qualitative knowledge, reinforcing how qualitative understanding is irreplaceable. 32

Thank you! Please contact me on liv@livlab.com with questions, follow me on twitter @livlab and visit by blog on http://livlab.com/thinkia 33

http://www.userglue.com	61
http://livlab.com	53
http://userexperience.pl	15
http://www.lmodules.com	14
http://feeds.feedburner.com	9
http://speakerrate.com	1
http://www.uxable.com	1

Ceci N'est Pas Une KPI (Interaction 10) with notes!

by Livia Labate on Feb 10, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Embed Views

Statistics

Ceci N’est Pas Une KPI (Interaction 10) with notes! — Document Transcript

Get analytics right now with SlideShare PRO

An effective way to find out who is viewing your content

Remove ads from all your pages with SlideShare PRO

Your viewers will engage with your content, not ads