2. Stephanie Slobodian
Senior UX Designer at NPR
Over 15 years with companies that include;
IBM Usability Engineering National
Practice, Critical Mass, and local PR and
consulting firms, designing user centered
experiences for Visa, American
Express, Gucci, Hallmark, Mercedes
Benz, USAA and Marriott
#facebookusertesting
5. ① Do we have enough participants for tests
to be meaningful
② Do we have the right users for the
product we’re testing.
③ Are we getting fresh users and opinions.
④ Is the effort in time worth the results
Are we doing this right?
22. ① Social … with people who have good taste.
② Music services focus on finding new music,
but 70% of the time they want to listen to
their favorites
③ Advertising is the price for free service
④ Still split on owning music vs. streaming
Key Findings.
24. ③ Recruiting is free and incentive costs are lower
④ More participants equals more data, giving greater
confidence in product decisions
⑤ Online method equalizes all users
Why it works.
Hello Colleagues, My Name is Stephanie Slobodian, I work at NPR on the Digital Media team as a Senior User Experience Designer. Part of my work includes conducting User Testing. I’ve been experimenting with using our existing social media communities to recruit and now conduct user testing and research. I’m going to walk you through how we conducted an on-line user testing event on facebook. I’ll also share a free facebook apps that you can use to conduct surveys and explain why I think social media can revolutionize user research and testing.
We all know how important user testing is. I’ve been working in this industry for about 15 years now and although there are a lot of great on-line tools that have helped us conduct surveys and card sorts but in my there has not been a lot of innovation in user testing. At NPR We recruit people from our listener panel, listeners sign up for the panel on our website, and although these people are from all over the country,the vast majority of user testing for all our digital products happens right here in D.C. We do test often, NPR is an agile shop so our scrum teams aim to test every 10 day cycle. We have people physically come in for an hour and our teams test their various projects. We pay these users 50$ for their time.
But the fact is we know very little about the people that come in for user testing. We know some demographics before they arrive but usually not much. If fact we might participants who won’t be users of the product that we are testing… At NPR we work hard to innovate whether it’s the waywe workor the products and services that we design. So we also want to think about user research andtesting in that same innovative spirit and conduct our testing with exactly the users that will use or we hope will use thee products. Ourcurrent method does a decent job at usability testing, but its not really giving us insights for future products or getting at user needs for specific products.
So some of the issues that we face include:The first issue is recruitment of the right users –You have to really love us because its really hard for users to figure out how to sign up for the listener panel! Also, over time the panel has gotten smaller, and we often get the same local people or we can’t get enough participants at allThee users may or may not be users of the products that we are testing – so basing product decisions on these tests can be dangerousConducting testing is time consuming, each test is one on one for an hour and then we spend time documenting or reviewing video.
So I hope we have some Planet Money fans in the crowd? ! I was lucky enough to work on designing the recent Planet Money iphone app. We knew that we wanted to do user testing but we also knew that Planet Money fans are a unique snowflake and in order for this test to be meaningful we wanted Planet Money Fans who had an iphone and listened to the podcasts on a regular basis. We just weren’t going to be able to get this specific with our existing NPR listener panel.
So we had Caitlin Kenney of Planet Money post a message on their planet money facebookand twitter pages. We wanted about 20 fans with iphones who were able to come to NPR for a 2 hour focus group. Within 4 hours we had well over 40 volunteers. That was when we thought we were on to something. All though we were still recruiting locally we had a group of people who were fans of the content and brand with iPhones willing to give us their insights and opinions for 2 hours.
So we knew we wanted to innovate in the way we conduct user testing, getting the community talking with each other not just one on one with an NPR staff. We knew that this dynamic would be different and were excited for the different insights we might capture. So we explored Facebook and the Event Pages that they have and thought this would be a great medium for our test. It would allow us to recruit from all over (the world). Test the right people for the product and it would allow us to gather a lot of quantifiable data for a lot of people in a short amount of time.
So from that Planet Money experience and the desire to innovate on user testing, we dreamed of a new social media universe.We now have a way to recruit and engage in conversation with real users and fans of individual NPR offerings. Many programs and blogs have their own social media pages as do various verticals like News and Music. If we could recruit from these social networks when we needto conduct testing for design on products we could also create a larger on-line … this would allow us to start gathering more forward leaning insights and help shape our product roadmap. We know the greater community will require lots of planning and seeding, like any new group. This community would give us the ability to get feedback almost instantly for work that we were currently doing or give is the ability to research ideas or spot gaps in the market place that we could explore. But. Baby steps. we wanted to start small.
So considering the this new social media universe.. We wanted to start with a smaller product test… NPR just launched a new streaming Alt Latino channel and considering the music industry is being revolutionized by services like Rdio and Spotify we thought this would be a good candidate for our first official facebook test.we also knew that the NPR Music facebook community what pretty active. We met with the Music product owner and she talked to us about some things that they were trying to understand but traditional research was not really giving them insight about. From this meeting and other research data we designed the test.
I used this FREE survey app within facebook,It generates a url that can post or email out… It does require that the user have a facebook account. You can pay to have an ad free version and get a few more analysis tools. Its very simple and you add questions one at a time, not unlike other survey tools. We asked questions like, Your age, Where you live, How much streaming music do you listen to and If you paid for a streaming music service. Also you had to be available between 11 and 1pm on the day we would conduct the online test.
Then we posted this message on the NPR Music Facebook Page and overnight we had well over 60 people respond to the survey. Reviewing the candidates, they were from all over the country (one from out of the country), a good cross section of male and female and various ages. Almost all streamed music somewhat through out the day and about half were subscribers of some sort of streaming music service free or paid.
We were essentially running this as a testing experiment, that would augmentthe in person testing we continue to do but this meant limited budget. We already recruited these people for free but we couldn’t ask them to give 2 hours of their time with out any incentive. Our idea was to give participants NPR swag, a t-shirt, a 2013 calendar and some pins and postcards. You’ll see further in the presentation how this wentbut when participants started to receive their swag in the mail they were taking pictures and posting back the facebook
We reviewed all of the 60+ surveys and picked 25 candidates, we created a facebook event page, and invited those 25 participants to the event. we quickly learned that we had to invite about 40 people to get 22 to commit to attending.
So now we have a user test designed and we know a lot about the community of people that we’ll be testing with. We can qualify and weigh comments.
We had 2 hours with our streaming music community and we wanted to keep the test page active. Our hope is that users would engage in conversation with each other as this would be a first for us- conducting testing that was not just a one way conversation between us an one user. We ask them to do three things during the test. 1.First of all we asked them five individual questions, one at a time and paced for time to answer and read comments. 2. Then we asked them to draw their ideal streaming music interface and post to the page. 3. Then we asked them to fill out another more in-depth survey about streaming music. Inaddition to those three things we wanted to spend some time talking about the test itself, what they liked or didn’t like and did they have any suggestions to fine tune it.
We were really amazed at what unfolded. Other colleagues logged into the room to watch the participation. Not only were participants answering the questions but they were reading others comments, liking the comments and engaging in real discussion around the questions. The first question we asked about their listening preferences around curated or genre listening, the second was to understand more about their listening habits.
Question 3 was a follow up regarding saving music for off-line listening and the 4th question was really interesting. We used a blog post from Bob Boilen as fodder – regarding how important it was for them to own their music. It sparked a healthy debate as almost all of the participants had read the blog pot.
The second assignment was to draw your idea streaming music interface then take a picture of it and post it on the event page. Lesson learned I wished I had used a drawing app like https://www.facebook.com/graffitiwall Graffiti where users could have engaged right in facebook. Some I think were also shy about posting but for the ones that did it also sparked some great discussion. One girl drew a pretty comprehensive wireframe!
At the end of the test, we thanked the participants and talked about the test itself, which they absolutely loved. For hours after the test ended participants remained in the room, reading and liking other users comments and posting more responses. They loved that NPR would conduct a test like this that lowered the barrier for their participant. Many of these users hang out in facebook already and they appreciated the chance to talk to other fans and be apart of product development. Some users suggested that in the future we could “post prototypes” for feedback – something that we absolutely are thinking about doing. So after 2 hours we had over 150 unique comments for the questions. We had several drawings andevery single user filled out the survey. All participants were fully engaged for 2 hours.
An unexpected benefit of this type of testing was the good will it caused in the community. Not only did the users have a good time and express how much they appreciated being able to participate in this type of dialog with NPR but they intern shared their experiences in their social networks and posted pictures of their NPR swag. They essentially became evangelists for NPR and NPR Music. We feel that long term, this will give us more exposure to listeners that currently aren’t our users or NPR fans.
Users wanted to control who they shared with and who they followed socially. They were only interested in friends with like taste or “taste makers” like NPR!Rdio and spotify do a great job at helping users discover new music but maybe there is an opportunity in advanced features around listening to your favorites?Users indicated they were ok with advertising and noted that they liked when ads had either something to do with the content they were consuming or was geographically localized. Because streaming music services are still spotty in their accountability of off-line listening users are still slip in owning vs streaming
In addition to the feedback and data that we collected during the test, we matched up the users with both the pre-qualifying and the survey they took during the test so when we look at the comments that users were making we can qualify them a little more as we knew quite a bit about these users now. From the survey tool itself there are some ways to look at the data in graphical ways. We ended up an excel file on each participants as well we could role up the data and look at it in terms of averages.
Our next steps at NPR include building the greater social universe - All the participants expressed desire to stay connected with the testing and participate in the future. We’ve got some big ideas at NPR about future products and we think our idea for the Facebook universe or test kitchen is going to be a great additional tool to help us build truly user centered NPR products.
