Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ted Boren
User Research Lead
The Church of Jesus Christ
of Latter-day Saints
THE BEST ONLINE
BENCHMARK YOU’VE NEVER
MEASUR...
Is our site successful?
GAPS IN
BENCHMARKING
What do visitors want to do?
(And can they do it?)
Where do
they go?
(How long
does it take?)
What do they
think of their
...
Remote
Usability Testing
Web
Analytics
Online
Surveys
True
Intent
What
you want
to know about
site visitors’
Experience
…
“True Intent” in a Nutshell
1. Intercept live visitors.
2. Ask them why they came to your site.
3. Ask them to go do that....
True Intent Studies are “Time-Aware”
“Remote research opens the door to conducting
research that also happens at the momen...
CASE STUDY: True Intent Study for LDS.org*
* The flagship website for the Church of Jesus Christ of Latter-day Saints, my ...
Give me the details on this
True Intent thing!
“WHEN, HOW, WHO?”
WHEN in the development cycle do I do this?
During planning…
… or after deployment.
BEWARE: Variation by hour, day, or special event
Thursday… Friday… Saturday…Sunday…Monday…
Personal study
Find something f...
Thursday… Friday… Saturday…Sunday…Monday…
Personal study
Find something for my family
Thursday… Friday… Saturday… Sunday… ...
TYPICAL STREAM: A Whole Week? Day? Month?
Thursday… Friday… Saturday…Sunday…Monday…
Search Intents Declining on the
Weeken...
So if in Doubt—Slow It Down and Spread It Out!
• If you invite 100% of your visitors, you may finish too
quickly, especial...
WHO do I invite? (And how?)
•Email from a pool?
•Link on the page?
•Pop-up on the page?
Sample Popup Invitation
Sample Instructions
Sample First Question
True Intent’s Golden Question:
Why did you come
to this website
today?
Please be as specific as you can.
Prepare a lesson.
80% of your responses may be
as exciting (and detailed) as
this one.
But the other 10%...
My grandmother died tonight
and one of my friends was trying
to comfort me. I wondered if
there was anything on the site
t...
its my homepage. helps
remind me who i am
everytime i get on the
computer.
~ LDS.org visitor
So when asking the Golden Question…
Beware of Qualifiers!
And Seize Today!
The One (and only) Task:
“Do What You Came To Do.”
Visitors make their way through your site…
?
… but sometimes they may be hard to track.
?
The tool continues tracking time and path…
… until they say they are done.
Primary Post-Test Questions
Goals
Satisfaction & Relevancy
Problems, Anything Else?
Secondary Post-Test Questions
•Key pivots for slicing and dicing
•Demographics / profile
•System Usability Scale (SUS) or ...
Q: How long do you run this thing?
A: “It Depends.”
• Site volume
• Percentage invited
• Survey length
• Audience engageme...
A FEW QUESTIONS?
300-400 responses later…
“ANALYZE THIS!”
Basic Quantitative Stats
Success Rate Average Satisfaction Ratings
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Abandoned
t...
Breakouts: Stated Goals, by Age Range
• Younger LDS.org
visitors more
likely looking for
personal study or
inspiration…
• ...
Benchmarks: LDS.org Search Success
Version 2 was better
overall:
• Especially for
preparing a lesson…
• But not so much fo...
ANALYZING QUALITATIVE DATA:
The Affinity Diagram
• A thematic, hierarchical view of qualitative data points.
• A way to se...
Sample Affinity Diagram Excerpt:
1. I visit LDS.org often for inspiration and information
1.1. I want to be inspired and s...
Preparing Data for the Affinity Diagram
•Export the data to a spreadsheet
•Create one row per comment*, including all data...
Meet Your New Best Friend…
•Post-It Notes for Printers
•Print a few merged notes onto regular
paper to check alignment
•Pr...
Organizing over
800 comments
Naming and grouping
169 categories
(11 at the top level)
Conduct an Affinity
Diagram Workshop
Finished diagram, with three levels of headings.
Record the Diagram
Headings Only
Record only the headings in a
document or spreadsheet
More Complete
Associate each data p...
A FEW QUESTIONS?
Analysis Steps LDS.org Example
1. Filter to those who a failed a
specific task. *
Looking for a location
2. Review all the...
Failure Cases for LDS.org
Almost all problem cases fell into one or more of the
following categories:
•Difficulty finding ...
Mitigating Challenges
It can be difficult to get enough context to conduct an
effective failure analysis.
This works best ...
What Next?
•Conduct interviews or tests on key areas.
•Adjust or create audience segments. Enrich personas.
•Prioritize to...
Archive It!
But don’t just forget
about it.
True Intent data has a
good shelf-life…
How Long Does This Take?
• For 300-400 intents:
• Creating the online study, collecting data, running quantitative stats:
...
“Thanks for leading this
exercise. I will admit I was a bit
skeptical going into these
meetings (who isn’t skeptical of
me...
“I mentioned this study in two
different high-level meetings
this week, and there is a great
deal of interest and enthusia...
“Thanks a ton for your work.
As I’ve said before, the results of
the study will help inform our
policies & site developmen...
No time for all of this?
Here are some ways to trim
time and budget.
SHORTCUTS
Shortcut #1: TRIM SCOPE
Skip 1 or 2 of the 3 main activities:
•quantitative benchmark
•affinity diagram
•failure analysis*...
Shortcut #2:
USE WORD CLOUDS
•One cloud per open-ended question
(or per set of closely related questions)
•Dump all commen...
“Please tell us why you came to
this website today.”
“What was the biggest problem you
encountered in your session today?”
“Anything else you'd like to tell us about
your experience today?”
Conduct a “True Intent” Study
1. Intercept live visitors.
2. Ask them why they came to your site.
3. Ask them to go do tha...
FINAL THOUGHTS
Evaluating success on an
actual, self-defined task
Large
numbers
of paths and
times on task
Feedback
& profile data
(quali...
We could call this:
A single-task
remote
unmoderated
interview-guided
usability test
with integrated analytics
and survey ...
But it’s really all about:
True Tasks
True Success
True
Paths
True Profiles
True Satisfaction
True
Intent
FINAL QUESTIONS?
References & Links
Official Website of the Church of Jesus
Christ of Latter-day Saints:
• http://lds.org
Nate Bolt and Ton...
True Intent: The Best Online Benchmark You've Never Measured
True Intent: The Best Online Benchmark You've Never Measured
Upcoming SlideShare
Loading in …5
×

4

Share

Download to read offline

True Intent: The Best Online Benchmark You've Never Measured

Download to read offline

UXPA 2013 Annual Conference - Wednesday July 10, 2013 by Ted Boren

Remote testing can tell you how easy your site is to use... for selected tasks. Surveys tell you how visitors feel... but lack performance data. Web analytics tell you where visitors go... but not whether they actually succeed.

"True intent" studies bridge these gaps and help your team learn what's really happening on your site, by asking real visitors why they came, tracking where they go, then allowing them to tell you if they succeeded. Work together to build affinity diagrams of intents and conduct a detailed failure analysis for even deeper insights that can shape your team's strategy for years.

Related Audiobooks

Free with a 30 day trial from Scribd

See all

True Intent: The Best Online Benchmark You've Never Measured

  1. 1. Ted Boren User Research Lead The Church of Jesus Christ of Latter-day Saints THE BEST ONLINE BENCHMARK YOU’VE NEVER MEASURED TRUE INTENT
  2. 2. Is our site successful? GAPS IN BENCHMARKING
  3. 3. What do visitors want to do? (And can they do it?) Where do they go? (How long does it take?) What do they think of their experience? (Who are they anyway?) What you want to know about site visitors’ Experience …
  4. 4. Remote Usability Testing Web Analytics Online Surveys True Intent What you want to know about site visitors’ Experience …
  5. 5. “True Intent” in a Nutshell 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions.
  6. 6. True Intent Studies are “Time-Aware” “Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest.” “Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.” ~ Nate Bolt and Tony Tulathimutte, “Remote Research,” page 10 (italics added).
  7. 7. CASE STUDY: True Intent Study for LDS.org* * The flagship website for the Church of Jesus Christ of Latter-day Saints, my employer
  8. 8. Give me the details on this True Intent thing! “WHEN, HOW, WHO?”
  9. 9. WHEN in the development cycle do I do this? During planning… … or after deployment.
  10. 10. BEWARE: Variation by hour, day, or special event Thursday… Friday… Saturday…Sunday…Monday… Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Prepare a lesson Prepare a talk Search Intents on LDS.org
  11. 11. Thursday… Friday… Saturday…Sunday…Monday… Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Prepare a lesson Prepare a talk BEWARE: Variation by hour, day, or special event Search Intents on LDS.org
  12. 12. TYPICAL STREAM: A Whole Week? Day? Month? Thursday… Friday… Saturday…Sunday…Monday… Search Intents Declining on the Weekend (spiking on Monday) Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Search Intents Spiking on the Weekend Prepare a lesson Prepare a talk
  13. 13. So if in Doubt—Slow It Down and Spread It Out! • If you invite 100% of your visitors, you may finish too quickly, especially with: • Heavy traffic • Enthusiastic audience So: • Meter the rate (preferred)* • Or turn your survey off and on
  14. 14. WHO do I invite? (And how?) •Email from a pool? •Link on the page? •Pop-up on the page?
  15. 15. Sample Popup Invitation
  16. 16. Sample Instructions
  17. 17. Sample First Question
  18. 18. True Intent’s Golden Question: Why did you come to this website today? Please be as specific as you can.
  19. 19. Prepare a lesson. 80% of your responses may be as exciting (and detailed) as this one. But the other 10%...
  20. 20. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. ~ LDS.org visitor My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
  21. 21. its my homepage. helps remind me who i am everytime i get on the computer. ~ LDS.org visitor
  22. 22. So when asking the Golden Question… Beware of Qualifiers! And Seize Today!
  23. 23. The One (and only) Task: “Do What You Came To Do.”
  24. 24. Visitors make their way through your site… ?
  25. 25. … but sometimes they may be hard to track. ?
  26. 26. The tool continues tracking time and path…
  27. 27. … until they say they are done.
  28. 28. Primary Post-Test Questions Goals Satisfaction & Relevancy Problems, Anything Else?
  29. 29. Secondary Post-Test Questions •Key pivots for slicing and dicing •Demographics / profile •System Usability Scale (SUS) or similar instruments… only if external comparisons are important and feasible •Be thinking ahead to future comparisons… how will you know you’ve improved? •What data will speak to your stakeholders? •But KEEP it SHORT!
  30. 30. Q: How long do you run this thing? A: “It Depends.” • Site volume • Percentage invited • Survey length • Audience engagement… and incentives* • I’ve finished in a day or two, or in 4-6 weeks. • Shoot for at least 300-400 responses: + More variety of audience and intents + More subgroups to slice and dice = More responses needed
  31. 31. A FEW QUESTIONS?
  32. 32. 300-400 responses later… “ANALYZE THIS!”
  33. 33. Basic Quantitative Stats Success Rate Average Satisfaction Ratings 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Abandoned task Completed task 1 2 3 4 5 6 7 I found what I wanted. It was easy to find what I wanted. Within the next few weeks, I'll probably study, share, or use the information I found… Strongly Agree Strongly Disagree
  34. 34. Breakouts: Stated Goals, by Age Range • Younger LDS.org visitors more likely looking for personal study or inspiration… • Older visitors come for these reasons too, but also more for utilitarian purposes. Personal Study Find something inspirational Find something for my calling Use the calendar, directory, maps, or leader tools 18 to 34 35 to 54 55 and older
  35. 35. Benchmarks: LDS.org Search Success Version 2 was better overall: • Especially for preparing a lesson… • But not so much for finding something inspirational… Prepare a lesson Find something inspirational Version 1 Version 2
  36. 36. ANALYZING QUALITATIVE DATA: The Affinity Diagram • A thematic, hierarchical view of qualitative data points. • A way to see what issues arise out of the data, instead of imposing a pre-determined structure.
  37. 37. Sample Affinity Diagram Excerpt: 1. I visit LDS.org often for inspiration and information 1.1. I want to be inspired and strengthened 1.1.1. I am looking for something to help me feel the Spirit 1.1.2. I want quick inspiration during a break in my day 1.1.3. I start on LDS.org to keep “spiritually minded” This is the data we start with... These are categories we generate.  Intent: “I try to read a quote or watch a Mormon Message every time I get on the internet. It helps me stay spiritually minded.” (Female, 35 yrs)  Intent: “I come every day as I start my browser, to keep me focused.” (Male, 62 yrs)  Intent: “LDS.org is my homepage. It helps me remember who I am and what I stand for before perusing the internet.” (Female, 17 yrs)  Intent: “Its my homepage. helps remind me who i am everytime i get on the computer” (Male, 25 yrs)
  38. 38. Preparing Data for the Affinity Diagram •Export the data to a spreadsheet •Create one row per comment*, including all data needed for context •Use “mail-merge” functionality to pull quotes and key contextual data (age, experience, success ratings, etc.) into a document formatted for printing the notes. •And then…
  39. 39. Meet Your New Best Friend… •Post-It Notes for Printers •Print a few merged notes onto regular paper to check alignment •Print a few at a time on manual feed to avoid jams
  40. 40. Organizing over 800 comments Naming and grouping 169 categories (11 at the top level) Conduct an Affinity Diagram Workshop
  41. 41. Finished diagram, with three levels of headings.
  42. 42. Record the Diagram Headings Only Record only the headings in a document or spreadsheet More Complete Associate each data point with its headings in a spreadsheet Formal Report Re-merge the headings and data back into a formatted document, with an introduction, findings, etc. REPORT Background Key Findings
  43. 43. A FEW QUESTIONS?
  44. 44. Analysis Steps LDS.org Example 1. Filter to those who a failed a specific task. * Looking for a location 2. Review all the data you have for each instance: • Stated intent • Path taken, search terms, etc. • Biggest problem • Other survey questions that might be relevant Trying to find the DC Temple Home > Maps < Home > Directory < Home > Temples > Find a Temple > Washington DC Temple Didn’t see Temples under Families Identify patterns and themes Temples not seen under “Families” Suggest solutions, or follow-up questions Broaden the heading. Conduct some tree testing. Failure Analysis
  45. 45. Failure Cases for LDS.org Almost all problem cases fell into one or more of the following categories: •Difficulty finding things in menus •Problems with lack of quotation marks in Search •Content lacking or spread across multiple sites
  46. 46. Mitigating Challenges It can be difficult to get enough context to conduct an effective failure analysis. This works best when you have: • Additional information derived from the URLs captured (e.g. search terms or other parameters) • Live pilot sessions to provide context for common experiences. • “Biggest problem” survey data. Be ready to accept that some paths will still be inexplicable.
  47. 47. What Next? •Conduct interviews or tests on key areas. •Adjust or create audience segments. Enrich personas. •Prioritize top intents for future development (or conduct a follow up survey to quantify themes from the affinity diagram). •Fix common problems from failure analysis. •Re-test after you’ve had new changes deployed for a while; compare quantitative results.
  48. 48. Archive It! But don’t just forget about it. True Intent data has a good shelf-life…
  49. 49. How Long Does This Take? • For 300-400 intents: • Creating the online study, collecting data, running quantitative stats: 40-60 hours for the facilitator* • Creating the affinity diagram: 20-40 hours for the facilitator* • Failure analysis: 20-60 hours for one analyst** • Total: 80-160 hours of your time, plus team time • Can be less with shortcuts • Should be less the second time around
  50. 50. “Thanks for leading this exercise. I will admit I was a bit skeptical going into these meetings (who isn’t skeptical of meetings that take you away from already-pressing deadlines?). But I’ve been really pleased with the process and results, and I think they will be of great help in determining our future strategy.” ~ Member of Team 1
  51. 51. “I mentioned this study in two different high-level meetings this week, and there is a great deal of interest and enthusiasm about the way the study was done and in seeing the results.” ~Member of Team 2
  52. 52. “Thanks a ton for your work. As I’ve said before, the results of the study will help inform our policies & site development going forward in big ways. We’ve learned a lot – I’ll share appropriate info with [other key stakeholders].” ~Member of Team 3
  53. 53. No time for all of this? Here are some ways to trim time and budget. SHORTCUTS
  54. 54. Shortcut #1: TRIM SCOPE Skip 1 or 2 of the 3 main activities: •quantitative benchmark •affinity diagram •failure analysis* Or collect the data for all 3, but wait on the analysis for 1 or more.
  55. 55. Shortcut #2: USE WORD CLOUDS •One cloud per open-ended question (or per set of closely related questions) •Dump all comments into a single text file •Standardize closely related word forms
  56. 56. “Please tell us why you came to this website today.”
  57. 57. “What was the biggest problem you encountered in your session today?”
  58. 58. “Anything else you'd like to tell us about your experience today?”
  59. 59. Conduct a “True Intent” Study 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions. Shortcut #3: Conduct a “True Intent” Study SURVEY 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions.
  60. 60. FINAL THOUGHTS
  61. 61. Evaluating success on an actual, self-defined task Large numbers of paths and times on task Feedback & profile data (qualitative & quantitative) True Intent
  62. 62. We could call this: A single-task remote unmoderated interview-guided usability test with integrated analytics and survey questions.
  63. 63. But it’s really all about: True Tasks True Success True Paths True Profiles True Satisfaction True Intent
  64. 64. FINAL QUESTIONS?
  65. 65. References & Links Official Website of the Church of Jesus Christ of Latter-day Saints: • http://lds.org Nate Bolt and Tony Tulathimutte, “Remote Research” • Book: http://amzn.to/18nEWtb • Supplementary Site:http://remoteresear.ch Jared Spool, “The KJ Technique”: • http://www.uie.com/articles/kj_tec hnique/ Caroline Jarrett, “Surveys That Work”: • http://rosenfeldmedia.com/books/s urvey- design/blog/do_incentives_help_to _improve/ Creative Commons Licenses: • Attribution, Non-commercial: http://creativecommons.org/lic enses/by-nc/2.0/ • Attribution, Non-commercial, No Derivatives: http://creativecommons.org/lic enses/by-nc-nd/2.0/
  • wakafrica

    Sep. 23, 2015
  • idasandersen35

    Feb. 9, 2015
  • TedBoren

    Sep. 19, 2013
  • StarReedyBradshaw

    Sep. 11, 2013

UXPA 2013 Annual Conference - Wednesday July 10, 2013 by Ted Boren Remote testing can tell you how easy your site is to use... for selected tasks. Surveys tell you how visitors feel... but lack performance data. Web analytics tell you where visitors go... but not whether they actually succeed. "True intent" studies bridge these gaps and help your team learn what's really happening on your site, by asking real visitors why they came, tracking where they go, then allowing them to tell you if they succeeded. Work together to build affinity diagrams of intents and conduct a detailed failure analysis for even deeper insights that can shape your team's strategy for years.

Views

Total views

4,015

On Slideshare

0

From embeds

0

Number of embeds

37

Actions

Downloads

39

Shares

0

Comments

0

Likes

4

×