Usability Professionals Association                                          International Conference, 2012New Methods to ...
Usability Professionals Association                                       International Conference, 2012Connect with Me   ...
Usability Professionals Association                                           International Conference, 2012Cross referenc...
Usability Professionals Association                                          International Conference, 2012New Methods to ...
Background        Planning         Data Collection   Analysis   ImpactAbout• Cleveland Clinic One of Top 4 Hospital in US ...
Background    Planning      Data Collection   Analysis   ImpactAbout•Sole IA for ClevelandClinic.org•Thought leadership: U...
Background     Planning   Data Collection   Analysis   ImpactInfluence Decision Makers• Who they are• How they think
Background     Planning   Data Collection   Analysis   ImpactInfluence Decision Makers• Who they are• How they think
Background     Planning   Data Collection   Analysis    ImpactInfluence Decision Makers• Who they are• How they think     ...
Background     Planning     Data Collection   Analysis   Impact State of Site Structure• Lack of IA governance• Inconsiste...
Background   Planning   Data Collection   Analysis   Impact Current State of Sites    Needs a standard      site map• Lack...
Background   Planning   Data Collection   Analysis   ImpactCreate a UX Opportunity• Started out as a design dilemma• Real ...
Background   Planning   Data Collection   Analysis   Impact If you have $300, one month, how would you design the study?
Background   Planning   Data Collection     Analysis       ImpactPlan the study• Quantitative             Corporate cultur...
Background    Planning     Data Collection   Analysis   ImpactResearch Questions• Elements of a sitemap   •Number of categ...
Background   Planning   Data Collection   Analysis   ImpactGenerate Cards• Full Content Inventory• 71 cards
Background   Planning   Data Collection   Analysis   ImpactOnline opencard sort• Sort all 71 cards• See card label &descri...
Background       Planning     Data Collection     Analysis         ImpactTroubling Response Rate• Pilot-tested with intern...
Background   Planning   Data Collection   Analysis   Impact Cross reference + Category labels  New Methods to Analyze Quan...
Background       Planning     Data Collection       Analysis          ImpactStatistical Analysis Methods• 179 completes, 7...
Background   Planning   Data Collection   Analysis   ImpactOptimal Sort Dendrogram
Background   Planning   Data Collection   Analysis   Impact
Background   Planning   Data Collection   Analysis   ImpactIn-house Cluster Analysis
Background      Planning    Data Collection       Analysis          ImpactStatistical Analysis MethodsCluster (Dendrogram)...
Background   Planning   Data Collection   Analysis   ImpactIn-house Multidimensional Scaling
Background                    Planning    Data Collection       Analysis          ImpactStatistical Analysis MethodsCluste...
Background   Planning   Data Collection   Analysis          ImpactCell Plot Analysis                                      ...
Background   Planning   Data Collection   Analysis                    ImpactCategory Labels                               ...
Background       Planning     Data Collection         Analysis           ImpactKeyword Frequency  ?          Participant #...
Background    Planning   Data Collection   Analysis   ImpactImpact of the study• Established a standard sitemap Accepted b...
Background   Planning   Data Collection   Analysis   ImpactApplication of Standard Sitemap
Background      Planning    Data Collection   Analysis   ImpactConclusions & Discussions• Tailor UX study to corporate cul...
Usability Professionals Association                                                                                       ...
Usability Professionals Association                                                                                       ...
Usability Professionals Association                           International Conference, 2012Special Thanks to•Amy Sternad•...
Background   Planning   Data Collection   Analysis   Impact
Upcoming SlideShare
Loading in …5
×

New Methods to Analyze Large Open Card Sort: UX Study Designed to Maximize Corporate Buy-In | User Experience Professionals' Association Intl' Conference 2012

1,834 views

Published on

Presented to Usability Professionals' Association International Conference (UPA/UXPA) on June 7, 2012 in Las Vegas.

Two main themes in the presentation:

1. Tailor UX study to get stakeholder buy-in, in the unique corporate culture of Cleveland Clinic.

2. We invented our own methods to extract insights from quantitative open card sort to establish corporate-wide standard sitemap.

Published in: Technology, Real Estate
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,834
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
23
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
  • So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
  • So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Qualitative data – emotional persuasion boss cube, first line of defense. Chair/manager making requests. In the past we let them drive. Now we have sitemap to guide them Department requests UX study. No proposal necessary. No struggle with participant incentives.
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • New Methods to Analyze Large Open Card Sort: UX Study Designed to Maximize Corporate Buy-In | User Experience Professionals' Association Intl' Conference 2012

    1. 1. Usability Professionals Association International Conference, 2012New Methods to Analyze Large OpenCard Sort:Cleveland Clinics First UX Study Kaitlan Chu | Information Architect | Cleveland Clinic
    2. 2. Usability Professionals Association International Conference, 2012Connect with Me Email ChuK@ccf.org Twitter @KaitlanChu LinkedIn linkedin.com/in/KaitlanChu SlideShare slideshare.net/KaitlanChu
    3. 3. Usability Professionals Association International Conference, 2012Cross reference + Category labelsNew Methods to Analyze QuantitativeOpen Card Sort:Cleveland Clinics First UX Study Formal, full-cycle, by in-house staff Kaitlan Chu | Information Architect | Cleveland Clinic
    4. 4. Usability Professionals Association International Conference, 2012New Methods to Analyze QuantitativeOpen Card Sort:Cleveland Clinics First UX Study Designed to maximize stakeholder buy-in Kaitlan Chu | Information Architect | Cleveland Clinic
    5. 5. Background Planning Data Collection Analysis ImpactAbout• Cleveland Clinic One of Top 4 Hospital in US 41,000+ Employees•Digital Marketing ClevelandClinic.org 13,000 Pages 120,000 visits/day Most visited US hospital site $5/visit in contribution margin
    6. 6. Background Planning Data Collection Analysis ImpactAbout•Sole IA for ClevelandClinic.org•Thought leadership: Usability / Navigation•Sign off on site changes•First to conduct UX studies in house•Academic research background•Analytics & front-end development, Fortune 100
    7. 7. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    8. 8. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    9. 9. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think Quantitative Studies Authoritative Sources
    10. 10. Background Planning Data Collection Analysis Impact State of Site Structure• Lack of IA governance• Inconsistent navigation Household caregiver• Friction navigating site
    11. 11. Background Planning Data Collection Analysis Impact Current State of Sites Needs a standard site map• Lack of governance applicable to all departments• Inconsistent navigation• Increased learning curve
    12. 12. Background Planning Data Collection Analysis ImpactCreate a UX Opportunity• Started out as a design dilemma• Real problem – inconsistent site structures• Solution – standardize IA – standard sitemap
    13. 13. Background Planning Data Collection Analysis Impact If you have $300, one month, how would you design the study?
    14. 14. Background Planning Data Collection Analysis ImpactPlan the study• Quantitative Corporate culture• Open card sort No presumption• Online, remote Content inventory• Put together a team Participant Recruiting: Panel• Budget $300 Statistical Analysis• Analysis guidelines Hard to find; Invent our own
    15. 15. Background Planning Data Collection Analysis ImpactResearch Questions• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
    16. 16. Background Planning Data Collection Analysis ImpactGenerate Cards• Full Content Inventory• 71 cards
    17. 17. Background Planning Data Collection Analysis ImpactOnline opencard sort• Sort all 71 cards• See card label &description• Name eachcategory
    18. 18. Background Planning Data Collection Analysis ImpactTroubling Response Rate• Pilot-tested with internal staff Don’t represent real users• Completion time: • Pilot test: 20 min • Panel: 45 min• Limited budget “That interactive study takes a long time to complete and we’re working for very little incentives.” “That was fun! Please contact me again for this type of study.”
    19. 19. Background Planning Data Collection Analysis Impact Cross reference + Category labels New Methods to Analyze Quantitative Open Card Sort: Cleveland Clinics First UX Study
    20. 20. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods• 179 completes, 71 cards => 12710 lines in excel New Method Cluster (Dendrogram) Multidimensional Scaling Cell PlotOptimal Sort: Built-in analysisUser Point Free Cluster Analysis Tool
    21. 21. Background Planning Data Collection Analysis ImpactOptimal Sort Dendrogram
    22. 22. Background Planning Data Collection Analysis Impact
    23. 23. Background Planning Data Collection Analysis ImpactIn-house Cluster Analysis
    24. 24. Background Planning Data Collection Analysis ImpactStatistical Analysis MethodsCluster (Dendrogram) Multidimensional Scaling Cell Plot
    25. 25. Background Planning Data Collection Analysis ImpactIn-house Multidimensional Scaling
    26. 26. Background Planning Data Collection Analysis ImpactStatistical Analysis MethodsCluster (Dendrogram) Multidimensional Scaling Cell Plot• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
    27. 27. Background Planning Data Collection Analysis ImpactCell Plot Analysis Cross reference
    28. 28. Background Planning Data Collection Analysis ImpactCategory Labels • Elements of a sitemap• What keywords to use •Number of categories •Content within categories• What NOT to use •Cross-reference •Category label ?
    29. 29. Background Planning Data Collection Analysis ImpactKeyword Frequency ? Participant #1 #2, #3, #4………… Participant #179 words words Patient 1378 Info 632 General 152 Care 81 Appointment 80 Resource 60 Become 17
    30. 30. Background Planning Data Collection Analysis ImpactImpact of the study• Established a standard sitemap Accepted by all departments• Stakeholders see the value Greatly increased budget line• Increase perceived value of myteam & UX studies
    31. 31. Background Planning Data Collection Analysis ImpactApplication of Standard Sitemap
    32. 32. Background Planning Data Collection Analysis ImpactConclusions & Discussions• Tailor UX study to corporate culture• Analyze data in as many methods as possible • Cluster • Multidimensional Scaling 3D • Cell Plot • Keyword Frequency• Pre-test online study in-person • - Mix qualitative & quantitative • - Test the usability of online study• Pilot test with participants represent real users• Provide decent participant incentives• Number of cards?
    33. 33. Usability Professionals Association International Conference, 2012 Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada. News & Information 0.40 C41 C9 C30 C44 C43 0.40 C75 C22 C4 C52 C38 C42 C36 C14 C78 C29 C23 C77 C46 C8 C27 C55 C32 C63 C64 Your Job- Technical C25 C54 C33 C81 C40 Aspects C26 C11 C47 C39 C49 Your Job- Business C85 C19 C74 C37 C69 C51 0.20 C61 C73 Aspects C57 C62 C7 C65 C34 0.15 C58 C67C87 C2 C48 C17 C70 C68 C59 C56 Your Job- Technical C76 C86 C18 C28 C84 HR & Benefits Your Job- Business C66 C21 C45 C3 Aspects C15 C35 C16 C82 C80 C24 Aspects C83 C60 C71 C1C6 C12 C10 C13 C50 C20 C79 C72 C79 C83 C72 C31 C20 C82 C5 C53 C66 C71 C10 C5 C31 0.00 C87 C86 C60 C13 C84 C18 C16 C21-0.10 C61 C35 C70 C28 C2 C12 C45 C24 C3 C53 C17 C68 C1 C80 C67 C6 C48 C15 C76 News & Information C58 C69 C73 HR & Benefits C51 C59 C7 C74 C81 C49 C56 C50 C19 C65 C37 C34 C40 -0.20-0.35 C46 C39 C57 C62 C38 C8 C85 C33 C63 C26 C32 C78 C47 C30 C11 C41 C9 C42 C36 C25 C44 C43 C22 C14 C64 C29 C55 C54 C52 C27 C23 C4 C77-0.60 -0.40 C75 -0.60 -0.30 0.00 0.30 0.60 -0.60 -0.35 -0.10 0.15 0.40
    34. 34. Usability Professionals Association International Conference, 2012References•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society AnnualMeeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 152011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –416•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: MorganKaufmann Publishers.•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the UsabilityProfessionals Association, June 24 - July 1, Montreal, QB, Canada.•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/
    35. 35. Usability Professionals Association International Conference, 2012Special Thanks to•Amy Sternad•Michelle Tackla Wallace•Amanda Beacher•Jieying Jane Chen•Zhijie Li•Scott Linabarger•Lynn Pecl
    36. 36. Background Planning Data Collection Analysis Impact

    ×