Usability Professionals Association                                                       International Conference, 2012Ch...
Usability Professionals Association                                       International Conference, 2012Connect with Me   ...
Usability Professionals Association                                         International Conference, 2012                ...
Background   Planning   Data Collection   Analysis   Impact
Background   Planning   Data Collection   Analysis   Impact
Background   Planning   Data Collection   Analysis   Impact
Background   Planning   Data Collection   Analysis   Impact
Background   Planning   Data Collection   Analysis   Impact
Background       Planning        Data Collection   Analysis   ImpactAbout Me• My organization One of Top 4 in US• My team ...
Background     Planning   Data Collection   Analysis   ImpactInfluence Decision Makers• Who they are• How they think
Background     Planning   Data Collection   Analysis   ImpactInfluence Decision Makers• Who they are• How they think
Background     Planning   Data Collection   Analysis    ImpactInfluence Decision Makers• Who they are• How they think     ...
Background   Planning   Data Collection   Analysis   Impact State of Site Structure• Lack of governance• Inconsistent navi...
Background   Planning   Data Collection   Analysis   Impact Current State of Sites    Needs a standard      site map• Lack...
Background   Planning   Data Collection   Analysis   ImpactCreate a UX Opportunity• Started out as a design dilemma• Real ...
Background   Planning   Data Collection   Analysis   Impact If you have $300, one month, how would you design the study?
Background   Planning   Data Collection     Analysis       ImpactPlan the study• Quantitative             Corporate cultur...
Background    Planning     Data Collection   Analysis   ImpactQuantitative Open Card Sort Analysis• Elements of a sitemap ...
Background       Planning     Data Collection     Analysis         ImpactTroubling Response Rate• 71 cards• Pilot-tested w...
Background        Planning      Data Collection      Analysis          ImpactStatistical Analysis Methods     179 complete...
Background   Planning   Data Collection   Analysis   ImpactCluster Analysis
Background      Planning    Data Collection       Analysis          ImpactStatistical Analysis Methods   179 completes, 71...
Background   Planning   Data Collection   Analysis   ImpactMultidimensional Scaling
Background                    Planning    Data Collection       Analysis              ImpactStatistical Analysis Methods  ...
Background   Planning   Data Collection   Analysis          ImpactCell Plot Analysis                                      ...
Background   Planning   Data Collection   Analysis                    ImpactCategory Labels                               ...
Background       Planning     Data Collection         Analysis           ImpactCategory Labels  ?          Participant #1 ...
Background    Planning   Data Collection   Analysis   ImpactImpact of the study• Established a standard sitemap Applicable...
Background   Planning   Data Collection   Analysis   ImpactApplication of Sitemap                 Compare to…
Background   Planning   Data Collection   Analysis   Impact• Lack of governance• Inconsistent navigation• Increased learni...
Background    Planning   Data Collection   Analysis   Impact Impact of the study•• Lack of governance governance    Inform...
Background                   Planning              Data Collection                 Analysis                    ImpactLesso...
Usability Professionals Association                                                                                       ...
Usability Professionals Association                           International Conference, 2012Special Thanks to•Amy Sternad•...
Background   Planning   Data Collection   Analysis   Impact
Upcoming SlideShare
Loading in …5
×

Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study: a quantitative open card sort analysis | NEOUPA April 25, 2012

6,801 views

Published on

Presented to Northeast Ohio Usability Professionals' Association (NEOUPA) on Wednesday, April 25, 2012 at Cleveland Clinic.

Two main themes in the presentation:

1. Tailor UX study to get stakeholder buy-in, in the unique corporate culture of Cleveland Clinic.

2. We invented our own methods to extract insights from quantitative open card sort to establish corporate-wide standard sitemap.

First, the new methods contribute to determine navigation menu naming convention and cross-reference points.

Secondly, I'd like to propose the UX & HCI community to include "Cell Plot Analysis" as a new statistical analysis when analyzing quantitative open card sorts.

I have not seen any of the above mentioned in prior literature before.


Feedback welcome.

The next appearance of this presentation will be UPA's International Conference in June 2012.

Published in: Design, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,801
On SlideShare
0
From Embeds
0
Number of Embeds
4,113
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Qualitative data – emotional persuasion boss cube, first line of defense. Chair/manager making requests. In the past we let them drive. Now we have sitemap to guide them Department requests UX study. No proposal necessary. No struggle with participant incentives.
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study: a quantitative open card sort analysis | NEOUPA April 25, 2012

    1. 1. Usability Professionals Association International Conference, 2012Challenges Faced & Lessons Learned ConductingCleveland Clinic’s First Usability Study:A quantitative open card sort Kaitlan Chu | Information Architect | Cleveland Clinic Northeast Ohio Usability Professionals Association Wednesday, April 25, 2012
    2. 2. Usability Professionals Association International Conference, 2012Connect with Me Email ChuK@ccf.org Twitter @KaitlanChu LinkedIn linkedin.com/in/KaitlanChu SlideShare slideshare.net/KaitlanChu
    3. 3. Usability Professionals Association International Conference, 2012 Formal, full-cycle, by in-house staffChallenges Faced & Lessons Learned ConductingCleveland Clinic’s First Usability Study:A quantitative open card sort New statistical analysis of large open card sort
    4. 4. Background Planning Data Collection Analysis Impact
    5. 5. Background Planning Data Collection Analysis Impact
    6. 6. Background Planning Data Collection Analysis Impact
    7. 7. Background Planning Data Collection Analysis Impact
    8. 8. Background Planning Data Collection Analysis Impact
    9. 9. Background Planning Data Collection Analysis ImpactAbout Me• My organization One of Top 4 in US• My team 13,000 Pages 120,000 visits/day Most visited US hospital site $5/visit in contribution margin• Me Usability / Navigation Sign off on changes Usability studies
    10. 10. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    11. 11. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    12. 12. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think Quantitative Studies Authoritative Sources
    13. 13. Background Planning Data Collection Analysis Impact State of Site Structure• Lack of governance• Inconsistent navigation• Increased learning curve
    14. 14. Background Planning Data Collection Analysis Impact Current State of Sites Needs a standard site map• Lack of governance applicable to all departments• Inconsistent navigation• Increased learning curve
    15. 15. Background Planning Data Collection Analysis ImpactCreate a UX Opportunity• Started out as a design dilemma• Real problem – inconsistent site structures• Solution – standardize IA – standard sitemap
    16. 16. Background Planning Data Collection Analysis Impact If you have $300, one month, how would you design the study?
    17. 17. Background Planning Data Collection Analysis ImpactPlan the study• Quantitative Corporate culture• Open card sort No presumption• Online, remote Content inventory• Put together a team Participant Recruiting: Panel• Budget $300 Statistical Analysis• Analysis guidelines Hard to find; Invent our own
    18. 18. Background Planning Data Collection Analysis ImpactQuantitative Open Card Sort Analysis• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
    19. 19. Background Planning Data Collection Analysis ImpactTroubling Response Rate• 71 cards• Pilot-tested with internal staff Don’t represent real users• Completion time: • Pilot test: 20 min • Panel: 45 min• Limited budget “That interactive study takes a long time to complete and we’re working for very little incentives.” “That was fun! Please contact me again for this type of study.”
    20. 20. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*Built-in analysis by most online card sorting tools
    21. 21. Background Planning Data Collection Analysis ImpactCluster Analysis
    22. 22. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*
    23. 23. Background Planning Data Collection Analysis ImpactMultidimensional Scaling
    24. 24. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ? * Jieying Jane Chen
    25. 25. Background Planning Data Collection Analysis ImpactCell Plot Analysis Cross reference
    26. 26. Background Planning Data Collection Analysis ImpactCategory Labels • Elements of a sitemap• What keywords to use •Number of categories •Content within categories• What NOT to use •Cross-reference •Category label ?
    27. 27. Background Planning Data Collection Analysis ImpactCategory Labels ? Participant #1 #2, #3, #4………… Participant #179 words words Patient 1378 Info 632 General 152 Care 81 Appointment 80 Resource 60 Become 17
    28. 28. Background Planning Data Collection Analysis ImpactImpact of the study• Established a standard sitemap Applicable to all departments• Stakeholders see the value Greatly increased budget line• Increase perceived value of myteam & UX studies
    29. 29. Background Planning Data Collection Analysis ImpactApplication of Sitemap Compare to…
    30. 30. Background Planning Data Collection Analysis Impact• Lack of governance• Inconsistent navigation• Increased learning curve
    31. 31. Background Planning Data Collection Analysis Impact Impact of the study•• Lack of governance governance Information Architecture•• Inconsistent navigation = Consistent site structures•• Increased learning curve Consistent navigation menu• Reduce learning curve
    32. 32. Background Planning Data Collection Analysis ImpactLessons Learned• Tailor to corporate culture• Create opportunities• Pre-test online study in-person Mix qualitative & quantitative - Collect qualitative data - Test the usability of online study• Pilot test with participants represent real users• Provide decent participant incentives Consumer Panel?• Analyze data in as many ways as possible• Number of cards?Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards?Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada.
    33. 33. Usability Professionals Association International Conference, 2012References•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society AnnualMeeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 152011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –416•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: MorganKaufmann Publishers.•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the UsabilityProfessionals Association, June 24 - July 1, Montreal, QB, Canada.•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/
    34. 34. Usability Professionals Association International Conference, 2012Special Thanks to•Amy Sternad•Michelle Tackla Wallace•Amanda Beacher•Jieying Jane Chen•Zhijie Li•Scott Linabarger•Lynn Pecl
    35. 35. Background Planning Data Collection Analysis Impact

    ×