Your SlideShare is downloading. ×
Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study: a quantitative open card sort analysis | NEOUPA April 25, 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Challenges Faced & Lessons Learned Conducting Cleveland Clinic's First UX Study: a quantitative open card sort analysis | NEOUPA April 25, 2012

3,759

Published on

Presented to Northeast Ohio Usability Professionals' Association (NEOUPA) on Wednesday, April 25, 2012 at Cleveland Clinic. …

Presented to Northeast Ohio Usability Professionals' Association (NEOUPA) on Wednesday, April 25, 2012 at Cleveland Clinic.

Two main themes in the presentation:

1. Tailor UX study to get stakeholder buy-in, in the unique corporate culture of Cleveland Clinic.

2. We invented our own methods to extract insights from quantitative open card sort to establish corporate-wide standard sitemap.

First, the new methods contribute to determine navigation menu naming convention and cross-reference points.

Secondly, I'd like to propose the UX & HCI community to include "Cell Plot Analysis" as a new statistical analysis when analyzing quantitative open card sorts.

I have not seen any of the above mentioned in prior literature before.


Feedback welcome.

The next appearance of this presentation will be UPA's International Conference in June 2012.

Published in: Design, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,759
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
11
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
  • Qualitative data – emotional persuasion boss cube, first line of defense. Chair/manager making requests. In the past we let them drive. Now we have sitemap to guide them Department requests UX study. No proposal necessary. No struggle with participant incentives.
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Qualitative data – emotional persuasion
  • Transcript

    • 1. Usability Professionals Association International Conference, 2012Challenges Faced & Lessons Learned ConductingCleveland Clinic’s First Usability Study:A quantitative open card sort Kaitlan Chu | Information Architect | Cleveland Clinic Northeast Ohio Usability Professionals Association Wednesday, April 25, 2012
    • 2. Usability Professionals Association International Conference, 2012Connect with Me Email ChuK@ccf.org Twitter @KaitlanChu LinkedIn linkedin.com/in/KaitlanChu SlideShare slideshare.net/KaitlanChu
    • 3. Usability Professionals Association International Conference, 2012 Formal, full-cycle, by in-house staffChallenges Faced & Lessons Learned ConductingCleveland Clinic’s First Usability Study:A quantitative open card sort New statistical analysis of large open card sort
    • 4. Background Planning Data Collection Analysis Impact
    • 5. Background Planning Data Collection Analysis Impact
    • 6. Background Planning Data Collection Analysis Impact
    • 7. Background Planning Data Collection Analysis Impact
    • 8. Background Planning Data Collection Analysis Impact
    • 9. Background Planning Data Collection Analysis ImpactAbout Me• My organization One of Top 4 in US• My team 13,000 Pages 120,000 visits/day Most visited US hospital site $5/visit in contribution margin• Me Usability / Navigation Sign off on changes Usability studies
    • 10. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    • 11. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think
    • 12. Background Planning Data Collection Analysis ImpactInfluence Decision Makers• Who they are• How they think Quantitative Studies Authoritative Sources
    • 13. Background Planning Data Collection Analysis Impact State of Site Structure• Lack of governance• Inconsistent navigation• Increased learning curve
    • 14. Background Planning Data Collection Analysis Impact Current State of Sites Needs a standard site map• Lack of governance applicable to all departments• Inconsistent navigation• Increased learning curve
    • 15. Background Planning Data Collection Analysis ImpactCreate a UX Opportunity• Started out as a design dilemma• Real problem – inconsistent site structures• Solution – standardize IA – standard sitemap
    • 16. Background Planning Data Collection Analysis Impact If you have $300, one month, how would you design the study?
    • 17. Background Planning Data Collection Analysis ImpactPlan the study• Quantitative Corporate culture• Open card sort No presumption• Online, remote Content inventory• Put together a team Participant Recruiting: Panel• Budget $300 Statistical Analysis• Analysis guidelines Hard to find; Invent our own
    • 18. Background Planning Data Collection Analysis ImpactQuantitative Open Card Sort Analysis• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
    • 19. Background Planning Data Collection Analysis ImpactTroubling Response Rate• 71 cards• Pilot-tested with internal staff Don’t represent real users• Completion time: • Pilot test: 20 min • Panel: 45 min• Limited budget “That interactive study takes a long time to complete and we’re working for very little incentives.” “That was fun! Please contact me again for this type of study.”
    • 20. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*Built-in analysis by most online card sorting tools
    • 21. Background Planning Data Collection Analysis ImpactCluster Analysis
    • 22. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*
    • 23. Background Planning Data Collection Analysis ImpactMultidimensional Scaling
    • 24. Background Planning Data Collection Analysis ImpactStatistical Analysis Methods 179 completes, 71 cards => 12710 lines in excel Cluster Multidimensional Scaling Cell Plot*• Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ? * Jieying Jane Chen
    • 25. Background Planning Data Collection Analysis ImpactCell Plot Analysis Cross reference
    • 26. Background Planning Data Collection Analysis ImpactCategory Labels • Elements of a sitemap• What keywords to use •Number of categories •Content within categories• What NOT to use •Cross-reference •Category label ?
    • 27. Background Planning Data Collection Analysis ImpactCategory Labels ? Participant #1 #2, #3, #4………… Participant #179 words words Patient 1378 Info 632 General 152 Care 81 Appointment 80 Resource 60 Become 17
    • 28. Background Planning Data Collection Analysis ImpactImpact of the study• Established a standard sitemap Applicable to all departments• Stakeholders see the value Greatly increased budget line• Increase perceived value of myteam & UX studies
    • 29. Background Planning Data Collection Analysis ImpactApplication of Sitemap Compare to…
    • 30. Background Planning Data Collection Analysis Impact• Lack of governance• Inconsistent navigation• Increased learning curve
    • 31. Background Planning Data Collection Analysis Impact Impact of the study•• Lack of governance governance Information Architecture•• Inconsistent navigation = Consistent site structures•• Increased learning curve Consistent navigation menu• Reduce learning curve
    • 32. Background Planning Data Collection Analysis ImpactLessons Learned• Tailor to corporate culture• Create opportunities• Pre-test online study in-person Mix qualitative & quantitative - Collect qualitative data - Test the usability of online study• Pilot test with participants represent real users• Provide decent participant incentives Consumer Panel?• Analyze data in as many ways as possible• Number of cards?Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards?Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada.
    • 33. Usability Professionals Association International Conference, 2012References•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society AnnualMeeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 152011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –416•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: MorganKaufmann Publishers.•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the UsabilityProfessionals Association, June 24 - July 1, Montreal, QB, Canada.•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/
    • 34. Usability Professionals Association International Conference, 2012Special Thanks to•Amy Sternad•Michelle Tackla Wallace•Amanda Beacher•Jieying Jane Chen•Zhijie Li•Scott Linabarger•Lynn Pecl
    • 35. Background Planning Data Collection Analysis Impact

    ×