Presented to Usability Professionals' Association International Conference (UPA/UXPA) on June 7, 2012 in Las Vegas. Two main themes in the presentation: 1. Tailor UX study to get stakeholder buy-in, in the unique corporate culture of Cleveland Clinic. 2. We invented our own methods to extract insights from quantitative open card sort to establish corporate-wide standard sitemap.

1. Usability Professionals' Association
International Conference, 2012
New Methods to Analyze Large Open
Card Sort:
Cleveland Clinic's First UX Study
Kaitlan Chu | Information Architect | Cleveland Clinic

2. Usability Professionals' Association
International Conference, 2012
Connect with Me
Email
ChuK@ccf.org
Twitter
@KaitlanChu
LinkedIn
linkedin.com/in/KaitlanChu
SlideShare
slideshare.net/KaitlanChu

3. Usability Professionals' Association
International Conference, 2012
Cross reference + Category labels
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study
Formal, full-cycle, by in-house staff
Kaitlan Chu | Information Architect | Cleveland Clinic

4. Usability Professionals' Association
International Conference, 2012
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study
Designed to maximize stakeholder buy-in
Kaitlan Chu | Information Architect | Cleveland Clinic

5. Background Planning Data Collection Analysis Impact
About
• Cleveland Clinic
One of Top 4 Hospital in US
41,000+ Employees
•Digital Marketing
ClevelandClinic.org
13,000 Pages
120,000 visits/day
Most visited US hospital site
$5/visit in contribution margin

6. Background Planning Data Collection Analysis Impact
About
•Sole IA for ClevelandClinic.org
•Thought leadership: Usability / Navigation
•Sign off on site changes
•First to conduct UX studies in house
•Academic research background
•Analytics & front-end development, Fortune 100

7. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

8. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

9. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think
Quantitative Studies Authoritative Sources

10. Background Planning Data Collection Analysis Impact
State of Site Structure
• Lack of IA governance
• Inconsistent navigation Household caregiver
• Friction navigating site

11. Background Planning Data Collection Analysis Impact
Current State of Sites
Needs a standard site map
• Lack of governance
applicable to all departments
• Inconsistent navigation
• Increased learning curve

12. Background Planning Data Collection Analysis Impact
Create a UX Opportunity
• Started out as a design dilemma
• Real problem – inconsistent site structures
• Solution – standardize IA – standard sitemap

13. Background Planning Data Collection Analysis Impact
If you have $300, one month,
how would you design the study?

14. Background Planning Data Collection Analysis Impact
Plan the study
• Quantitative Corporate culture
• Open card sort No presumption
• Online, remote Content inventory
• Put together a team Participant Recruiting: Panel
• Budget $300 Statistical Analysis
• Analysis guidelines Hard to find; Invent our own

15. Background Planning Data Collection Analysis Impact
Research Questions
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ?

16. Background Planning Data Collection Analysis Impact
Generate Cards
• Full Content Inventory
• 71 cards

17. Background Planning Data Collection Analysis Impact
Online open
card sort
• Sort all 71 cards
• See card label &
description
• Name each
category

18. Background Planning Data Collection Analysis Impact
Troubling Response Rate
• Pilot-tested with internal staff Don’t represent real users
• Completion time:
• Pilot test: 20 min
• Panel: 45 min
• Limited budget
“That interactive study takes a long time to complete and
we’re working for very little incentives.”
“That was fun! Please contact me again for this type of study.”

19. Background Planning Data Collection Analysis Impact
Cross reference + Category labels
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study

20. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
• 179 completes, 71 cards => 12710 lines in excel New Method
Cluster (Dendrogram) Multidimensional Scaling Cell Plot
Optimal Sort: Built-in analysis
User Point Free Cluster Analysis Tool

21. Background Planning Data Collection Analysis Impact
Optimal Sort Dendrogram

22. Background Planning Data Collection Analysis Impact

23. Background Planning Data Collection Analysis Impact
In-house Cluster Analysis

24. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
Cluster (Dendrogram) Multidimensional Scaling Cell Plot

25. Background Planning Data Collection Analysis Impact
In-house Multidimensional Scaling

26. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
Cluster (Dendrogram) Multidimensional Scaling Cell Plot
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ?

27. Background Planning Data Collection Analysis Impact
Cell Plot Analysis
Cross reference

28. Background Planning Data Collection Analysis Impact
Category Labels
• Elements of a sitemap
• What keywords to use •Number of categories
•Content within categories
• What NOT to use •Cross-reference
•Category label ?

29. Background Planning Data Collection Analysis Impact
Keyword Frequency
? Participant #1 #2, #3, #4………… Participant #179
words words
Patient 1378
Info 632
General 152
Care 81
Appointment 80
Resource 60
Become 17

30. Background Planning Data Collection Analysis Impact
Impact of the study
• Established a standard sitemap
Accepted by all departments
• Stakeholders see the value
Greatly increased budget line
• Increase perceived value of my
team & UX studies

31. Background Planning Data Collection Analysis Impact
Application of Standard Sitemap

32. Background Planning Data Collection Analysis Impact
Conclusions & Discussions
• Tailor UX study to corporate culture
• Analyze data in as many methods as possible
• Cluster
• Multidimensional Scaling 3D
• Cell Plot
• Keyword Frequency
• Pre-test online study in-person
• - Mix qualitative & quantitative
• - Test the usability of online study
• Pilot test with participants represent real users
• Provide decent participant incentives
• Number of cards?

33. Usability Professionals' Association
International Conference, 2012
Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards?
Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada.
News & Information
0.40 C41
C9
C30 C44
C43
0.40 C75
C22 C4
C52
C38 C42 C36
C14 C78
C29 C23 C77 C46 C8
C27 C55 C32 C63
C64
Your Job- Technical C25 C54
C33 C81
C40
Aspects C26
C11
C47 C39
C49
Your Job- Business C85
C19
C74 C37
C69 C51 0.20
C61 C73 Aspects C57
C62 C7 C65
C34
0.15 C58
C67C87
C2
C48
C17
C70
C68
C59 C56 Your Job- Technical C76
C86 C18
C28
C84 HR & Benefits Your Job- Business
C66 C21 C45
C3 Aspects C15
C35 C16
C82
C80
C24
Aspects
C83 C60 C71 C1C6
C12 C10
C13 C50 C20 C79
C72
C79 C83 C72 C31
C20 C82 C5
C53 C66 C71
C10 C5 C31 0.00 C87 C86 C60 C13 C84
C18 C16
C21
-0.10 C61 C35 C70 C28
C2
C12
C45 C24
C3 C53
C17 C68 C1 C80
C67 C6
C48 C15
C76 News & Information C58 C69
C73
HR & Benefits
C51
C59 C7
C74
C81 C49 C56 C50
C19 C65
C37 C34
C40 -0.20
-0.35 C46 C39
C57
C62
C38 C8 C85
C33 C63 C26
C32
C78 C47
C30
C11
C41 C9 C42
C36 C25
C44 C43 C22 C14
C64
C29 C55
C54 C52
C27 C23 C4 C77
-0.60 -0.40 C75
-0.60 -0.30 0.00 0.30 0.60 -0.60 -0.35 -0.10 0.15 0.40

34. Usability Professionals' Association
International Conference, 2012
References
•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society Annual
Meeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.
•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 15
2011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).
•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –
416
•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.
•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (
http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).
•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.
•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (
http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).
•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: Morgan
Kaufmann Publishers.
•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability
Professionals Association, June 24 - July 1, Montreal, QB, Canada.
•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/

35. Usability Professionals' Association
International Conference, 2012
Special Thanks to
•Amy Sternad
•Michelle Tackla Wallace
•Amanda Beacher
•Jieying Jane Chen
•Zhijie Li
•Scott Linabarger
•Lynn Pecl

36. Background Planning Data Collection Analysis Impact