Presented to Northeast Ohio Usability Professionals' Association (NEOUPA) on Wednesday, April 25, 2012 at Cleveland Clinic. Two main themes in the presentation: 1. Tailor UX study to get stakeholder buy-in, in the unique corporate culture of Cleveland Clinic. 2. We invented our own methods to extract insights from quantitative open card sort to establish corporate-wide standard sitemap. First, the new methods contribute to determine navigation menu naming convention and cross-reference points. Secondly, I'd like to propose the UX & HCI community to include "Cell Plot Analysis" as a new statistical analysis when analyzing quantitative open card sorts. I have not seen any of the above mentioned in prior literature before. Feedback welcome. The next appearance of this presentation will be UPA's International Conference in June 2012.

1. Usability Professionals' Association
International Conference, 2012
Challenges Faced & Lessons Learned Conducting
Cleveland Clinic’s First Usability Study:
A quantitative open card sort
Kaitlan Chu | Information Architect | Cleveland Clinic
Northeast Ohio Usability Professionals' Association
Wednesday, April 25, 2012

2. Usability Professionals' Association
International Conference, 2012
Connect with Me
Email
ChuK@ccf.org
Twitter
@KaitlanChu
LinkedIn
linkedin.com/in/KaitlanChu
SlideShare
slideshare.net/KaitlanChu

3. Usability Professionals' Association
International Conference, 2012
Formal, full-cycle, by
in-house staff
Challenges Faced & Lessons Learned Conducting
Cleveland Clinic’s First Usability Study:
A quantitative open card sort
New statistical analysis
of large open card sort

4. Background Planning Data Collection Analysis Impact

5. Background Planning Data Collection Analysis Impact

6. Background Planning Data Collection Analysis Impact

7. Background Planning Data Collection Analysis Impact

8. Background Planning Data Collection Analysis Impact

9. Background Planning Data Collection Analysis Impact
About Me
• My organization
One of Top 4 in US
• My team
13,000 Pages
120,000 visits/day
Most visited US hospital site
$5/visit in contribution margin
• Me
Usability / Navigation
Sign off on changes
Usability studies

10. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

11. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

12. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think
Quantitative Studies Authoritative Sources

13. Background Planning Data Collection Analysis Impact
State of Site Structure
• Lack of governance
• Inconsistent navigation
• Increased learning curve

14. Background Planning Data Collection Analysis Impact
Current State of Sites
Needs a standard site map
• Lack of governance
applicable to all departments
• Inconsistent navigation
• Increased learning curve

15. Background Planning Data Collection Analysis Impact
Create a UX Opportunity
• Started out as a design dilemma
• Real problem – inconsistent site structures
• Solution – standardize IA – standard sitemap

16. Background Planning Data Collection Analysis Impact
If you have $300, one month,
how would you design the study?

17. Background Planning Data Collection Analysis Impact
Plan the study
• Quantitative Corporate culture
• Open card sort No presumption
• Online, remote Content inventory
• Put together a team Participant Recruiting: Panel
• Budget $300 Statistical Analysis
• Analysis guidelines Hard to find; Invent our own

18. Background Planning Data Collection Analysis Impact
Quantitative Open Card Sort Analysis
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ?

19. Background Planning Data Collection Analysis Impact
Troubling Response Rate
• 71 cards
• Pilot-tested with internal staff Don’t represent real users
• Completion time:
• Pilot test: 20 min
• Panel: 45 min
• Limited budget
“That interactive study takes a long time to complete and
we’re working for very little incentives.”
“That was fun! Please contact me again for this type of study.”

20. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
179 completes, 71 cards => 12710 lines in excel
Cluster Multidimensional Scaling Cell Plot*
Built-in analysis by most online card sorting tools

21. Background Planning Data Collection Analysis Impact
Cluster Analysis

22. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
179 completes, 71 cards => 12710 lines in excel
Cluster Multidimensional Scaling Cell Plot*

23. Background Planning Data Collection Analysis Impact
Multidimensional Scaling

24. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
179 completes, 71 cards => 12710 lines in excel
Cluster Multidimensional Scaling Cell Plot*
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ? * Jieying Jane Chen

25. Background Planning Data Collection Analysis Impact
Cell Plot Analysis
Cross reference

26. Background Planning Data Collection Analysis Impact
Category Labels
• Elements of a sitemap
• What keywords to use •Number of categories
•Content within categories
• What NOT to use •Cross-reference
•Category label ?

27. Background Planning Data Collection Analysis Impact
Category Labels
? Participant #1 #2, #3, #4………… Participant #179
words words
Patient 1378
Info 632
General 152
Care 81
Appointment 80
Resource 60
Become 17

28. Background Planning Data Collection Analysis Impact
Impact of the study
• Established a standard sitemap
Applicable to all departments
• Stakeholders see the value
Greatly increased budget line
• Increase perceived value of my
team & UX studies

29. Background Planning Data Collection Analysis Impact
Application of Sitemap
Compare to…

30. Background Planning Data Collection Analysis Impact
• Lack of governance
• Inconsistent navigation
• Increased learning curve

31. Background Planning Data Collection Analysis Impact
Impact of the study
•• Lack of governance governance
Information Architecture
•• Inconsistent navigation =
Consistent site structures
•• Increased learning curve
Consistent navigation menu
• Reduce learning curve

32. Background Planning Data Collection Analysis Impact
Lessons Learned
• Tailor to corporate culture
• Create opportunities
• Pre-test online study in-person Mix qualitative & quantitative
- Collect qualitative data
- Test the usability of online study
• Pilot test with participants represent real users
• Provide decent participant incentives Consumer Panel?
• Analyze data in as many ways as possible
• Number of cards?
Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards?
Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada.

33. Usability Professionals' Association
International Conference, 2012
References
•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society Annual
Meeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.
•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 15
2011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).
•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –
416
•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.
•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (
http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).
•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.
•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (
http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).
•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: Morgan
Kaufmann Publishers.
•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability
Professionals Association, June 24 - July 1, Montreal, QB, Canada.
•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/

34. Usability Professionals' Association
International Conference, 2012
Special Thanks to
•Amy Sternad
•Michelle Tackla Wallace
•Amanda Beacher
•Jieying Jane Chen
•Zhijie Li
•Scott Linabarger
•Lynn Pecl

35. Background Planning Data Collection Analysis Impact