Uploaded byKaitlan Chu
New Methods to Analyze Large Open Card Sort: UX Study Designed to Maximize Corporate Buy-In | User Experience Professionals' Association Intl' Conference 2012
The document outlines a UX study conducted by Kaitlan Chu at the Cleveland Clinic, focusing on new methods to analyze large open card sorting. It details the challenges faced in site navigation and IA governance, leading to the creation of a standardized sitemap based on quantitative research and stakeholder engagement. The study utilized various statistical analysis methods to assess participant responses and ultimately enhanced the perceived value of UX efforts within the organization.
More Related Content
Similar to New Methods to Analyze Large Open Card Sort: UX Study Designed to Maximize Corporate Buy-In | User Experience Professionals' Association Intl' Conference 2012
New Methods to Analyze Large Open Card Sort: UX Study Designed to Maximize Corporate Buy-In | User Experience Professionals' Association Intl' Conference 2012
- 1. Usability Professionals' Association International Conference, 2012 New Methods to Analyze Large Open Card Sort: Cleveland Clinic's First UX Study Kaitlan Chu | Information Architect | Cleveland Clinic
- 2. Usability Professionals' Association International Conference, 2012 Connect with Me Email ChuK@ccf.org Twitter @KaitlanChu LinkedIn linkedin.com/in/KaitlanChu SlideShare slideshare.net/KaitlanChu
- 3. Usability Professionals' Association International Conference, 2012 Cross reference + Category labels New Methods to Analyze Quantitative Open Card Sort: Cleveland Clinic's First UX Study Formal, full-cycle, by in-house staff Kaitlan Chu | Information Architect | Cleveland Clinic
- 4. Usability Professionals' Association International Conference, 2012 New Methods to Analyze Quantitative Open Card Sort: Cleveland Clinic's First UX Study Designed to maximize stakeholder buy-in Kaitlan Chu | Information Architect | Cleveland Clinic
- 5. Background Planning Data Collection Analysis Impact About • Cleveland Clinic One of Top 4 Hospital in US 41,000+ Employees •Digital Marketing ClevelandClinic.org 13,000 Pages 120,000 visits/day Most visited US hospital site $5/visit in contribution margin
- 6. Background Planning Data Collection Analysis Impact About •Sole IA for ClevelandClinic.org •Thought leadership: Usability / Navigation •Sign off on site changes •First to conduct UX studies in house •Academic research background •Analytics & front-end development, Fortune 100
- 7. Background Planning Data Collection Analysis Impact Influence Decision Makers • Who they are • How they think
- 8. Background Planning Data Collection Analysis Impact Influence Decision Makers • Who they are • How they think
- 9. Background Planning Data Collection Analysis Impact Influence Decision Makers • Who they are • How they think Quantitative Studies Authoritative Sources
- 10. Background Planning Data Collection Analysis Impact State of Site Structure • Lack of IA governance • Inconsistent navigation Household caregiver • Friction navigating site
- 11. Background Planning Data Collection Analysis Impact Current State of Sites Needs a standard site map • Lack of governance applicable to all departments • Inconsistent navigation • Increased learning curve
- 12. Background Planning Data Collection Analysis Impact Create a UX Opportunity • Started out as a design dilemma • Real problem – inconsistent site structures • Solution – standardize IA – standard sitemap
- 13. Background Planning Data Collection Analysis Impact If you have $300, one month, how would you design the study?
- 14. Background Planning Data Collection Analysis Impact Plan the study • Quantitative Corporate culture • Open card sort No presumption • Online, remote Content inventory • Put together a team Participant Recruiting: Panel • Budget $300 Statistical Analysis • Analysis guidelines Hard to find; Invent our own
- 15. Background Planning Data Collection Analysis Impact Research Questions • Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
- 16. Background Planning Data Collection Analysis Impact Generate Cards • Full Content Inventory • 71 cards
- 17. Background Planning Data Collection Analysis Impact Online open card sort • Sort all 71 cards • See card label & description • Name each category
- 18. Background Planning Data Collection Analysis Impact Troubling Response Rate • Pilot-tested with internal staff Don’t represent real users • Completion time: • Pilot test: 20 min • Panel: 45 min • Limited budget “That interactive study takes a long time to complete and we’re working for very little incentives.” “That was fun! Please contact me again for this type of study.”
- 19. Background Planning Data Collection Analysis Impact Cross reference + Category labels New Methods to Analyze Quantitative Open Card Sort: Cleveland Clinic's First UX Study
- 20. Background Planning Data Collection Analysis Impact Statistical Analysis Methods • 179 completes, 71 cards => 12710 lines in excel New Method Cluster (Dendrogram) Multidimensional Scaling Cell Plot Optimal Sort: Built-in analysis User Point Free Cluster Analysis Tool
- 21. Background Planning Data Collection Analysis Impact Optimal Sort Dendrogram
- 22. Background Planning Data Collection Analysis Impact
- 23. Background Planning Data Collection Analysis Impact In-house Cluster Analysis
- 24. Background Planning Data Collection Analysis Impact Statistical Analysis Methods Cluster (Dendrogram) Multidimensional Scaling Cell Plot
- 25. Background Planning Data Collection Analysis Impact In-house Multidimensional Scaling
- 26. Background Planning Data Collection Analysis Impact Statistical Analysis Methods Cluster (Dendrogram) Multidimensional Scaling Cell Plot • Elements of a sitemap •Number of categories •Content within categories •Cross-reference ? •Category label ?
- 27. Background Planning Data Collection Analysis Impact Cell Plot Analysis Cross reference
- 28. Background Planning Data Collection Analysis Impact Category Labels • Elements of a sitemap • What keywords to use •Number of categories •Content within categories • What NOT to use •Cross-reference •Category label ?
- 29. Background Planning Data Collection Analysis Impact Keyword Frequency ? Participant #1 #2, #3, #4………… Participant #179 words words Patient 1378 Info 632 General 152 Care 81 Appointment 80 Resource 60 Become 17
- 30. Background Planning Data Collection Analysis Impact Impact of the study • Established a standard sitemap Accepted by all departments • Stakeholders see the value Greatly increased budget line • Increase perceived value of my team & UX studies
- 31. Background Planning Data Collection Analysis Impact Application of Standard Sitemap
- 32. Background Planning Data Collection Analysis Impact Conclusions & Discussions • Tailor UX study to corporate culture • Analyze data in as many methods as possible • Cluster • Multidimensional Scaling 3D • Cell Plot • Keyword Frequency • Pre-test online study in-person • - Mix qualitative & quantitative • - Test the usability of online study • Pilot test with participants represent real users • Provide decent participant incentives • Number of cards?
- 33. Usability Professionals' Association International Conference, 2012 Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada. News & Information 0.40 C41 C9 C30 C44 C43 0.40 C75 C22 C4 C52 C38 C42 C36 C14 C78 C29 C23 C77 C46 C8 C27 C55 C32 C63 C64 Your Job- Technical C25 C54 C33 C81 C40 Aspects C26 C11 C47 C39 C49 Your Job- Business C85 C19 C74 C37 C69 C51 0.20 C61 C73 Aspects C57 C62 C7 C65 C34 0.15 C58 C67C87 C2 C48 C17 C70 C68 C59 C56 Your Job- Technical C76 C86 C18 C28 C84 HR & Benefits Your Job- Business C66 C21 C45 C3 Aspects C15 C35 C16 C82 C80 C24 Aspects C83 C60 C71 C1C6 C12 C10 C13 C50 C20 C79 C72 C79 C83 C72 C31 C20 C82 C5 C53 C66 C71 C10 C5 C31 0.00 C87 C86 C60 C13 C84 C18 C16 C21 -0.10 C61 C35 C70 C28 C2 C12 C45 C24 C3 C53 C17 C68 C1 C80 C67 C6 C48 C15 C76 News & Information C58 C69 C73 HR & Benefits C51 C59 C7 C74 C81 C49 C56 C50 C19 C65 C37 C34 C40 -0.20 -0.35 C46 C39 C57 C62 C38 C8 C85 C33 C63 C26 C32 C78 C47 C30 C11 C41 C9 C42 C36 C25 C44 C43 C22 C14 C64 C29 C55 C54 C52 C27 C23 C4 C77 -0.60 -0.40 C75 -0.60 -0.30 0.00 0.30 0.60 -0.60 -0.35 -0.10 0.15 0.40
- 34. Usability Professionals' Association International Conference, 2012 References •Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society Annual Meeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society. •Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 15 2011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp). •Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 – 416 •Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30. •Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point ( http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/). •Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media. •Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com ( http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet). •Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: Morgan Kaufmann Publishers. •Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability Professionals Association, June 24 - July 1, Montreal, QB, Canada. •Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/
- 35. Usability Professionals' Association International Conference, 2012 Special Thanks to •Amy Sternad •Michelle Tackla Wallace •Amanda Beacher •Jieying Jane Chen •Zhijie Li •Scott Linabarger •Lynn Pecl
- 36. Background Planning Data Collection Analysis Impact
Editor's Notes
- #17 So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
- #18 So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
- #19 So we set up the study in optimalsort, Market research department were the ones who did the actual data collection.
- #21 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #22 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #23 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #24 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #25 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #26 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #27 Spencer’s spreadsheet accommodates 40 cards Built-in tools: cluster analysis
- #31 Qualitative data – emotional persuasion boss cube, first line of defense. Chair/manager making requests. In the past we let them drive. Now we have sitemap to guide them Department requests UX study. No proposal necessary. No struggle with participant incentives.
- #32 Qualitative data – emotional persuasion
- #33 Qualitative data – emotional persuasion
- #34 Qualitative data – emotional persuasion
- #35 Qualitative data – emotional persuasion
- #36 Qualitative data – emotional persuasion