Talk Presented by Shannon Halgren at UXPA 2023 in Austin Texas.
Session Abstract:
What would you do if you were tasked with developing a new categorization for your company’s products and services? Under the best circumstances, this is a hefty undertaking; but, to add to the complexity, consider that your company is a massive hiring website with varied products and services, and they want to change the names of their products and services based on UX research that revealed the names are confusing. So, now you are tasked with establishing a new categorization before these new names are developed. This task was a recent research challenge of ours. In this presentation, we will discuss how we utilized card sorting and treejack testing to iteratively test categorizations. We will share our methodology and discuss how we overcame usability and technical limitations of the Optimal Workshop treejack tool. We will share our hard-earned tips and tricks for the audience to utilize in their own research.
Evaluating the Impact of Design Thinking in ActionDavid Allan Chin
Design thinking offers a problem-solving approach widely adopted by the most innovative companies and organizations - but how do we truly measure its impact?
Professor Jeanne Liedtka of the University of Virginia’s Darden School of Business reports on the results of research conducted at UVA over the past 6 years of over 30 organizations using design thinking in practice.
This presentation shared during a MURAL webinar hosted by Jeanne Liedtka on 12/24/18.
User Research 101: DIY Quick Course - CodeMash 2.0.1.1.Carol Smith
Tight budget and short on time? This CodeMash 2.0.1.1. session taught attendees how to get a quick start on user research. Three discount user research methods were covered: observations; interviews; and card sorting. These quick and inexpensive methods will provide you with rich information about users including their goals, needs and abilities. This session also introduced ways to effectively share and communicate this information such as through personas and mental models.
User research: Uncovering compelling insights through interviewsSteve Portigal
Interviewing users is undeniably one of the most valuable and commonly used user research tools. Yet sometimes we forget that it’s a skill we need to learn, because it’s based on skills we think we have (talking or even listening) and it’s not taught or reflected on.
In this workshop, we’ll consider how to frame the problem, when to use research in the design process, and the tactics for setting up a successful study. We will focus in detail on the interview itself, reviewing detailed techniques for listening and asking questions, then conclude with an engaging exercise to bring these best practices to life.
This workshop will show you how to:
Integrate mixed methods of research into solving a problem.
Develop increased empathy, a critical facet to meaningful interviews.
Derive useful results from interviews once you get participants to open up.
On Wednesday, July 13th at 11:30am PDT, UserZoom and partner Key Lime Interactive hosted a complimentary webinar where they discussed the benefits of two POWERFUL research methods: CARD SORTING & TREE TESTING.
This is the full slide deck from the webinar. Enjoy!
Taxonomy of Spices and Pantries - Optimal Workshop RoadshowGrace G. Lau
An introduction to open card sorting and tree testing with a case study from Grace's kitchen/pantries.
Sponsored by Optimal Workshop as part of #owroadshow in Los Angeles, hosted at Philosophie. This was presented on Thursday, June 15, 2017 by Chris Chandler and Grace Lau
**Follow along with the digital card sort**
Kitchen card sort: http://bit.ly/owla-sort
Results from the kitchen card sort: http://bit.ly/owla-sort-results
Note: The workshop focused on a manual card sort. Data was not saved back to the OptimalSort tool.
**Tree test:**
Kitchen tree test: http://bit.ly/owla-tree
Results from Kitchen tree test: http://bit.ly/owla-tree-results
Sitemap that was used for this tree test: http://bit.ly/owla-tree-mindmap
Evaluating the Impact of Design Thinking in ActionDavid Allan Chin
Design thinking offers a problem-solving approach widely adopted by the most innovative companies and organizations - but how do we truly measure its impact?
Professor Jeanne Liedtka of the University of Virginia’s Darden School of Business reports on the results of research conducted at UVA over the past 6 years of over 30 organizations using design thinking in practice.
This presentation shared during a MURAL webinar hosted by Jeanne Liedtka on 12/24/18.
User Research 101: DIY Quick Course - CodeMash 2.0.1.1.Carol Smith
Tight budget and short on time? This CodeMash 2.0.1.1. session taught attendees how to get a quick start on user research. Three discount user research methods were covered: observations; interviews; and card sorting. These quick and inexpensive methods will provide you with rich information about users including their goals, needs and abilities. This session also introduced ways to effectively share and communicate this information such as through personas and mental models.
User research: Uncovering compelling insights through interviewsSteve Portigal
Interviewing users is undeniably one of the most valuable and commonly used user research tools. Yet sometimes we forget that it’s a skill we need to learn, because it’s based on skills we think we have (talking or even listening) and it’s not taught or reflected on.
In this workshop, we’ll consider how to frame the problem, when to use research in the design process, and the tactics for setting up a successful study. We will focus in detail on the interview itself, reviewing detailed techniques for listening and asking questions, then conclude with an engaging exercise to bring these best practices to life.
This workshop will show you how to:
Integrate mixed methods of research into solving a problem.
Develop increased empathy, a critical facet to meaningful interviews.
Derive useful results from interviews once you get participants to open up.
On Wednesday, July 13th at 11:30am PDT, UserZoom and partner Key Lime Interactive hosted a complimentary webinar where they discussed the benefits of two POWERFUL research methods: CARD SORTING & TREE TESTING.
This is the full slide deck from the webinar. Enjoy!
Taxonomy of Spices and Pantries - Optimal Workshop RoadshowGrace G. Lau
An introduction to open card sorting and tree testing with a case study from Grace's kitchen/pantries.
Sponsored by Optimal Workshop as part of #owroadshow in Los Angeles, hosted at Philosophie. This was presented on Thursday, June 15, 2017 by Chris Chandler and Grace Lau
**Follow along with the digital card sort**
Kitchen card sort: http://bit.ly/owla-sort
Results from the kitchen card sort: http://bit.ly/owla-sort-results
Note: The workshop focused on a manual card sort. Data was not saved back to the OptimalSort tool.
**Tree test:**
Kitchen tree test: http://bit.ly/owla-tree
Results from Kitchen tree test: http://bit.ly/owla-tree-results
Sitemap that was used for this tree test: http://bit.ly/owla-tree-mindmap
1. Always question what they say There’s a saying among researchers, ‘Do a quant survey with 100 people and learn ten things. Do a qual study with ten people and learn 100 things.’ You will learn when research is only providing you a false sense of security.
2. How to be a better consumer of research results If these insights were rabbits, how do stakeholders know which insights (rabbits) are worth chasing? What leads to the best business and design opportunities?
3. Get the best value from a qualitative and quantitative study In this Masterclass, we will be going over which methodology to apply to a research question. We will discuss how to weigh appropriately, act upon, or be critical of the resulting research findings.
4. Tools to help you Know when research is only providing you a false sense of security. Be a better consumer of research results.
5. And many more strategies Learn how to catch the best rabbits!
This presentation covers validation techniques for testing taxonomy and metadata with users. Four approaches are covered: Delphi card sorting, online card sorting, usability testing, and search term analysis. The presentation also contains a list of online card sorting tools.
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...BrittanyRubinstein
As part of Denver's 2019 Startup Week, Crownpeak's Director of UX, Ari Weissman and Lys Maitland, Experience Research Manager at a national healthcare organization, presented a joint session on "Choosing a direction: Learning how to test ideas and designs."
UX 101: Making Great Human Experiences at Pittsburgh PodCamp 9Carol Smith
Carol Smith provides the tools you need to get started doing User Experience (UX) work right away. She introduces three quick and inexpensive UX research methods that will provide you with rich information about users and designs: interviews; card sorting; and usability testing. You'll learn how this work will influence your design and ways to effectively share and communicate what has been learned to increase stakeholders understandings of customers.
EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description:
Test early, test often.
It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies.
This talk will cover:
What is user testing and why is it important
How to plan for user testing
What are ways to make testing cheaper
What are ways to make testing quicker
How to test with different fidelities of concept and design
How to collect data more frequently
Opportunities for getting the whole team engaged
What to do with the insights/outcomes of research
1. Test Assumptions, Not Ideas The Lean Startup popularized the idea of testing the assumptions that need to be true in order for your ideas to work.
2. Build it and test it It can be challenging, however, to see our own assumptions let alone test them.
3. Discovery process In this Masterclass, you’ll learn a structured approach for how to take an idea, break it down into its underlying assumptions, and quickly prioritize the ones that need to be tested.
4. Tools to help you Finally, you’ll learn a simple framework for how to design fast and better.
5. And many more strategies Such as effective assumption tests so that you can quickly identify what to build.
Providing a compelling user experience is pivotal to developing a successful product. As a product manager, you are often tasked with difficult decisions that require a deep understanding of customer needs and how to deliver the best experience possible. User research is an effective way to both generate insights and validate direction.
In this workshop you will learn:
* The skills to effectively integrate user research into the product development process with a strong return on investment.
* How foundational user research can help product teams understand user goals, generate insights, and narrow focus.
* How to use research to evaluate and iterate on product concepts.
* How to validate design and product decisions to ready your product for launch.
This presentation grew out of my experience working with an information architecture that was incomplete and lacked generative research to understand the users. I was tasked with finding ways to validate the IA while providing useful user insight for future design changes. To meet the goal, I studied card sorts in depth and developed useful techniques for problems that often arise but are seldom discussed.
I also explored the use of other IA validation techniques such as task-based tree tests and click tests to complement and cross-validate card sort findings. Through the use of mixed methodology within-subject study design, card sort data can be more easily understood and validated by the outcomes of tree tests and click tests.
This presentation will provide practitioners with information about card sorting in advanced situations. Specifically it will cover card sorting when applied to complex hierarchies, multiple audiences, and information architecture validation.
Presented by Caron Garstka at UXPA 2018
Getting Started With User Research, Presented at Agile2010Carol Smith
The gap between a good product and a great one can be bridged by understanding your users.
This presentation shared how better systems are built by taking small, iterative steps to understand the users desires, needs and abilities.
Attendees will learn how to get information about users quickly and cheaply. For those that have more time (and perhaps a small budget) Carol introduced methods to use to get more detailed information from your users. Carol also introduced ways to effectively share and communicate this information.
Getting Started with User Research - Stir Trek 2011Carol Smith
Presented at Stir Trek: Thor Edition, in Columbus, Ohio on May 6, 2011.
Once you know who uses your product, all sorts of new questions start to emerge. How are they using the product? Why are they using it? What else might they want? In this session you will learn about three quick and easy methods to understand the users desires, needs and abilities. The basics of observations, interviews and card sorting will be covered. You will also learn ways to effectively share and communicate what you learn with your team.
Leveraging the Latest in Brain Science to Deliver the Next Generation of E-Le...Human Capital Media
With the explosion in brain research during the past 10 years, scientists know more about the brain than ever before. Consumer brain training software designed to improve memory, attention, cognitive ability and more has hit the market with a vengeance. How is this same research set to change the world of corporate learning and e-learning solutions? What does the next generation of e-learning look like? Join Dr. Alice Kim from the Rotman Research Institute, a premier international center for the study of human brain, along with Carol Leaman, the CEO of Axonify, to learn about:
Three key areas of brain research that are expected to have the most impact on corporate e-learning.
How the latest research is being applied in e-learning tools in the workplace.
Real corporate examples of improvements in knowledge retention, learning transfer and the bottom line.
Dive into the innovative world of smart garages with our insightful presentation, "Exploring the Future of Smart Garages." This comprehensive guide covers the latest advancements in garage technology, including automated systems, smart security features, energy efficiency solutions, and seamless integration with smart home ecosystems. Learn how these technologies are transforming traditional garages into high-tech, efficient spaces that enhance convenience, safety, and sustainability.
Ideal for homeowners, tech enthusiasts, and industry professionals, this presentation provides valuable insights into the trends, benefits, and future developments in smart garage technology. Stay ahead of the curve with our expert analysis and practical tips on implementing smart garage solutions.
More Related Content
Similar to Hijacking Treejack UXPA 2023 Talk.pptx.pdf
1. Always question what they say There’s a saying among researchers, ‘Do a quant survey with 100 people and learn ten things. Do a qual study with ten people and learn 100 things.’ You will learn when research is only providing you a false sense of security.
2. How to be a better consumer of research results If these insights were rabbits, how do stakeholders know which insights (rabbits) are worth chasing? What leads to the best business and design opportunities?
3. Get the best value from a qualitative and quantitative study In this Masterclass, we will be going over which methodology to apply to a research question. We will discuss how to weigh appropriately, act upon, or be critical of the resulting research findings.
4. Tools to help you Know when research is only providing you a false sense of security. Be a better consumer of research results.
5. And many more strategies Learn how to catch the best rabbits!
This presentation covers validation techniques for testing taxonomy and metadata with users. Four approaches are covered: Delphi card sorting, online card sorting, usability testing, and search term analysis. The presentation also contains a list of online card sorting tools.
Denver Startup Week 2019: Choosing a Direction Learning How to Test Ideas and...BrittanyRubinstein
As part of Denver's 2019 Startup Week, Crownpeak's Director of UX, Ari Weissman and Lys Maitland, Experience Research Manager at a national healthcare organization, presented a joint session on "Choosing a direction: Learning how to test ideas and designs."
UX 101: Making Great Human Experiences at Pittsburgh PodCamp 9Carol Smith
Carol Smith provides the tools you need to get started doing User Experience (UX) work right away. She introduces three quick and inexpensive UX research methods that will provide you with rich information about users and designs: interviews; card sorting; and usability testing. You'll learn how this work will influence your design and ways to effectively share and communicate what has been learned to increase stakeholders understandings of customers.
EffectiveUI's Ari Weissman (Lead Experience Architect) and Lys Maitland (Senior Experience Planner) spoke at Denver Startup Week 2016. Discussion description:
Test early, test often.
It’s a mantra that’s been proven successful time and again when it comes to innovation and design. So why aren’t you doing it? In the start-up world, when everything is moving so quickly, it can be easy to overlook or postpone collecting feedback from real people because of cost, time, or lack of preparation. Don’t let those things stop you. Valid data can be captured cheaply, quickly, and with half-finished products and strategies.
This talk will cover:
What is user testing and why is it important
How to plan for user testing
What are ways to make testing cheaper
What are ways to make testing quicker
How to test with different fidelities of concept and design
How to collect data more frequently
Opportunities for getting the whole team engaged
What to do with the insights/outcomes of research
1. Test Assumptions, Not Ideas The Lean Startup popularized the idea of testing the assumptions that need to be true in order for your ideas to work.
2. Build it and test it It can be challenging, however, to see our own assumptions let alone test them.
3. Discovery process In this Masterclass, you’ll learn a structured approach for how to take an idea, break it down into its underlying assumptions, and quickly prioritize the ones that need to be tested.
4. Tools to help you Finally, you’ll learn a simple framework for how to design fast and better.
5. And many more strategies Such as effective assumption tests so that you can quickly identify what to build.
Providing a compelling user experience is pivotal to developing a successful product. As a product manager, you are often tasked with difficult decisions that require a deep understanding of customer needs and how to deliver the best experience possible. User research is an effective way to both generate insights and validate direction.
In this workshop you will learn:
* The skills to effectively integrate user research into the product development process with a strong return on investment.
* How foundational user research can help product teams understand user goals, generate insights, and narrow focus.
* How to use research to evaluate and iterate on product concepts.
* How to validate design and product decisions to ready your product for launch.
This presentation grew out of my experience working with an information architecture that was incomplete and lacked generative research to understand the users. I was tasked with finding ways to validate the IA while providing useful user insight for future design changes. To meet the goal, I studied card sorts in depth and developed useful techniques for problems that often arise but are seldom discussed.
I also explored the use of other IA validation techniques such as task-based tree tests and click tests to complement and cross-validate card sort findings. Through the use of mixed methodology within-subject study design, card sort data can be more easily understood and validated by the outcomes of tree tests and click tests.
This presentation will provide practitioners with information about card sorting in advanced situations. Specifically it will cover card sorting when applied to complex hierarchies, multiple audiences, and information architecture validation.
Presented by Caron Garstka at UXPA 2018
Getting Started With User Research, Presented at Agile2010Carol Smith
The gap between a good product and a great one can be bridged by understanding your users.
This presentation shared how better systems are built by taking small, iterative steps to understand the users desires, needs and abilities.
Attendees will learn how to get information about users quickly and cheaply. For those that have more time (and perhaps a small budget) Carol introduced methods to use to get more detailed information from your users. Carol also introduced ways to effectively share and communicate this information.
Getting Started with User Research - Stir Trek 2011Carol Smith
Presented at Stir Trek: Thor Edition, in Columbus, Ohio on May 6, 2011.
Once you know who uses your product, all sorts of new questions start to emerge. How are they using the product? Why are they using it? What else might they want? In this session you will learn about three quick and easy methods to understand the users desires, needs and abilities. The basics of observations, interviews and card sorting will be covered. You will also learn ways to effectively share and communicate what you learn with your team.
Leveraging the Latest in Brain Science to Deliver the Next Generation of E-Le...Human Capital Media
With the explosion in brain research during the past 10 years, scientists know more about the brain than ever before. Consumer brain training software designed to improve memory, attention, cognitive ability and more has hit the market with a vengeance. How is this same research set to change the world of corporate learning and e-learning solutions? What does the next generation of e-learning look like? Join Dr. Alice Kim from the Rotman Research Institute, a premier international center for the study of human brain, along with Carol Leaman, the CEO of Axonify, to learn about:
Three key areas of brain research that are expected to have the most impact on corporate e-learning.
How the latest research is being applied in e-learning tools in the workplace.
Real corporate examples of improvements in knowledge retention, learning transfer and the bottom line.
Similar to Hijacking Treejack UXPA 2023 Talk.pptx.pdf (20)
Dive into the innovative world of smart garages with our insightful presentation, "Exploring the Future of Smart Garages." This comprehensive guide covers the latest advancements in garage technology, including automated systems, smart security features, energy efficiency solutions, and seamless integration with smart home ecosystems. Learn how these technologies are transforming traditional garages into high-tech, efficient spaces that enhance convenience, safety, and sustainability.
Ideal for homeowners, tech enthusiasts, and industry professionals, this presentation provides valuable insights into the trends, benefits, and future developments in smart garage technology. Stay ahead of the curve with our expert analysis and practical tips on implementing smart garage solutions.
You could be a professional graphic designer and still make mistakes. There is always the possibility of human error. On the other hand if you’re not a designer, the chances of making some common graphic design mistakes are even higher. Because you don’t know what you don’t know. That’s where this blog comes in. To make your job easier and help you create better designs, we have put together a list of common graphic design mistakes that you need to avoid.
Expert Accessory Dwelling Unit (ADU) Drafting ServicesResDraft
Whether you’re looking to create a guest house, a rental unit, or a private retreat, our experienced team will design a space that complements your existing home and maximizes your investment. We provide personalized, comprehensive expert accessory dwelling unit (ADU)drafting solutions tailored to your needs, ensuring a seamless process from concept to completion.
Can AI do good? at 'offtheCanvas' India HCI preludeAlan Dix
Invited talk at 'offtheCanvas' IndiaHCI prelude, 29th June 2024.
https://www.alandix.com/academic/talks/offtheCanvas-IndiaHCI2024/
The world is being changed fundamentally by AI and we are constantly faced with newspaper headlines about its harmful effects. However, there is also the potential to both ameliorate theses harms and use the new abilities of AI to transform society for the good. Can you make the difference?
Top 5 Indian Style Modular Kitchen DesignsFinzo Kitchens
Get the perfect modular kitchen in Gurgaon at Finzo! We offer high-quality, custom-designed kitchens at the best prices. Wardrobes and home & office furniture are also available. Free consultation! Best Quality Luxury Modular kitchen in Gurgaon available at best price. All types of Modular Kitchens are available U Shaped Modular kitchens, L Shaped Modular Kitchen, G Shaped Modular Kitchens, Inline Modular Kitchens and Italian Modular Kitchen.
1. Highjacking Treejack
Using reverse card-sort to determine
both the feature names and the
categories used to cluster those features -
an iterative process
2. About the speakers
2
PAUL JAYE
Senior Director, User
Research
SHANNON HALGREN, PhD
President + Sr UX Consultant
3. This was a team effort!
PAUL JAYE
Senior Director, User Research
SHANNON HALGREN, PhD
President + Sr UX|HF Consultant
AGATHA KALINCHENKO
Sr UX|HF Consultant
BILL GOINS
Director of Messaging
and Content
3
4. About Sage Research + Design
We are a team of experienced user experience (UX) and human factors (HF)
consultants who help our clients create effective products with positive user
experiences across a wide range of industries
4
5. About Sage Research + Design
5
We offer a wide variety of user-experience research and design services, but our
small size keeps us flexible to meet the individualized needs of our clients.
Some of our UX Services
9. 14,600+ Indeedians worldwide are
working to help people get jobs.
Over 450 UX professionals,
including designers, researchers,
content designers, and design
engineers ensure that the user is
part of every conversation.
Want to know more? Check out indeed.com/about for more about Indeed
and indeed.design to learn about design at Indeed.
10. We are a great team!
10
+
● Over 7 years of partnership
● Nearly 100 research studies!
○ Remote Usability Testing
○ Rapid Iterative Testing
○ Journey Mapping
○ Diary Studies
○ Card Sort/Treejack Testing
● Over 20 product teams
● Over 2,000 participants!
11. But this research effort was our BIGGEST EVER!
11
● Over 2 years
● Countless meetings + hours
● 13 research projects
● Mostly card sort and tree testing methods
+
12. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks
13. What is card sorting?
13
Card sorting uncovers users’
mental models, revealing the
way they think about items and
categories
17. Card sort variations
17
Open Sort
Users can create and
name any categories that
they choose
Closed Sort
Users must sort cards
into predetermined
categories
Hybrid Sort
One or more categories
are pre-determined, but
users can create and
name any number of new
categories they choose
18. 18
Example card sort data
Lettuce
Spinach
Apples
Bananas
Peaches
Plumbs
Mangos
Pineapple
Coconut
Kiwi
Peanuts
Pretzels
Similarity Matrix
Provides a quantitative
measurement of the
similarity between two
items
These 2 items are highly similar
These 2 items are not very similar
19. 19
Example card sort data
Pickles
Salsa
Sushi ginger
Hummus
Tapioca
Baking Powder
Black beans
Navy beans
Lettuce
Carrots
Apples
Bananas
Peaches
Plumbs
Hotdogs
Hamburgers
Popcorn
Pretzels
Peanuts
Almonds
Dendogram
A visual representation
of the output of a cluster
analysis
Data-based groupings
are represented by color
By looking at the graph
at different points along
the X-axis, a different
number of groupings is
presented
20. What is Treejack testing?
20
“Tree testing helps you
evaluate the findability and
hierarchy of content on your
website, app or intranet.”
- Optimal Workshop
21. What is Treejack testing?
21
Helps answer questions like:
● Do my labels make sense to
people?
● Is my content grouped
logically to people?
● Can people find the
information they want easily
and quickly? If not, what’s
stopping them?
22. Example Treejack
task
1. Task is
presented
2. Top-level menu
is presented
3. User navigates to
content
described in task
26. 26
Example Treejack data
The correct path was the
most popular
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
27. 27
Example Treejack data
But, several took the wrong path
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
28. 28
Example Treejack data
Many went directly to the
correct location
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
29. 29
Example Treejack data
Some were in the correct
location, but then backed out
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
30. 30
Example Treejack data
Some realized they took the
wrong path and went back
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
31. 31
Example Treejack data
Some thought they had found
the correct location here
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles
32. What is Treejack testing?
32
Most commonly, tree testing is
performed as a validation after
the content has been named and
organized
But, in the world of UX research,
sometimes we need to adapt to
uncommon use cases… and
sometimes you can can take
tools and methods and do
interesting things with them!
34. Indeed naming project
34
Goal: Create a set of names for current Indeed products and features and develop a
naming philosophy such that we can create coherent brand identity for all new
feature and product launches
Key Result: Names are understandable and connect to product functionality
35. Indeed naming project
35
Key Result: Names are understandable and connect to product functionality
How do we measure this?
36. Indeed naming project
36
Key Result: Names are understandable and connect to product functionality
How do we measure this?
Tree testing!
Questions Treejack studies help answer:
Do my labels make sense to people?
42. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
1 2 3 4
42
43. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
Name: “Pickle”
Descriptor: “A small cucumber
preserved in vinegar, brine, or a
similar solution”
1 2 3 4
43
44. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
1 2 3 4
Results
High variability
Some themes
No strong winner 44
45. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack Usability Test
1 2 3 4
Results
High variability
Some themes
No strong winner 45
46. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack Usability Test
1 2 3 4
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 46
47. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack
3 categorizations
3 Treejack studies
Names
300 users
Usability Test
1 2 3 4
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 47
48. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack
3 categorizations
3 Treejack studies
Names
300 users
Usability Test
1 2 3 4
Results
All categorizations
performed poorly
This is challenging!
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 48
49. 49
Back to the drawing board…
What themes have emerged?
User groups have differences!
50. Phase 2 Research - From the beginning, with user groups!
Card Sort #1
User Group #1
Open
Names
30 users
Card Sort #2
User Group #2
Open
Names
30 users
Card Sort #3
User Group #3
Open
Names
30 users
1 2 3
50
55. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks
56. The research project impact
56
● Indeed settled on a proven new way of categorizing all of its products
● Indeed validated a new naming philosophy that appropriately names (and doesn’t
name) products now and into the future
● Indeed is finalizing market testing for the launch of its first new set of product and
feature names
57. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks
59. 59
Unmoderated Testing PRO
A useful way to collect large
quantities of data without the time
and monetary costs of moderated
testing
Use moderated pilot sessions
Hack #1
Unmoderated Testing CON
No moderator to witness testing tool
usability, technical issues, or a user’s
confusion regarding the instructions
or activity
The Solution? Moderated Pilot Sessions
Treejack testing is typically conducted during unmoderated sessions.
Unmoderated testing, however, comes with some pros and cons
60. 60
Use moderated pilot sessions
Hack #1
Why pilot test?
Work iteratively between sessions to resolve and refine
Uncover programming bugs
or mistakes that would have
had serious data collection
consequences if not fixed
Uncover confusing
instructions, tasks, or
menu item descriptors
Uncover unexpected
usability issues with the
tool
61. 61
Use moderated pilot sessions
Hack #1
Why pilot test?
Uncover programming bugs
or mistakes that would have
had serious data collection
consequences if not fixed
Uncover confusing
instructions, tasks, or
menu item descriptors
Uncover unexpected
usability issues with the
tool
One Treejack usability issues you
should anticipate…
Work iteratively between sessions to resolve and refine
62. 62
Use moderated pilot sessions
Hack #1
Where would you go to find pickles?
What this means:
X User is forced to make a
selection they know is
wrong
X User is prohibited from
exploring the menu structure
naturally which will affect
overall task performance
Some users did not know they could backtrack in the menus
63. 63
Use moderated pilot sessions
Hack #1
Welcome!
Welcome to this Treejack study, and thank you for agreeing to
participate!
The activity shouldn't take longer than 10 to 15 minutes to
complete.
Your response will help us to organize the content on our website.
Find out how on the next page...
Instructions
Here's how it works:
1. You will be asked to find a certain item and presented with a list of links.
2. Click through the list until you arrive at one that you think helps you complete
the task.
3. If you take a wrong turn, you can go back by clicking one of the links above.
This is not a test of your ability, there are no right or wrong answers.
That's it, let's get started!
Things we tried:
1. Addressing the issue in the instructions
64. 64
Use moderated pilot sessions
Hack #1
Welcome! Instructions
Things we tried:
1. Addressing the issue in the instructions
“Go back by clicking on a previous
level. Your browser’s back button
will NOT work.”
65. 65
Use moderated pilot sessions
Hack #1
Things we tried:
2. Providing a training video
Instructions
67. 67
Include practice questions
Hack #2
Include practice questions to allow users to learn how to use the Treejack
tool and to get introduced to the menu structure
69. Include practice questions
Hack #2
Task 1 of 20
Practice Task: Where would you go to find pickles?
Practice Answer:
Click Condiments
Click Pickles
Click the green “I’d find it here box”
Tip: Click on Home or a previous level to go back. You’re browser’s button with
not work.
Task 1 of 20
Let’s try a warm up (unscored) task to allow you to explore the menus first.
Practice Task: Where would you go to find pickles?
Practice Answer:
Click Condiments
Click Pickles
Click the green “I’d find it here box”
Tip: Click on Home or a previous level to go back. You’re browser’s button with
not work.
Things we tried:
1. Walk users through a practice session
70. 70
Include practice questions
Hack #2
70
Task 1 of 20
Let’s start by getting you oriented to this menu structure.
There are 4 levels to this menu. Take a few minute to go back and forth between these
4 levels and see what options there are to select.
- When you are done exploring, Please go and find the following menu item:
“Pickles”
- When you have found it, click the green “I’d find it here box”
Tip #1: If you are struggling to find the exact word, keep on exploring and digging
through ALL the menu levels.
Tip #2: Click on Home or a previous level to go back.
Task 3 of 20
Let’s try a warm up (unscored) task to allow you to explore the menus first.
Practice Task: Where would you go to find greek yogurt?
(if needed) Cheatsheet:
Click Dairy
Click Yogurt
Click Greek Yogurt
Click the green “I’d find it here box”
Tip #1: If you are struggling to find the exact word, keep on exploring and digging
through ALL the menu levels.
Tip #2: Click on Home or a previous level to go back.
Things we tried:
2. Provide a directive to explore 3. Provide multiple practice questions
71. 71
Include practice questions
Hack #2
Tips for using practice questions:
1. Remember to remove the practice task(s) from your dataset!
○ If the practice tasks are kept in the same Treejack study as the
test tasks, this removal process must be done manually outside
of Optimal Workshop
2. To save time, place practice tasks in a separate study from the test
tasks and link the two studies together
74. 74
Consider headers cannot be targets
Hack #3
● One unfortunate limitation of the Treejack tool is that menu headers
cannot be end destinations
We cared about this and this in our studies
Where would you go
to find toppings for
your hamburger?
Where would you go
to find pickles?
75. 75
Consider headers cannot be targets
Hack #3
Things we tried:
1. Duplicating the header as the first item in its group
Where would you go
to find toppings for
your hamburger?
76. 76
Consider headers cannot be targets
Hack #3
Things we tried:
2. Create a separate Treejack study for each level of the hierarchy.
77. 77
Consider headers cannot be targets
Hack #3
Treejack #1a:
Level 1
Treejack #1b:
Level 2
Where would you go to find
toppings for your hamburger?
Where would you go
to find pickles?
79. 79
Summary
● A multi-phased approach was utilized to develop and
validate a new data-driven Indeed categorization and set of
product names - simultaneously
● We pushed the card sort and tree test methodologies to
the edge to meet our research needs and key objectives
● Along the way we were forced to discover several Treejack
Hacks
○ Use moderated pilot sessions
○ Include practice questions
○ Consider headers cannot be targets