Talk Presented by Shannon Halgren at UXPA 2023 in Austin Texas. Session Abstract: What would you do if you were tasked with developing a new categorization for your company’s products and services? Under the best circumstances, this is a hefty undertaking; but, to add to the complexity, consider that your company is a massive hiring website with varied products and services, and they want to change the names of their products and services based on UX research that revealed the names are confusing. So, now you are tasked with establishing a new categorization before these new names are developed. This task was a recent research challenge of ours. In this presentation, we will discuss how we utilized card sorting and treejack testing to iteratively test categorizations. We will share our methodology and discuss how we overcame usability and technical limitations of the Optimal Workshop treejack tool. We will share our hard-earned tips and tricks for the audience to utilize in their own research.

1. Highjacking Treejack
Using reverse card-sort to determine
both the feature names and the
categories used to cluster those features -
an iterative process

2. About the speakers
2
PAUL JAYE
Senior Director, User
Research
SHANNON HALGREN, PhD
President + Sr UX Consultant

3. This was a team effort!
PAUL JAYE
Senior Director, User Research
SHANNON HALGREN, PhD
President + Sr UX|HF Consultant
AGATHA KALINCHENKO
Sr UX|HF Consultant
BILL GOINS
Director of Messaging
and Content
3

4. About Sage Research + Design
We are a team of experienced user experience (UX) and human factors (HF)
consultants who help our clients create effective products with positive user
experiences across a wide range of industries
4

5. About Sage Research + Design
5
We offer a wide variety of user-experience research and design services, but our
small size keeps us flexible to meet the individualized needs of our clients.
Some of our UX Services

6. The Sage team
6

7. About Indeed

9. 14,600+ Indeedians worldwide are
working to help people get jobs.
Over 450 UX professionals,
including designers, researchers,
content designers, and design
engineers ensure that the user is
part of every conversation.
Want to know more? Check out indeed.com/about for more about Indeed
and indeed.design to learn about design at Indeed.

10. We are a great team!
10
+
● Over 7 years of partnership
● Nearly 100 research studies!
○ Remote Usability Testing
○ Rapid Iterative Testing
○ Journey Mapping
○ Diary Studies
○ Card Sort/Treejack Testing
● Over 20 product teams
● Over 2,000 participants!

11. But this research effort was our BIGGEST EVER!
11
● Over 2 years
● Countless meetings + hours
● 13 research projects
● Mostly card sort and tree testing methods
+

12. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks

13. What is card sorting?
13
Card sorting uncovers users’
mental models, revealing the
way they think about items and
categories

14. Example card sort
14
Cookies
Lettuce
Eggs
BBQ
Sauce
Carrots
Yogurt
Garlic
Chips
Apples
Squash
Celery
Popcorn

15. Example card sort
15
Cookies
Lettuce
Eggs
BBQ
Sauce
Carrots
Yogurt
Garlic
Produce
Apples
Squash
Celery
Cookies
Lettuce
Eggs
BBQ
Sauce
Carrots
Yogurt
Garlic
Chips
Squash Popcorn

16. Example card sort
16
Cookies
Lettuce
Eggs
Yogurt
Garlic
Produce
Apples
Squash
Celery
Cookies
Lettuce
Eggs
BBQ
Sauce
Carrots
Yogurt
Garlic
Chips
Squash Popcorn
Snacks
Chips Popcorn
Condiments

17. Card sort variations
17
Open Sort
Users can create and
name any categories that
they choose
Closed Sort
Users must sort cards
into predetermined
categories
Hybrid Sort
One or more categories
are pre-determined, but
users can create and
name any number of new
categories they choose

18. 18
Example card sort data
Lettuce
Spinach
Apples
Bananas
Peaches
Plumbs
Mangos
Pineapple
Coconut
Kiwi
Peanuts
Pretzels
Similarity Matrix
Provides a quantitative
measurement of the
similarity between two
items
These 2 items are highly similar
These 2 items are not very similar

19. 19
Example card sort data
Pickles
Salsa
Sushi ginger
Hummus
Tapioca
Baking Powder
Black beans
Navy beans
Lettuce
Carrots
Apples
Bananas
Peaches
Plumbs
Hotdogs
Hamburgers
Popcorn
Pretzels
Peanuts
Almonds
Dendogram
A visual representation
of the output of a cluster
analysis
Data-based groupings
are represented by color
By looking at the graph
at different points along
the X-axis, a different
number of groupings is
presented

20. What is Treejack testing?
20
“Tree testing helps you
evaluate the findability and
hierarchy of content on your
website, app or intranet.”
- Optimal Workshop

21. What is Treejack testing?
21
Helps answer questions like:
● Do my labels make sense to
people?
● Is my content grouped
logically to people?
● Can people find the
information they want easily
and quickly? If not, what’s
stopping them?

22. Example Treejack
task
1. Task is
presented
2. Top-level menu
is presented
3. User navigates to
content
described in task

23. Example Treejack task
23

24. 24
Example Treejack data

25. 25
Example Treejack data
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

26. 26
Example Treejack data
The correct path was the
most popular
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

27. 27
Example Treejack data
But, several took the wrong path
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

28. 28
Example Treejack data
Many went directly to the
correct location
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

29. 29
Example Treejack data
Some were in the correct
location, but then backed out
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

30. 30
Example Treejack data
Some realized they took the
wrong path and went back
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

31. 31
Example Treejack data
Some thought they had found
the correct location here
Task: Where would you go to find pickles?
Answer: Home > Condiments > Pickles

32. What is Treejack testing?
32
Most commonly, tree testing is
performed as a validation after
the content has been named and
organized
But, in the world of UX research,
sometimes we need to adapt to
uncommon use cases… and
sometimes you can can take
tools and methods and do
interesting things with them!

33. Outline
1. Treejack Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks

34. Indeed naming project
34
Goal: Create a set of names for current Indeed products and features and develop a
naming philosophy such that we can create coherent brand identity for all new
feature and product launches
Key Result: Names are understandable and connect to product functionality

35. Indeed naming project
35
Key Result: Names are understandable and connect to product functionality
How do we measure this?

36. Indeed naming project
36
Key Result: Names are understandable and connect to product functionality
How do we measure this?
Tree testing!
Questions Treejack studies help answer:
Do my labels make sense to people?

37. “
Let’s rethink our
categories.”
Chris Hyams
CEO, Indeed
Key Result: Categorizations are intuitive
and allow items to be easily
found

38. Names
Categories
38

39. Card sort to the rescue!
39

40. 40
Visibility was high and stakeholders had opinions

41. Phase 1 Research
Card Sort #1 Card Sort #2 Treejack Usability Test
1 2 3 4
41

42. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
1 2 3 4
42

43. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
Name: “Pickle”
Descriptor: “A small cucumber
preserved in vinegar, brine, or a
similar solution”
1 2 3 4
43

44. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2 Treejack Usability Test
1 2 3 4
Results
High variability
Some themes
No strong winner 44

45. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack Usability Test
1 2 3 4
Results
High variability
Some themes
No strong winner 45

46. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack Usability Test
1 2 3 4
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 46

47. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack
3 categorizations
3 Treejack studies
Names
300 users
Usability Test
1 2 3 4
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 47

48. Phase 1 Research
Card Sort #1
Open
Descriptors
40 users
Card Sort #2
Hybrid
Descriptors
40 users
Treejack
3 categorizations
3 Treejack studies
Names
300 users
Usability Test
1 2 3 4
Results
All categorizations
performed poorly
This is challenging!
Results
High variability
A “winning”
category emerged
Results
High variability
Some themes
No strong winner 48

49. 49
Back to the drawing board…
What themes have emerged?
User groups have differences!

50. Phase 2 Research - From the beginning, with user groups!
Card Sort #1
User Group #1
Open
Names
30 users
Card Sort #2
User Group #2
Open
Names
30 users
Card Sort #3
User Group #3
Open
Names
30 users
1 2 3
50

51. Phase 2 Results
51

52. Phase 3 Research - Back to Treejack!
Treejack #1 Treejack #2 Treejack #3 Treejack #4
1 2 3 4
52

53. Phase 3 Research - Back to Treejack!
Treejack #1
4 categorizations
2 user groups
8 Treejack studies
Names
160 users
Treejack #2
2 categorizations
2 user groups
4 Treejack studies
Descriptors
80 users
Treejack #3
2 categorizations
2 user group
4 Treejack studies
Names
40 users
Treejack #4
2 categorizations
1 user group
2 name variations
(long v short)
4 Treejack studies
120 users
1 2 3 4
53

54. The finish line!
54

55. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks

56. The research project impact
56
● Indeed settled on a proven new way of categorizing all of its products
● Indeed validated a new naming philosophy that appropriately names (and doesn’t
name) products now and into the future
● Indeed is finalizing market testing for the launch of its first new set of product and
feature names

57. Outline
1. Card Sort & Treejack
Introduction
2. The Research Project
3. The Impact
4. Treejack Hacks

58. 58
Use moderated pilot sessions
Hack #1

59. 59
Unmoderated Testing PRO
A useful way to collect large
quantities of data without the time
and monetary costs of moderated
testing
Use moderated pilot sessions
Hack #1
Unmoderated Testing CON
No moderator to witness testing tool
usability, technical issues, or a user’s
confusion regarding the instructions
or activity
The Solution? Moderated Pilot Sessions
Treejack testing is typically conducted during unmoderated sessions.
Unmoderated testing, however, comes with some pros and cons

60. 60
Use moderated pilot sessions
Hack #1
Why pilot test?
Work iteratively between sessions to resolve and refine
Uncover programming bugs
or mistakes that would have
had serious data collection
consequences if not fixed
Uncover confusing
instructions, tasks, or
menu item descriptors
Uncover unexpected
usability issues with the
tool

61. 61
Use moderated pilot sessions
Hack #1
Why pilot test?
Uncover programming bugs
or mistakes that would have
had serious data collection
consequences if not fixed
Uncover confusing
instructions, tasks, or
menu item descriptors
Uncover unexpected
usability issues with the
tool
One Treejack usability issues you
should anticipate…
Work iteratively between sessions to resolve and refine

62. 62
Use moderated pilot sessions
Hack #1
Where would you go to find pickles?
What this means:
X User is forced to make a
selection they know is
wrong
X User is prohibited from
exploring the menu structure
naturally which will affect
overall task performance
Some users did not know they could backtrack in the menus

63. 63
Use moderated pilot sessions
Hack #1
Welcome!
Welcome to this Treejack study, and thank you for agreeing to
participate!
The activity shouldn't take longer than 10 to 15 minutes to
complete.
Your response will help us to organize the content on our website.
Find out how on the next page...
Instructions
Here's how it works:
1. You will be asked to find a certain item and presented with a list of links.
2. Click through the list until you arrive at one that you think helps you complete
the task.
3. If you take a wrong turn, you can go back by clicking one of the links above.
This is not a test of your ability, there are no right or wrong answers.
That's it, let's get started!
Things we tried:
1. Addressing the issue in the instructions

64. 64
Use moderated pilot sessions
Hack #1
Welcome! Instructions
Things we tried:
1. Addressing the issue in the instructions
“Go back by clicking on a previous
level. Your browser’s back button
will NOT work.”

65. 65
Use moderated pilot sessions
Hack #1
Things we tried:
2. Providing a training video
Instructions

66. 66
Include practice questions
Hack #2

67. 67
Include practice questions
Hack #2
Include practice questions to allow users to learn how to use the Treejack
tool and to get introduced to the menu structure

68. 68
Include practice questions
Hack #2

69. Include practice questions
Hack #2
Task 1 of 20
Practice Task: Where would you go to find pickles?
Practice Answer:
Click Condiments
Click Pickles
Click the green “I’d find it here box”
Tip: Click on Home or a previous level to go back. You’re browser’s button with
not work.
Task 1 of 20
Let’s try a warm up (unscored) task to allow you to explore the menus first.
Practice Task: Where would you go to find pickles?
Practice Answer:
Click Condiments
Click Pickles
Click the green “I’d find it here box”
Tip: Click on Home or a previous level to go back. You’re browser’s button with
not work.
Things we tried:
1. Walk users through a practice session

70. 70
Include practice questions
Hack #2
70
Task 1 of 20
Let’s start by getting you oriented to this menu structure.
There are 4 levels to this menu. Take a few minute to go back and forth between these
4 levels and see what options there are to select.
- When you are done exploring, Please go and find the following menu item:
“Pickles”
- When you have found it, click the green “I’d find it here box”
Tip #1: If you are struggling to find the exact word, keep on exploring and digging
through ALL the menu levels.
Tip #2: Click on Home or a previous level to go back.
Task 3 of 20
Let’s try a warm up (unscored) task to allow you to explore the menus first.
Practice Task: Where would you go to find greek yogurt?
(if needed) Cheatsheet:
Click Dairy
Click Yogurt
Click Greek Yogurt
Click the green “I’d find it here box”
Tip #1: If you are struggling to find the exact word, keep on exploring and digging
through ALL the menu levels.
Tip #2: Click on Home or a previous level to go back.
Things we tried:
2. Provide a directive to explore 3. Provide multiple practice questions

71. 71
Include practice questions
Hack #2
Tips for using practice questions:
1. Remember to remove the practice task(s) from your dataset!
○ If the practice tasks are kept in the same Treejack study as the
test tasks, this removal process must be done manually outside
of Optimal Workshop
2. To save time, place practice tasks in a separate study from the test
tasks and link the two studies together

72. 72
Include practice questions
Hack #2

73. 73
Consider headers cannot be targets
Hack #3

74. 74
Consider headers cannot be targets
Hack #3
● One unfortunate limitation of the Treejack tool is that menu headers
cannot be end destinations
We cared about this and this in our studies
Where would you go
to find toppings for
your hamburger?
Where would you go
to find pickles?

75. 75
Consider headers cannot be targets
Hack #3
Things we tried:
1. Duplicating the header as the first item in its group
Where would you go
to find toppings for
your hamburger?

76. 76
Consider headers cannot be targets
Hack #3
Things we tried:
2. Create a separate Treejack study for each level of the hierarchy.

77. 77
Consider headers cannot be targets
Hack #3
Treejack #1a:
Level 1
Treejack #1b:
Level 2
Where would you go to find
toppings for your hamburger?
Where would you go
to find pickles?

78. 78
Consider headers cannot be targets
Hack #3

79. 79
Summary
● A multi-phased approach was utilized to develop and
validate a new data-driven Indeed categorization and set of
product names - simultaneously
● We pushed the card sort and tree test methodologies to
the edge to meet our research needs and key objectives
● Along the way we were forced to discover several Treejack
Hacks
○ Use moderated pilot sessions
○ Include practice questions
○ Consider headers cannot be targets

80. Thank you!
Shannon Halgren
shannon@sage-research.com
Paul Jaye
pjaye@indeed.com