Diving Deeper into
Data-Driven
Product Design
Graham Davis
Sr. Product Manager

Donald Wysocki
Product Director
I help
people
get
jobs.
Our guiding principle
Do what's best
for job seekers
Simple
Fast
Comprehensive
Relevant
Data-Driven Product Design
Data-Driven Product Design
1. Vision
Data-Driven Product Design
1. Vision
2. V1
Data-Driven Product Design
1. Vision
2. V1
3. Test and measure
Data-Driven Product Design
1.
2.
3.
4.

Vision
V1
Test and measure
and test and measure and...
Company Reviews
Graham Davis
Indeed Apply
In the past 12 months...
500 million jobs
on Indeed
39 billion job searches
6 billion jobs viewed
Estimated
720 million applications
~20 minutes to apply
10,000,000 days
How it works
Applications
● Email
Interested parties, please email your resume
with the subject line "PHP Web Developer" to

jobs@smallworldlabs.com
Applications
● Email
● Simple resume upload
Applications
● Email
● Simple resume upload
● Account + complex forms
Applications
●
●
●
●

Email
Simple resume upload
Account + complex forms
Mobile
Applications
●
●
●
●

Email
Simple resume upload
Account + complex forms
Mobile
Applications
●
●
●
●

Email
Simple resume upload
Account + complex forms
Mobile
Vision
Easily apply to all jobs
on all devices.
So, where do we start?
Nearly every job
application was completed
on another site
2,000 jobs posted to Indeed
by small businesses
V1: How can we
make it easier?
What if job seekers who
previously applied could reuse their resume?
-10%
0%
Why?
Maybe job seekers didn't
understand they have a saved
resume
Can an Indeed Resume make
it easier to apply?
Applications Per Week
+0%
For job seekers who are
signed in & have an Indeed
Resume, the application rate
increased by 2x
Promising, but...
2x a small number is still a
small number
We need more jobs
Opportunities
Applications
● Email
● Simple resume upload
● Mobile
Applications
● Email
● Simple resume upload
● Mobile
Jobs
9,500
5,000

Posted
on Indeed

Apply
with Email
A new test:
1. US only
2. Signed-in
3. Indeed Resume
Applications Per Week
+33%
Can we make this available to
more job seekers?
Expanded test:
1. US only
2. Signed-in
3. Indeed Resume
Total Job Applications
+70%
Can we get more jobs?
Expand Internationally
207,000

14,500
US

INTL
Applications per Week
+58%
Can we get even more jobs?
Applications
● Email
● Simple resume upload
● Mobile
Applications
● Email
● Simple resume upload
● Mobile
Apply on the
original website
Trusted,
Everywhere
First (and fast) partner test:
ZipRecruiter
Applications from Indeed
+109%
More applications from the
same number of job seekers
How can we make it easier
for job seekers to find these
jobs?
+2% applications
Does branding actually help
job seekers?
+6% applications
Progress:
Easier apply process
More jobs
Easier to find
●

●

●
Applications
● Email
● Simple resume upload
● Mobile
Applications
● Email
● Simple resume upload
● Mobile
The big win...
> 1/3 of searches

Mobile
Web
<5% of the Fortune 500
accept applications on
their mobile sites
3rd party apply
mobile apply
more resumes
more posted jobs...
Applications per Week

+584%
Data driven design isn't just
about numbers and charts...
I help
people
get
jobs.
Company Reviews
Graham Davis
I help
people
get
jobs.
I help
people
get < the best
jobs.
What is it like to work at...
typical day
typical day
management
typical day
management
co-workers
647
important decision
opportunity to help
job seekers
Vision
Help job seekers find the
best employers.
3,000,000
0
Jan 2011: V1
Not perfect
We don't know anything
yet
Good enough
to start testing
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
Remove anything that's
not absolutely necessary
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
●
●
●
●
●
●
●
●
●
●
●

"write a review" button
review form
search for companies
see reviews on company page
see reviews on...
Built it
August 2011: 8,000
August 2011: 8,000
only 0.2% of jobs
August 2011: 8,000
only 0.2% of jobs
1 of every 50 search pages
August 2011: 8,000
only 0.2% of jobs
1 of every 50 search pages
NOT good enough
Goal:
every search page
500,000
300
10,000
300
10,000

+3000%
300
So we started testing
Why aren't people
submitting reviews?
How can we make it
easier?
What can we learn from
successful review sites?
+27%
387
What can the data tell us?
How can we improve
company search?
-45%
Step back and assess
What works for one site
does not necessarily work
for another site
Indeed is not a reviews site
Indeed is a job site
How can we engage our job
seekers where they are?
The flow: Oct 2011
What can the data tell us?
2 ways to improve a funnel
2 ways to improve a funnel
Improve conversion at step
2 ways to improve a funnel
Improve conversion at step
Remove step entirely
Can we remove steps?
Yes,
if we target job seekers with
Indeed resumes.
New flow: Nov 2011
+343%
1,507
What can the data tell us?
Avg reviews per reviewer: 1
Avg reviews per reviewer: 1
Avg jobs per resume: 3.6
How can we increase
reviews per reviewer?
Before
After
+46%
3,719
1.5
Avg reviews per reviewer: 1
Avg jobs per resume: 3.6
why not 3.6?
1.5
Avg reviews per reviewer: 1
Avg jobs per resume: 3.6
What can the data tell us?
Indeed
?
Zoominfo

42%
Indeed
My Perfect Gig

Zoominfo
+57%
5,834
What would happen if we
had more targeted
promos for job seekers
with resumes?
+36%
7,919
Can we remove more steps?
Before
After
+104%
13,010
Can we remove anything
from the review form?
Ratings
Review Title
Review
Pros
Cons
Job Title
Job Location
Start/End Date
Ratings
Review Title
Review
Pros
Cons
Job Title
Job Location
Start/End Date

Resume
+24%
17,168
Today...
3,000,000
1.7M companies
60,000 / week
60% of jobs
6 out of 10
we moved the needle
more by removing than
adding
Simple
Perfection is finally attained
not when there is no longer
anything to add, but when
there is no longer anything to
take a...
Q&A
Donald Wysocki
donald@indeed.com

Graham Davis
gdavis@indeed.com
Next @IndeedEng Talk
Engineering Velocity: Building Great
Software Through Fast Iteration
Dan Heller, Engineering Manager
...
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
[@IndeedEng Talk] Diving deeper into data-driven product design
Upcoming SlideShare
Loading in …5
×

[@IndeedEng Talk] Diving deeper into data-driven product design

6,595
-1

Published on

Video available at: http://www.youtube.com/watch?v=i8MGTZ3KWmc

At April’s @IndeedEng Talk we introduced Indeed’s philosophy and practice of A/B testing. In this talk, two Indeed product managers will discuss how we used data-driven opportunity analysis and iterative testing to build two products. From product vision to product success, we’ll describe what we tested, how it performed, and what we learned from it. Product managers, designers, and engineers who want to learn how to prioritize product and feature ideas, iterate through tests, or optimize a funnel will find valuable insights to apply to their own products.

Graham Davis is a Senior Product Manager for Employer Products at Indeed. Prior to Indeed, Graham previously worked for several startups and got an MBA from Harvard Business School.

Donald Wysocki is Product Director for Job Search at Indeed. Prior to Indeed, Donald worked at frog design and Microsoft.

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,595
On Slideshare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
33
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

[@IndeedEng Talk] Diving deeper into data-driven product design

  1. 1. Diving Deeper into Data-Driven Product Design Graham Davis Sr. Product Manager Donald Wysocki Product Director
  2. 2. I help people get jobs.
  3. 3. Our guiding principle
  4. 4. Do what's best for job seekers
  5. 5. Simple
  6. 6. Fast
  7. 7. Comprehensive
  8. 8. Relevant
  9. 9. Data-Driven Product Design
  10. 10. Data-Driven Product Design 1. Vision
  11. 11. Data-Driven Product Design 1. Vision 2. V1
  12. 12. Data-Driven Product Design 1. Vision 2. V1 3. Test and measure
  13. 13. Data-Driven Product Design 1. 2. 3. 4. Vision V1 Test and measure and test and measure and...
  14. 14. Company Reviews Graham Davis
  15. 15. Indeed Apply
  16. 16. In the past 12 months...
  17. 17. 500 million jobs on Indeed
  18. 18. 39 billion job searches
  19. 19. 6 billion jobs viewed
  20. 20. Estimated 720 million applications
  21. 21. ~20 minutes to apply
  22. 22. 10,000,000 days
  23. 23. How it works
  24. 24. Applications ● Email
  25. 25. Interested parties, please email your resume with the subject line "PHP Web Developer" to jobs@smallworldlabs.com
  26. 26. Applications ● Email ● Simple resume upload
  27. 27. Applications ● Email ● Simple resume upload ● Account + complex forms
  28. 28. Applications ● ● ● ● Email Simple resume upload Account + complex forms Mobile
  29. 29. Applications ● ● ● ● Email Simple resume upload Account + complex forms Mobile
  30. 30. Applications ● ● ● ● Email Simple resume upload Account + complex forms Mobile
  31. 31. Vision Easily apply to all jobs on all devices.
  32. 32. So, where do we start?
  33. 33. Nearly every job application was completed on another site
  34. 34. 2,000 jobs posted to Indeed by small businesses
  35. 35. V1: How can we make it easier?
  36. 36. What if job seekers who previously applied could reuse their resume?
  37. 37. -10%
  38. 38. 0%
  39. 39. Why?
  40. 40. Maybe job seekers didn't understand they have a saved resume
  41. 41. Can an Indeed Resume make it easier to apply?
  42. 42. Applications Per Week +0%
  43. 43. For job seekers who are signed in & have an Indeed Resume, the application rate increased by 2x
  44. 44. Promising, but...
  45. 45. 2x a small number is still a small number
  46. 46. We need more jobs
  47. 47. Opportunities
  48. 48. Applications ● Email ● Simple resume upload ● Mobile
  49. 49. Applications ● Email ● Simple resume upload ● Mobile
  50. 50. Jobs 9,500 5,000 Posted on Indeed Apply with Email
  51. 51. A new test: 1. US only 2. Signed-in 3. Indeed Resume
  52. 52. Applications Per Week +33%
  53. 53. Can we make this available to more job seekers?
  54. 54. Expanded test: 1. US only 2. Signed-in 3. Indeed Resume
  55. 55. Total Job Applications +70%
  56. 56. Can we get more jobs?
  57. 57. Expand Internationally 207,000 14,500 US INTL
  58. 58. Applications per Week +58%
  59. 59. Can we get even more jobs?
  60. 60. Applications ● Email ● Simple resume upload ● Mobile
  61. 61. Applications ● Email ● Simple resume upload ● Mobile
  62. 62. Apply on the original website
  63. 63. Trusted, Everywhere
  64. 64. First (and fast) partner test: ZipRecruiter
  65. 65. Applications from Indeed +109%
  66. 66. More applications from the same number of job seekers
  67. 67. How can we make it easier for job seekers to find these jobs?
  68. 68. +2% applications
  69. 69. Does branding actually help job seekers?
  70. 70. +6% applications
  71. 71. Progress: Easier apply process More jobs Easier to find ● ● ●
  72. 72. Applications ● Email ● Simple resume upload ● Mobile
  73. 73. Applications ● Email ● Simple resume upload ● Mobile
  74. 74. The big win...
  75. 75. > 1/3 of searches Mobile Web
  76. 76. <5% of the Fortune 500 accept applications on their mobile sites
  77. 77. 3rd party apply mobile apply more resumes more posted jobs...
  78. 78. Applications per Week +584%
  79. 79. Data driven design isn't just about numbers and charts...
  80. 80. I help people get jobs.
  81. 81. Company Reviews Graham Davis
  82. 82. I help people get jobs.
  83. 83. I help people get < the best jobs.
  84. 84. What is it like to work at...
  85. 85. typical day
  86. 86. typical day management
  87. 87. typical day management co-workers
  88. 88. 647
  89. 89. important decision
  90. 90. opportunity to help job seekers
  91. 91. Vision Help job seekers find the best employers.
  92. 92. 3,000,000
  93. 93. 0
  94. 94. Jan 2011: V1
  95. 95. Not perfect
  96. 96. We don't know anything yet
  97. 97. Good enough to start testing
  98. 98. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  99. 99. Remove anything that's not absolutely necessary
  100. 100. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  101. 101. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  102. 102. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  103. 103. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  104. 104. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  105. 105. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  106. 106. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  107. 107. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  108. 108. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  109. 109. ● ● ● ● ● ● ● ● ● ● ● "write a review" button review form search for companies see reviews on company page see reviews on job pages see reviews on search results best places to work filter by location filter by job function review commenting review voting
  110. 110. Built it
  111. 111. August 2011: 8,000
  112. 112. August 2011: 8,000 only 0.2% of jobs
  113. 113. August 2011: 8,000 only 0.2% of jobs 1 of every 50 search pages
  114. 114. August 2011: 8,000 only 0.2% of jobs 1 of every 50 search pages NOT good enough
  115. 115. Goal: every search page
  116. 116. 500,000
  117. 117. 300
  118. 118. 10,000 300
  119. 119. 10,000 +3000% 300
  120. 120. So we started testing
  121. 121. Why aren't people submitting reviews?
  122. 122. How can we make it easier?
  123. 123. What can we learn from successful review sites?
  124. 124. +27% 387
  125. 125. What can the data tell us?
  126. 126. How can we improve company search?
  127. 127. -45%
  128. 128. Step back and assess
  129. 129. What works for one site does not necessarily work for another site
  130. 130. Indeed is not a reviews site
  131. 131. Indeed is a job site
  132. 132. How can we engage our job seekers where they are?
  133. 133. The flow: Oct 2011
  134. 134. What can the data tell us?
  135. 135. 2 ways to improve a funnel
  136. 136. 2 ways to improve a funnel Improve conversion at step
  137. 137. 2 ways to improve a funnel Improve conversion at step Remove step entirely
  138. 138. Can we remove steps?
  139. 139. Yes, if we target job seekers with Indeed resumes.
  140. 140. New flow: Nov 2011
  141. 141. +343% 1,507
  142. 142. What can the data tell us?
  143. 143. Avg reviews per reviewer: 1
  144. 144. Avg reviews per reviewer: 1 Avg jobs per resume: 3.6
  145. 145. How can we increase reviews per reviewer?
  146. 146. Before
  147. 147. After
  148. 148. +46% 3,719
  149. 149. 1.5 Avg reviews per reviewer: 1 Avg jobs per resume: 3.6
  150. 150. why not 3.6? 1.5 Avg reviews per reviewer: 1 Avg jobs per resume: 3.6
  151. 151. What can the data tell us?
  152. 152. Indeed ? Zoominfo 42%
  153. 153. Indeed My Perfect Gig Zoominfo
  154. 154. +57% 5,834
  155. 155. What would happen if we had more targeted promos for job seekers with resumes?
  156. 156. +36% 7,919
  157. 157. Can we remove more steps?
  158. 158. Before
  159. 159. After
  160. 160. +104% 13,010
  161. 161. Can we remove anything from the review form?
  162. 162. Ratings Review Title Review Pros Cons Job Title Job Location Start/End Date
  163. 163. Ratings Review Title Review Pros Cons Job Title Job Location Start/End Date Resume
  164. 164. +24% 17,168
  165. 165. Today...
  166. 166. 3,000,000
  167. 167. 1.7M companies
  168. 168. 60,000 / week
  169. 169. 60% of jobs
  170. 170. 6 out of 10
  171. 171. we moved the needle more by removing than adding
  172. 172. Simple
  173. 173. Perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away. Antoine de Saint Exupéry
  174. 174. Q&A Donald Wysocki donald@indeed.com Graham Davis gdavis@indeed.com
  175. 175. Next @IndeedEng Talk Engineering Velocity: Building Great Software Through Fast Iteration Dan Heller, Engineering Manager 7pm Wednesday, August 28th
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×