Video available at: http://www.youtube.com/watch?v=i8MGTZ3KWmc At April’s @IndeedEng Talk we introduced Indeed’s philosophy and practice of A/B testing. In this talk, two Indeed product managers will discuss how we used data-driven opportunity analysis and iterative testing to build two products. From product vision to product success, we’ll describe what we tested, how it performed, and what we learned from it. Product managers, designers, and engineers who want to learn how to prioritize product and feature ideas, iterate through tests, or optimize a funnel will find valuable insights to apply to their own products. Graham Davis is a Senior Product Manager for Employer Products at Indeed. Prior to Indeed, Graham previously worked for several startups and got an MBA from Harvard Business School. Donald Wysocki is Product Director for Job Search at Indeed. Prior to Indeed, Donald worked at frog design and Microsoft.

1. Diving Deeper into
Data-Driven
Product Design
Graham Davis
Sr. Product Manager
Donald Wysocki
Product Director

2. I help
people
get
jobs.

3. Our guiding principle

4. Do what's best
for job seekers

7. Simple

8. Fast

9. Comprehensive

10. Relevant

11. Data-Driven Product Design

12. Data-Driven Product Design
1. Vision

13. Data-Driven Product Design
1. Vision
2. V1

14. Data-Driven Product Design
1. Vision
2. V1
3. Test and measure

15. Data-Driven Product Design
1.
2.
3.
4.
Vision
V1
Test and measure
and test and measure and...

16. Company Reviews
Graham Davis

17. Indeed Apply

18. In the past 12 months...

19. 500 million jobs
on Indeed

20. 39 billion job searches

21. 6 billion jobs viewed

22. Estimated
720 million applications

23. ~20 minutes to apply

24. 10,000,000 days

25. How it works

26. Applications
● Email

28. Interested parties, please email your resume
with the subject line "PHP Web Developer" to
jobs@smallworldlabs.com

29. Applications
● Email
● Simple resume upload

32. Applications
● Email
● Simple resume upload
● Account + complex forms

37. Applications
●
●
●
●
Email
Simple resume upload
Account + complex forms
Mobile

43. Applications
●
●
●
●
Email
Simple resume upload
Account + complex forms
Mobile

44. Applications
●
●
●
●
Email
Simple resume upload
Account + complex forms
Mobile

45. Vision
Easily apply to all jobs
on all devices.

46. So, where do we start?

47. Nearly every job
application was completed
on another site

48. 2,000 jobs posted to Indeed
by small businesses

50. V1: How can we
make it easier?

51. What if job seekers who
previously applied could reuse their resume?

54. -10%

55. 0%

56. Why?

57. Maybe job seekers didn't
understand they have a saved
resume

60. Can an Indeed Resume make
it easier to apply?

62. Applications Per Week
+0%

63. For job seekers who are
signed in & have an Indeed
Resume, the application rate
increased by 2x

64. Promising, but...

65. 2x a small number is still a
small number

66. We need more jobs

67. Opportunities

68. Applications
● Email
● Simple resume upload
● Mobile

69. Applications
● Email
● Simple resume upload
● Mobile

73. Jobs
9,500
5,000
Posted
on Indeed
Apply
with Email

74. A new test:
1. US only
2. Signed-in
3. Indeed Resume

77. Applications Per Week
+33%

78. Can we make this available to
more job seekers?

79. Expanded test:
1. US only
2. Signed-in
3. Indeed Resume

80. Total Job Applications
+70%

81. Can we get more jobs?

82. Expand Internationally
207,000
14,500
US
INTL

83. Applications per Week
+58%

84. Can we get even more jobs?

85. Applications
● Email
● Simple resume upload
● Mobile

86. Applications
● Email
● Simple resume upload
● Mobile

87. Apply on the
original website

90. Trusted,
Everywhere

91. First (and fast) partner test:
ZipRecruiter

93. Applications from Indeed
+109%

94. More applications from the
same number of job seekers

95. How can we make it easier
for job seekers to find these
jobs?

98. +2% applications

99. Does branding actually help
job seekers?

101. +6% applications

102. Progress:
Easier apply process
More jobs
Easier to find
●
●
●

103. Applications
● Email
● Simple resume upload
● Mobile

104. Applications
● Email
● Simple resume upload
● Mobile

105. The big win...

106. > 1/3 of searches
Mobile
Web

107. <5% of the Fortune 500
accept applications on
their mobile sites

112. 3rd party apply
mobile apply
more resumes
more posted jobs...

113. Applications per Week
+584%

114. Data driven design isn't just
about numbers and charts...

115. I help
people
get
jobs.

116. Company Reviews
Graham Davis

117. I help
people
get
jobs.

118. I help
people
get < the best
jobs.

120. What is it like to work at...

121. typical day

122. typical day
management

123. typical day
management
co-workers

125. 647

128. important decision

129. opportunity to help
job seekers

130. Vision
Help job seekers find the
best employers.

131. 3,000,000

132. 0

133. Jan 2011: V1

134. Not perfect

135. We don't know anything
yet

136. Good enough
to start testing

137. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

138. Remove anything that's
not absolutely necessary

139. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

140. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

141. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

142. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

143. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

144. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

145. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

146. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

147. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

148. ●
●
●
●
●
●
●
●
●
●
●
"write a review" button
review form
search for companies
see reviews on company page
see reviews on job pages
see reviews on search results
best places to work
filter by location
filter by job function
review commenting
review voting

151. Built it

152. August 2011: 8,000

153. August 2011: 8,000
only 0.2% of jobs

154. August 2011: 8,000
only 0.2% of jobs
1 of every 50 search pages

155. August 2011: 8,000
only 0.2% of jobs
1 of every 50 search pages
NOT good enough

156. Goal:
every search page

157. 500,000

158. 300

159. 10,000
300

160. 10,000
+3000%
300

161. So we started testing

162. Why aren't people
submitting reviews?

163. How can we make it
easier?

167. What can we learn from
successful review sites?

175. +27%
387

176. What can the data tell us?

178. How can we improve
company search?

181. -45%

182. Step back and assess

184. What works for one site
does not necessarily work
for another site

185. Indeed is not a reviews site

186. Indeed is a job site

187. How can we engage our job
seekers where they are?

188. The flow: Oct 2011

193. What can the data tell us?

199. 2 ways to improve a funnel

200. 2 ways to improve a funnel
Improve conversion at step

201. 2 ways to improve a funnel
Improve conversion at step
Remove step entirely

202. Can we remove steps?

203. Yes,
if we target job seekers with
Indeed resumes.

208. New flow: Nov 2011

212. +343%
1,507

213. What can the data tell us?

214. Avg reviews per reviewer: 1

215. Avg reviews per reviewer: 1
Avg jobs per resume: 3.6

216. How can we increase
reviews per reviewer?

217. Before

220. After

223. +46%
3,719

224. 1.5
Avg reviews per reviewer: 1
Avg jobs per resume: 3.6

225. why not 3.6?
1.5
Avg reviews per reviewer: 1
Avg jobs per resume: 3.6

226. What can the data tell us?

227. Indeed
?
Zoominfo
42%

228. Indeed
My Perfect Gig
Zoominfo

229. +57%
5,834

230. What would happen if we
had more targeted
promos for job seekers
with resumes?

233. +36%
7,919

234. Can we remove more steps?

237. Before

240. After

243. +104%
13,010

244. Can we remove anything
from the review form?

245. Ratings
Review Title
Review
Pros
Cons
Job Title
Job Location
Start/End Date

246. Ratings
Review Title
Review
Pros
Cons
Job Title
Job Location
Start/End Date
Resume

248. +24%
17,168

249. Today...

251. 3,000,000

252. 1.7M companies

253. 60,000 / week

254. 60% of jobs

255. 6 out of 10

259. we moved the needle
more by removing than
adding

260. Simple

261. Perfection is finally attained
not when there is no longer
anything to add, but when
there is no longer anything to
take away.
Antoine de Saint Exupéry

262. Q&A
Donald Wysocki
donald@indeed.com
Graham Davis
gdavis@indeed.com

263. Next @IndeedEng Talk
Engineering Velocity: Building Great
Software Through Fast Iteration
Dan Heller, Engineering Manager
7pm Wednesday, August 28th