Your SlideShare is downloading. ×
0
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
[@IndeedEng] Data-Driven Product Design
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

[@IndeedEng] Data-Driven Product Design

7,534

Published on

@IndeedEnd April: Wednesday, April 24, 2013 …

@IndeedEnd April: Wednesday, April 24, 2013
Video available: http://youtu.be/Y6NI8y21xhg

Advertising pioneer John Wanamaker famously said, "Half the money I spend on advertising is wasted; the trouble is I don't know which half." At Indeed, our experience shows that more than half of product features are not only wasted but actually have a negative impact on user experience. Thanks to rigorous A/B testing we know precisely which half, so we can ensure that every feature we add delivers measurable value.

With 100 million monthly unique visitors and 3 billion monthly searches, our data provides empirical answers to (nearly) all of our product questions. In this talk, we will cover:

* a motivation for why we A/B test everything
* the tools and systems we have developed to measure and analyze test results and the key metrics we focus on, and
* multiple case studies illustrating how years of testing has shaped Indeed's design and product thinking.

We will include specific examples of how small, overlooked changes have impacted our metrics in significant and unexpected ways, and how implementing simple "no-brainer" updates on our site decreased key business metrics and caused us to re-evaluate our solutions.

This talk is for designers, product managers, and engineers from companies of any size who want to understand how data-driven design makes better products.

Speakers

J Christopher Garcia is the User Experience Lead at Indeed. Chris ensures Indeed's products are easy to use for millions of people all over the world. He has focused on user experience at Indeed for over 6 years. As the company has grown from start-up underdog to the world's leading job search engine, he has contributed to every major product Indeed offers.

Chris Hyams is SVP of Product & Engineering at Indeed. Prior to Indeed, Chris was founder of B-Side, an Austin-based company that provided digital distribution, marketing, and analytics technology for the film industry. Prior to B-Side, he was VP of Engineering at Trilogy Software.

Published in: Technology, Business
1 Comment
13 Likes
Statistics
Notes
  • Note: the full video from this presentation is also available at https://engineering.indeed.com/blog/2013/05/indeedeng-data-driven-product-design-slides-video/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
7,534
On Slideshare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
51
Comments
1
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Data-DrivenProduct DesignJ Christopher GarciaChris Hyams
  • 2. DesignJ Christopher GarciaUser Experience Lead, Indeed
  • 3. 2008
  • 4. 2013
  • 5. 114 tests244 variations84,000 combinations
  • 6. Direct observationcan be inefficient
  • 7. Before and afterdoesnt work
  • 8. Traffic doesnt look like this
  • 9. Hourly traffic looks like this...
  • 10. ...and daily traffic looks like this
  • 11. ...and monthly traffic looks like this
  • 12. Local maximum
  • 13. What weve learnedUI Principles
  • 14. Clickable items shouldlook clickable
  • 15. Users read
  • 16. a word is worth athousand icons
  • 17. +14%more searchresults pages
  • 18. BEWARE MULTIPLECHANGES
  • 19. Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%
  • 20. Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%
  • 21. Really?
  • 22. Really.
  • 23. ProductChris HyamsSVP, Product & Engineering
  • 24. I helppeopleget jobs.
  • 25. Whats best for the jobseeker?
  • 26. How do we know whatsbest for job seekers?
  • 27. Test and measureEVERYTHING
  • 28. Measurement
  • 29. 1. Log everything
  • 30. jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpodsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtimeojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojcrojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllongsqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize
  • 31. acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyopsfeedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasxorganicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin
  • 32. 16 million jobs100 million UV3 billion searches
  • 33. 1TB per day
  • 34. Everything you ever wanted to knowabout logging but were afraid to askLogrepo: EnablingData-Driven Decisions
  • 35. 2. Index data
  • 36. 3. Build tools toquery indexes
  • 37. [another tech talk]
  • 38. Tools answer questions
  • 39. Mobile searches per hour inJP vs. UK?
  • 40. Resume creation by country?
  • 41. Impressions in job alerts byemail domain?
  • 42. Percent of app downloads fromiOS, Android, Windows?
  • 43. How quickly does a datacenter takeon traffic after a failover?
  • 44. Test EVERYTHING
  • 45. TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]Everything
  • 46. 5% 5%grp:bluebtntst0 grp:bluebtntst1split:log:
  • 47. Measure
  • 48. statistical relevance
  • 49. p-value
  • 50. Big changes require less dataConversion Improvement p-value Trials10% +30% 0.038 1,000source: http://go.indeed.com/jyusg
  • 51. Small changes require more dataConversion Improvement p-value Trials10% +30% 0.038 1,00010% +1% 0.042 750,000source: http://go.indeed.com/ommgd
  • 52. Product Principles
  • 53. Theres no such thing asa no-brainer
  • 54. -30%
  • 55. Corollary: if a test fails,it means the test failed
  • 56. -30%
  • 57. +25%
  • 58. Little things can makea big difference
  • 59. organic
  • 60. sponsored
  • 61. +34%
  • 62. Corollary: be skepticalof big changes
  • 63. Does color really matterthat much?
  • 64. a Aa Ba C+34%+10%
  • 65. Are people just boredof blue?
  • 66. +29% +22%
  • 67. Will the effect wear off?
  • 68. Get it right,then do it right
  • 69. Location backfill: job clicks+8%
  • 70. Location backfill: search time+1,500%
  • 71. Location backfill: performance fix+1,500%
  • 72. Location backfill: performance fix+1,500% +0%
  • 73. Location backfill: job clicks+8%
  • 74. Location backfill: clicks after fix+8% +9%
  • 75. Measure everything(aka, Beware of unintended consequences)
  • 76. April Fools 2010
  • 77. Organic clicks+1.8%
  • 78. Job alert signups-17%
  • 79. Revenue+15.5%
  • 80. Revenue+15.5%
  • 81. Job alert signups: top v. bottom-17% +6%-21%
  • 82. Corollary: attention is azero-sum game(usually)
  • 83. SearchClickRevenueEmail signupResume create...Relative value of actions
  • 84. Different for everyone
  • 85. Location auto-complete
  • 86. Zero-results page
  • 87. Zero-results pages-2.7%
  • 88. Organic clicks+0%
  • 89. Clicks w/ auto-complete+8%
  • 90. Keyword ad revenue+1,410%
  • 91. Keyword ad revenue+1,410%
  • 92. keyword ads
  • 93. Keyword ad revenue after fix
  • 94. Measure everything,every time
  • 95. Getting Started
  • 96. Fewer tests
  • 97. Focus on big wins
  • 98. Conversion Improvement p-value Trials10% +30% 0.038 1,00010% +1% 0.042 750,000source: http://go.indeed.com/ommgd
  • 99. Straw manDetermine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on
  • 100. What We Get
  • 101. Quality
  • 102. Risk tolerance
  • 103. Culture of measurement
  • 104. (Empirically)better products
  • 105. fin

×