Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Design for Continuous                    Experimentation                                Dan McKinley                      ...
www.   .comSunday, December 2, 12and I’m here from etsy.com
The world’s handmade        and vintage marketplace.Sunday, December 2, 12Etsy is the world’s handmade and vintage marketp...
Sunday, December 2, 12Etsy’s a place where you can buy all kinds of things, including handmade crafts like thissampler
Sunday, December 2, 12... or this vintage credenza ...
Sunday, December 2, 12... and rhinestone-studded underwear made of beef jerky ...
Sunday, December 2, 12Beef jerky underwear is reasonably popular apparently. we’re on track to sell between $800MM and900M...
OCTOBER 2012                                         1.5 billion page views                                         55 mil...
We love experiments.Sunday, December 2, 12At Etsy, we love experiments and A/B testing. And that’s the main thing I want t...
Tons of active A/B tests and rampups.Sunday, December 2, 12Here’s a screenshot of an internal view of the various tests an...
Sunday, December 2, 12We’ve invested plenty of time and effort into tooling to support this work. This is ascreenshot of o...
Sunday, December 2, 12We’ve built tools that protect us from some gnarly statistics. This wizard does the math foryou and ...
Continuous Experimentation                             Small, measurable changes.                                  Keeps u...
Seung yun Yoo                                                                Seoul, South Korea                           ...
Etsy Sales: Two Scenarios                                                 Good product release                            ...
Another reason we measure:Sunday, December 2, 12
Another reason we measure:                         Experimental results are surprising!Sunday, December 2, 12
“When I am comparison shopping, I open items in new             tabs. We should do that on Etsy.”                         ...
Sunday, December 2, 12So we eventually stopped arguing about this and just tried it. We ran an A/B test that opened up ite...
The Horrible Sound of Epic Failure credit: EmbroideryEverywhere.etsy.comSunday, December 2, 12When we tried that, 70% more...
Surprise!Sunday, December 2, 12Surprise! We don’t argue about that anymore.
One big thing we’ve learned                               from experiments:Sunday, December 2, 12We’ve been at this for a ...
Design and product process                         must change to accommodate                               experimentatio...
Removing the Search                         Infinite Scroll                                                                ...
Removing the Search                         Infinite Scroll                                                           Dropd...
Infinite Scroll                                      So hot right now.Sunday, December 2, 12First I’ll go through our deplo...
WoahSunday, December 2, 12If anyone doesn’t know what I mean by infinite scroll: I mean that we changed search results so t...
Seeing more items faster is                         presumed to be a better experience.Sunday, December 2, 12The reason we...
Infinite Scroll: Release Plan                                            (Implied)              1. Build infinite scroll.   ...
Infinite Scroll: ResultsSunday, December 2, 12so the results,
Infinite Scroll: Results                          Spoiler: they were not expected.Sunday, December 2, 12not to spoil the su...
Infinite Scroll: Results                             Median item impressions:                                      Infinite ...
Infinite Scroll: Results                         Visitors seeing infinite scroll clicked                            fewer re...
Infinite Scroll: Results                         Visitors seeing infinite scroll saved                              fewer it...
Infinite Scroll: Results                         Visitors seeing infinite scroll purchased                                 f...
Initial reaction:                         “something’s broken.”Sunday, December 2, 12The first thing that occurred to us is...
Gradual, horrible realization:             “we changed many things at once.”Sunday, December 2, 12Eventually we came to te...
Premise-validating Experiments                          Or: “things we should have                           done in the fi...
Are more items in search results better?Sunday, December 2, 12First of all, is it true that more items is better?
Sunday, December 2, 12We ran a test where we just varied the number of results in normal search results.
Are more items in search results better?                         Barely, maybe: more people get to an item                ...
Are faster results better?Sunday, December 2, 12The other major premise was that faster search results would stop people f...
Sunday, December 2, 12We ran a test where we slowed down search artificially, by adding sleeps().
Are faster results better?                                               MehSunday, December 2, 12Absolutely nothing happe...
credit: LunaLetterpress.etsy.comSunday, December 2, 12In the end the expected benefits to infinite scroll just didn’t seem t...
Infinite Scroll: Release Plan                                             (Implied)              1. Build infinite scroll.  ...
A Slightly Better Infinite                            Scroll Release Plan              1. Validate premise: more items is b...
Throwing out work sucks.Sunday, December 2, 12Throwing out work feels really horrible. Most of the time this is a really d...
Infinite scroll: not stupid.Sunday, December 2, 12My point is not that infinite scroll is stupid. It may be great on your we...
Removing the Search Dropdown                         A much better experience                            for me, personall...
2007Sunday, December 2, 12Pretty early on, we added this dropdown to the header, mainly to pick between handmade items and...
2012Sunday, December 2, 12But as these things always do, it got way out of hand. It looked like this five years later.
Kill the Dropdown: Project Plan              1. Redesign marketplace facets.              2. Default to “all items.”      ...
Kill the Dropdown: Project Plan              1. Redesign marketplace facets.              2. Default to “all items.”      ...
Kill the Dropdown: Project Plan              1. Redesign marketplace facets.              2. Default to “all items.”      ...
Kill the Dropdown: Project Plan              1. Redesign marketplace facets.              2. Default to “all items.”      ...
Sunday, December 2, 12So, the first thing we had to address was the fact that the dropdown was used to cut the marketplaceb...
HYPOTHESIS:                             Most users of the site don’t                             know anything about this....
Sunday, December 2, 12First we introduced this faceting on the left side of search results, and made it more obvious. This...
Sunday, December 2, 12But still, relatively few people noticed that. So we also built faceting into our autosuggest. We ma...
Sales of Vintage Items:                                               +3.7%Sunday, December 2, 12After we did this, sales ...
VERIFIED HYPOTHESIS:                             Casual users of the site don’t                              know anything...
Context-sensitive!Sunday, December 2, 12Another horrible behavior of the search dropdown was that it was context-sensitive...
HYPOTHESIS:                            Casual users of the site don’t                                    realize this.Sund...
Sunday, December 2, 12To contend with this we introduced a secondary search box on shop pages so that people could do asea...
Sunday, December 2, 12We also tried adding this search bar to the item page. But few people used it and those who didperfo...
Sunday, December 2, 12So we took that part out.If we had done the whole project all at once, we probably would not have no...
Sunday, December 2, 12Another thing the search dropdown could be used for was searching for shops. Nobody used it.
Sunday, December 2, 12So we added shops suggestions to item results and made sure more people could find shops
...plus five or ten other things                                on the same scale.Sunday, December 2, 12So you more or less...
Data was involved at every step.Sunday, December 2, 12
Infinite Scroll                         Design          Develop                      Measure              ಠ_ಠ       Dropdow...
Some AdviceSunday, December 2, 12I want to leave you with some parting advice.
Experiment with minimal                              versions of your idea.Sunday, December 2, 12Experiment with a minimal...
Plan on being wrong.Sunday, December 2, 12Plan on being wrong. If you measure, you’ll encounter many counterintuitive resu...
Prefer incremental redesigns.Sunday, December 2, 12
This will not always work.                             Occasionally, you may                            need to make big b...
...but it usually does.Sunday, December 2, 12But if you’re throwing this card down all the time you’re probably doing it w...
Thank you.                           Dan McKinley                          dan@etsy.comSunday, December 2, 12thanks
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Enchanted Objects: The next wave of the web
Next
Upcoming SlideShare
Enchanted Objects: The next wave of the web
Next
Download to read offline and view in fullscreen.

Share

Infinite Scroll: Results Visitors seeing Design for Continuous Experimentation

Download to read offline

Infinite Scroll: Results Visitors seeing infinite scroll clicked fewer results than the control.Sunday, December 2, 12they clicked on fewer items.

Related Books

Free with a 30 day trial from Scribd

See all
  • keilakomori

    Jun. 9, 2021
  • saharrabadi

    Mar. 5, 2021
  • FilipeFilibox

    Aug. 5, 2020
  • LeaSefer

    Dec. 14, 2019
  • michelevaughan5

    Aug. 30, 2019
  • swag

    Aug. 9, 2019
  • JustinaEng

    May. 16, 2019
  • MarcRueda1

    May. 13, 2019
  • jroselln

    Mar. 8, 2019
  • pwucker

    Oct. 16, 2018
  • HannahWestberg

    Jul. 31, 2018
  • lexshkurko

    Jul. 12, 2018
  • Vhjkjh

    Jun. 29, 2018
  • BrandieRDavis

    Apr. 20, 2018
  • timothompson

    Apr. 10, 2018
  • annhudspeth

    Apr. 10, 2018
  • EllaEfrat

    Feb. 6, 2018
  • BrittanyMederos

    Jan. 26, 2018
  • julianbarkow

    Dec. 16, 2017
  • BinnurAlKazily

    Sep. 6, 2017

Infinite Scroll: Results Visitors seeing infinite scroll clicked fewer results than the control.Sunday, December 2, 12they clicked on fewer items.

Views

Total views

262,734

On Slideshare

0

From embeds

0

Number of embeds

107,644

Actions

Downloads

1,123

Shares

0

Comments

0

Likes

409

×