Transforming Search in the Digital Marketplace


Published on

Late last summer, Etsy made a seemingly innocuous change to its search engine that had far reaching impact. Historically, when a user came to the marketplace and issued a search, the default sort order ranked items by listing date with most recently listed items first. While
users could easily switch to other sort options, e.g. sorted by price or sorted by relevancy, the default behavior was one sorted by recency. On August 9 of last year, Etsy changed the default sort order of the marketplace from recency to relevancy. While at a technical level this change involved only a single line of code and a few altered pixels on the site, it introduced a fundamental shift in the dynamics of the marketplace.

This talk provides some insight into the many months of development, planning, and goals around the change, and will cover aspects from business strategy to product development to marketing and communications. Specifically, we will outline risk posed to the business and marketplace. We will discuss product issues and changes to search made before and after the switch, including the launch of a keyword based search advertising product called Search Ads. Finally, we will overview how these changes and improvements were communicated to Etsy's sellers, leading up to a big Etsy milestone: surpassing $1 billion in total marketplace sales last October.

About Etsy

Etsy is the world's handmade marketplace. Etsy is headquartered in Brooklyn and has over 14 million members, 800,000 active shops, over $525 million in annual sales and 1.4 billion page views per month.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • Last summer, Etsy changed its default sort ranking from recency sorting to relevancy sorting. In many levels, the change was seemingly innocuous, consisting of a few changed pixels and a couple of lines of code.\n
  • Behind the scenes, the change involved a substantial effort, including collaboration between engineering, product, marketing, support, and finance.\nA big part of the change was the launch of Etsy’s first performance-based keyword ad system called search ads. Search ads inventory today is sold out.\n
  • An overview of what I’ll be covering today.\n
  • Etsy is the world’s handmade marketplace\n
  • \n
  • Etsy was founded in June of 2005. The site offered basic functionality to buy, sell, and search.\n
  • Etsy’s initial search technology wasn’t too much different from this. While simple, this form of searching could easily be gamed by sellers by simply “renewing” their item, which updated their creation_tsz timestamp.\n
  • An entire culture and cottage industry grew around keeping your creation_tsz as recent as possible.\n
  • As Etsy grew, we encountered scaling issues. Our search was modernized in late 2008. However, seller behavior around renewing and expectations around sorting by recency were set at this point. The new technology launched largely behind the scenes; the sorting behavior from a user’s perspective was unchanged.\n
  • \n
  • Relevancy Thursdays was a fun science experiment. We learned a lot, and not all sellers found it to be very fun.\n
  • \n
  • Sorting by recency provides a sub-optimal buyer experience.\n
  • Renewing a listing costs $0.20, and the effect of renewing was highly unpredictable. It could vary based on category, time of day, and season.\n
  • Other than search on, sellers had very little visibility into their shop’s performance.\n
  • I’m here today to talk about strategy....\n
  • Improving search relevancy would lead to a better buyer experience. Moving towards black-box search sorting enabled us to take control of search and set direction.\n
  • We wanted to build an advertising product that was performance based, simple to understand, and measurable.\n
  • Understanding shop performance from an analytics point of view is critical\n
  • So, we launched a new ad product, new seller analytics, and switched the default sort order of search. What order did we do this in? \n
  • Search ads was a large undertaking and took many months of development effort. Components include keyword selection, keyword pricing, statistical problems around scheduling, billing, and the buying console\n
  • An overview of what I’ll be covering today.\n
  • Before making the switch to relevancy, we made several key improvements to our search ranking algorithms. \n
  • We used human raters to judge improvements in search quality via a side by side testing tool that we developed in house.\n
  • \n
  • We ran dozens of 1% search experiments on and carefully measured changes in response\n
  • Our first major release was shop stats. Before making any changes to search or ad campaigns, we felt it was critically important for sellers to understand the data behind their shop.\n
  • When we announced our intentions to make the switch, our communications spanned several sources, including the blog and the forums. This was an important announcement and we wanted to make sure it was heard.\n
  • When we announced our intentions to make the switch, our communications spanned several sources, including the blog and the forums. This was an important announcement and we wanted to make sure it was heard.\n
  • Affirmation from Chad was an important part of our communications strategy. It stressed the importance of this initiative to our sellers.\n\n
  • We made the switch to relevancy before releasing an advertising alternative. The lack of clear search advertising options for sellers also represented lost revenue for Etsy. We felt that this reinforced our message that this change was best for the marketplace.\nPriori to the switch, over 90% of all searches were sorted by recency. After the switch, over 90% of searches were sorted by relevance.\n
  • \n
  • \n
  • \n
  • \n
  • What did we learn?\n
  • Doing this wasn’t easy. It was disruptive to the marketplace. But we felt it was necessary.\n
  • At Etsy, we have two primary customers.\nSellers are much more outspoken. We listened to them very carefully, and thought very hard. Some sellers were very unhappy with the change.\nBuyers are much quieter. They spoke through quantitative improvements we saw with our search changes.\n
  • We communicated with our sellers early and often. We set direction both through our standard channels (blogs, forums, etc.), but also through our actions.\n
  • Without great products, this would have all failed.\n
  • \n
  • \n
  • Transforming Search in the Digital Marketplace

    1. 1. Transforming Search in theDigital Marketplace Dr. Jason Davis Director of Search & Data / @jasondavis
    2. 2. (the change)
    3. 3. 6 months from inception tolaunch3 engineering teams3 product launches5 different functions1 startup acquisition4 months to realize success
    4. 4. About EtsyHistory of Etsy searchThe pitfalls of recencyRelevancy: strategy and goalsTakeaways: what we learned
    5. 5. By the Numbers1.5B pageviews/month$525MM sales in 201140MM unique visitors/month800k shops / 150 countries
    6. 6. The beginnings June 2005 the birth of Etsy2005 2006 2007 2008 2009 2010 2011 2012
    7. 7. SELECT * FROM listings WHERE title LIKE %silver%ORDER BY creation_tsz;
    8. 8. Text December 2008 Solr / Lucene2005 2006 2007 2008 2009 2010 2011 2012
    9. 9. Text December 2009 Jason starts at Etsy2005 2006 2007 2008 2009 2010 2011 2012
    10. 10. Relevancy“For the next three Thursdays, we will change thedefault sorting method for all searches on Etsy,from recency to relevancy.” Text January 20102005 2006 2007 2008 2009 2010 2011 2012
    11. 11. The recency problem
    12. 12. Buyer experienceI was looking for a pink bonnet with bluepolka dots. I searched for "polka dotbonnet" but all I got were vintage polkarecords. Then I wanted a classic rock albumand searched for "rock and roll" and gotnothing but circular stones.
    13. 13. Predictability
    14. 14. Visibility
    15. 15. Strategy
    16. 16. Better buyer experienceImproved search relevancy
    17. 17. Betteradvertising for Search Ads
    18. 18. More searchinsight forShop Stats
    19. 19. Sequencing
    20. 20. Search AdsdevelopmentApril May June July August September October
    21. 21. Core system and architectureKeyword pricing & analysisAd buying experienceAd qualityIntegration with search
    22. 22. SearchimprovementsApril May June July August September October
    23. 23. Side by sidetesting
    24. 24. Tweaking &tuning
    25. 25. Experimentalanalysis
    26. 26. July 21, 2011 Shop Stats with referralsApril May June July August September October
    27. 27. “We plan to move search by relevancy to the default search setting once we feel this is the strongest site experience.” Natalie on the Blog search/ July 28, 2011 Announcing the switch April May June July August September October
    28. 28. “As we continue to develop searchsorted by relevancy, our goal is tomake this the default site Frank in the Forumsexperience.” July 28, 2011 Announcing the switch April May June July August September October
    29. 29. “With more buyers coming to the site over the holidays, we know it’s critical that we improve the search experience on Etsy. We want to make those changes well in advance of the holiday season to give you plenty of time to prepare, so we’ve begun communicating what changes you may need to make to your Notes from Chad listings” August 1, 2011 A note from our CEO April May June July August September October
    30. 30. August 9, 2011 Switch flipped to relevancyApril May June July August September October
    31. 31. September 1, 2011 Surprise!April May June July August September October
    32. 32. September 20, 2011 Search ads available for purchaseApril May June July August September October
    33. 33. September 28, 2011 Search ads go liveApril May June July August September October
    34. 34. Takeaways
    35. 35. Make harddecisions
    36. 36. Understand yourcustomers
    37. 37. Communicate Set
    38. 38. Shipawesome stuff
    39. 39. ResourcesCode as Craft blog (including these) in the fun