Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy

1,615 views

Published on

video available here: https://blueprintdigital.com/ia-summit-2017/jenny-benevento-giovanni-fernandez-kincade-jill-fruchter/

This was a talk given at IA Summit 2017 in Vancouver, BC by Jenny Benevento, Gio Fernandez-Kincade, and Jill Fruchter.

Etsy is a marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. Etsy is also a tech company that invests in the craft of coding and data-driven product development as a strategic priority. Etsy has employed AI and machine learning to tackle personalization, recommendations, image understanding, item similarity, search relevance, spelling correction, and many other tasks. We’ll talk through several examples of how Etsy leverages data, where it’s excelled, and where this hammer hasn’t quite hit the nail on the head.

We will be asking ourselves hard questions, recognizing the limitations of decisions driven purely by big data:
- Who are we satisfying? Our customers or our mathematical models?
- Are those models even an accurate reflection of the outcomes we want?
- In a dual marketplace, where complex changes depend on interactions between both sides of the market, can one metric or measure of success tell the full story?
- How do we consider the impact our models are having on our users?
- Are we even addressing real human needs and motivations in the first place?
- How do we inform and enrich AI with expert created & applied taxonomy & metadata?

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

A Taxonomist, a Software Engineer, and a UX Researcher Walk Into a Bar: Bridging AI and User Experience Methods at Etsy

  1. 1. Bridging AI and UX Methods at Etsy _________ IA Summit March 25, 2017
  2. 2. Gio Staff Engineer Jenny Taxonomist Jill Staff UX Researcher
  3. 3. Etsy is a global marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. 3
  4. 4. By the Numbers 1.7M active sellers AS OF DECEMBER 31, 2016 28M active buyers AS OF DECEMBER 31, 2016 $2.84B annual GMS IN 2016 45+M items for sale AS OF DECEMBER 31, 2016 Photo by Kirsty-Lyn Jameson
  5. 5. SELLER Pursues craft, grows business ETSY Invests in the platform and delivers a global base of buyers ETSY Facilitates the transaction BUYER Finds unique goods that are hard to find elsewhere ETSY EMPOWERMENT LOOP ETSY EMPOWERMENT LOOP 6
  6. 6. SELLER Pursues craft, grows business ETSY Invests in the platform and delivers a global base of buyers ETSY Facilitates the transaction BUYER Finds unique goods that are hard to find elsewhere ETSY EMPOWERMENT LOOP ETSY EMPOWERMENT LOOP 7 SEARCH
  7. 7. “e-Search” A case study 8
  8. 8. Frame the problem “E-Search” - A case study WINTER 2015 Iterative evaluation Concept testing SPRING 2014 SUMMER 2014
  9. 9. “E-Search” - A Case Study
  10. 10. Parallel streams of effort UX RESEARCH
  11. 11. UX Research 13
  12. 12. Etsy.com What are people shopping for on Etsy?
  13. 13. People shop for ideas on Etsy 17 GIFTS LIFE EVENTS SEASONAL CELEBRATIONS STYLE/EMOTION POP CULTURE/ TRENDS NOT “known item” searches
  14. 14. People shop for ideas on Etsy 18 GIFTS LIFE EVENTS STYLE/EMOTION NOT “known item” searches Often, using broad, thematic search terms SEASONAL CELEBRATIONS POP CULTURE/ TRENDS
  15. 15. Framing the problem 19
  16. 16. Existing “browse”pages were low performers, i.e. not converting. PROBLEM HOW MIGHT WE… Make finding on Etsy easier with a combination of search & browse improvements? 20
  17. 17. Customers are doing broad searches and getting “overwhelmed” with search results. PROBLEM HOW MIGHT WE… Turn broad search queries into satisfying experiences of finding and discovery? 21
  18. 18. UX research as the new kid in town FEB 2014 MARCH 2014 MARCH 2015 JUNE 2015 JULY 2014 AUGUST 2014 ● Etsy was founded in 2005 ● No UX Research until 2014 (Thick data) ● Since 2010, data-driven design and evaluation (Big data) RESEARCH BRAND DESIGN PRODUCT DESIGN
  19. 19. Etsy almost solely relied on AI and experimentation to frame and solve problems. PROBLEM HOW MIGHT WE… Use this new research capacity to help frame the problem and build a better product? 23
  20. 20. Jewelry 8M items
  21. 21. Necklaces 3M items
  22. 22. GOAL: Observe “jewelry” search behavior on- and off- Etsy to introduce product teams to shoppers’ expectations and needs TEST STIMULI: - The Internet - Self-guided searches Contextual observation (2x)
  23. 23. Goals 1 2 3 UX Insights Searching on Etsy is conceptually different
  24. 24. Goals 1 2 3 Highly visual UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye”
  25. 25. Goals 1 2 3 Highly visual Use search to browse UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye” “6PM is like running into the drugstore. On Etsy, I’m always finding surprises.” “It’s like watching Reality TV. I like the process.”
  26. 26. Goals 1 2 3 Highly visual Use search to browse UX Insights Searching on Etsy is conceptually different “I’m not on track” “I’m looking for something to catch my eye” “6PM is like running into the drugstore. On Etsy, I’m always finding surprises.” “It’s like watching Reality TV. I like the process.” FOMO “Even if it were $10,000. I want to know about it.” “Patience on Etsy is rewarded.”
  27. 27. Guiding Principles Visual Exploratory Surprise JOMO
  28. 28. E-Search SINGLE CATEGORY: JEWELRY > NECKLACE CROSS-CATEGORY: MID-CENTURY MODERN, MOTHER’S DAY Concept testing (2x)
  29. 29. Goals 1 2 3 UX Insights Until items feel more similar than different, the more categorization the better
  30. 30. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better
  31. 31. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better More intuitive “I’m making more progress with minimal effort.”
  32. 32. Goals 1 2 3 Less FOMO UX Insights "It lets me see how it's organized and go on. It guides you." "It's nice the categories are right here. It prompts me to narrow down an extremely broad search." Until items feel more similar than different, the more categorization the better More intuitive “I’m making more progress with minimal effort.” More “Etsy” “Expands for me what is on Etsy because I haven't thought of that being there.”
  33. 33. UX Vision Find more, search less
  34. 34. Ye Olde Librarian Knowledge
  35. 35. Berrypicking “the query is satisfied...by a series of selections of individual references and bits of information at each stage of the ever-modifying search...retrieval of this sort is here called berrypicking.” Source: Marcia Bates (1989). "The Design of Browsing and Berrypicking Techniques for the Online Search Interface."
  36. 36. UX Vision Question: Can we build that?
  37. 37. Jousting Windmills 43
  38. 38. Jewelry
  39. 39. “silver necklace” AI Jewelry! Query Classification
  40. 40. Query Classification
  41. 41. Document Classification
  42. 42. Jewelry > Necklaces > Charm Necklaces Document Classification
  43. 43. 49
  44. 44. Disaster 50
  45. 45. Goals 1 2 3 Data is Noisy Disaster
  46. 46. AI Data is Noisy
  47. 47. AI Data is Noisy
  48. 48. 54 “turtle costume”
  49. 49. 55 “turtle costume”
  50. 50. 56 “hose”
  51. 51. 57 “hose”
  52. 52. 58 “chair”
  53. 53. 59 “chair”
  54. 54. Goals 1 2 3 Data is Noisy Disaster Expensive False Positives
  55. 55. “sunglasses”
  56. 56. “shoes”
  57. 57. Goals 1 2 3 Data is Noisy Disaster Expensive False Positives Sellers Last
  58. 58. Sellers last 64
  59. 59. Jousting Windmills UX RESEARCH AI
  60. 60. Reboot GOOD DATA USER AGENCY
  61. 61. Reboot TAXONOMY
  62. 62. Reboot UX RESEARCH AI UX+AI+TAXO
  63. 63. Reboot 69
  64. 64. A taxonomy to serve both user groups 70
  65. 65. TAXONOMY MERCH MARKETING POLICY ENG MAKERS LEGAL DATAUX
  66. 66. Hat?
  67. 67. ● Handmade > Clothing > Dresses ● Handmade > Women > Clothing > Dresses ● Handmade>Weddings>Dresses ● Vintage > Clothing > Dresses ● Vintage > Women > Clothing > Dresses ● Weddings > Dresses ● Women > Clothing > Dresses
  68. 68. title Knitting Technique Glass Material Geekery Subculture Vintage Age Children Who Holidays When Old Categories Aspect
  69. 69. Seller Control is Good UX
  70. 70. Not One. Two Taxonomies
  71. 71. Seller Control
  72. 72. Metadata is Emotional
  73. 73. Sellers Buyers
  74. 74. Communication is Product
  75. 75. • Transparent • Art not SEO • Translatable • Flexible • Evolves • Buyer Friendly • Powers Structured Data 85 The Power of Two Taxonomies
  76. 76. AI + Inventory Data Taxonomy AI + Search
  77. 77. Epilogue 87
  78. 78. Takeaways 89
  79. 79. Data is queen. 90 TAKEAWAYS
  80. 80. AI is not enough. 91 TAKEAWAYS
  81. 81. Holistic solutions require multi-disciplinary thinking. 92 TAKEAWAYS
  82. 82. Respect and trust your users. 93 TAKEAWAYS
  83. 83. Thanks!
  84. 84. Gio giokincade@gmail.com @giokincade Jenny jbenevento@etsy.com @jennybento Jill jfruchter@etsy.com @jillfruchter

×