eBay Search Science: Leveraging Behavioral Data Analysis for Effective Query Reformulation Brian will talk about combing through behavioral log files with Scala on Hadoop in order to generate large data sets used to drive dynamic, online query rewrites at eBay. He’ll cover the product/feature pipeline from ideation to data mining, prototyping, statistical analysis, offline side by side analysis, human judgment, online experimentation, and finally launch.

1. Welcome To
Director of Engineering
Search Science Recall & Spam
April 3, 2015
BRIAN JOHNSON
With more than 100 million active users globally, eBay is
the world's largest online marketplace, where practically
anyone can buy and sell practically anything. Founded in
1995, eBay connects a diverse and passionate community
of individual buyers and sellers, as well as small
businesses. Their collective impact on ecommerce is
staggering: In 2014, the total value of goods sold on eBay
was $82 billion -- more than $2,500 every second.

2. http://www.ebayinc.com/who

7. DATA DRIVEN
DECISIONS
METRICS & TESTING

8. Why?
www.wallpapertimes.com
$’s per year in
incremental revenue

9. Data Collection
• Cell Tracking
• Eye Tracking
• Much Richer and more Detailed online
• Behavioral Log Files

10. We Are Data

11. Trends
Intelligence:Human  Machine
Data: Small  Big
Sources: Few  Many
Context: Aggregate  Detailed

12. SEARCH QUERY INTENT
Users
+
Documents

13. What is Special for eBay Search?
•Commercial Intentions
– Both the sellers and buyers have strong and clear intention
– Transactions happen on eBay, hence more behavior data
•Listings (Supply Side)
– Given by sellers
– Semi-structured data
•Buyers (Demand Side)
– Relevance matters (Browser vs. Searching)
– Price matters
– Seller trust/credit matters
– 70% eBay revenue starts with Search

14. Fish Sticks
Demand Category
42%
Business & Industrial > Electrical & Test Equipment > Electrical Equipment & Tools >
Electrical Tools > Cable Pullers
10% Business & Industrial > Construction > Building Materials & Supplies > Electrical
9% Home & Garden > Tools > Other
7% Pet Supplies > Fish & Aquariums > Fish Pond Supplies
6% Business & Industrial > Light Equipment & Tools > Air Tools > Staplers

15. Fishing Sticker
Demand Category
41%
eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License
Frames > Decals & Stickers > Graphics Decals
28%Home & Garden > Home Decor > Decals, Stickers & Vinyl Art
12%Sporting Goods > Fishing > Novelties & Gifts
11%Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches
3%Collectibles > Transportation > Automobilia > Decals & Stickers

16. Fish Stickers
Demand Category
53%Home & Garden > Home Decor > Decals, Stickers & Vinyl Art
24%Crafts > Scrapbooking & Paper Crafts > Embellishments > Stickers
16%
eBay Motors > Parts & Accessories > Car & Truck Parts > Decals/Emblems/License
Frames > Decals & Stickers > Graphics Decals
3%Sporting Goods > Fishing > Novelties & Gifts
1%Sporting Goods > Fishing > Fishing Equipment > Decals, Stickers & Patches

17. Keyword Expansion Test
Key Expansion Top Query
fishing sticker fish stickers fishing sticker
diaries diary vampire diaries
baggies baggie patagonia baggies
baggies baggy patagonia baggies
cranberries cranberry the cranberries
jogging jog jogging stroller
Key Expansion Top Query
fishing sticker fish stickers fishing sticker
diaries diary vampire diaries
baggies baggie patagonia baggies
baggies baggy patagonia baggies
cranberries cranberry the cranberries
jogging jog jogging stroller

18. Context Matters
cowboys hats ≠ cowboy hats
Plastic toy cowboys = plastic toy cowboy

19. Context & Specificity
Context
ATC  Armored Troop Carrier in Toys and Hobbies
ATC  Artist trading card in ART
ATC  Automatic Tool Change in Business and Industrial
Specificity/Directionality
Old  Antique
Yoga towels/mats  Yogitoes

20. Category Expansion
Stamps > Commonwealth/ British Colonial > Bermuda
Before After

21. Compound/Decompound Expansions
acidwash  acid wash
Before After

22. German Compounds
•Syntactically, words can be combined and split in many ways
•Multiple candidates
Granitpflastersteine (granite paving stones)
Granit(granite) pflastersteine(cobblestones)
Granit(granite) pflaster(paving/band-aid) steine(stones)
•Binding characters
Hochzeitsschuhe (grammatically correct, 593 hits on ebay.de)
Hochzeitschuhe (129 hits on ebay.de)
•Some words shouldn’t be de-compounded.
beiden (both) – bei(at) den(the)

23. Intent Preserving Query Relaxation

24. Hadoop Graph/Session Analysis
Bipartite Graphs
Keyword | Keyword Synonym Expansion
Keyword | Attribute Aspect Expansions
Keyword | Category Category Expansions, Related Search Diversity
Query | Query Related Search
Query | Item Related Search
Query Session Analysis
Successive Queries Synonyms, Related Search
Query Substitutions Synonyms
Same Session Correlation Related Search
Query Metrics
Click Through Rate
Purchase Attribution
Time to First Click, View Item, Purchase
Query Pair Price & Category Divergence
Query Pair Result Set Overlap
Result Count

25. Why we’re excited about data mining…
•We’re at an inflection point – customers are defining how they shop
– We are a data company
– 40+ Pb of data (listings, pictures, queries, clicks, sales, feedback, …)
– Many tests running orthogonally (in parallel with overlapping user slices)
– Nearly all users in one of more tests
– Many users per test, often millions
•Find patterns and insights drives our customer experience
•We’ve built successful teams of data scientists

26. THANKS!
QUESTIONS?
WE’RE HIRING
BJOHNSON@EBAY.COM

27. METRICS
•What should we optimize
–Page Views
–Time on Site
–Click Through Rate
–Normalized Discounted Cumulative Gain
–Purchases per User per Session/Day/Week
–Revenue per User per Session/Day/Week
–Net Promoter Score
•How likely would you be to recommend …?

28. REVENUE
Every business focuses on and measures revenue.
Every business focuses on profit and loss.
And they should.

29. Experimental Variation By Day

30. Experimental Variation By Metric

33. Query Rewrites at eBay
Query Rewrite
Search
User Query
eBay Results
Search Query
User Query: pilzlampe {mushroom lamp}
Search Query: OR(pilzlampe, PHRASE(OR(pilz,pilze),OR(lampe,lampen)))

34. Example Query Services/Rewrites
• Stemming (ipod OR ipods)
• Spelling (cannon OR canon)
• Condition (new OR condition=new)
• Synonyms (boat carpet OR marine carpet)
• Space Synonyms (MarioKart OR Mario-Kart)
• Item Specifics (blue OR color=blue)
• Acronyms (hp OR hewlett-packard OR horsepower)
• Category (shoes OR Category=Shoes)
• Cross Border (site=0 AND category =123) OR (site=3 AND
category=456)
• Fitment (fits model=corolla)
• Term Removal (Harry Potter and the Order of the Phoenix (daily deal))

35. Acronym/Abbreviation Mining
•Acronyms/Abbreviations mined from raw text and query logs
•Look for patterns of text:
long form (short form)
short form (long form)
• Employ intelligent matching algorithms to mine candidates
• Schwartz et al: Greedy Match Algorithm
new cheap Playstation portable (PSP)
PlayStation 3 (PS3)
• Acronym discovered
PSP => PlayStation Portable
PS3 => PlayStation 3
• Candidates mined are fed to an ML classifier to remove false positives