Hadoop World 2011 Keynote: Ebay - Hugh Williams

by Cloudera, Inc. on Nov 11, 2011

17,105 views

Hugh Williams will discuss building Cassini, a new search engine at eBay which processes over 250 million search queries and serves more than 2 billion page views each day. Hugh will trace the genesis and building of Cassini as well as highlight and demonstrate the key features of this new search platform. He will discuss some of the challenges in scaling arguably the world’s largest real-time search problem, including the unique considerations associated with e-commerce and eBay’s domain, and how Hadoop and HBase are used to solve these problems

Less

18 Embeds 13,460

http://ebaystrategies.blogs.com	10652
http://blog.channeladvisor.com	1732
http://www.cloudera.com	578
http://www.ebaystrategies.blogs.com	240
http://d.hatena.ne.jp	93
http://feeds2.feedburner.com	92
http://www.csestrategies.com	35
http://translate.googleusercontent.com	17
http://www.ecommercetactics.com	5
http://webcache.googleusercontent.com	5
http://www.hanrss.com	4
http://www.streetcommentary.com	1
http://s1ici01	1
http://cloudera.brian.dev	1
http://127.0.0.1	1
http://cloudera.matt.dev	1
http://105.144.21.164	1
http://74.6.238.254	1

More...

Statistics

Favorites: 8
Downloads: 0
Comments: 1
Embed Views: 13,460
Views on SlideShare: 3,645
Total Views: 17,105

Hadoop World 2011 Keynote: Ebay - Hugh Williams — Presentation Transcript

Project Cassini: ’sNew Search Engine Vice President of Search, Experience, and Platforms eBay Marketplaces

$2.63millionfor a lunch withWarren Buffett

$40,668for Justin Bieber’sjust-cut hair

$130Kfor PrincessBeatrice’s hat

$62billionin merchandise sold in 2010

97 millionactive buyers and sellers worldwide250 million querieseach day to our search engine200+ million itemslive in more than 50,000 categories

9 petabytes of datain our Hadoop and Teradata clusters2 billion page viewseach day75 billion database callseach day

Huge Opportunity: Taking the “e” out of ecommerce Yesterday Today Tomorrow Online Online 4% 6% Web- influenced Online offline + Offline 37% Offline Offline 96% 2008 = $325B 2013 = $10T Source: Forrester, Euromonitor and Economist Intelligence Unit Source: Forrester Source: Economist Intelligence Unit

Voyager: our current search engine

Voyager: our current search engine ► Reliable, critical, proven workhorse

Voyager: our current search engine ► Circa-2002 textbook design ► Basic ranking functionality ► Title-only match by default ► Very literal search

Voyager: our current search engine ► Inflexible & Manual ► The next wave of innovation requires a new search platform…

Project Cassini at eBayOur new search engine

Project Cassini at eBay Our most ambitious core engineering project

Project Cassini at eBay Our most ambitious core engineering project ► Entirely new codebase ► World-class, from a world-class team ► Platform for ranking innovation ► Uses all data by default ► Flexible ► Automated ► Four major tracks, 100+ engineers ► Complete in less than 18 months

Project Cassini at eBay Beginning tests, likely launch in 2012

A Short Primer on Indexing When a user types a query, it isn’t practical to exhaustively scan 200+ million items Instead, we create an inverted index, and use it to rank the items and find the best matches An inverted index is similar to the index in the back of a book:  A set of searchable terms  For each term, a list of locations

An Inverted Index cat 3: 1, 2, 7 1 cat on the mat fat cat 2 3 4 wild cat 5 6 7 8

Distributed Index Construction

 Larger index than Voyager  Descriptions, Seller data, other metadata, …  Much more history in our indexes More computationally expensive work at index- time (and less at query-time) Ability to rescore or reclassify entire site inventory

 Hadoop:  Distributed indexing – platform for hourly index refreshes  Fault tolerance through HDFS replication  Better utilization of hardware – can generate different index types with one cluster

 HBase:  Column-oriented data store on top of HDFS  Used to store eBay’s items  Bulk and incremental item writes  Fast item reads for index construction  Fast item reads and writes for item annotation

 Everyone is still learning Some issues only appear at scale Production cluster configuration is challenging  Hardware issues  Tuning cluster configuration to our work loads HBase stability Monitoring health of HBase Managing workflows – many step map/reduce jobs

Hadoop World 2011 Keynote: Ebay - Hugh Williams

by Cloudera, Inc. on Nov 11, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

18 Embeds 13,460

Statistics

Hadoop World 2011 Keynote: Ebay - Hugh Williams — Presentation Transcript