Apache Hadoop India Summit 2011 talk "Hadoop Avatar at eBay" by Srinivasan Rengarajan and Mohit Soni

1
Avatar at eBay
Srinivasan Rengarajan (srengarajan@ebay.com)
Mohit Soni (mosoni@ebay.com)
Courtesy
Anil Madan (amadan@ebay.com)

2
2007 Research Team Builds a 4 node Cluster
Subset of Click Stream and EDW data
Innovation with Mobius Query Language
Visualization and Click Path analysis
2009 Sept Search Clusters
Machine Learning Ranking cluster of 28 nodes
Search relevance cluster of 10 nodes
Subset of Click Stream and EDW Data
2010 May – Athena* Exploratory Cluster of 532 nodes
Platform Teams join hands with Search/Research to build a larger cluster .
Build it as a core competency for advanced insights for complex data
Rapid build-out with timelines pulled in by couple of months
* Athena, is the goddess of civilization, wisdom, strength, strategy, craft, justice and skill in Greek mythology
MIT's Athena ushered the world in a new era of distributed systems when it started in the mid 80s.
2

Infrastructure
3
Enterprise Nodes
Sun 64bit , Red Hat Linux
2 Quad Core Nehalem, 72GB RAM, 4TB
Servers
NameNode(s)
Job Tracker
Zookeeper
HBaseMaster
Ganglia Server
eBay (Cloudera) HUE
Data Nodes
SGI-Rackables, Cent OS, 1U , 5.3PB
2 Quad Core Nehalem, 36GB RAM, 10TB
Hbase on 20 nodes
Network
TOR 1Gbps
Core Switches uplink 40Gbps
3

Ecosystem
4
Monitoring & Alerting
Ganglia, Nagios
Tools
HUE/Mobius – lifecycle of user jobs UC4 - scheduling Oozie – user workflow and data pipelines
Mahout – data mining
Monitoring & Alerting
(Ganglia, Nagios)
Tools & Libraries
(HUE,UC4,Oozie.Mobius,Mahout)
Data Access Frameworks
Hbase - for EDWdata
Pig – data piplelines
Hive – Adhoc queries MQL – Mobius Query Language
Data Access
(Hbase, Pig, Hive)
MapReduce
(Java, Streaming, Pipes,Scala)
Hadoop Core
(HDFS,Common)
MapReduce
Sourcing data primarily Java Applications using Perl, Scala, Python…
4

Administration
Groups
Built to support multiple groups
Job invocation uses the group name
Fair Scheduler
Allocations based on investment
Weights
Minimum share of mappers and reducers
poolMaxJobsDefault
userMaxJobsDefault
defaultMinSharePreemptionTimeout
fairSharePreemptionTimeout
Auth & Auth
HUE – custom module to use corp. credentials
CLI*– PAM custom module
Security* - Implement token interface to replace Kerberos with SAML.
* Work in Progress
5

Data Sourcing Patterns
6
Click Stream
Search Indices
EDW
Analytics Reporting
Description
Acquisition
Algorithmic Models
Images

Search Use Case – Machine Learned Ranking
7
ClickStream
Items
Users
Feedback
Classifiers
Ranking Function
Great Search Results
Goal
Enhance search relevance for eBay’s items.
Hadoop Usage
Build a ranking function that takes multiple factors into account like price, listing format, seller track record, relevance.
Ability to add new factors to validate hypothesis
.

Research Use Case – Description Data Mining
8
BARBIE
1999 "PREMIERE NIGHT"
Home Shopping Special Edition
Gorgeous Doll With Beautiful Blond Hair / In A Gown Of Purple And Silver
New / Never Removed From Box / Doll Is In Mint Condition / Remember This Beauty Is 11 Years Old
Free Shipping To US Only / Will Ship International / Please E-mail For Cost
Feel Free To Ask Me Any Questions Or Concerns
Smoke - Free Environment
Free Shipping
Year: 1999
Model: premiere night
Edition: home shopping special
Hair: blond
Gown: purple and silver
Condition: new / never removed from box / mint
Goal
Extend catalog coverage
Hadoop Usage
Leverage data mining/machine learning techniques to create inventory into name value pairs
in an completely unsupervised way

10
Acknowledgments
Athena Team
Cloudera Inc.
Community

http://d.hatena.ne.jp	203
http://www.mohitsoni.com	64
http://presenyuzuru.posterous.com	4
http://paper.li	1

Apache Hadoop India Summit 2011 talk "Hadoop Avatar at eBay" by Srinivasan Rengarajan and Mohit Soni

by Yahoo! Developer Network

Accessibility

Upload Details

Usage Rights

4 Embeds 272

Statistics

Apache Hadoop India Summit 2011 talk "Hadoop Avatar at eBay" by Srinivasan Rengarajan and Mohit Soni Presentation Transcript