2.5B+ ids, 2ms latency, 15K+ TPS and Petabytes of data.These numbers outline the challenges with eBay’s Entity Resolution Service (ERS). ERS provides a temporal map between anyid-anyid. The technology stack of ERS has Hadoop as the batch layer, Couchbase as cache layer, Spring Batch to load data to Couchbase and Rest API at Service layer. In our presentation we will take you through the journey from conceptual to production release. It’s a great story and we would like to share with you!

1. Entity Resolution Service
Bringing Petabytes of Data Online for Instant Access
Gurpreet Singh, eBay
Ying Zhang, eBay

2. Problem Statement
2
eBay maintains hundreds of millions of accounts* across our properties
and partners, that are sometimes unstructured, in different formats,
different character sets, and are changing over time. Identifying which
accounts belong to the same person enables us to personalize each
customer's experience, deliver great customer service, and fight fraud.
ERS brings xID along with device information online.

3. Background
3
xIDpii xIDguid
ERS

4. xID
4
Links User Credentials and transient identities to recognize a
customer:
–Hundreds of millions of user accounts.
–20+ different linking strategies with the flexibility of adding more.
–Billions of cookies, browser fingerprints, all mobile identities.
–Customer recognition across sessions, sites, browsers, devices.

5. xID Modular Solution
5
Sources Edges Graph Table
Account Entity
102832 10921
236896 10921
786273 10921
324324 23987
349709 73652
152631 73652
543273 37726

6. SOURCES – Overall Data Flow
6

7. Architecture
7
Monitoring

8. Technology Stack
8

9. ERS
9
Enables access to consolidated customer Identity
–RESTFul service with ~2ms response time
–Optimized access to customer identities
–10K+ requests per second
– 70 Nodes Couchbase Cluster with 11 TB memory in 3 data centers

10. Challenges
10
LARGE DATASET
SKILLSET
COMPLEX
TIMELINE
FUTURE GROWTH
Raw clickstream + PII ~ in PB
20+ edge types
100+ billion edges

11. Challenge
11
LARGE DATASET
SKILLSET
COMPLEX
TIMELINE
FUTURE GROWTH
Graph Join
Process
Geo Redundancy
Requirement

12. Challenge
12
LARGE DATASET
SKILLSET
COMPLEX
TIMELINE
FUTURE GROWTH
Downstream waiting
Infrastructure
Deliver it ASAP!

13. Challenge
13
LARGE DATASET
SKILLSET
COMPLEX
TIMELINE
FUTURE GROWTH
NoSQL knowledge
Near Real Time
Processing

14. Challenge
14
LARGE DATASET
SKILLSET
COMPLEX
TIMELINE
FUTURE GROWTH
More business Usecases
Data Volume
Data Variability
Data Velocity

15. Key Take Away’s
1
5
• Optimize CB data load to prevent negative service impact
• Reduce JSON size to increase CB Active Resident Ratio
• Geo-Redundancy by data replication
• Enterprise BigData solutions are complex (Velocity, Volume and
Variability)

16. THE END