eBay has been using MongoDB to power several large scale applications due to its ability to scale horizontally and handle high volumes of data and queries. Some key applications summarized include search suggestions which handles high volume lookups, a cloud manager which tracks resource states, and a media metadata store which will store tens of terabytes of image data partitioned across multiple shards. MongoDB provides benefits like flexible schemas, automatic shard rebalancing, and proximity-aware replication which help eBay meet performance and scalability needs for these systems.

1. Yuri Finkelstein
eBay Platform Architect
yfinkelstein@ebay.com
May 2012

2. DB Scalability @ eBay
 eBay is one of the first and largest BASE
environments based on Oracle DB App1 App2
• Basic Availability
• Soft-state Business Business
• Eventual consistency Logic Logic
 Every database we use is shared and partitioned
Hint (shard key) Hint (shard key)
• N logical hosts names are defined for each use case ahead
of time
DAL DAL
• These logical hosts are mapped to physical based on static
mapping tables which are controlled by DBAs Framework Framework
• A common ORM framework called DAL provides powerful
and consistent patterns for data scalability
Applications
 If the client provides a hint along with every DB F1(Hint) F2(Hint)
query:
• DAL maps the hint to a logical host using one of N mapping Logical DB
schemes (ex: modulus, lookup table, range, etc)
… hosts …
• Logical host is then mapped to a physical using L-to-Ph map (shards)
• The query is sent to just one shard
 If the client does not have a hint, the query is sent to Config
all shards and the results are joined on the client Physical
with the help of DAL framework … Master DB
 Side-effects: hosts
• Hint is not part of the query; client has to manage it Physical
• Logical to Physical mapping scheme becomes extra piece of … standby
client configuration
• Shard rebalancing is “DBA magic” DB hosts

3. Key desired improvements
 All eBay site-facing applications use the scheme outlined above
 It’s proven to scale to tens of thousands of developers, petabytes of data, hundreds
of millions of SQL queries per day
 But there is always room for improvements and new ideas
• ORM is not the fastest way to develop; how do we achieve faster development cycles and reduce
schema mapping frictions?
• How do we add new attributes to tables faster and without DBA’s involvement? Schema free approach
sounds interesting.
• Can we make the hint transparent, ex: auto-extract it from queries?
• Can we rebalance the data seamlessly and automatically?
• Can we add shards faster in order to scale out on demand and transparently to applications?
• How do we deploy new DBs to the cloud on demand?
 And what about performance? Can we use RAM more aggressively and
seamlessly to speed up queries?

4. Enters MongoDB
 We are playing with MongoDB since 2010.
Why? Business
Logic Document
 Its scalability scheme is very similar to how
we shard RDBMS Morphia/Mongo
• Single master for writes, eventually consistent slaves for
Driver
reads Dynamic
• Horizontal partitioning of data sets is a norm at eBay Config
• MongoS is performing familiar scatter-gather and client-
side merge-sorts MongoS
F(Shard Key)
 We don’t use distributed transactions since
day 1; transactional updates of multiple tables
…
that we do use can be simulated by atomic
<- Replicas ->
updates of a single Mongo document
 MongoDB offers a number of features that …
help address our goals mentioned earlier:
• Developers love document model and schema-free
persistence
…
• Hints are embedded into the queries
• MongoDB has automatic shard rebalancing
• Shards can be added on demand without application
restart and data will be auto-rebalanced ---------- Shards -------
• We can easily bring it up in the cloud since cloud
machines have storage

5. Case study #1: eBay Search Suggestions
 Search suggestion list is a MongoDB document
indexed by word prefix as well as by some
metadata: product category, search domain,
etc.
 Must have < 60-70msec round trip end to end
 MongoDB query < 1.4msec
 Data set fits in RAM; 100-s M documents
 Data is bulk loaded once a day from Hadoop,
but can be tweaked on demand during sale
promotions, etc
 Single replica set, no shards in this case
 MongoDB benefits:
• Multiple indexes allow flexible lookups
• In-memory data placement ensures lookup speed
• Large data set is durable and replicated

6. Case Study #2: Cloud Manager “State Hub”
Query  State Hub powers eBay Cloud
Provision
Resources
Resources  Every resource provisioned by the cloud is
and Topology
represented by a single Mongo document
 Documents contain highly structured
metadata reflecting roles and grouping of
the resources
 Lookup by both primary and secondary
State Hub indexes
Mongo  Several GB data sets, easily fit in RAM
Update
 Documents are not uniform
resource
state  All resources have “State” field which is
updated periodically to reflect health state
of the underlying resource
 Mixed workload: lots of in-place writes, but
also lots of read queries

7. Case Study #3: eBay Merchandizing Info Cache
 Merchandizing backend powers eBay product/item
classification and categorization
 Each MongoDB document represents a cluster of similar
products
 Numerous relationships between clusters are modeled as
R1 document attributes
Cluster1 Cluster2
 Relationship hierarchy traversal is achieved by issuing a
R3 number of queries on “edge” attributes
R2
 Each instance of such a hierarchy is called a model; there
Cluster3 are lots of models
 Again, data set fits in RAM, single replica set
 Replica set members are located in 3 different data
centers (3+2+2) with all members in a single data center
having higher weight to avoid moving master away
 MongoDB benefits:
• Schema-free design and declarative indexes are perfect for this use
case where new attributes and new queries are constantly being
added
• Async replication across multiple data centers
• MongoDB Java Driver ensures automatic detection of proximity
of clients to replica set members; reads with slaveOK=true are
served from local data center nodes which insures low
response latency

8. Case Study #4: Zoom – Media Metadata Store
 This is a new mega project which is a work in progress
 MongoDB is being evaluated as a storage backend for all media-related
metadata on the site (example: picture IDs with lots attributes)
 Requirements:
• Tens of TBs data set, Millions of documents: data set must be partitioned; this is our
first use case where MongoDB sharding is used
• System of record for picture info; data can not be lost!
• Replication/DR across 2 data centers; local DC reads are required
• Queries are from site-facing flows; <10msec response time SLA
• Mixed workload: both inserts and reads are happening concurrently all the time
 Can MongoDB do it ??

9. Zoom: Data Model
 2 main collections: Item and Image
• Item references multiple Images
 Item represents eBay Item:
• _id in Item is external ID of the item in eBay site DB
• These IDs are already sharded in balanced across N
logical DB hosts using ID ranges
• We use MongoDB pre-split points for initial
mapping our N site DB shards to M MongoDB shards
• This ensures good balance between the shards;
 Image represents a picture attached to an
Item
• _id in Image is md5 of the image content
• This ensures good distribution across any number of
shards
• Md5 is also used to find duplicate images
 Our choice of document IDs in both
collections ensures good balance across
Mongo shards
 We never query both collections in a single
service request to ensure data consistency
and to have only one index lookup

10. Zoom: Service Topology and Configuration
 MongoS is deployed on app servers
• Ensures network IO on MongoS won’t become a bottleneck
• This is a very familiar pattern in eBay as was explained in the
>
--- DC1(Primary)---
beginning of this presentation
 M shards; each replica set has 6 members
M M M M • 3 + 3 in 2 data centers
• Master can be only in one DC during automatic failover; manual
failover may activate another DC
--- Replicas ---> • One slave in the secondary DC is invisible for reads and is
dedicated to periodic backups/snapshots (more on this later)
 For reads, client first sets SlaveOK=true and if
required document is not found flips to
SlaveOK=false to read from Master
-- DC2(Secondary)-->
 Home-grown MongoDB configuration and monitoring
agent is running on every node
• Fetches MongoD configuration from a central configuration store
and saves it to local config file
• Manages lifecycle of MongoD
B B B B • Monitors state and metrics
---- Shards -----

11. Zoom: Data Backup and Restore strategy
 Goals:
• Take periodic backups of the entire data set
Application • Be able to recover from backup
• Do not loose any writes that have happened after last snapshot
• Briefly service unavailability during recovery is better than data
Dual-write loss …
to capped
M collection C  Dual writes on the client
• Regular write to main cluster
…
• Second write to another Mongo cluster: single replica set,
capped collection, the data written is similar to REDO log record
Recovery
B Agent  Hidden slave in each shard has volume mounted on a
remote storage appliance capable of instant file
system snapshot; captures both DB files and journal
files
 If DB recovery is activated:
• All MongoD on primary cluster are shutdown
• NFS slave is remounted to snapshot volume
Instant • MongoD on this machine is started as a master
Shapshot • MongoD on other replica set members are started cold
• Full sync-up from master
Capable • Master is switched to a regular member
device • Writes that occurred since time when the backup was taken
are replayed from the REDO log capped collection in the
secondary cluster
•

12. Key Learning
 MongoDB can be a very powerful tool but use it wisely
 Deletes can be slow; automatic balancer is dangerous; use it only when you
must (example: be careful when adding new shards)
 Use explain for every query; disable full scans to discover inefficiencies
early
 Query profiler is great
 Retry every failed query at least once; long tail in response times is possible
when data set > RAM size

13. Questions?
 Thank you!