Kiril Savino, CTO of GameChanger Media, discusses his experience growing MongoDB from 10 years with Oracle and MySQL to now using MongoDB for 4 years. His company's MongoDB deployment serves over 865 million documents totaling 3 terabytes of data across 16 nodes with 240GB of RAM and 8TB of SSD storage, sustaining 120,000 operations per second. He provides learnings around schemas, concurrency, availability, and firefighting for large-scale MongoDB deployments.

1. Growing up
MongoDB
Kiril Savino - CTO GameChanger
@kirilnyc

2. About Me.
Lead Engineer, Higher One
Lead Engineer, DoubleClick
Lead Engineer -> CTO, ShopWiki
Director Engineering, Conductor
Founder & CTO, GameChanger Media
10 years Oracle and MySQL, 4 MongoDB

3. pre-$not

4. About GameChanger.

5. Growing up.
865,443,426+
3 terabytes
16-nodes
240GB RAM, 8TB SSD storage
120,000 ops/s sustained

6. Learnings.
Schemas
Concurrency
Availability
Firefighting

7. 1. Schemas!

8. Schema-less.
I do not think that word means
what you think it means.

9. Be abnormal.

10. Schema-less does mean not having
to separate data for modeling
reasons
Focus on data usage patterns along
with semantics
You're going to have to do this
anyway: start now and scale up
easy

11. Go monolithic.
Learn to query, then forget and
pretend MongoDB is a really
fancy, full-featured KV store.

12. Querying secondary data is a waste
Scans & indexed queries are slow
Memory fragmentation can kill you
Use MongoDB's strengths

13. Garbage in...

14. Validation is your problem
Don't let inconsistency linger
Know what parts of schema are
flexible

15. Index wisely.
Data size
Insert/Update speed
Bad schema smell

16. 2. Concurrency!

17. (A)CID.

18. Good schema design provides for
basic atomicity at the document
level
Obviates the need for transactions
in many trivial cases

19. Your friends.
$set/$unset
$push/$pull
$addToSet
findAndModify

20. Two phased commits.

21. Optimistic locking.

22. External transactions.

23. Eventual Consistency.
Write canonical data first
Ensure queuing of propagation
Guarantee queue entry completeness

24. 3. Availability!
“/dev/null is web scale”

25. Durability.
Journaling.
OK?

26. Replication.

27. Moar = better?

28. 4. Firefighting!

29. Test your capacity.
Naïve throughput testing with real
hardware
Clone prod configuration
Consider copying data or subset
Start with crude approximations
Get as close to “real load” as
makes sense

30. Model your growth.
db.stats()
db.<collection>.stats()
avg doc size, avg index size / doc
growth rates / collection
approx active portion / collection

31. Read your logs.
{...}
ntoreturn:1
keyUpdates:0
numYields: 136
locks(micros) r:368727
reslen:78 199ms

32. Don’t scan.

33. So...
Schema-less != no schema
ACID overrated; concurrency not
High availability is up to you
Understand the mechanics

34. Thanks!
Kiril Savino
CTO, GameChanger Media
www.GameChanger.io
@kirilnyc
kirilsavino.com/blog

35. Next Sessions at 3:40
5th Floor:
West Side Ballroom 3&4: Advanced Replication Internals
West Side Ballroom 1&2: Building a High-Performance Distributed Task
Queue on MongoDB
Juilliard Complex: WhiteBoard Q&A
Lyceum Complex: Ask the Experts
7th Floor:
Empire Complex: Managing a Maturing MongoDB Ecosystem
SoHo Complex: MongoDB Indexing Constraints and Creative Schemas