Scaling a stateless system like web servers & application servers are pretty well understood, but scaling a stateful system has its own set of challenges. In this presentation, we hope to present our learnings & challenges faced while scaling a NoSQL database used within LinkedIn. Scaling a data system involves significant movement & replication of data within in a cluster. This can put considerable load on a system that is already running hot, affecting the service experience We will look at some of the challenges & the approaches that we took.

1. Scaling distributed Data
Systems: a LinkedIn Case
study
By
Sai Kiran Kanuri

2. Agenda
• Brief intro to Espresso
• Challenges & Approaches
• Additional reading

3. Espresso
• A distributed, fault tolerant NoSQL DB
• ~ 7 PB of data footprint in a colo
• Multi Master cross colo support
• Bridges gap between RDBMS & k-v stores

4. Espresso in LinkedIn

5. Espresso Architecture

6. Espresso Storage Node

7. Challenges

8. Fault
Tolerance
Challenges
• Nodes fail all the time
• Failover time
• Data replication
• Partition movement during failures
• Service Discovery

9. Approach to
Fault
Tolerance
• Minimize resource sizes
• Distribute data movement across cluster
• Throttle data migration
• Minimize latency in control events
• Load balancing with error back-off
• Read after write consistency requirements
• Enable client re-tries based on response codes

10. Multi-Tenancy
• Security
• Data growth
• Hot partitions
• Service Discovery

11. Approach to
Multi-Tenancy
• Ensure tenant isolation
• Quotas
• Have a mechanism to transparently migrate
data from one cluster to another
• Be able to re-partition existing data
• Data movement should be transparent to
customer.

12. Future
Changes
• Improving on MTTR with P2P communication
• Cross cluster data balancer
• Dynamic data re-partitioning
• Weighted nodes & partitions

13. Additional Reading
• Espresso - https://engineering.linkedin.com/espresso/introducing-espresso-
linkedins-hot-new-distributed-document-store
• R2D2 - https://github.com/linkedin/rest.li
• Brooklin - https://engineering.linkedin.com/blog/2017/10/streaming-data-
pipelines-with-brooklin
• Helix - https://engineering.linkedin.com/apache-helix/apache-helix-framework-
distributed-system-development