Downloaded 25 times
More Related Content
Similar to Using Kafka to scale database replication
Using Kafka to scale database replication
- 1. How LinkedIn usedKafka to scale Database Infrastructure Basavaiah Thambara(Basu) Staff SRE https://www.linkedin.com/in/basavaiaht
- 2. Today’s agenda Introduction to Espresso- DataStore Espresso - Replication needs Limitations of using MySQL Replication Espresso Replication using Kafka Advantages of using Kafka How Kafka based replication works Conclusion & References
- 3. Espresso ● Document store ●Built on top of MySQL ● Bridges gap between RDBMS & k-v stores ● Features ■ Multi-colo ■ Secondary Indexing ■ Schema evolution ■ Change data capture ■ ETL to and from Hadoop ● Use cases ■ Profiles, Invitations, InMails, etc.
- 4. Scale of usage ●80% of site facing databases ● No.of clusters - 145 , servers - ~19k ● No.of databases - ~300 ● Data size - ~12PB ● Peak qps - ~3.4 M on single data store
- 5. Database Sharding ● Database ●Shard or partition
- 6. Espresso Basic Architecture ●Storage node ● Apache Helix ● Zookeeper ● Router ● Client/application
- 7. The need ofreplication ● Read scaling ● High availability ● Disaster Recovery ● Multi-colo support ● Backups
- 8. Espresso - localreplication ● MySQL replication ● Per node replication ● Master ● Slave ● Master serves ● Node failure old design
- 9. Espresso - crosscolo replication ● Multi-colo writes ● Last writer win ● Databus ● Data Replicator ● Colo failure old design
- 10. Limitations of usingMySQL replication ● Poor resource utilization
- 11. Limitations of usingMySQL replication ● Cluster expansion is complex
- 12. Limitations of usingMySQL replication ● Upon master failure, single node gets traffic ● Human intervention to bring up slaves ● Slave less situation might lead to outage When master goes down When slaves go down
- 13. Limitations of usingMySQL replication ● Databus operational complexity ● Databus maintenance cost
- 14. Espresso : Replicationusing kafka ● Per partition replication ● Flexible partition placement ● Every node serves traffic ● Data replicator uses kafka New design
- 15. Advantages of usingkafka ● Better h/w utilization ● Cluster expansion is easy as ■ add node(s) to cluster ■ rebalance ● No human intervention
- 16. Advantages of usingkafka ● Node failure ■ parallel mastership handoff ■ parallel restore of slaves ● Databus complexity eliminated ● Huge cost savings ● Single platform for ■ internal replication ■ cross colo replication
- 17. Kafka based replication ●Delivery must be ■ guaranteed ■ In-Order ■ Exactly Once
- 18. GTIDs and SCNs ●Global transaction identifier ● Unique
- 19.
- 20. Espresso Kafka Producer ●part of storage node ● Uses Open Replicator ● Single Threaded
- 21.
- 22. Message protocol -Mastership Handoff
- 23. Message protocol -Mastership Handoff
- 24. Message protocol -Mastership Handoff
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. Producer configuration ● acks= “all” ● Infinite retries ● block.on.buffer.full = true ● max.in.flight.requests.per.connection = 1 ● linger = 0 ● on non-retryable exception ■ destroy producer ■ create new producer ■ resume from last checkpoint
- 31. Espresso Kafka eventConsumer
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. Kafka broker configand spec ● Kafka broker config ■ replication factor =3 ■ min.isr = 2 ■ Disabled unclean leader elections ● Kafka broker node spec ■ 256GB RAM ■ 8 core, intel @ 2.00GHz ■ 19TB HDD with RAID ■ os - RHEL 6
- 43. Kafka replication stats ●Kafka cluster in each colo ● No.of kafka brokers - 336 ● Peak 500MB per sec , 36 TB per day ● Peak 1.5M messages per sec, 34 billion per day
- 44. Conclusion ● Kafka isused for database replication at scale ● LinkedIn leveraged Kafka to scale Espresso ● Kafka helped to Unify data pipelines ● Saved $$$
- 45.
- 46.