Download as PDF, PPTX
![What is an event?
• message with technical
and business data
• Sereal encoded
• srl([ $e, $e, $e, … ])](https://image.slidesharecdn.com/solvingsomeofthescalabilityproblemsatbooking-170811155447/75/Solving-some-of-the-scalability-problems-at-booking-com-4-2048.jpg)
More Related Content
![[231] the simplicity of cluster apps with circuit](https://cdn.slidesharecdn.com/ss_thumbnails/213thesimplicityofclusterappswithcircuit-150915022150-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Solving some of the scalability problems at booking.com
Solving some of the scalability problems at booking.com
- 1. Solving some ofthe scalability problems at Booking.com Ivan Kruglov YAPC::EU 2017 based on Oleg Sidorov’s slides
- 2.
- 3.
- 4. What is anevent? • message with technical and business data • Sereal encoded • srl([ $e, $e, $e, … ])
- 5. WEB CRON e-mail FAX MySQL VoIP Transport monitoring hadoop/hiveA/B testing elastic search
- 6. • Distributed, dc-faulttolerant • Generates graphite metrics from events • Runs user-defined code (~260 monitors) • Processes events second-by-second (epochs) • upto ~500k events per second Event Graphite Processor.
- 7. Event Graphite Processor. epoch_of_events= get_events_for_epoch(now) foreach monitor : monitors result = monitor.run(epoch_of_events) graphite.send(result)
- 8. Event Graphite Processor. •The dataset is huge and it’s growing • Every second of events takes 10–15GB of RAM • Monitors are split into groups to run faster • Every group runs in a fork • Forking provokes COW • RAM is being saturated • No RAM = the box is being kicked out
- 9.
- 10. Processing. Thinking different. epoch_of_events= sereal_decoder.parse(srl_events) foreach (event : epoch_of_events) ... vs iterator = sereal_decoder.iterator(srl_events) while event = iterator.next() ...
- 11. Processing. Thinking different. epoch_of_events= get_events_for_epoch(now) iterator = epoch_of_events.iterator while event = iterator.next() foreach monitor : monitors monitor.process_event(event) foreach monitor : monitors result = monitor.post_process() graphite.send(result) Problem: need to rewrite user code
- 12. epoch_of_events = get_events_for_epoch(now) foreachmonitor : monitors result = monitor.run(epoch_of_events) graphite.send(result) epoch_of_events = get_events_for_epoch(now) iterator = epoch_of_events.iterator while event = iterator.next() foreach monitor : monitors monitor.process_event(event) foreach monitor : monitors result = monitor.post_process() graphite.send(result) vs
- 13. How? • Sereal::Path::Iterator a. iterateover objects (scalar/arrayref/hashref/blessed/etc) b. decode from current position • no streaming • check limitations
- 14. Still need torewrite user code!
- 15. A proof ofconcept. FlogCron
- 17. 1. Identical stack 2.Same problems 3. Easy to migrateA proof of concept. FlogCron
- 18. First results: promising. •CPU: no changes • Processing time: 60sec vs 30sec • # of boxes: 20 vs 8 • RAM: 10GB vs 100MB
- 19. 1. RAM isan issue 2. New user monitors 3. New systems 4. More events every day EGP: Time to act!
- 20. 1. Implement aproof of concept 2. Freeze EGP development 3. Migrate all monitors 4. Full-scale test 5. Roll out the new system 6. Profit! EGP migration TODO
- 21. 1. 8 people 2.All done in 1 day 3. 260 monitors 4. 317 files changed, 10336 insertions, 11288 deletions 5. Ready to run a full-scale test Migration. Hackathon. Results
- 22.
- 23. Chasing the problem. foreach(@events) { … } while (my $event = $iterator->next()) { … } POSIX::exit POSIX::exit undef $event # ~500k times, ~15GB in total
- 25. 1. Xeon E5-2690 2.56 logical cores The new hardware.
- 26. 1. No moreRAM constraints 2. More aggressive forking 3. 3x more groups/forks Parallelization. Fork me!
- 27.
- 28. 1. Processing time:80sec vs 40sec 2. RAM: 16GB vs 500MB 3. # of boxes: 80 vs 30 The results. The new system
- 29. 1. Engineering isthe king, collaboration is the queen 2. The ideas that failed individually might work together 3. Challenge everything Lessons learned.
- 30.