This document summarizes how Booking.com solved scalability issues with their Event Graphite Processor (EGP) system. The EGP processes large volumes of event data to generate metrics but was limited by high RAM usage. A new approach was developed that uses event streaming and parallelization, reducing processing time from over 120 seconds to 80 seconds while using much less RAM. This was achieved through a hackathon that rewrote 260 monitors in one day. The new system uses 56-core servers, processes events in parallel groups, and requires only 500MB of RAM compared to the previous 15GB.

1. Solving some of the scalability
problems at Booking.com
Ivan Kruglov
YAPC::EU 2017
based on Oleg Sidorov’s slides

2. graphite processor

3. event graphite processor

4. What is an event?
• message with technical
and business data
• Sereal encoded
• srl([ $e, $e, $e, … ])

5. WEB
CRON
e-mail FAX
MySQL
VoIP
Transport
monitoring hadoop/hive A/B testing elastic search

6. • Distributed, dc-fault tolerant
• Generates graphite metrics from events
• Runs user-defined code (~260 monitors)
• Processes events second-by-second (epochs)
• upto ~500k events per second
Event Graphite Processor.

7. Event Graphite Processor.
epoch_of_events = get_events_for_epoch(now)
foreach monitor : monitors
result = monitor.run(epoch_of_events)
graphite.send(result)

8. Event Graphite Processor.
• The dataset is huge and it’s growing
• Every second of events takes 10–15GB of RAM
• Monitors are split into groups to run faster
• Every group runs in a fork
• Forking provokes COW
• RAM is being saturated
• No RAM = the box is being kicked out

9. Processing.
Think different.

10. Processing. Thinking different.
epoch_of_events = sereal_decoder.parse(srl_events)
foreach (event : epoch_of_events) ...
vs
iterator = sereal_decoder.iterator(srl_events)
while event = iterator.next() ...

11. Processing. Thinking different.
epoch_of_events = get_events_for_epoch(now)
iterator = epoch_of_events.iterator
while event = iterator.next()
foreach monitor : monitors
monitor.process_event(event)
foreach monitor : monitors
result = monitor.post_process()
graphite.send(result)
Problem: need to rewrite user code

12. epoch_of_events = get_events_for_epoch(now)
foreach monitor : monitors
result = monitor.run(epoch_of_events)
graphite.send(result)
epoch_of_events = get_events_for_epoch(now)
iterator = epoch_of_events.iterator
while event = iterator.next()
foreach monitor : monitors
monitor.process_event(event)
foreach monitor : monitors
result = monitor.post_process()
graphite.send(result)
vs

13. How?
• Sereal::Path::Iterator
a. iterate over objects
(scalar/arrayref/hashref/blessed/etc)
b. decode from current
position
• no streaming
• check limitations

14. Still need to rewrite
user code!

15. A proof of concept.
FlogCron

17. 1. Identical stack
2. Same problems
3. Easy to migrateA proof of concept.
FlogCron

18. First results: promising.
• CPU: no changes
• Processing time: 60sec vs 30sec
• # of boxes: 20 vs 8
• RAM: 10GB vs 100MB

19. 1. RAM is an issue
2. New user monitors
3. New systems
4. More events every day
EGP: Time to act!

20. 1. Implement a proof of concept
2. Freeze EGP development
3. Migrate all monitors
4. Full-scale test
5. Roll out the new system
6. Profit!
EGP migration TODO

21. 1. 8 people
2. All done in 1 day
3. 260 monitors
4. 317 files changed,
10336 insertions,
11288 deletions
5. Ready to run a full-scale test
Migration.
Hackathon.
Results

22. 80s
vs
120+s

23. Chasing the problem.
foreach (@events) {
…
}
while (my $event = $iterator->next()) {
…
}
POSIX::exit
POSIX::exit
undef $event # ~500k times, ~15GB in total

25. 1. Xeon E5-2690
2. 56 logical cores
The new hardware.

26. 1. No more RAM constraints
2. More aggressive forking
3. 3x more groups/forks
Parallelization.
Fork me!

27. 2 times faster.

28. 1. Processing time: 80sec vs 40sec
2. RAM: 16GB vs 500MB
3. # of boxes: 80 vs 30
The results.
The new system

29. 1. Engineering is the king,
collaboration is the queen
2. The ideas that
failed individually might
work together
3. Challenge everything
Lessons learned.

30. Ivan Kruglov
ivan.kruglov@booking.com
Thank you!