The document discusses the implementation of event storage and real-time analysis at Booking.com using Riak, detailing the structure and flow of events generated from different subsystems. It highlights the need for efficient data serialization and aggregation methods, as well as the architecture chosen for scalability, security, and performance. The presentation concludes with insights on real-time processing, network bandwidth challenges, and the efficacy of using Riak as a robust solution for handling large volumes of event data.

1. Event storage and real-time analysis at
Booking.com with Riak
Damien Krotkine

2. Damien Krotkine
• Software Engineer at Booking.com
• github.com/dams
• @damsieboy
• dkrotkine

3. • 800,000 room nights reserved per day
WE ARE HIRING

4. INTRODUCTION

9. www APImobi

10. www APImobi

11. www APIfrontendbackend
mobi
events storage
events: info about
subsystems status

12. backend
web mobi api
databases
caches
load
balancersavailability
cluster
email
etc…

13. WHAT IS AN EVENT ?

14. EVENT STRUCTURE
• Provides info about subsystems
• Data
• Deep HashMap
• Timestamp
• Type + Subtype
• The rest: specific data
• Schema-less

15. EXAMPLE 1: WEB APP EVENT
• Large event
• Info about users actions
• Requests, user type
• Timings
• Warnings, errors
• Etc…

16. { timestamp => 12345,
type => 'WEB',
subtype => 'app',
action => { is_normal_user => 1,
pageview_id => '188a362744c301c2',
# ...
},
tuning => { the_request => 'GET /display/...'
bytes_body => 35,
wallclock => 111,
nr_warnings => 0,
# ...
},
# ...
}

17. EXAMPLE 2: AVAILABILITY CLUSTER EVENT
• Small event
• Cluster provides availability info
• Event: Info about request types and timings

18. { type => 'FAV',
subtype => 'fav',
timestamp => 1401262979,
dc => 1,
tuning => {
flatav => {
cluster => '205',
sum_latencies => 21,
role => 'fav',
num_queries => 7
}
}
}

19. EVENTS FLOW PROPERTIES
• Read-only
• Schema-less
• Continuous, ordered, timed
• 15 K events per sec
• 1.25 Billion events per day
• peak at 70 MB/s, min 25MB/s
• 100 GB per hour

20. SERIALIZATION
• JSON didn’t work for us (slow, big, lack features)
• Created Sereal in 2012
• « Sereal, a new, binary data serialization format that
provides high-performance, schema-less serialization »
• Added Sereal encoder & decoder in Erlang in 2014

21. USAGE

22. ASSESS THE NEEDS
• Before thinking about storage
• Think about the usage

23. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

24. GRAPHS
• Graph in real-time ( few seconds lag )
• Graph as many systems as possible
• General platform health check

25. GRAPHS

26. GRAPHS

27. DASHBOARDS

28. META GRAPHS

29. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

30. DECISION MAKING
• Strategic decision ( use facts )
• Long term or short term
• Technical / Non technical Reporting

31. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

32. SHORT TERM ANALYSIS
• From 10 sec ago -> 8 days ago
• Code deployment checks and rollback
• Anomaly Detector

33. USAGE
1. GRAPHS
2. DECISION MAKING
3. SHORT TERM ANALYSIS
4. A/B TESTING

34. A/B TESTING
• Our core philosophy: use facts
• It means: do A/B testing
• Concept of Experiments
• Events provide data to compare

35. EVENT AGGREGATION

36. EVENT AGGREGATION
• Group events
• Granularity we need: second

37. event
event
events storage
event
event
event
event
event
event
event
event
event
event
event

38. e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
LOGGER
e
e

39. web api
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
ee
e

40. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
ee
e

41. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
1 sec
e
e

42. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e e
e e
e
e
e e
e
e
e
e
e
ee
e
e
e
1 sec
e
e

43. web api dbs
e
e
e
e
e
e
e
e
e
e
e
e
1 sec
events storage

44. web api dbs
e
e
e
e
e
e
e
e
e
1 sec
events storage
ee e reserialize
+ compress

45. events storage
LOGGER …LOGGER LOGGER

46. STORAGE

47. WHAT WE WANT
• Storage security
• Mass write performance
• Mass read performance
• Easy administration
• Very scalable

48. WE CHOSE RIAK
• Security: cluster, distributed, very robust
• Good and predictable read / write performance
• The easiest to setup and administrate
• Advanced features (MapReduce, triggers, 2i, CRDTs …)
• Riak Search
• Multi Datacenter Replication

50. CLUSTER
• Commodity hardware
• All nodes serve data
• Data replication
• Gossip between nodes
• No master
Ring of servers

51. hash(key)

52. KEY VALUE STORE
• Namespaces: bucket
• Values: opaque or CRDTs

53. RIAK: ADVANCED FEATURES
• MapReduce
• Secondary indexes
• Riak Search
• Multi DataCenter Replication

54. MULTI-BACKEND
• Bitcask
• Eleveldb
• Memory

55. BACKEND: BITCASK
• Log-based storage backend
• Append-only files
• Advanced expiration
• Predictable performance
• Perfect for reading sequential data

56. CLUSTER CONFIGURATION

57. DISK SPACE NEEDED
• 8 days
• 100 GB per hour
• Replication 3
• 100 * 24 * 8 * 3
• Need 60 T

58. HARDWARE
• 16 nodes
• 12 CPU cores ( Xeon 2.5Ghz)
• 192 GB RAM
• network 1 Gbit/s
• 8 TB (raid 6)
• Cluster space: 128 TB

59. RIAK CONFIGURATION
• Vnodes: 256
• Replication: n_val = 3
• Expiration: 8 days
• 4 GB files
• Compaction only when file is full
• Compact only once a day

60. DISK SPACE RECLAIMED
one day

61. DATA DESIGN

62. web api dbs
e
e
e
e
e
e
e
e
e
1 sec
events storage
1 blob per EPOCH / DC / CELL / TYPE / SUBTYPE
500 KB max chunks

63. DATA
• Bucket name: “data“
• Key: “12345:1:cell0:WEB:app:chunk0“
• Value: serialized compressed data
• About 120 keys per seconds

64. METADATA
• Bucket name: “metadata“
• Key: epoch-dc “12345-2“
• Value: list of data keys:
[ “12345:1:cell0:WEB:app:chunk0“,
“12345:1:cell0:WEB:app:chunk1“
…
“12345:4:cell0:EMK::chunk3“ ]
• As pipe separated value

65. PUSH DATA IN

66. PUSH DATA IN
• In each DC, in each cell, Loggers push to Riak
• 2 protocols: REST or ProtoBuf
• Every seconds:
• Push data values to Riak, async
• Wait for success
• Push metadata

67. JAVA
Bucket DataBucket = riakClient.fetchBucket("data").execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk0", Data1).execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk1", Data2).execute();
DataBucket.store("12345:1:cell0:WEB:app:chunk2", Data3).execute();
Bucket MetaDataBucket = riakClient.fetchBucket("metadata").execute();
MetaDataBucket.store("12345-1", metaData).execute();
riakClient.shutdown();

68. Perl
my $client = Riak::Client->new(…);
$client->put(data => '12345:1:cell0:WEB:app:chunk0', $data1);
$client->put(data => '12345:1:cell0:WEB:app:chunk1', $data2);
$client->put(data => '12345:1:cell0:WEB:app:chunk2', $data3);
$client->put(metadata => '12345-1', $metadata, 'text/plain' );

69. GET DATA OUT

70. GET DATA OUT
• Request metadata for epoch-DC
• Parse value
• Filter out unwanted types / subtypes
• Fetch the data keys

71. Perl
my $client = Riak::Client->new(…);
my @array = split '|', $client->get(metadata => '12345-1');
@filtered_array = grep { /WEB/ } @array;
$client->get(data => $_) foreach @array;

72. REAL TIME PROCESSING OUTSIDE OF RIAK

73. STREAMING
• Fetch 1 second every second
• Or a range ( last 10 min )
• Client generates all the epochs for the range
• Fetch all epochs from Riak

74. EXAMPLES
• Continuous fetch => Graphite ( every sec )
• Continuous fetch => Anomaly Detector ( last 2 min )
• Continuous fetch => Experiment analysis ( last day )
• Continuous fetch => Hadoop
• Manual request => test, debug, investigate
• Batch fetch => ad hoc analysis
• => Huge numbers of fetches

75. events storage
graphite
cluster
Anomaly
detector
experiment
cluster
hadoop
cluster
mysql
analysis
manual
requests
50 MB/s
50 MB/s
50
M
B/s
50
M
B/s
50 MB/s
50 MB/s

76. REALTIME
• 1 second of data
• Stored in < 1 sec
• Available after < 1 sec
• Issue : network saturation

77. REAL TIME PROCESSING INSIDE RIAK

78. THE IDEA
• Instead of
• Fetching data, crunch data, small result
• Do
• Bring code to data

79. WHAT TAKES TIME
• Takes a lot of time
• Fetching data out
• Decompressing
• Takes almost no time
• Crunching data

80. MAPREDUCE
• Send code to be executed
• Works fine for 1 job
• Takes < 1s to process 1s of data
• Doesn’t work for multiple jobs
• Has to be written in Erlang

81. HOOKS
• Every time metadata is written
• Post-Commit hook triggered
• Crunch data on the nodes

83. Riak post-commit hook
REST serviceRIAK service
key key
socket
new data sent for storage
fetch, decompress
and process all tasks
NODE HOST

84. HOOK CODE
metadata_stored_hook(RiakObject) ->
Key = riak_object:key(RiakObject),
Bucket = riak_object:bucket(RiakObject),
[ Epoch, DC ] = binary:split(Key, <<"-">>),
Data = riak_object:get_value(RiakObject),
DataKeys = binary:split(Data, <<"|">>, [ global ]),
send_to_REST(Epoch, Hostname, DataKeys),
ok.

85. send_to_REST(Epoch, Hostname, DataKeys) ->
Method = post,
URL = "http://" ++ binary_to_list(Hostname)
++ ":5000?epoch=" ++ binary_to_list(Epoch),
HTTPOptions = [ { timeout, 4000 } ],
Options = [ { body_format, string },
{ sync, false },
{ receiver, fun(ReplyInfo) -> ok end }
],
Body = iolist_to_binary(mochijson2:encode( DataKeys )),
httpc:request(Method, {URL, [], "application/json", Body},
HTTPOptions, Options),
ok.

86. REST SERVICE
• In Perl, using PSGI, Starman, preforks
• Allow to write data cruncher in Perl
• Also supports loading code on demand

87. ADVANTAGES
• CPU usage and execution time can be capped
• Data is local to processing
• Two systems are decoupled
• REST service written in any language
• Data processing done all at once
• Data is decompressed only once

88. DISADVANTAGES
• Only for incoming data (streaming), not old data
• Can’t easily use cross-second data
• What if the companion service goes down ?

89. FUTURE
• Use this companion to generate optional small values
• Use Riak Search to index and search those

90. THE BANDWIDTH PROBLEM

91. • PUT - bad case
• n_val = 3
• inside usage =
3 x outside usage

92. • PUT - good case
• n_val = 3
• inside usage =
2 x outside usage

93. • GET - bad case
• inside usage =
3 x outside usage

94. • GET - good case
• inside usage =
2 x outside usage

95. • network usage ( PUT and GET ):
• 3 x 13/16+ 2 x 3/16= 2.81
• plus gossip
• inside network > 3 x outside network

96. • Usually it’s not a problem
• But in our case:
• big values, constant PUTs, lots of GETs
• sadly, only 1 Gbit/s
• => network bandwidth issue

97. THE BANDWIDTH SOLUTIONS

98. THE BANDWIDTH SOLUTIONS
1. Optimize GET for network usage, not speed
2. Don’t choose a node at random

99. • GET - bad case
• n_val = 1
• inside usage =
1 x outside

100. • GET - good case
• n_val = 1
• inside usage =
0 x outside

101. WARNING
• Possible only because data is read-only
• Data has internal checksum
• No conflict possible
• Corruption detected

102. RESULT
• practical network usage reduced by 2 !

103. THE BANDWIDTH SOLUTIONS
1. Optimize GET for network usage, not speed
2. Don’t choose a node at random

104. • bucket = “metadata”
• key = “12345”

105. • bucket = “metadata”
• key = “12345”
Hash = hashFunction(bucket + key)
RingStatus = getRingStatus
PrimaryNodes = Fun(Hash, RingStatus)

106. hashFunction()
getRingStatus()

107. hashFunction()
getRingStatus()

108. WARNING
• Possible only if
• Nodes list is monitored
• In case of failed node, default to random
• Data is requested in an uniform way

109. RESULT
• Network usage even more reduced !
• Especially for GETs

110. CONCLUSION

111. CONCLUSION
• We used only Riak Open Source
• No training, self-taught, small team
• Riak is a great solution
• Robust, fast, scalable, easy
• Very flexible and hackable
• Helps us continue scaling

112. Q&A
@damsieboy