We will be talking about components of our architecture, how do data flow via those components and what is the real life use case applied to it. Evolution and vision as critical aspect of building such platform. Technology stack that we will talk about includes Kafka, Hadoop stack, Spark, Legacy MongoDB, IMF.

1. Analy&cs for the fastest growing companies

2. λ architecture applied
at Exponea
Mar&n Strýček, 10.3.2016

4. Choose your stack

5. Intuitive approach
• Collect data to (no)SQL DB
• Running live queries against (no)SQL database
• UI will be like SQL generator
• But may/will eventually result in slow queries
• Batch preprocessing of data
• Continues change of report definitions
• Delays / Over night results / no more night

6. Conversion Funnel

7. Valid solutions - SQL
FROM events e1
LEFT JOIN events e2 ON e1.customer_id = e2.customer_id
AND e2.type = 'view_item' AND e1.?mestamp <
e2.?mestamp
LEFT JOIN events e3 ON e2.customer_id = e3.customer_id
AND e3.type = 'add_to_cart' AND e2.?mestamp <
e3.?mestamp

8. Valid solutions - NoSQL
var map = func?on () {
var steps = ['view_item', 'add_to_cart', 'buy’];
var counts = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
var i = 0;
for (var j in this.value.events) {
var event = this.value.events[j];
if (event['type'] == steps[i]) {
counts[i]++;
i++;
if (i === steps.length) break;
}
}
if (i > 0) emit('funnel', {'counts': counts});
};

9. Valid solutions - NoSQL
var reduce = func?on (key, values) {
var counts = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
for (var i in values) {
for (var j in values[i].counts) {
counts[j] += values[i].counts[j];
}
}
return {'counts': counts};
};
db.embeded_customers.mapReduce(map, reduce, {
out: 'customers_matched_funnel_1'
}).find();

10. Alternative solutions - custom in memory database

11. IMF – Customer data structure

12. IMF – Basic structure
project1
customer1
event1
?mestamp
proper?es
property1, value1
...
event2
...
proper?es
property1, value1
...
…

13. IMF
• Sharding
• Customer Id as sharding key
• Replication
• IMF –master knows how many shards and
replicas are connected
• Loading
• From a stream of data

14. App architecture

15. λ architecture

16. λ architecture

17. We have speed we need volume
• Fast layer is solved
• Big data requirements
• Loading old data into fast layer
• 0 data expiration
• Access to data from BI tools
• Custom queries

18. λ architecture

19. Map-R
• Map-R filesystem
• Direct access to files that are stored within cluster
• Faster than HDFS
• Map-R distribution
• No dependency hell

20. λ architecture

21. Data collec?on API : Real?me vs Async
• Realtime
• Customer segments
• Website customization
• Recommendations
• personalization
• Async
• Do not lose data
• Event driven campaigns

22. Data Collection

23. Real ?me – web customiza?on

24. Even trigger campaign

25. λ architecture at Exponea

26. Takeaways
• Lambda solves two contradictory challenges
• Process data fast
• Process very big data
• Apache Spark is good choice for both speed & batch
layer, anyway our IMF is way faster :-)

27. Thank you.