Presentation of how Exponea uses Kafka and Hadoop. What features are based on it.

1. Kafka and Hadoop as components of
architecture - Martin Strycek

2. Kafka
Kafka is a distributed streaming platform.

3. Hadoop
The Apache™ Hadoop® project develops
open-source software for reliable, scalable,
distributed computing

4. How Kafka and Hadoop got into Exponea?

5. How Kafka and Hadoop got into Exponea?

6. How Kafka and Hadoop got into Exponea
● We had our in memory database super fast,
but in memory
● Our customers were scared that they will have to pay a lot
● They want to have freedom to run analyses on all data
● We had some troubles processing data

7. Kafka + MapR

8. Kafka + MapR
● We were appending data to files that contain jsons
○ HDFS does not support append
● We started using Kafka 0.8.2.1
● We had no idea how to monitor the whole stack

9. Where we are now

10. How Kafka and Hadoop got into Exponea
● We are using Kafka to stream data to
● We have first Sparks jobs that are part of application
○ Recommendation
○ Predictions
○ Campaigns overview
○ Loading data to

11. ● We are using Oryx 2
○ But we need multitenancy
● We have MapR
○ But it ships with different Spark version that Oryx 2
● We are using Oryx 2
○ But it works with different version with Kafka
Recommendation

13. ● How about we use
● How about we create better local storage for
● We need another cluster for testing
○ Bare metal? AWS? Google Cloud?
● library to be usable in
● We want to do a workshop for all of you that want to
try it out, but don’t have a place where.
What next?

14. ● Freedom & responsibility
● Big impact
● Team
● Proficiency
Exponea Culture
}
Global ambitions,
best company to work for

15. Thanks!