A talk for the Breda Dev meetup in which I showed what challenges microservices architectures bring for data analysis and how you can tackle these challenges with Apache Spark on Azure.

1. Big data streaming
Willem Meints

2. Microservices & analytics

3. Event bus
Micro services
• Multiple smaller services that scale independedely
• Each service his own data store
• Data flows between services through the event bus

4. Rapids
Rivers
Lakes

5. Data analytics challenges with microservices
• A complete picture is there, but spread over a vast landscape
• Most data doesn’t come in a database
• Data changes rapidly

6. Exploring some scenarios

7. Scenario 1: Get a annual sales report
• The goal is to get a complete picture of the situation
• Data based on business events

8. OrdersInvoices
Event bus
Data analytics
Data Lake

9. OrdersInvoices
Event bus
Data analytics
Data Lake

10. Scenario 2: Detect anomalies
• The goal is to detect anomalies on the website and prevent abuse
• Machine learning needed to detect the anomalies
• Data based on the data lake

11. Click stream
collector
Event bus
Data analytics
Data Lake

12. Click stream
collector
Event bus
Data analytics
Data Lake
Model

13. Analytics tools

14. vs

15. Event bus
Data
processing tool
Distributed
database
Alerting
Dashboarding

16. Event bus
Data
processing tool
Distributed
database
Alerting
Dashboarding
Flow control
logic
Cluster
Manager

17. The Azure based solution

18. Azure Event Hub HDInsight
Azure Data Lake
Alerting
Dashboarding
Azure App
Services
Cluster
Manager

19. Demo
A short introduction into Apache Spark

20. Spark SQL Spark Streaming
Machine
Learning
GraphX
Apache Spark Core

22. Resilient Distributed Data Sets
Resilient Distributed Dataset
Partition
Record Record
Partition
Record Record

23. Stream Batches Processed data
Streams with Spark

24. Stream Batches Processed data
Streams with Spark
Lists of RDDs

25. Demo
Deploying Spark to Azure using HDInsight

26. Azure Event Hubs
• Capable of streaming large
volumes of data
• SDK available in many languages
• Ruby
• Python
• Java/Scala
• C#
• Apache Spark

27. Hoe werkt een Azure Event Hub?
Partition
Partition
Partition
Consumer
group
Consumer
group

28. Demo
Using Azure Event Hub with Spark

29. Tips for going in production
• When using streams, always have n+1 worker nodes
• More partitions = more speed
• Longer intervals is slower, but sometimes better

30. Thanks!
Willem Meints
Technical Evangelist/Microsoft MVP
@willem_meints