Big Data & Data Engineering

1. Using Kafka and Azure Event hub
together for streaming Big data
Sergii Bielskyi
Cloud Architect at Eleks
https://medium.com/@sergiibielskyi
https://www.facebook.com/groups/iot.ua/

2. How big your data?

3. Data history

4. Before you can process
Big Data you must get the data

5. Typical data collection pipeline
Data
producers
Collection/
staging
Ingress/
capture
Processing Storage Visualization

6. How people stream today
Google cloud dataflow

7. Azure Event Hub for Big Data (use cases)

8. Azure Event Hub architecture

9. • Integration to Azure Stream Analytics, Functions, LogicApps, Azure Monitor, etc
• Cloud native service
• Using core features like RBAC, Vnet, MSI
• IP Filtering
• Capture
• Auto - scale
• NiFi
• Spark
• More …
Features of using Azure Event Hubs

10. • Automatically send Event Hubs data to your storage
• Better serve batching and archival scenarios
• Easy to configure, no code
• Enable batch & real time on the same stream
Event Hubs Capture – easy way to archive

11. How capture works

12. Compare IoT hub and Event Hubs

13. Azure Event Hubs is expensive?

14. DEMO

15. Kafka architecture

16. Architecture of using Kafka (sample)

17. Deployment
• Buy hardware
• Install physical network
• Ensure reliable power
• Provide physical security
• Configure IP network
• Install OS
• Install / Configure Zookeeper
• Install / Configure Kafka
Running Kafka on premises

18. Deployment
• Configure virtual network
• Install / Configure Zookeeper
• Install / Configure Kafka
Running Kafka on Azure

19. Difference between Kafka and Event Hubs
• Event Hubs is multi tenant managed service, kafka is not
• On premises support for kafka
• Protocols support. Kafka has HTTP rest clients, event hubs supports AMQP, HTTPS, Kafka clients
• Languages. Kafka (Java), Event hubs (C#, .Net) and also Java
• Disaster recovery. Event Hubs supports Geo Redundant Storage, Kafka supports replication only into
the cluster
• Scalability. Event hubs auto scale the storage but kafka is IaaS
• Integration with Stream Analytics is only possible with Event Hubs
• Message size. Event hubs limit is 256Kb, kafka – any limits

20. Azure Event Hubs loves Kafka

21. How integration works

22. Configure integration
1. Enable integration. Only Azure Event hubs standard
2. Edit Kafka client
bootstrap.servers={<strong>YOUR.EVENTHUBS.FQDN</strong>}:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=”$ConnectionString” password=”{<strong>YOUR.EVENTHUBS.CONNECTION.STRING</strong>}”;

23. Kafka on HDInsight

24. Kafka on HDInsight
• SLA 99.9%
• Azure Managed Disks as the backing store for Kafka. Managed Disks can provide up to
16 TB of storage per Kafka broker
• Azure Log Analytics can be used to monitor Kafka on HDInsight

25. DEMO

26. Use cases

27. Use cases
• Messaging. Message broker
• Activity tracking. Provides in-order logging of records, it can be used to track and re-
create activities
• Aggregation. Using stream processing, aggregates information from different streams
to combine and centralize the information into operational data.
• Transformation. Using stream processing, combines the data from multiple input
topics into one or more output topics

28. Дякую