Author: Vladimir Bychkov, www.eastbanctech.com The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.

1. Getting started with Azure Event Hubs
and Stream Analytic Services

3. Processing streaming data – use cases
• Automotive - fleet tracking, operations telemetry
• Medical - vital signs, fitness and health monitoring
• Meteorology - weather and environmental data
• Software - applications performance and instrumentation data
• Home automation – energy efficiency, home security
• Agriculture/farming – crops and cattle monitoring

4. Connected cow

5. End-to-end stream processing architecture

6. Producing sensor data with Raspberry Pi

7. Azure Service Bus
Source: https://channel9.msdn.com/Events/TechEd/Europe/2014/CDP-B307

8. Azure Event Hub
Cloud Services
Storage &
Analytics
Custom Code
& 3rd Party
Services
Web/Mobile
User
Interfaces
Integration
Services
Event Hub
- Hyper Scale -
- Fully Managed -
- Interoperable -
- Secure -
- Cost Effective -

9. Azure Event Hub (cont.)

10. DEMO
• Create Azure Event Hub
• Read events using console windows app
• Send events using Python script on Raspberry Pi

11. Azure Stream Analytics
• Managed service in Azure Cloud
• Build for hyper-scale
• Cost effective
• Real-time data processing
• Developer productivity

12. Azure Stream Analytics: Basic Job Topology

13. Inputs
• Data Stream
• Reference data
• Supported data formats:
• JSON
• CSV
• Apache Avro (binary JSON)

14. Outputs
• Azure Tables or Blob
• SQL Database
• Event Hub
• Service Bus Queue/Topic
• PowerBI dataset
• DocumentDB

15. Query - SQL like language with built in temporal semantics

16. Azure Stream Analytics: Grouping data - Windowing
Tumbling window
Aggregate per time interval
Hopping window
Schedule overlapping windows
Sliding window
Window constantly re-evaluated

17. DEMO
• Create Azure Stream Analytics job
• Add input/output to job topology
• Write pass-through query to archive events to Azure Table
• Write aggregate query to save temp averages only to Azure Table
• Web-dashboard demo

18. ANATOMY OF A H4CK3R ATTACK

19. DEMO
• Review CurcuitBreaker app
• Review CurcuitBreaker ASA query
• Run real-time Dashboard
• Hack into Steel Mountain!

20. Q&A

21. THANK YOU
Vladimir Bychkov
Technical Team Lead
vbychkov@eastbanctech.com
eastbanctech.com