Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building High-scalable Enterprise Solutions,

200 views

Published on

This session shows you how you can use Microsoft Azure to build a high-scalable solution for event-processing. You can use this approach for classical IoT-scenarios or if you want for example to capture telemetry-data of a widely distributed application. Then each application-instance will send data to Azure’s Event Hub. In this session you will not only get some insights into the Event Hub, but also into Stream Analytics. Stream Analytics is used to aggregate the millions of events coming from the Event Hub by using a SQL-like syntax. From Stream Analytics the data can be pushed into a database or for example into a Live Dashboard in Microsoft’s Power BI.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Building High-scalable Enterprise Solutions,

  1. 1. BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH #SDF16 Building High-scalable Enterprise Solutions with Microsoft Azure and PowerBI Thomas Claudius Huber – Principal Consultant @thomasclaudiush
  2. 2. #SDF16 Thomas Claudius Huber Principal Consultant – Microsoft Most Valuable Professional (MVP) Focus on C#, TypeScript, Angular and Azure – Part of Cloud-Unit @Trivadis Book- and Pluralsight-Author – See also www.thomasclaudiushuber.com
  3. 3. #SDF16 Agenda The Big Picture Azure Event Hubs Stream Analytics and Power BI
  4. 4. #SDF16 The Big Picture
  5. 5. #SDF16 The big picture
  6. 6. #SDF16 Agenda The big picture Azure Event Hubs Stream Analytics and Power BI
  7. 7. #SDF16 Azure Event Hubs
  8. 8. #SDF16 Azure Event Hubs Scales up to millions of events Great approach for telemetry data Great approach for sensor-data
  9. 9. #SDF16
  10. 10. #SDF16 Azure Event Hubs: Partitions Event stream is segmented by partitions for scale – Scaling for event consumers – Scaling for event submitters You can have between 2 and 32 partitions Azure Support can enable up to 1024 or even more «under special conditions»
  11. 11. #SDF16 Azure Event Hubs: Throughput Units Throughput units (TUs) are important for billing. 1 Unit means – Write maximum 1MB/sec and 1000 messages per second – Read maximum 2MB/sec Number of partitions has to be equal or greater than number of TUs TUs are applied on namespace level – You can have up to 10 event hubs in a namespace – Namespace is used for the URI, for example https://<yourNamespace>.servicebus.windows.net
  12. 12. #SDF16 Azure Event Hubs: Consumer Groups Views on the Event Stream – Similar to topic subscriptions that are also a «view» on an event stream – Organizational anchor for checkpointing $Default consumer group is always there – Up to 20 named consumer groups are supported Limitations – 5 concurrent reading connections per consumer group • It’s recommended to use a separate consumer group for each job
  13. 13. #SDF16 Azure Event Hub vs. Azure IoT Hub Quote: «It is not uncommon to use both IoT Hub and Event Hubs in the same solution» Official docmentation: https://azure.microsoft.com/en-us/documentation/articles/iot-hub- compare-event-hubs/ Event Hub IoT Hub Communication Device to cloud Device to cloud and cloud to device Security Event hub wide shared access policy Per device identity Scale 5000 simultaneous connections Millions of simultaneous connections SDKs Supported on .NET and C, also has AMQP and HTTP (REST) interfaces Device SDKs for several platforms.
  14. 14. #SDF16 But why not always using IoT Hub? IoT Hub scales on devices – a lot of concurrent connections – but is expensive for a few thousand devices and a lot events Event Hub scales on events – but is limited to 5000 concurrent connections
  15. 15. #SDF16 So what about pricing? Event Hub CHF 0.0253 per million events 1 Throughput Unit = CHF 0.0135/hr (~CHF 10/month) – 1 Throughput Unit = 1MB / 1000 Messages Ingress, 2MB read
  16. 16. #SDF16 Creating an Event Hub: Step 1 Go to the https://portal.azure.com Select «New  Internet of Things  Event Hubs» Enter Namespace-Name and Resource Group and click «Create»
  17. 17. #SDF16 Creating an Event Hub: Step 2 So far you’ve created a namespace – the namespace can have up to 10 event hubs When you navigate to the namespace (via «Resources»), you can add a new event hub
  18. 18. #SDF16 Demo: Create an Event Hub
  19. 19. #SDF16 Connect to the Event Hub To push events to an event hub, you have several options: – From a .NET application you can use the NuGet-package WindowsAzure.ServiceBus and the EventHubClient-class • Samples: https://github.com/thomasclaudiushuber/EventHub.Clients – You can also use a REST-API that is available automatically for each event hub • https://github.com/thomasclaudiushuber/EventHub.RestClientGenerator You need to setup a Shared Access policy for your event hub To access the event hub, you need the URI and the shared access key – both is part of a Connection String that you find in the Shared Access policy
  20. 20. #SDF16 Pushing Events to the Event Hub from a .NET App Use the NuGet-package WindowsAzure.Service – It has an EventHubClient-class Events are JSON-data items
  21. 21. #SDF16 Consuming Events from the Event Hub in a .NET App Same EventHubClient-class Grab the consumerGroup and create the receivers for the partitionIds
  22. 22. #SDF16 Demo: Access the Event Hub
  23. 23. #SDF16 Agenda The big picture Azure Event Hubs Stream Analytics and Power BI
  24. 24. #SDF16 The big picture
  25. 25. #SDF16 What is Stream Analytics it’s a real-time event processing engine Use a SQL-variant to query the event stream High scalable
  26. 26. #SDF16 Stream Analytics: Inputs and Outputs
  27. 27. #SDF16 Stream Analytics: Windowed Aggregates There are different windowed aggregates, one is the Tumbling Window select DeviceName,max(value),count(*) into [SDFPBOutput] from [SDFEventHub99] Timestamp by Readtime group by devicename, TumblingWindow( Duration(second, 5), Offset(millisecond, -1))
  28. 28. #SDF16 Demo: Setting up Stream Analytics
  29. 29. #SDF16 Power BI as an Output BI-tool to visualize your data – Access it via https://www.powerbi.com Stream Analytics can directly push its outupt to Power BI In Power BI you can work with the Stream Analytics Data
  30. 30. #SDF16 Build a Live Dashboard in Power BI
  31. 31. #SDF16 Agenda The big picture Azure Event Hubs Stream Analytics and Power BI
  32. 32. #SDF16 The big picture
  33. 33. #SDF16 Questions Thomas Claudius Huber Principal Consultant thomas.huber@trivadis.com @thomasclaudiush

×