Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure IOT: EVENT HUB & STREAM ANALYTICS & POWER BI

1,707 views

Published on

The Internet of Things (IOT) starts with your things: the things that matter most to your business. It`s the Internet of your Things, it means that you are able to connect your device to a solution without any difficulties, find and rely on a comprehensive set of technologies to connect to and analyze data, or even build new intelligent devices.

Published in: Software

Azure IOT: EVENT HUB & STREAM ANALYTICS & POWER BI

  1. 1. Azure IOT: EVENT HUB, STREAM ANALYTICS & POWER BI By Chourouk Hjaiej February 2016
  2. 2. IOT with Azure Services Chourouk HJAIEJ 2 Table of content Who should read this book? Introduction Azure event hubs, Stream Analytics & Power BI Understanding the solution Solution Overview with Practical Cases Setting Things Up Azure Event Hub Azure Stream Analytics Power BI Examples of dashboard Conclusion Keywords References
  3. 3. IOT with Azure Services Chourouk HJAIEJ 3 Who should read this book? This paper is a practical look at IOT and different services in Azure which help to solve these problems. It assumes the reader has a basic understanding of C#. It is written for both developers and operations professionals who want to improve their team’s or organization’s ability to deliver robust, highly available applications successfully. One of the biggest challenges IOT projects face is how to reliably and securely connect devices to the solution backend. Introduction The Internet of Things (IOT) starts with your things: the things that matter most to your business. It`s the Internet of your Things, it means that you are able to connect your device to a solution without any difficulties, find and rely on a comprehensive set of technologies to connect to and analyze data, or even build new intelligent devices. One of the biggest challenges IOT projects face is how to reliably and securely connect devices to the solution backend. In this paper, we will demonstrate how Azure is able to ensure these functionalities. At WPC 2014, Microsoft announced the preview of Azure Event Hubs which will enable the collection of event streams at high throughput which makes it perfect for IOT, data processing, scenarios. In this paper, I'll start with a simple example to give you a first hands-on experience with Event Hubs. We'll be building a simple example that publishes events to an Event Hub which can be processed by one or more worker processed. I'm not fond of these examples, but in this case this could really be useful. Think of all those applications deployed on hundreds or even thousands of machines. Some of those applications are probably deployed on laptops, POS devices, … that don't have a direct link with the main office, so being related to SQL Server is probably not an option. Wouldn't it be useful if all these applications could send their information (performance…) to a single 'hub' instead of local files or the event log, even if they're not in the main office? And what if we could then process all those events for reporting, performance, …? These are the scenarios that are made possible by Event Hubs. IOT raises important questions which we will address in this paper within the context of a set of technologies. The most important one is Microsoft Azure, named by Gartner as a leading cloud platform (2015 Magic Quadrant). This paper will discuss the topic which is the real-time branch of the Lambda architecture and we will implement the following components:  An Event Hub  A Stream Analytics with Power BI as an output.  We will use a program in C # to simulate sending information in real time. Regards, Chourouk HJAIEJ, DX Technical Evangelist Intern
  4. 4. IOT with Azure Services Chourouk HJAIEJ 4 Azure event hubs, Stream Analytics & Power BI Understanding the solution A complete IOT solution is made of several parts. First of all, we need to ingest all the events and data sent from devices and it’s a such big problem because in the Internet of Things, we have to think at “scale” with hundreds, thousands, million and billions of devices. In this scenario, we have to use an ingestion system that is able to scale and handle incoming data very quickly without slowing down the entire process. After acquiring data, the ingestion system must provide them to the business engine of our solution that could have a path for analyzing data as a stream in real time. This path could expose these informations to the end user that can monitor the devices and what they are “saying” from the real world. The same information is very useful as input for a machine learning system that can help us for predictive analysis to understand how data could evolve in the future based on current incoming values and take preventive actions. Solution Overview with Practical Cases The previous section described how to manage these informations with the Microsoft Azure by using its services. This section illustrates those concepts in a real world scenario and demonstrates the usefulness of Microsoft Azure services. Among these services are event hub which include ….. and stream analytics. The aim here is to show how the combination of an event hub and Stream analytics provides an efficient way tomanage the huge number of events. The figure below shows the scenario which we will use in this part.
  5. 5. IOT with Azure Services Chourouk HJAIEJ 5 The scenario we will use in this post builds upon previous posts that I have written so far: We are simulating potentially millions of devices sending telemetry data. The data is sent to an Event Hub, and processed in parallel by 3 different processes: Instead of chaining processes together, making one process wait on the completion of a previous process, we can scale our solution by having many independent processors work with the data in parallel. This is incredibly powerful. We will focus in this paper on using Stream Analytics to process streaming event data from Azure Event Hubs at scale. We will show some pieces of code, but this code sample will show all the project. You can build a Universal App with service bus (event hub) on Azure using AMQP. This sample explain it very well. Event Source  Use Azure Event Hub with Visual Studio  Device Simulator for Event Hubs Collect  Use Azure Event Hubs with Visual Studio Filter / Analyze / Aggregate  Scaling Azure Event Hubs Processing  Use the Azure Stream Analytics Dashboard  Use Power BI in order to have significant results
  6. 6. IOT with Azure Services Chourouk HJAIEJ 6 Setting Things Up Before we start using Stream Analytics, we will need an Event Hub, something that sends data to the Event Hub. Creating the Event Hub is very easy using the Azure Management Portal. The figure below describes the steps to create an event hub. Azure Event Hub Provide a name and region and a namespace. Provide a partition count and message retention period.
  7. 7. IOT with Azure Services Chourouk HJAIEJ 7 As you can see, this is the list of event hubs created by me and this is the last one. I will create “DevicePolicy” : You can see the primary and the secondary key of this shared access policy: Our Event hub is ready. We will now create and configure a service which will consume post events in our Event Hub. This service is Azure Stream Analytics. Azure Stream Analytics From the Azure portal, click the "+" button at the bottom of the window. Select "Data Services", "Stream Analytics," "Quick Create". Fill in the different fields and click "Create Stream Analytics job".
  8. 8. IOT with Azure Services Chourouk HJAIEJ 8 After a few seconds, my Azure Stream Analytics service will be created. Setting the Azure Service Stream Analytics Now that our Azure Stream Analytics service was created, we'll set it to receive information from our Event Hub (then possibly turn), then send them to a Power BI portal. Click the name of your Azure Stream Analytics ("streamanalyticsjob" in this example). Then click "INPUTS". Click on "ADD AN INPUT". We will select "Events Hub" (Event Hub).
  9. 9. IOT with Azure Services Chourouk HJAIEJ 9 In the Alias Entry field, enter the name you want to give your input in the Azure Stream Analytics. For this demonstration, we'll select "Use Event Hub from current subscription."
  10. 10. IOT with Azure Services Chourouk HJAIEJ 10 Finally, we will choose the serialization settings. We have the choice between "Avro", "CSV" or "JSON". We will select JSON because the program which I will use to send the information in the JSON format. You will have an inputs for this Stream Analytics. Setting a query Now that we have defined an entry, we will define a query to retrieve the data. This query can be a simple "select" or a more complex query to transform data or perform operations on time windows (to average every 5 seconds). Click on "Query" to define the data processing request. Note that the language used is very similar to T-SQL. For this example, I will realize just a select all the fields that are present in the entrance to my service:
  11. 11. IOT with Azure Services Chourouk HJAIEJ 11 SELECT * FROM Inputs Click "Save" and confirm registration. Setting an output We will now set the output of our service to a Power BI portal.
  12. 12. IOT with Azure Services Chourouk HJAIEJ 12 Set in the "Add a Microsoft Power output BI" the name of the database and the table name. If the table already exists Power BI side, it will be replaced with the new one. Confirm your changes. You must obtain a new outlet for your Azure Stream Analytics service.
  13. 13. IOT with Azure Services Chourouk HJAIEJ 13 It remains for us to start the service. At the bottom of the page, click the Start button. Simulating data injection Now, we have to interest in how it sends data. To do this, I used a small program in C # that can simulate sending data, but above all it shows how to connect to an Event Hub precisely to send the data. First, in my "Program.cs" file, don`t forget to replace the values with that of your Event Hub. When running the program, here is an example of the data sent to the Event Hub:
  14. 14. IOT with Azure Services Chourouk HJAIEJ 14 Power BI Now log in to the portal via the BI Power http://app.powerbi.com address. In the left pane, in the "DataSets", a new data source, "DeviceDataset" has just appeared. It is recognizable with the small green asterisk.
  15. 15. IOT with Azure Services Chourouk HJAIEJ 15 Examples of Dashboard
  16. 16. IOT with Azure Services Chourouk HJAIEJ 16 Azure Stream Analytics Monitoring The Azure Stream Analytics also provides a monitoring dashboard as well as the possibility to refer to the events related to the service logs. This can be very useful to understand what happens (especially when it does not work). In your Azure Stream Analytics service, click "Dashboard" to see the events related to your service (you can also click on "Monitor"). In the dashboard, you default service supervisory common indicators and access to the operations log. Click on the link "operations logs" to open the Services management window.
  17. 17. IOT with Azure Services Chourouk HJAIEJ 17 Keywords: Azure Event Hub, Azure Stream Analytics, Scalability, Power BI, Dashboard, Cloud, Query, Azure, References Azure > What is Microsoft Azure? https://azure.microsoft.com/en-us/overview/what-is-azure/ > Microsoft – the only vendor named a leader in Gartner Magic Quadrants for IaaS, Application PaaS and Cloud Storage https://azure.microsoft.com/en-us/blog/microsoft-the-only-vendor-named-a-leader-in-gartner- magic-quadrants-for-iaas-application-paas-cloud-storage-and-hybrid/ > Azure Preview Portal https://azure.microsoft.com/en-us/features/azure-portal/ Azure Event Hub > What is Azure Event hub? https://azure.microsoft.com/en-gb/documentation/articles/event-hubs-overview/ Azure Stream Analytics > What is Azure Stream analytics? https://azure.microsoft.com/en-us/services/stream-analytics/ Power BI > What is Power BI? https://powerbi.microsoft.com/fr/ Azure IOT vs Amazon aws IOT > What is the differences between Azure IOT and Amazon aws IOT? https://paolopatierno.wordpress.com/2015/10/13/an-iot-platforms-match-microsoft-azure-iot-vs- amazon-aws-iot/
  18. 18. IOT with Azure Services Chourouk HJAIEJ 18 Microsoft Azure

×