Downloaded 24 times
![Apache Spark
Great momentum in the industry
Active and large community
Supported by all major big data vendors
Fast release cadence
Upcoming in Spark 2.0
Dataset is the new unifying API
Tungsten Phase 2 (3-10x speedup)
Structured Streams [ALPHA]](https://image.slidesharecdn.com/hall3thur1210julianleefailaifinal-160906150901/85/Creating-the-Internet-of-Your-Things-8-320.jpg)
Uploaded byDataWorks Summit/Hadoop Summit
Creating the Internet of Your Things
The document discusses Microsoft's Azure IoT platform for connecting, managing, and analyzing Internet of Things devices and data. It provides an overview of the key components of Azure IoT including Azure IoT Hub for device connectivity and management, analytics services like Azure Machine Learning and Stream Analytics, and connectivity to other Azure services. It also highlights aspects of Azure IoT like its open ecosystem, support for open standards, and global infrastructure running on Microsoft's Azure cloud.
More Related Content
Similar to Creating the Internet of Your Things
More from DataWorks Summit/Hadoop Summit
Creating the Internet of Your Things
- 1. Creating the Internetof Your Things Fai Lai hoongfai@microsoft.com Global Technology Specialist – IoT Microsoft Cloud + Enterprise – Enterprise Partner Group
- 2. Simplifying IoT Analytics & OperationalizedInsights Presentation & Business Connectivity Device Connectivity & Management • Azure IoT Hub • Azure IoT Device SDK w/Cross Platform OS Support • Azure IoT Gateway SDK w/Cross Platform OS Support • Azure IoT Hub Device Management • Cortana Intelligence Suite • Azure Data Lake • Azure Machine Learning • Azure Stream Analytics • Azure HDInsight Storm • Power BI • Azure App Service • Azure Mobile Services • Azure API Management • Azure Logic Apps
- 3. Simplifying IoT World ClassFoundation Hybrid & On-Premises Azure Stack will enable you to run IoT workloads wherever you want Hyperscale Cloud 30 regions worldwide More than any other hyperscale cloud vendor Enterprise Proven Largest Number of Compliance Certifications ISV and System Integrator Partnerships End to End Security & Privacy Industry Standard Support Rich Security Services Data & Analytics Ecosystem Cortana Intelligence Suite World Class Support Rich, Open Ecosystem Azure Certified for IoT Azure IoT OSS SDKs
- 4. Simplifying IoT Openness &Interoperability Open Source Software OPC-UA Support for Windows 10 and Azure IoT Eclipse Foundation / Kura Connectors Azure IoT Device SDK Azure IoT Gateway SDK OpenT2T Open Standards & Consortiums Open Mobile Alliance (OMA) / LWM2M OASIS / MQTT Industrial Internet Consortium (IIC) Internet Engineering Task Force (IETF) Open Connectivity Foundation (OCF) Open Fog Consortium Eclipse Foundation OPC Foundation
- 5. Connected device AzureIoT Hub Device-to-Cloud telemetry Cloud-to-Device commands
- 6.
- 7. Analyzing data withSpark Streaming Julian Lee julian.lee@microsoft.com Global Technology Specialist – Advanced Analytics Microsoft Cloud + Enterprise – Enterprise Partner Group
- 8. Apache Spark Great momentumin the industry Active and large community Supported by all major big data vendors Fast release cadence Upcoming in Spark 2.0 Dataset is the new unifying API Tungsten Phase 2 (3-10x speedup) Structured Streams [ALPHA]
- 9. Spark on HDInsight FullyManaged Service 100% open source Apache Spark and Hadoop bits Latest releases of Spark (currently 1.6.1, 1.6.2 and 2.0 are coming) Fully supported by Microsoft and Hortonworks 99.9% Azure Cloud SLA Certifications: PCI, ISO 27018, SOC, HIPAA, EU-MC Optimized for data exploration, experimentation and development Jupyter Notebooks (scala, python, automatic data visualizations) IntelliJ and Eclipse plugins (job submission, remote debugging) ODBC connector for Power BI, Tableau, Qlik, SAP, Excel, etc
- 10. Integrated with AzureData Pipelines Batch pipelines Azure Data Factory orchestrates Spark ETL jobs PowerBI connector for Spark HA of job submission service Streaming pipelines High Availability for Spark Streaming using Yarn Azure connectors: EventHub, Power BI, Azure SQL, DW, Data Lake Github: https://github.com/hdinsight/spark-powerbi-connector Blog: https://blogs.msdn.microsoft.com/azuredatalake/2016/03/09/ saving-spark-dataframe-to-powerbi/
- 11. Brief Introduction toSpark 2.0 • Unifying DataFrames and Datasets in Scala/Java. DataFrame is just a type alias for Dataset of Row. • SparkSession - a new entry point that supersedes SQLContext and HiveContext. • DataFrame-based Machine Learning API emerges as the primary ML API. Original spark.mllib preserved but development on DataFrame- based API • Machine learning pipeline persistence – save and load ML pipelines • Updated support in R for GLMs, Naïve Bayes, Survival Regression, Kmeans Source: Databricks
- 12.
- 13.
- 14.
- 15. Connected device AzureIoT Hub Device-to-Cloud telemetry Cloud-to-Device commands
- 16.
Editor's Notes
- #7 Demo the remote monitoring preconfigured solution in Azure IoT Suite.
- #12 Both the typed methods (e.g. map, filter, groupByKey) and the untyped methods (e.g. select, groupBy) are available on the Dataset class. For users of the DataFrame API, a common source of confusion for Spark is which “context” to use. (SQLContext or HiveContext?) Spark.ml and spark.mllib packages
- #17 Demo the remote monitoring preconfigured solution in Azure IoT Suite.