![Group By – Windowing - Tumbling
SELECT count(*), Topic FROM Tweets
GROUP BY Topic, TumblingWindow(second, 5)
TUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] )
TUMBLINGWINDOW ( Duration( timeunit , windowsize ), [Offset(timeunit , offsetsize)] )](https://image.slidesharecdn.com/20160316-techstolica-cloudanalytics-160316151741/85/20160316-techstolica-cloudanalytics-tidk-20-320.jpg)
![Group By – Windowing - Hoppingwindow
SELECT System.TimeStamp, Topic, COUNT(*)
FROM Tweets BY EntryTime
GROUP BY Topic, HoppingWindow(second, 10, 5)
HOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] )
HOPPINGWINDOW ( Duration( timeunit , windowsize ) , Hop (timeunit , windowsize ), [Offset(timeunit , offsetsize)])](https://image.slidesharecdn.com/20160316-techstolica-cloudanalytics-160316151741/85/20160316-techstolica-cloudanalytics-tidk-21-320.jpg)
Uploaded byŁukasz Grala
20160316 techstolica - cloudanalytics -tidk
Wprowadzenie do analizy danych w chmurze. Między innymi o Azure Stream Analytics, Azure Data Lake Analytics, Azure Machine Learning, ale też i o rozwiazaniach OpenSource (Spark, Yupiter, Storm, Zepelin)
More Related Content
Similar to 20160316 techstolica - cloudanalytics -tidk
![[PL] WDI22-Adrian Chodkowski - Azure SQL Skalowalna baza w świecie chmury.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/plwdi22-adrianchodkowski-azuresqlskalowalnabazawwieciechmury-220410131943-thumbnail.jpg?width=640&height=640&fit=bounds)
20160316 techstolica - cloudanalytics -tidk
- 1. Wprowadzenie do Cloud Analytics ŁukaszGrala Architect Data Platform & Advanced Analytics & BI Solutions Data Platform MVP
- 2. @Łukasz Grala –lukasz@tidk.pl • Architekt rozwiązań Platformy Danych & Business Intelligence & Zaawansowanej Analityki w TIDK • Certyfikowany trener Microsoft i wykładowca na wyższych uczelniach • Autor zaawansowanych szkoleń i warsztatów, oraz licznych publikacji i webcastów • Od 2010 roku wyróżniany nagrodą Microsoft Data Platform MVP • Doktorant Politechnika Poznańska – Wydział Informatyki (obszar bazy danych, eksploracja danych, uczenie maszynowe) • Prelegent na licznych konferencjach w kraju i na świecie • Posiada liczne certyfikaty (MCT, MCSE, MCSA, MCITP,…) • Członek Polskiego Towarzystwa Informatycznego • Członek i lider Polish SQL Server User Group (PLSSUG) • Pasjonat analizy, przechowywania i przetwarzania danych, miłośnik Jazzu
- 3.
- 4. Data (Big Data) •72 hours of video are uploaded per minute on YouTube (1 terabyte every 4 minutes) • 500 terabytes of new data per day are ingested in Facebook databases • Sensors from a Boeing jet engine create 20 terabytes of data every hour • The proposed Square Kilometer Array telescope will generate “a few Exabytes of data per day” (single beam) lukasz@tidk.pl
- 5. Internet of Things(IoT) lukasz@tidk.pl
- 6.
- 7. New BI Solutions ETLTool (SSIS, etc) EDW (SQL Server, Teradata, etc) Extract Original Data Load Transformed Data Transform BI Tools Ingest (EL) Original Data Scale-out Storage & Compute (HDFS, Blob Storage, etc) Transform & Load Data Marts Data Lake(s) Dashboards Apps Streaming data lukasz@tidk.pl
- 8.
- 9.
- 10. lukasz@tidk.pl Azure Data LakeStorage & Analytics
- 11. lukasz@tidk.pl Analytics Storage HDInsight (“managed clusters”) Azure DataLake Analytics Azure Data Lake Storage Azure Data Lake
- 12. lukasz@tidk.pl WebHDFS YARN U-SQL Analytics Service HDInsight (managedHadoop Clusters) Analytics Store Azure Data Lake
- 13. Built on ApacheYARN Scales dynamically with the turn of a dial Pay by the query Supports Azure AD for access control, roles, and integration with on-prem identity systems Built with U-SQL to unify the benefits of SQL with the power of C# Processes data across Azure Azure Data Lake Analytics Services 13
- 14. Work across allcloud data Azure Data Lake Analytics Azure SQL DW Azure SQL DB Azure Storage Blobs Azure Data Lake Store SQL DB in an Azure VM
- 15.
- 16. lukasz@tidk.pl Azure Stream Analytics Pointof Service Devices Self Checkout Stations Kiosks Smart Phones Slates/ Tablets PCs/ Laptops Servers Digital Signs Diagnostic EquipmentRemote Medical Monitors Logic Controllers Specialized DevicesThin Clients Handhelds Security POS Terminals Automation Devices Vending Machines Kinect ATM
- 17. lukasz@tidk.pl SELECT count(*), TopicFROM Tweets GROUP BY Topic, TumblingWindow(second, 5) Let’s count tweets by topic…
- 18. Stream Analytics QueryLanguage Built-in Functions Data Types Query Language Elements Time Management
- 19. Group By -Windowing TUMBLING WINDOW HOPPING WINDOW SLIDING WINDOW
- 20. Group By –Windowing - Tumbling SELECT count(*), Topic FROM Tweets GROUP BY Topic, TumblingWindow(second, 5) TUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] ) TUMBLINGWINDOW ( Duration( timeunit , windowsize ), [Offset(timeunit , offsetsize)] )
- 21. Group By –Windowing - Hoppingwindow SELECT System.TimeStamp, Topic, COUNT(*) FROM Tweets BY EntryTime GROUP BY Topic, HoppingWindow(second, 10, 5) HOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] ) HOPPINGWINDOW ( Duration( timeunit , windowsize ) , Hop (timeunit , windowsize ), [Offset(timeunit , offsetsize)])
- 22.
- 23. Big Data -HDInsight • HDInsight is a Hadoop-based service that brings 100% Apache Hadoop solution running on the Microsoft Azure platform • Based on the Hortonworks Data Platform (HDP) • Scalable, on-demand service lukasz@tidk.pl
- 24.
- 25.
- 26. HDInsight & SQLServer 2016 Query relational and non-relational data, on-premises and in Azure Apps T-SQL query SQL Server Hadoop lukasz@tidk.pl
- 27. Advanced Analytics -Language and Tools • Language R and Python • Microsoft R Open, Microsoft R Server, R Services, CARN R, Revolution • Mahout • SparkR • MLLib • Azure Machine Learning lukasz@tidk.pl
- 28.
- 29. Key Concept –Machine Learning Data Model Parameters Learning Prediction Decision Making Utility Function lukasz@tidk.pl
- 30. Class Learning Problems •Classification: Assign a category to each item (Chinese | French | Indian | Italian | Japanese restaurant). • Regression: Predict a real value for each item (stock/currency value, temperature). • Ranking: Order items according to some criterion (web search results relevant to a user query). • Clustering: Partition items into homogeneous groups (clustering twitter posts by topic). • Dimensionality reduction: Transform an initial representation of items into a lower-dimensional representation while preserving some properties (preprocessing of digital images). lukasz@tidk.pl
- 31. Steps to BuildMachine Learning Solution lukasz@tidk.pl
- 32.
- 33.
- 34. Summary • PowerBI • AzureData Lake Storage & Analytics • Azure Stream Analytics • Azure Machine Learning • Azure HDInsight (Hadoop, Spark, Hbase, Storm + MLLib, Mahout, SparkR…) lukasz@tidk.pl
- 35.
- 36. • 16-18 maj2016 • Wrocław Centrum Konferencyjne • 3 dni, 6 warsztatów, 4 ścieżki, ponad 30 prelegentów, 50 sesji • 600 uczestników + sponsorzy + prelegenci + organizatorzy • Goście między innymi z USA, Anglii, Niemiec, Ukrainy, Bułgarii, Słoweni • Premiera techniczna SQL Server 2016 sqlday.pl @sqlday lukasz@tidk.pl W tym warsztat Big Data Analytics – Łukasz Grala & Marcin Szeliga
- 38. Masterclass: Cloud Storage 23-25.05.2016,Warszawa Azure SQL Server i Azure SQL Database, Skalowanie bazy relacyjnej w chmurze, Hurtownia danych w chmurze PowerShell i bazy danych w Azure, Azure BLOB Storage, Bazy dokumentowe, Big Data z HDInsight, Hadoop, Apache Spark, Pozostałe komponenty HDInsight i Hadoop, Wirtualne maszyny Masterclass: Cloud Analytics 20-22.06.2016, Warszawa Data Catalog, Data Factory, Data Lake, PowerBI i dane relacyjne w chmurze, Hadoop, Apache Spark, Analiza danych strumieniowych, Analiza z baz danych dokumentowych i grafowych, Uczenie maszynowe, Polybase w SQL Server 2016 Łukasz Grala Data Platform MVP, MCT, MCSE, MCSA, MCITP, MCSA, MCP, MTA Łukasz o szkoleniach: „Danych produkowanych jest więcej niż kiedykolwiek, pochodzą z sieci Internet, z portali społecznościowych, z urządzeń. Bardzo duży rozwój Internetu Rzeczy (IoT) ilość tych danych jeszcze bardziej zwiększa. Dlatego przygotowaliśmy dwa specjalne kursy Cloud Storage i Cloud Analytics, przedstawiające mechanizmy składowania, przetwarzania i analizy danych z wykorzystaniem chmury.” Big Data, BI, Analityka, SQL Standard -25% na hasło TechStolicawww.hexcode.pl
Editor's Notes
- #14 A new distributed analytics service Built on Apache YARN Dynamically scales Handles jobs of any scale instantly by simply setting the dial for how much power you need. You only pay for the cost of the query Supports Azure Active Directory for Access Control, Roles, Integration with on-premises identity systems It also includes U-SQL, a language that unifies the benefits of SQL with the expressive power of C# U-SQL’s scalable runtime processes data across multiple Azure data sources
- #15 ADLA allows you to compute on data anywhere and a join data from multiple cloud sources.