SQL Server 2014 Faster Insights from Any Data


Published on

Presented @ Ottawa SQL Server Day

SQL Server 2014’s mission is to deliver for our customers mission critical performance for the most demanding database applications, hitting on all aspects of mission critical criteria from performance to security, scalability and high availability along with the mission critical support.

When it comes to business intelligence the mission is to deliver faster insights into any data big data, small data, all data and most importantly deliver BI in a consumable manner for business users through familiar tools.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SQL Server 2014 Faster Insights from Any Data

  1. 1. SQL Server 2014 Faster Insight from Any Data Stéphane Fréchette Friday May 9, 2014
  2. 2. Email: stephanefrechette@ukubu.com Twitter: @sfrechette Blog: stephanefrechette.com Stéphane Fréchette Founder, CEO | Strategic consultant Microsoft SQL Server MVP
  3. 3. Session Overview
  4. 4. Excel BI | Capabilities
  5. 5. Microsoft Power BI for Office 365 1 in 4 enterprise customers on Office 3651 Billion Office Users Analyze Visualize Share Find Q&A MobileDiscover Scalable | Manageable | Trusted
  6. 6. Extend with Hybrid Cloud Solutions
  7. 7. Extend with Hybrid Cloud Solutions
  8. 8. Extend with Hybrid Cloud Solutions
  9. 9. Power Query, PowerPivot, Power View, and Power Map
  10. 10. Powerful Self-Service BI with Excel 2013
  11. 11. Power Query Enable self-service data discovery, query, transformation and mashup experiences for Information Workers, via Excel and PowerPivot Discovery and connectivity to a wide range of data sources, spanning volume as well as variety of data. Highly interactive and intuitive experience for rapidly and iteratively building queries over any data source, any size. Consistency of experience, and parity of query capabilities over all data sources. Joins across different data sources; ability to create custom views over data that can then be shared with team/department.
  12. 12. Power Query Discover, combine, and refine Big Data, small data, and any data with Data Explorer for Excel.
  13. 13. S Data Sources Windows Azure Marketplace Windows Active Directory Azure SQL Database Azure HDInsight
  14. 14. Powerful Self-Service BI with Excel 2013
  15. 15. Introducing PowerPivot
  16. 16. PowerPivot for SharePoint
  17. 17. Powerful Self-Service BI with Excel 2013
  18. 18. Introducing Power View
  19. 19. Power View for Multidimensional Models • Power View on Analysis Services via BISM • Native support for DAX in Analysis Services • Better flexibility: Choice of DAX on Tabular or Multidimensional (cubes)
  20. 20. Architecture Internet Explorer Analysis Services BI Semantic Model Tabular SharePoint (2010 or 2013) Reporting Services Power View Analysis Services BI Semantic Model Multidimensional SQL Server Data Tools SQL Server Data Tools 1 2 35 6 4
  21. 21. BI Semantic Model: ArchitectureThird-party applications Reporting Services (Power View) Excel PowerPivot Databases LOB Applications Files OData Feeds Cloud Services SharePoint Insights
  22. 22. BISM-MD Object Tabular Object Cube Model Cube Dimension Table Attributes (Key(s), Name) Columns Measure Group Table Measure Measure Measure without MeasureGroup Within Table called “Measures” MeasuregroupCube Dimension relationship Relationship Perspective Perspective KPI KPI User/Parent-Child Hierarchies Hierarchies Multidimensional-Tabular Mapping
  23. 23. Powerful Self-Service BI with Excel 2013
  24. 24. Power Map for Microsoft Excel enables information workers to discover and share new insights from geographical and temporal data through three-dimensional storytelling. What Is Power Map?
  25. 25. Map Data • Data in Excel • Geo-Code • 3D and 3 Visuals Discover Insights • Play over Time • Annotate points • Capture scenes Share Stories • Cinematic Effects • Interactive Tours • Share Workbook Power Map: Steps to 3D insights
  26. 26. Map Data •
  27. 27. Discover Insights • • • •
  28. 28. Share Stories • • • • Export to Video for Viral!
  29. 29. Power Map Excel Add-in to Enhance Data Visualization
  30. 30. Power BI Site
  31. 31. Power BI for Office 365 | Capabilities
  32. 32. Power BI for Office 365 | Capabilities
  33. 33. Power BI for Office 365 | Capabilities
  34. 34. Power BI for Office 365 | Capabilities Corporate Data Sources
  35. 35. Data Management Gateway Enabling Corporate OData Feeds Enabling Excel Workbook Data Refresh using SharePoint Online Enabling Discovery in Power Query capabilities Power BI Admin Center Data Management Gateway
  36. 36. Data Management Gateway - Conceptual Power BI Admin Center Allows IT to configure, manage and monitor access to corporate data sources. Data Management Gateway Connects to corporate data sources and sends data to Microsoft cloud services through a secure channel (Service Bus). Corporate Data Sources The Gateway can connect to a variety of data sources. Secure Credential Store All credentials used by the gateway are stored on-premises.
  37. 37. Data Management Gateway Network Topology MICROSOFT DATA CENTERINTERNET PERIMETER NETWORK INTRANET Data Management Gateway Data Management Gateway Cloud Services Customer network Power Query Outgoing connection to cloud services (Registration, Regular Heartbeat, Data Source definition requests) Connect to Corporate OData feed Data Per Machine: Single gateway installed Credential Management Saves credentials
  38. 38. Corporate OData Feeds and Data Management Gateway Data Management Gateway Power Query (1) Using Power Query Anna connects to OData feed (URL: http://feedgwMyDB ) Example: ContosoAnna (2) The Data Management Gateway connects to SQL Server using either Windows account or Database account setup by Patrick when creating the feed Example: DB1_Reader (3) Returns Result (4) Returns OData feed
  39. 39. Scheduled Refresh Scenario: workbook is refreshed on schedule as configured by the author in BI Sites • Scheduler runs in BI Azure and triggers refresh as configured in the BI Sites application • The flow assumes the workbook has been added to Power BI, thus save back is done directly to SPO • When refresh is called by BI Azure, SPO rehydrates the user identity and calls WAC in a back channel (i.e. redirect equivalent) 3. Refresh workbook BI Azure Office Web Apps Service (WAC) Excel Services 5. Get shadow workbook refresh Data Model SPO Azure Active Directory(AAD) OrgID, MSODS,ACS Excel Service s SOAP API 1. Verify user existence and license in MSODS and get access token to target URL in SPO from ACS 2. Construct the user part of the access token, and trigger refresh for a workbook on behalf of the scheduled refresh user On-Prem Data Sources Cloud Data Sources 6. Get data from cloud/on-prem sources and re- process the data model 7. Save updated workbook to SPO 4. Power BI workbook?
  40. 40. On-premises Data Access from BI Azure Scenario: Interactive refresh from Excel Web Access where the data source is on-premises • For interactive refresh, shared data sources are configured in advance in the Power BI Admin Center • For scheduled refresh, personal data sources can be configured by the workbook owner Azure Active Directory (AAD) OrgID, MSODS, ACS BI Azure Hybrid Proxy ADO.NET Provider Discovery API Tenant Configuration SQL Azure Hybrid Data Integration Service Hybrid Proxy Hybrid Delivery 1. Determine whether data source is cloud or on-prem, and retrieve registered ID 2. Authenticate & retrieve tenant information 3. Get registered data source info On-Prem Cloud 4. Issue refresh query Data Management Gateway Windows Azure Service Bus 5. Send request to Gateway (via Service Bus) Hybrid Delivery Client API 6. Read query request from Service Bus queue 7. Retrieve data source credentials Credential Manager8. Run query and retrieve the data 9. Coordinate transfer job Azure Storage (temporary) 10. Compress & stream data in multiple chunks 11 . Receive & decompress data Azure Active Directory (AAD) OrgID, MSODS, ACS BI Azure Hybrid Proxy ADO.NET Provider Discovery API Hybrid Data Integration Service Hybrid Proxy Hybrid Delivery
  41. 41. Data Refresh in SPO– How does it work? Data Management Gateway Excel Workbook in SharePoint Online Gateway Cloud Service (1) Excel workbook uploaded to SharePoint Online (2) Click Data Refresh for Excel workbook (3) Connects to Gateway Cloud Service (4) Checks whether user is authorized to perform a refresh (5) Sends command (SQL statement, connection string) to on-premise Data Management Gateway (6) Sends SQL to SQL Server (7) Return Results (8) Efficiently transfer this to cloud service (9) Returns data to Excel Workbook
  42. 42. Data Management Gateway - OData
  43. 43. Power BI for Office 365 | Capabilities
  44. 44. Engage customers with smart, contextual mobile experiences Boost agility with real-time access to apps and data from anywhere Enable Deep Business and Customer Connections Virtually Anytime, Anywhere
  45. 45. Stay Productive on the Go Deliver Familiar, Connected Experiences to a Mobile Workforce …while ensuring enterprise security, manageability, and compliance
  46. 46. Mobile BI Capabilities Available Today Browser-based corporate BI solutions on iOS, Android and Windows: • SharePoint Mobile enhancements • PerformancePoint Services • Excel Services • SQL Server Reporting Services “Ultimately, the new Microsoft mobile BI solution leads to more revenue for Recall and gives us deeper customer insight, helping us stay ahead of our competitors.” Recall Records Management Company Gets Real-Time BI, Boosts Sales with Mobile Solution case study. Full Case study.
  47. 47. Excel Web App
  48. 48. Excel Web App Quick Explore
  49. 49. Mobile-Friendly Apps for Office
  50. 50. Power BI for Office 365 | Capabilities
  51. 51. Tabular models for Power BI
  52. 52. Datasources
  53. 53. Creating & managing models in Power BI
  54. 54. Reliable Persistent Storage (RPS) Power BI Tabular Model Architecture SSDT SQL Azure HDInsight Azure Tables External Data Sources AS Instance AS Instance AS Instance AS Instance … On Prem SQL Gateway Power BI Portal in O365 Excel XMLA REST
  55. 55. Service Health Monitoring At a glance view of the health of IT managed gateways
  56. 56. Enabler of Self Service BI Varying levels of control across data sources, departments Oversight and monitoring of cloud data access Ability to make corporate data sources easier to discover, and easier to access Role of the IT Admin in Power BI
  57. 57. https://itadmin.clouddatahub.net/ Power BI Admin Center
  58. 58. Power BI Admin Portal & Data Management Gateway Power BI Admin CenterPower BI Admin Center
  59. 59. HDInsight, Polybase, and StreamInsight
  60. 60. Key Trends
  61. 61. Big Data Analytics
  62. 62. Internet of things Audio / Video Log Files Text/Image Social Sentiment Data Market Feeds eGov Feeds Weather Wikis / Blogs Click Stream Sensors / RFID / Devices Spatial & GPS Coordinates WEB 2.0Mobile Advertising CollaborationeCommerce Digital Marketing Search Marketing Web Logs Recommendations ERP / CRM Sales Pipeline Payables Payroll Inventory Contacts Deal Tracking Terabytes (10E12) Gigabytes (10E9) Exabytes (10E18) Petabytes (10E15) Velocity - Variety - variability Volume 1980 190,000$ 2010 0.07$ 1990 9,000$ 2000 15$ Storage/GB ERP / CRM WEB 2.0 Internet of things What Is Big Data?
  63. 63. Modern Data Warehousing
  64. 64. Hadoop Distributed Architecture
  65. 65. MapReduce: Move Code to the Data
  66. 66. So How Does It Work?
  67. 67. Distributed Storage (HDFS) Query (Hive) Distributed Processing (MapReduce) ODBC Legend Red = Core Hadoop Blue = Data processing Gray= Microsoft integration points and value adds Orange = Data Movement Green = Packages HDInsight and Hadoop Ecosystem
  68. 68. Record reader Map Combiner Partitioner Shuffle and sort Reduce Output format
  69. 69. MapReduce Summary
  70. 70. Programming HDInsight Hive, Pig, Mahout, Cascading, Scalding, Scoobi, Pegasus… C#, F# Map/Reduce, LINQ to Hive, Microsoft .NET management clients JavaScript Map/Reduce, browser hosted console, Node.js management clients PowerShell, cross-platform CLI tools
  71. 71. RDBMS vs. Hadoop
  72. 72. Microsoft Hadoop Vision Insights to all users by activating new types of data
  73. 73. Polybase 76 DBHDFS SQL Server PDW querying HDFS data, in-situ =
  74. 74. Polybase in PDW V2 77 Hadoop HDFS DB (a) PDW query in, results out Hadoop HDFS DB (b) PDW query in, results stored in HDFS
  75. 75. Sensor & RFID Web Apps Unstructured data Structured data Traditional schema- based DW applications RDBMS Hadoop Social Apps Mobile Apps How to overcome the “impedance mismatch” Increasingly massive amounts of unstructured data driven by new sources At the same time, vast amounts of corporate data and data sources, and the bulk of their data analysis Polybase addresses this challenge for advanced data analytics by allowing native query across PDW and Hadoop, integrating structured and unstructured data Native Query Across Hadoop and PDW
  76. 76. • Querying data in Hadoop from PDW using regular SQL queries, including • Full SQL query access to data stored in HDFS, represented as ‘external tables’ in PDW • Basic statistics support for data coming from HDFS • Querying across PDW and Hadoop tables (joining ‘on the fly’) • Fully parallelized, high performance import of data from HDFS files into PDW tables • Fully parallelized, high performance export of data in PDW tables into HDFS files • Integration with various Hadoop distributions: Hadoop on Windows Server, Hortonwork and Cloudera. • Supporting Hadoop 1.0 and 2.0 Native Query Across Hadoop and PDW Polybase Features in SQL Server PDW
  77. 77. Native Query Across Hadoop and PDW Creating “External Tables” • Internal representation of data residing in Hadoop/HDFS (delimited text files only) • High-level permissions required for creating external tables • ADMINISTER BULK OPERATIONS & ALTER SCHEMA • Different than ‘regular SQL tables’: essentially read only (no DML support) CREATE EXTERNAL TABLE table_name ({<column_definition>} [,...n ]) {WITH (LOCATION =‘<URI>’,[FORMAT_OPTIONS = (<VALUES>)])} [;] Indicates “External” Table 1 Required location of Hadoop cluster and file 2 Optional Format Options associated with data import from HDFS 3
  78. 78. Native Query Across Hadoop and PDW Querying Unstructured Data 1. Querying data in HDFS and displaying results in table form (using external tables) 2. Joining data from HDFS with relational PDW data Example – Creating external table ‘ClickStream’: CREATE EXTERNAL TABLE ClickStream(url varchar(50), event_date date, user_IP varchar(50)), WITH (LOCATION =‘hdfs://MyHadoop:5000/tpch1GB/employee.tbl’, FORMAT_OPTIONS (FIELD_TERMINATOR = '|')); Text file in HDFS with | as field delimiter SELECT top 10 (url) FROM ClickStream where user_IP = ‘’ Filter query against data in HDFS SELECT url.description FROM ClickStream cs, Url_Description url WHERE cs.url = url.name and cs.url=’www.cars.com’; Join data coming from files in HDFS (Url_Description is a second text file in HDFS) Query Examples 1 2 SELECT user_name FROM ClickStream cs, Users u WHERE cs.user_IP = u.user_IP and cs.url=’www.microsoft.com’; 3 Join data from HDFS with relational PDW table (Users is a distributed PDW table)
  79. 79. Native Query Across Hadoop and PDW Parallel Data Import from HDFS into PDW Persistently storing data from HDFS in PDW tables Fully parallelized via CREATE TABLE AS SELECT (CTAS) with external tables as source table and PDW tables (either distributed or replicated) as destination CREATE TABLE ClickStream_PDW WITH DISTRIBUTION = HASH(url) AS SELECT url, event_date, user_IP FROM ClickStream Retrieval of data in HDFS “on-the-fly” Enhanced PDW query engine CTAS Results External Table DMS Reader 1 DMS Reader N … HDFS bridge Parallel HDFS Reads Parallel Importing Sensor & RFID Web Apps Unstructured data Hadoop Social Apps Mobile Apps Structured data Traditional DW applications PDW
  80. 80. Sensor & RFID Web Apps Unstructured data Social Apps Mobile Apps HDFS data nodes Native Query Across Hadoop and PDW Parallel Data Export from PDW into HDFS • Fully parallelized via CREATE EXTERNAL TABLE AS SELECT (CETAS) with external tables as destination table and PDW tables as source • ‘Round-trip of data’ possible with first importing data from HDFS, joining it with relational data, and then exporting results back to HDFS CREATE EXTERNAL TABLE ClickStream (url, event_date, user_IP) WITH (LOCATION =‘hdfs://MyHadoop:5000/users/outputDir’, FORMAT_OPTIONS (FIELD_TERMINATOR = '|')) AS SELECT url, event_date, user_IP FROM ClickStream_PDW Enhanced PDW query engine CETAS Results External Table DMS Writer 1 DMS Writer N … HDFS bridge Parallel HDFS Writes Parallel Reading Structured data Traditional DW applications PDW
  81. 81. In-Memory for big data analytics Interactive Analytics over “Big Data” 84 • SQL Server Analysis Services scaled out to very large data volumes • Sourced from “Big Data” sources, e.g. • Hadoop, Isotope, etc. • Enterprise data sources (SQL Server, Oracle, SAP, etc.) • Built upon the In-Memory Analytics engine • In-memory, column-store, 10x compression • Deployment vehicles: Box, Appliance, Cloud • Customers: • Skype, Klout, Halo 4, UBS, AdCenter, Windows Update XMLAWeb services External Data Sources GW Mgmt Deploy Monitor AS Instance AS Instance AS Instance Reliable Persistent Storage Excel, PV 3rd party apps, tools, etc.
  82. 82. StreamInsight Managing Streaming Data In-Memory • • • Customer benefits • • • • 85 Event Output stream Input stream
  83. 83. Complete and Consistent Data Platform
  84. 84. What Questions Do You Have?
  85. 85. Thank You for attending this session