SQL Server Workshop for Developers - Visual Studio Live! NY 2012

SQL Server Workshop for DevelopersAndrew Brust Leonard LobelCEO and Founder CTO, Sleek TechnologiesBlue Badge Insights Principal Consultant, Tallan Level: Intermediate May 14, 2012

Meet Andrew • CEO and Founder, Blue Badge Insights • CTO, Tallan, Inc. • Member, Microsoft BI Partner Advisory Council • Microsoft Regional Director, MVP • Co-chair VSLive! and over 15 years as a speaker • Founder, Microsoft BI User Group of NYC – http://www.msbinyc.com • Co-moderator, NYC .NET Developers Group – http://www.nycdotnetdev.com • “Redmond Review” columnist for Visual Studio Magazine and Redmond Developer News • brustblog.com, Twitter: @andrewbrust

Andrew’s New Blog (bit.ly/bigondata)

Meet Lenni sleek Leonard Lobel technologies • CTO & Co-Founder – Sleek Technologies, Inc. • Principal Consultant – Tallan, Inc. • Microsoft MVP – SQL Server • .NET consultant and trainer • Speaker • Author • Programming since 1979 Contact • Email: lenni.lobel@sleektech.com • Blog: lennilobel.wordpress.com • Twitter: @lennilobel

Read All About It!

Agenda • Part I – Overview • Part II – T-SQL Enhancements • Part III – Business Intelligence • Part IV – Beyond Relational • Demos, demos, demos! –http://sdrv.ms/VSLiveNY2012SQL

Download Slides and Code http://sdrv.ms/ VSLiveNY2012SQL (case sensitive!)

PART I OVERVIEW

SQL Server Versions• SQL Server 2012 (Version 11) – Newest version of SQL Server – Released March 7, 2012• SQL Server 2008 R2 (Version 10.5) – Adds powerful BI features to SQL Server 2008 – Released April 21, 2010• SQL Server 2008 (Version 10) – Released August 6, 2008• SQL Server 2005 – Revolutionary upgrade from SQL Server 2000 – Integration of multiple services (SSRS, SSAS, SSIS) with the RDBMS• Before SQL Server 2005… – The relational database engine was the product – Added value features (Reporting, OLAP, DTS) through a patchwork of optional add-ons

Introducing SQL Server 2012• The latest major release of SQL Server• Mission Critical Platform – HADR (High-Availability Disaster Recovery)• IT and Developer Productivity – SQL Server Data Tools (SSDT) – T-SQL enhancements – Beyond Relational enhancements (FileTable, FTS improvements) – Geospatial improvements (circular data, full globe, performance)• Pervasive Insight – Columnstore Indexing (xVelocity) – BI Semantic Model (PowerPivot technology) comes to Analysis Services – Real ad hoc reporting and data visualization in Power View – Data lineage and data quality (DQS)

BI Foundation • Stack Review – The MS BI Stack – The SQL Server 2008 R2 “sub-stack” – New SQL Server 2012 Components • Analysis Services and OLAP – Dimensional Concepts – Analysis Services cube design – Overview of ADO MD.NET and other APIs

BI Delivery • Presentation Layer – Excel BI – PowerPivot and Excel Services Including new PowerPivot features in SQL Server 2012 – Analysis Services tabular databases (SQL Server 2012) – Power View (SQL Server 2012) – Reporting Services and Report Builder – PerformancePoint Services (brief) • Overview: Other New R2/2012 Components – Master Data Services, Data Quality Services – StreamInsight

Introducing SQL Server Data Tools• SSDT – Code-named “Juneau” – Next generation IDE for SQL Server development – Declarative, model-based development in connected, offline, and cloud scenarios – New project type: SQL Server Database Project – Visual Studio shell, solutions, projects + (partial) SSMS/Azure functionality + full BIDS• Not intended to replace SSMS – SSMS still the primary dba tool for managing SQL Server• Intended to replace VS DbPro (aka “data dude”) – But not ready to, as it still lacks total feature parity – Missing data generation, data compare, unit testing• Separate Web Platform Installer Download – Updates to ship out-of-band with VS and SQL Server releases

Declarative Model-Based Development SQL Server Data Tools (SSDT) Offline Dev/TestOn-Premise CloudDataCenter Database Model SQL Server Database Project SQL Server 2005, Local Database 2008, 2008 R2, Runtime SQL Azure 2012 (LocalDB) Database Snapshot File (.dacpac)VersionHistory

SSDT Projects SSDT Relational Analysis Reporting Integration Engine Services Services Services SQL Server Static Analysis Database T-SQL Language Power BufferObject Explorer and Validation Publish Services Editing Schema Local Database T-SQLTable Designer SQL CLR Compare Runtime Debugging

PART II T-SQL ENHANCEMENTS

TABLE-VALUEDPARAMETERS

Table-Valued Parameters• Process a set of rows as a single entity – Similar to temp tables, table variables and CTEs – Example: INSERT an entire order (header & details) with only two parameters and one stored procedure call• Populate a TVP, and then pass it around – It’s a single parameter – Pass from procedure to procedure on the server – Pass from client to server across the network• Based on User-Defined Table Types – Defines the schema, just like an ordinary table – Simply declare a variable as the type to get a TVP• Stored in tempdb – Created and destroyed automatically behind the scenes – Can be indexed

Creating a User-Defined Table Type CREATE TYPE CustomerUdt AS TABLE (Id int, CustomerName nvarchar(50), PostalCode nvarchar(50)) DECLARE @BestCustomers AS CustomerUdt

Table-ValuedParameters

TVP Limitations• TVPs are read-only, once populated and passed – You must apply the READONLY keyword when declaring TVPs in stored procedure signatures – OUTPUT keyword cannot be used – You cannot update, insert or delete• No ALTER TABLE…AS TYPE statement – To change the schema, it must be dropped and re-created – All dependent objects must also be dropped and re-created• Statistics are not maintained on TVPs

MERGE• Four statements in one – SELECT – INSERT – UPDATE – DELETE• And even more… – OUTPUT clause – INSERT OVER DML• Operates on a join – Between source and target – Type of join based on merge clause• Start using it now – 100% compatible with existing business logic – Existing triggers continue to work

MERGE Syntax MERGE target USING source ON join WHEN MATCHED UPDATE | DELETE WHEN NOT MATCHED [BY TARGET] INSERT WHEN NOT MATCHED BY SOURCE UPDATE | DELETE ;

INSERT OVER DML

DML Output• INSERT, UPDATE, DELETE, and MERGE all support the OUTPUT clause – Captures before-and-after snapshots of modified data via INSERTED and DELETED pseudo-tables (just like triggers) – MERGE adds $action virtual column (returning INSERT, UPDATE or DELETE)• OUTPUT INTO can capture the change data to a table or table variable – Suffers from one limitation – no filtering – Solution – use INSERT OVER DML

INSERT OVER DML Syntax INSERT INTO target(columns) SELECT columns FROM (DML statement with OUTPUT) CHANGES(columns) WHERE filter

DATE AND TIME TYPES

Improved Date and Time Support• Start using these four data types – date – time – datetime2 – datetimeoffset• Stop using these two data types – datetime – smalldatetime• Enhanced storage, portability and functionality – Greater range and precision – More efficient compacted storage – Time zone awareness – All traditional functions support the new types – ODBC, OLE-DB, ADO.NET, SSIS, SSAS, SSRS, Replication

Separate Date and Time Types• Only use what you need – Eliminate extraneous storage when only date or time is needed – Better performance for date-only manipulations and calculations• For example, DECLARE @DOB AS date DECLARE @MedsAt AS time(0)

More Portable Dates and Times• Value ranges align with .NET and Windows• Date Values – From 1/1/0001 (DateTime.MinValue) – Through 12/31/9999 (DateTime.MaxValue) – Legacy datetime type limited to 1/1/1753 through 12/31/9999• Time Values – Up to 100ns (10-millionth of a second) precision – datetime accurate only to roughly hundredth of millisecond, smalldatetime to the minute

Time Zone Awareness• datetimeoffset type – Same range and precision as datetime2• Includes the time zone – Stores an offset ranging from -14:00 to +14:00 – Does not support DST (daylight savings time)• Store local date/time in different regions of the world – Values appear to go in and out as local dates/times• Internally stored in UTC – Automatically converted and treated as UTC for comparisons, sorting and indexing• You append the time zone… – …and SQL Server handles the conversions to and from UTC under the covers automatically

Date/Time Accuracy and Storage• Date values compacted into 3 bytes – One byte less than 4-byte date portion of datetime – Greater range in less space!• Time values consume 5 bytes at most – Supports 100-ns accuracy• Pay less for less – Reduced storage for times that don’t require high accuracy – Specify a scale parameter (0-7) on time, datetime2 or datetimeoffset types – 0 = No decimal precision in 3 bytes – 7 = Greatest decimal precision (100-ns) in 5 bytes – Example: DECLARE @FeedingTime time(0) – Differing scales compatible for comparison

Date and Time Types

T-SQL Enhancements(SQL Server 2012)• Windowing (OVER Clause) Enhancements• New T-SQL Functions in SQL Server 2012• The THROW Statement• Server-Side Paging• The SEQUENCE Object• Metadata Discovery• Contained Databases

Introducing OVER• OVER Clause – Exposes a window over the result set for each row – Added in SQL Server 2005, along with the ranking functions ROW_NUMBER, RANK, DENSE_RANK, NTILE• Can also be used with aggregate functions – SUM, COUNT, MIN, MAX, AVG – Doesn’t require GROUP BY

OVERAcctId TxnDate Amount AcctId TxnDate Amount1 3/10/2012 500 1 3/10/2012 5001 3/22/2012 250 1 3/22/2012 2501 3/24/2012 75 1 3/24/2012 751 3/26/2012 125 1 3/26/2012 1252 3/11/2012 500 2 3/11/2012 5002 3/15/2012 50 2 3/15/2012 502 3/22/2012 5000 2 3/22/2012 50002 3/24/2012 550 2 3/24/2012 5502 3/27/2012 95 2 3/27/2012 953 3/15/2012 600 3 3/15/2012 6003 3/22/2012 25 3 3/22/2012 253 3/23/2012 125 3 3/23/2012 125

OVER with PARTITION BYAcctId TxnDate Amount AcctId TxnDate Amount1 3/10/2012 500 1 3/10/2012 5001 3/22/2012 250 1 3/22/2012 2501 3/24/2012 75 1 3/24/2012 751 3/26/2012 125 1 3/26/2012 1252 3/11/2012 500 2 3/11/2012 5002 3/15/2012 50 2 3/15/2012 502 3/22/2012 5000 2 3/22/2012 50002 3/24/2012 550 2 3/24/2012 5502 3/27/2012 95 2 3/27/2012 953 3/15/2012 600 3 3/15/2012 6003 3/22/2012 25 3 3/22/2012 253 3/23/2012 125 3 3/23/2012 125

OVER with PARTITION BY and ORDER BYAcctId TxnDate Amount AcctId TxnDate Amount1 3/10/2012 500 1 3/24/2012 751 3/22/2012 250 1 3/26/2012 1251 3/24/2012 75 1 3/22/2012 2501 3/26/2012 125 1 3/10/2012 5002 3/11/2012 500 2 3/11/2012 502 3/15/2012 50 2 3/15/2012 952 3/22/2012 5000 2 3/22/2012 5002 3/24/2012 550 2 3/24/2012 5502 3/27/2012 95 2 3/27/2012 50003 3/15/2012 600 3 3/15/2012 253 3/22/2012 25 3 3/22/2012 1253 3/23/2012 125 3 3/23/2012 600

Window Partitioning and Ordering• OVER with PARTITION BY – Optionally groups the result set into multiple windows• OVER with ORDER BY – Specifies the row sequence in each window – Required for the ranking functions – Not previously supported for the aggregate functions• Now in SQL Server 2012 – OVER with ORDER BY now supported with aggregate functions – Window “framing” with ROWS and RANGE – Eight new analytic windowing functions

Windowing(OVER Clause)

New T-SQL Analytic Functions• FIRST_VALUE – Returns a column value from the first row of the window• LAST_VALUE – Returns a column value from the last row of the window• LAG – Returns a column value from a previous row of the window• LEAD – Returns a column value from a subsequent row of the window• PERCENT_RANK – Calculate percentile as (RANK – 1) / (N – 1)• CUME_DIST – Calculate percentile as RANK / N• PERCENTILE_DISC – Returns a discreet column value at the specified percentile• PERCENTILE_CONT – Returns a value based on the scale of column values at the specified percentile

More New T-SQL Functions• Logical • Date/Time – CHOOSE – DATEFROMPARTS – IIF – TIMEFROMPARTS• String – DATETIME2FROMPARTS – CONCAT – DATETIMEOFFSETFROMPARTS – FORMAT – DATETIMEFROMPARTS• Conversion – SMALLDATETIMEFROMPARTS – TRY_CONVERT – EOMONTH – PARSE • Math – TRY_PARSE – LOG (changed)

New SQL Server 2012T-SQL Functions

The THROW Statement• SQL Server 2005 added TRY/CATCH – Borrowed from .NET’s try/catch model – Vast improvement over repeatedly testing @@ERROR – Still used RAISERROR for generating errors• SQL Server 2012 adds THROW – Recommended alternative way to generate your own errors – Does not entirely replace RAISERROR• Two usages for THROW – With error code, description, and state parameters (like RAISERROR) – Inside a CATCH block with no parameters (re-throw)

THROW vs. RAISERRORTHROW RAISERRORCan only generate user exceptions Can generate user (>= 50000) and(unless re-throwing in CATCH block) system (< 50000) exceptionsSupplies ad-hoc text; doesn’t utilize Requires user messages defined insys.messages sys.messages (except for code 50000)Doesn’t support token substitutions Supports token substitutionsAlways uses severity level 16 (unless Can set any severity levelre-throwing in a CATCH block)Can re-throw original exception caught Always generates a new exception; thein the TRY block original exception is lost to the clientError messages are buffered, and don’t Supports WITH NOWAIT to immediateappear in real-time flush buffered output on error

Server-Side Paging• New result paging keywords • Append to ORDER BY clause • Limits the number of rows returned• OFFSET @start ROWS • The first result row to return (zero-based)• FETCH NEXT @count ROWS – The number of rows to return

Server-Side Paging

Sequences• Sequential Number Generator – As found in Oracle and DB2 – Alternative to using IDENTITY for assigning new primary keys• Advantages over IDENTITY – SET IDENTITY INSERT ON/OFF not needed for inserts – Can obtain next sequence value without performing an insert• Create a Sequence Object CREATE SEQUENCE MySequence START WITH 1 INCREMENT 1 MINVALUE 1 NO MAXVALUE• Retrieve Sequence Values INSERT INTO MyTable(Id, ...) VALUES(NEXT VALUE FOR MySequence, ...)

Sequences

Metadata Discovery• New system stored procedures and data management views• sys.sp_describe_first_result_set – Analyzes a T-SQL statement and returns information describing the schema of the statement’s result set• sys.dm_exec_describe_first_result_set – Similar, but implemented as a table-valued function to support filtering• sys.dm_exec_describe_first_result_set_for_object – Similar, but accepts an OBJECT_ID of a T-SQL object in the database to be analyzed, rather than a T-SQL statement• sys.sp_describe_undeclared_parameters – Analyzes a T-SQL statement and returns information describing the parameter(s) required by the statement

Metadata Discovery

Contained Databases• Databases are not entirely self-contained – They have external instance-level dependencies – Logins, collations, tempdb, linked servers, endpoints, etc.• SQL Server 2012 provides partial containment – Makes databases more portable – Enables (but does not enforce) containment• Can store logins (with passwords) in the database – Users authenticate directly at the database level• sys.dm_db_contained_entities – Discovers threats to containment

Contained Databases

PART III BUSINESS INTELLIGENCE

MICROSOFT’S BI STACKAND SQL SERVER’S ROLEIN IT

Microsoft Business Intelligence Business User Experience Familiar User Experience Self-Service access & insight Data exploration & analysis Business Collaboration Predictive analysis Platform Data visualization Dashboards & ScorecardsBusiness Collaboration Platform Contextual visualization Excel Services Databased forms & Web Infrastructure workflow and BI Platform Collaboration Analysis Services Search Reporting Services Information Platform Content Management Integration Services LOB data integration Master Data Services Data Mining Data Warehousing

SQL Server 2008 BI Components

But Wait, There’s More!• R2: PowerPivot• R2: Report Parts in SSRS• 2012: Analysis Services Tabular mode – And corresponding improvements in PowerPivot• 2012: Power View• 2012: Data Quality Services• How to get through it all? Here’s the menu…

The Appetizer• Learn Data Warehousing/BI terms and concepts.

The Main Course• Build a multidimensional cube – Query in Excel• Build a PowerPivot model – Query that from Excel too. – Publish to SharePoint• Upsize the PowerPivot model to SSAS tabular model – Add advanced features – Query from Excel• Analyze tabular model from Power View

Dessert• Reporting Services – Report Builder and Report Parts• PerformancePoint Services• Overviews of – Data Quality Services – Master Data Services – StreamInsight

BI CONCEPTS, ANALYSISSERVICES

Business Intelligence

Preparing For BusinessIntelligence Data • Transactions Warehouse • Process • Data • Relationships • Analysis Transaction Database

Dimensional Model• Measure• Dimension• Hierarchy• Grain• Star Schema

Star Schemas Country• Physical data model• Central fact table• Multiple dimension Shipper Year tables – Used to constrain fact table Total queries Sales Sales Product Person

Example Data Request• Get Total Sales By State, By Month for a Calendar Year For Country = USA and Calendar Year = 1996

Data Warehouse Query

Data Migration Data• Transactions Warehouse • Multi-• Process • Data dimensional • Relationships • Hierarchical • Analysis Transaction OLAP Database Database

SQL Server Analysis Services• Built for analysis• It is free with SQL Server• And you can use the Microsoft stack that you know and love

From Data Warehouse to OLAP Dimensions• Measure Dimension•Aggregations – Can have Hierarchies Fact Table• Cube Measures

Building OLAP Cube With BIDS• Business Intelligence Development Studio – AKA Visual Studio• Business Intelligence Projects – Analysis Services Project Type Add Data Source Add Data Source View Add Cube Add Dimensions Add Measures Deploy the Cube

Creating anSSAS ProjectThe Basics

Advanced OLAP• Calculated Members• Key Performance Indicators• Perspectives• And: – MDX – Actions – Partitions – Aggregations – Translations

QUERYING CUBES FROMYOUR APPLICATIONS

SSAS Interfaces C++ App VB App .NET App Any AppOLEDB for OLAP/DM ADO/DSO ADOMD.NET AMO Any Platform, Any Device WAN XMLA XMLA Over TCP/IP Over HTTP Analysis Server (msmdsrv.exe) OLAP Data Mining Server ADOMD.NET DM Interfaces .NET Stored Procedures Microsoft Algorithms Third Party Algorithms

ADOMD.NET Client

PRESENTATIONTECHNOLOGIES

Presenting Your Cube Excel Services PerformancePoint Services Reporting Services Excel SQLServerOracle DB2 Tera- Data

The SSAS/Excel/SharePoint Loop Build models with SSAS Multidim’l, Tabular or PowerPivot Visualize + Query from Analyze with Excel SSRS/ PivotTables and Excel Services/ PerformancePoint Charts Services OR Power View Publish to SharePoint (via Excel Services) and query in the browser

EXCEL BI

PivotStuff• PivotTable, and linked charts (sometimes referred to as PivotCharts) work extremely well with OLAP cubes• How to create a PivotTable: – Ribbon’s Data tab (From Other Sources button/From Analysis Services option or Existing Connections button) – Insert tab (PivotTable “split button”)• How to insert a chart – PivotChart button on PivotTable Tools/Options tab – Several others

PivotCharts andPivotTables

Formula Language CUBE Functions• CUBEMEMBER and CUBEVALUE – Also CUBEKPIMEMBER, CUBEMEMBERPROPERTY, CUBERANKEDMEMBER, CUBESET and CUBESETCOUNT• IntelliSense style support – In a cell, type “=CU” and all CUBE formulas will display – Select one with arrow keys and hit Tab – When prompted for connection, type " and view pop-up list – Other pop-ups activate on " or "."

At Your Service• “Range Drag” and relative formula support on CUBEVALUE• CUBEVALUE and Data Bars go great together• Ability to convert PivotTables to formulas

CUBExxx Formulas

POWERPIVOT AND EXCELSERVICES

Self-Service BI with PowerPivot• Excel + Analysis Services + SharePoint• Enables the working in Excel but mitigates the “spreadmart” pitfalls: – Use Analysis Services (AS) as a hidden engine Instead of no engine – Share via SharePoint, accessible by all AS clients Instead of “deploying” via email – Formal data refresh on server So data doesn’t get stale, and users don’t have to make effort at updating – Allow IT to monitor So it’s not all rogue – Provide path to more rigorous implementations Can be upsized to Analysis Services

Column-Oriented Stores• Imagine, instead of: Employee ID Age Income 1 43 90000 2 38 100000 3 35 100000• You have: Employee ID 1 2 3 Age 43 38 35 Income 90000 100000 100000• Perf: values you wish to aggregate are adjacent• Efficiency: great compression from identical or nearly- identical values in proximity• Fast aggregation and high compression means huge volumes of data can be stored and processed, in RAM

Data Import• Relational databases – SQL Server (including SQL Azure!), Access – Oracle, DB2, Sybase, Informix – Teradata – “Others” (OLE DB, including OLE DB provider for ODBC)• OData feeds, incl. R2/2012 Reporting Services, Azure DataMarket, WCF Data Services (Astoria), SharePoint 2010 lists, Visual Studio LightSwitch• Excel via clipboard, linked tables• Filter, preview, friendly names for tables/columns

Calculated Columns and DAX• Formula-based columns may be created• Formula syntax is called DAX (Data Analysis eXpressions). – Not to be confused with MDX or DMX. Or DACs.• DAX expressions are similar to Excel formulas – Work with tables and columns; similar to, but distinct from, worksheets and their columns (and rows)• =FUNC(table name[column name])• =FUNCX(table name, <filter expression>)• FILTER(Resellers,[ProductLine] = "Mountain")• RELATED(Products[EnglishProductName])• DAX expressions can be heavily nested

Import data from almost anywherePowerPivot Guidebook View data in Excel Calculated column entry Sort and filter DAX formula bar Relationship indicator Table tabs

Data andWhat’s New? Diagram views KPIs Measures Measure grid Sort one column by another Measure formula

Perspectives Default Aggregations Special Advanced ModeDiagram View Reporting properties Hierarchies Hide specific columns and tables Create relationships visually Measures KPIs

PowerPivot Client

Excel Services• A component of SharePoint Server 2007/2010; requires Enterprise CAL• Allows export of workbook, worksheet, or individual items to SharePoint report library – Works great for PivotTables and Charts! – Also for sheets with CUBExxx formulas or conditional formatting-driven “scorecards”• Content can be viewed in browser – Excel client not required – Drilldown interactivity maintained – Rendered in pure HTML and JavaScript – Parameterization supported

PowerPivot Server• Publish to Excel Services• Viewing and interacting• Data Refresh• Treating as SSAS cube – 2008 R2 version: URL to .xlsx as server name – 2012 version: use POWERPIVOT named instance and treat just like SSAS Db name is GUID-based; best to discover it – Use Excel, Reporting Services as clients And now Power View too…more later

The IT DashboardIncrease IT efficiency: Familiar Technologies for Authoring, Sharing, Security, and Compliance Customizable IT Dashboard Visualize usage with animated charts Simplify management of SSBI content using IT Operations Dashboard for SharePoint

PowerPivot Server

Analysis Services Tabular Mode• SSAS Tabular Mode is the enterprise/server implementation of PowerPivot• You must have a dedicated tabular mode SSAS instance• Tabular SSAS projects: BI Developer Studio (BIDS) gone PowerPivot – Implements equivalent tooling to PowerPivot Window – Can create an SSAS tabular database project by importing an Excel workbook with PowerPivot model• SSAS tabular models support partitions and roles

SSAS Tabular Project in BIDS SSAS tabular project menus and toolbar Measure grid and formula bar Reporting properties in Properties window

DirectQuery Mode• In DQ mode, model defines schema, but is not used for data• Queries issued directly against source• Similar to ROLAP storage for conventional cubes• Combine with xVelocity ColumnStore indexes for fast, real-time querying

SSAS Tabular Mode

POWER VIEW

What is Power View?• Ad hoc reporting. Really!• Analysis, data exploration• Data Visualization• In Silverlight, in the browser, in SharePoint• Feels a little like Excel BI• Is actually based on SSRS – Power View makes a special RDL file

Power View Data Sources• Power View works only against PowerPivot/SSAS tabular models – DirectQuery mode supported, however• For PowerPivot, click “Create Power View Report” button or option on workbook in SharePoint PowerPivot Gallery• For SSAS tabular model, create BISM data source, then click its “Create Power View Report” button or option – BISM data sources can point to PowerPivot workbooks too, if you want.

In the browser,Power View! in Silverlight Ribbon, like Excel Variety of visualizations and data formats Field list, like Excel Data regions pane, like Excel

View Modes Maximize one chart, fit report to window, put whole report in Reading Mode or Full Screen Create multiple pages (views)

Power View Basics

Constraining Your DataIn Power View• Tiles – A filtering mechanism within a visualization• Highlighting – Selection in one visualization affects the others• Slicers – Similar to Excel against PowerPivot• True Filters – Checked drop-down list; very Excel-like – Right-hand filter pane, similar to SSRS and Excel Services

Power View Filtering

Scatter/Bubble Charts• Allow for several measures• Features a “play” axis which can be manipulated through a slider or animated• Excellent way to visualize trends over time

Multipliers• Multiple charts within a chart, in columns, rows, or a matrix – Horizontal and vertical multipliers• Allows for visualizing 1 or 2 additional dimensions

Advanced Properties• Setting the representative column and image tells Power View how to summarize your data, and show stored images• Other properties tell it about key attributes, default aggregations and more• Reminder: “DirectQuery” mode tells Power View to get data from relational data source instead of columnar cache – Use with columnstore indexes to have the best of both worlds – columnstore indexes require Enterprise Edition, available in BI Edition

Power ViewAdvanced Features

Vocabulary• MOLAP: Multidimensional OLAP• UDM: Unified Dimensional Model• Cube: unit of schema in a dimensional database• xVelocity Columnstore Technology: PowerPivot/SSAS’ column store engine• VertiPaq: Old name for xVelocity• BISM: BI Semantic Model• IMBI: In-Memory BI engine• Tabular: a column store-based model – Because it uses tables, not cubes

xVelocity Columnstore Indexes• Implementation of xVelocity columnar technology engine for SQL Server relational databases – Code name was: “Apollo”• Use it by creating a columnstore index – CREATE COLUMNSTORE INDEX index ON table (col1, Col2, …)• Can ignore it in a SELECT, too: – OPTION (IGNORE_NONCLUSTERED_COLUMNSTORE_INDEX)• Can significantly increase performance of star join queries (i.e. aggregating queries with dimension lookups).• Must enable “batch” mode as well – look @ query plan to confirm!• Not as good as SSAS, but better than plain old GROUP BY

REPORTING SERVICESAND REPORT BUILDER

Scalable, Enterprise Reporting Platform• Author Impactful Reports • Powerful Designers • Flexible Report Layout • Rich Visualizations• Manage any Workload • Enterprise Scale Platform • Central Deployment • Strong Manageability• Deliver Personalized Reports • Interactive Reports • Rendering in the Format Users Want • Delivery to Location Users Want

Self-Service Report AuthoringReport Builder 3.0 Familiar Microsoft Office Interface Powerful Query Designer Powerful Wizards Flexible Layout Options w/ rich visualizations SharePoint list as data source PowerPivot as data source

Rich Visualizations

SSRS + SSAS = YES(S)• Reporting Services can query Analysis Services databases, including multidimensional cubes and tabular models, directly. PowerPivot too.• Uses MSOLAP OLE DB provider and issues MDX queries – Has its own query designer for MDX• Beware: SSRS essentially “flattens” SSAS data – Conforming multidimensional data to relational structures

Self-Service Reporting?• Fact: Reporting Services is powerful• Fact: the data visualizations in SSRS are some of the best in the BI stack• Fact: None of that makes SSRS report design end-user-friendly• Building SSRS reports, and especially charts, gauges, etc. from scratch is tedious• Until now, best end-user option has been to copy an existing report and tweak it. Yech!• There must be a better way…

Report Parts to the Rescue• Skilled SSRS designers can publish report parts – From Report Builder 3.0 or VS report projects• End users can pick them from a gallery – A task pane, in Report Builder 3.0, with search capability – Cannot select from VS report designer• What can be published?: – Tablixes (i.e. tables, matrices) – Rectangles – Images, Charts, Gauges, Maps – Parameters and Lists• All aided by new ability to share Datasets and original ability to share Data Sources

Easy to Publish; Easy to Select

SSRS Report Parts

PERFORMANCEPOINTSERVICES

PerformancePoint Services (PPS) Scorecards Analytics Cascading Multi-dimensional scorecards with slice and dice for interactive charts advanced analytics and data from including multiple sources decomposition tree, performance map, and perspective view

PPS Capabilities• Analytic Grids & Charts – PPS’ own data visualizations – AJAX-based interactive analytics capabilities• Scorecards – Key Performance Indicators – Objectives – Dimensions• Dashboards – Bird’s eye view of business activities – Display Analytic Grids and Charts; Scorecards – Integrate SSRS, Excel Services (including PowerPivot) content – Add Filters

PerformancePoint ServicesA Finished Dashboard

MASTER DATA SERVICES,DATA QUALITY SERVICESAND STREAMINSIGHT

MDS: Microsoft’s Master DataManagement (MDM) tool• Examples: – Sales states, countries, currencies, customer types – Customers, products – Think of “lookup tables” or just think of dimensions! – Slowly changing non-transactional entities in your data• What gets stored: – Schemas – Any hierarchies – The data!• Other features: – Collections, business rules, security, workflows – Versioning

Other Facts• Result of acquisition of Stratature• v1 is an ASP.NET application; UI is “different”• New in v2 (SQL Server 2012): • Now Silverlight-based; UI is still “different” • Excel add-in for data entry; creation of entities and attributes • Perform matching with DQS before loading• Includes .NET and Web Services APIs for reading/writing data and creating/editing models• Does not integrate with Analysis Services tools even though many of its features and concepts mirror those of dimension designer• Catalog kept in SQL Server database• Deployment packages can be created, shared and deployed

Objects in MDS• Models – Entities (like tables or SSAS dimensions) Attributes (like columns/fields or SSAS attributes) Common attributes are Name and Code Attribute Groups Used to taxonomize attributes within tabs in UI Members (like rows/records or SSAS members) Hierarchies (like SSAS hierarchies) Derived or Explicit Collections (like SSAS named sets) – Versions – Business rules – Workflows

Data Quality Services• Data Cleansing Tool• New to SQL Server 2012• Result of Zoomix acquisition• Uses Artificial Intelligence algorithms to detect invalid data and perform matching (for de-duplication)• Allows manual intervention, too• Can integrate with MDS and SSIS• Cleaner data = better adoption of your BI project

DQS Concepts• Knowledge Bases – Domains “semantic representation[s] of a type of data in a data field…[contain] a list of trusted values, invalid values, and erroneous data.” – Mapping• Data Quality Projects – Cleansing (i.e. correcting) Validate Using Reference Data Services and Azure DataMarket (or 3rd party providers) – Matching (i.e. de-duping) – Confidence – Profiling, Monitoring

StreamInsight• Microsoft’s Complex Event Handling (CEP) Product• Processes data streams that are fast and high- volume• Highly parallel C++ code assures low latency, high throughput• Not based on SQL Server, though that is its “ship vehicle”• Interesting collaborative potential with BizTalk and SSIS

StreamInsight Concepts• No UI. All interaction is programmatic.• Based on adapter architecture – Input and output adapters – Buy or build – Sensors, RFID readers, Web logs, market data streams are possible event sources• StreamInsight applications – Streams and events can be queried via LINQ from .NET – Server can run in-process, or shared

StreamInsight v1.2:New in SQL Server 2012• Resiliency: can take snapshots and restore to that saved state after an outage• Dev experience: LINQ enhancements – Multiple FROM clauses – Nested types• Extensibility: user-defined stream operators now supported• Admin features: server-wide and query- specific perf counters, Windows event logging• Included with Web and Standard Editions

Hadoop on Windows• Hadoop is the open source implementation of Google’s MapReduce distributed processing engine• MS working with Hortonworks to implement it on Windows – A full “distro” – Window Server, Windows Azure DIY, Windows Azure self- serve cluster provisioning• Also: ODBC driver for Hive – Works with SSRS, SSAS Tabular, PowerPivot and plain Excel.• Also: JavaScript console in the browser• Come to my session on Thursday!

PART IV BEYOND RELATIONAL

FILESTREAM

The Need To Stream• Data explosion accelerating the creation and management of unstructured binary large objects (BLOBs) – Photos – Audio – Video – Email messages – Spreadsheets – Documents – Etc.

BLOBs And The DatabaseTwo choices• Store them in the database• Store them outside the database, either in the file system or in a dedicated BLOB storeBLOBs in the database• varbinary(max) column• Integrated management• Transactional• Simplified programming• Bloats the structured file groups

BLOBs Inside The Database T-SQL CustomerId FirstName LastName Picture varbinary(max) 235 John Doe 0x3B0E95AE3B2F02292F0B… 236 Sally Smith 0xF3000EEF293039A2002C…

BLOBs And The DatabaseTwo choices• Store them in the database• Store them outside the database, either in the file system or in a dedicated BLOB storeBLOBs in the database BLOBs outside the database• varbinary(max) column • Path references to file system• Integrated management • Separate from the database• Transactional • Not transactional• Simplified programming • Complex programming• Bloats the structured file • Doesn’t interfere with groups performance

BLOBs Outside The Database T-SQL File I/O CustomerId FirstName LastName Picture varchar(max) 235 John Doe C:filesjdoe.jpg 236 Sally Smith C:filesssmith.jpg

BLOBs Using FILESTREAM• Transparently store varbinary(max) data in the file system – Declare column as “varbinary(max) FILESTREAM”• Integrated management – BLOBs are logically part of the database (backup, restore, etc.), but stored physically separate as a file group mapped to the file system• Simplified programming – SQL Server transparently links rows in relational tables to BLOBs in the file system – No developer effort required to maintain references between structured and unstructured data• Transactional – SQL Server integrates with the NTFS file system – Database transactions wrap NTFS transactions• Performant – File system is optimized for streaming

BLOBs Using FILESTREAM T-SQL CustomerId FirstName LastName Picture varbinary(max) FILESTREAM 235 John Doe 0x3B0E95AE3B2F02292F0B… 236 Sally Smith 0xF3000EEF293039A2002C…

Enabling the Service for FILESTREAM• Security concern of the Windows administrator• Set to one of four levels – Disabled – T-SQL only – T-SQL + file system (local only) – T-SQL + file system (remote)• Enable it either: – During setup – With SQL Server Configuration Manager – No way to script with T-SQL, but VBScript file is available on CodePlex that provides command-line alternative

Enabling the Instance for FILESTREAM• Security concern of the SQL Server administrator – Windows and SQL admins must agree! • Set to one of three levels – Disabled, T-SQL only, T-SQL + file system• Enable it either: – In SSMS Server Properties dialog – In T-SQL, with: EXEC sp_configure filestream_access_level, n RECONFIGURE

Creating a FILESTREAM Database• CREATE DATABASE PhotoLibrary ON PRIMARY (NAME = PhotoLibrary_data, FILENAME = C:DBPhotoLibrary_data.mdf), FILEGROUP FileStreamGroup1 CONTAINS FILESTREAM (NAME = PhotoLibrary_group2, FILENAME = C:DBPhotos) LOG ON (NAME = PhotoLibrary_log, FILENAME = C:DBPhotoLibrary_log.ldf)

Creating FILESTREAM Columns• Table requires ROWGUIDCOL column – Attribute applied to a uniqueidentifier (GUID) column – Must be primary key or have unique constraint – Only one ROWGUIDCOL column is permitted per table• Define BLOB columns as “varbinary(max) FILESTREAM” – Any number of BLOB columns are permitted per table

Getting Started withFILESTREAM

Introducing SqlFileStream• It is not easy, practical, or efficient to manipulate BLOBs in T-SQL• Build a streaming client with SqlFileStream – System.Data.SqlTypes.SqlFileStream – Part of System.Data.dll in .NET 3.5 SP1 and higher – Inherits from Stream – Constructor takes logical path and transaction context – Wraps OpenSqlFilestream SQL Server native client API – Consumes no SQL Server memory during processing

The SqlFileStream Recipe• Begin Transaction• INSERT/SELECT row• Retrieve BLOB PathName()• Retrieve GET_FILESTREAM_TRANSACTION_CONTEXT()• Create and use SqlFileStream• Commit Transaction

BLOBs Using SqlFileStream T-SQL SqlFileStream CustomerId FirstName LastName Picture varbinary(max) FILESTREAM 235 John Doe 0x3B0E95AE3B2F020B… 236 Sally Smith 0xF3000EEF293039A2…

Using SqlFileStream

FILESTREAM Limitations &Considerations• Mirroring/HADR – Not supported with mirroring – Supported with HADR (SQL Server 2012 “AlwaysOn”)• Transparent Data Encryption (TDE) – Does not encrypt files• Replication – Supported with some restrictions, see BOL• Log Shipping – Fully supported – Primary and secondary servers require SQL Server 2008 or higher• Full-Text Search (FTS) – Fully supported

FILESTREAM Limitations &Considerations (cont.)• Database Snapshots – Not supported for FILESTREAM filegroups• Snapshot Isolation Level – Wasn’t supported in SQL Server 2008, supported in 2008 R2 and 2012• Local NTFS File System – Requires local NTFS file system – RBS (Remote BLOB Store) API makes SQL Server act as a dedicated BLOB store• Security – Requires mixed-mode (integrated) security• SQL Server Express Edition – Fully supported – Database size limit (4GB in SQL Server 2008, 10GB in 2008 R2 and 2012) does not include FILESTREAM data

HIERARCHYID

Hierarchical Data Is Not Relational• Today’s most common form of hierarchical data is XML• XML support added in SQL Server 2005 is great, if: – You want to store and retrieve an entire hierarchy at one time – The data is consumed in XML by client applications• Parent-child relationships define rigid hierarchies – Can’t support unlimited breadth and depth

Hierarchical Storage Scenarios• Forum and mailing list threads• Business organization charts• Content management• Product categories• File/folder management • FileTable in SQL Server 2012• Many more… – All typically iterated recursively

Traditional Self-Join Approach• One table – Each row is linked to its parent• Works, but has limitations – CTEs help with recursive queries – Still your job to manage updates – Manually maintain structure – Complex to reparent entire sub-trees – Difficult to query – Difficult to control precise ordering of siblings

Introducing hierarchyid• System CLR data type – Extremely compact variable-length format – Does not require SQL CLR to be enabled on the server• Enables a robust hierarchical structure over a self-joining table – Each row is a node with a unique hierarchyid value – Contains the path in the hierarchy to the node… down to the sibling ordinal position• Invoke methods in T-SQL – Efficiently query the hierarchy – Arbitrarily insert, modify, and delete nodes – Reparent entire sub-trees with a single update

hierarchyid Methods• GetAncestor• GetDescendant• GetLevel• GetReparentedValue• GetRoot• IsDescendantOf• Parse• ToString

Indexing hierarchyid Columns• Two types of indexes: Depth-First Index – Use one, the other, or both as your needs dictate• Depth-First – Create a primary key or unique index Breadth-First Index• Breadth-First – Create a composite index that includes a level column

hierarchyid

hierarchyid Example

FILETABLE

Introducing FileTable• Builds on FILESTREAM and hierarchyid• A “semi”-ordinary table that houses a logical file system – Fixed column schema – Each row represents a “folder” or “file” Column Name Data Type Description stream_id uniqueidentifier Unique row identifier ROWGUIDCOL file_stream varbinary(max) BLOB content (NULL if directory) FILESTREAM name nvarchar(255) Name of file or directory path_locator hierarchyid Location of file or directory within the file system hierarchy – Plus 10 storage attributes columns (e.g., is directory, created, modified, archive bit)• Windows file/directory management API support – A Windows file share exposes the FileTable – Bi-directional – changes to the table are reflected in the file share and vice versa

BLOBs Using FileTable hierarchyid column varbinary(max) Name Database Rows FileTable Name Server Machine Instance Name defines logical file Name FILESTREAM column FILESTREAM Share and folder paths contents holds each file’s T-SQL T-SQL SqlFileStreamstream_id name path_locator file_stream27D8D4AD-D100-39… Financials 0xFF271A3562… NULL78F603CC-0460-73… ReadMe.docx 0xFF59345688… 0x3B0E956636AE3B2F020B…207D4A96-E854-01… Budget.xlsx 0xFD0011039A… 0xF3F359000EEF293039A2…

FileTable Prerequisites• Prerequisites at the instance level – FILESTREAM must be enabled for the instance• Prerequisites at the database level – Enable non-transactional FILESTREAM access for the database (is still transactional internally) – Set a root directory name for all FileTables in the database (this will become a child in the Windows file share) – Use this T-SQL statement: CREATE DATABASE … WITH – or – ALTER DATABASE … SET – …followed by… FILESTREAM( NON_TRANSACTED_ACCESS=FULL|READ, DIRECTORY_NAME=DatabaseRootDirectory)

Creating a FileTable• Create a FileTable in T-SQL – CREATE TABLE TableName AS FileTable• FileTable has a fixed schema – You don’t (can’t) supply a column list• Creates logical directory – Logical root directory for the FileTable – Created beneath the root directory for the database – Named after the table, can override by specifying: WITH(FileTable_Directory=TableRootDirectory) – Exposes a functional Win32 file system – Does not support memory-mapped files (does not affect remote clients)

GEO-SPATIAL TYPES

SQL Server Spaces Out • Integrate location awareness into any application – Long been the domain of sophisticated GIS applications • GIS – A system for capturing, storing, analyzing, and managing data and associated attributes which are spatially referenced to the earth • Allow a user to interact with information that is relevant to locations that they care about: – Home, work, school, or vacation destinations • Two geospatial models – Planar – Geodetic

Spatial Data Types • Two spatial models = Two system CLR types • geometry – Planar (flat) model – Flat 2-dimensional Cartesian Coordinate system – X and Y coordinates with uniform units of measure – Use for mapping small areas • geography – Geodetic (round-earth) model – Latitude and longitude – Use for larger mapping where land mass is too big to fit on one planar projection

Planar Spatial Model • Two-Dimensional Surface – X and Y coordinates on an arbitrary plane • Flat Earth Projection – To work with geospatial data on a 2D surface, a projection is created to flatten the geographical objects on the spheroid – Example: Planar Model based on Mercator Projection Greenland Square KM: North - Antarctica = 13 million America - Greenland = 2 million Africa - N. America = 24 million - Africa = 30 million Antarctica

Geodetic Spatial Model • Accurate geographic measurements – Locations on planet surface described by latitude and longitude angles • Ellipsoidal sphere – Latitude = angle N/S of the equator – Longitude = angle E/W of the Prime Meridian

Spatial Data Standards • Open Geospatial Consortium (OGC) – International standards body • Microsoft belongs to the OGC – SQL Server 2008 uses the OGC’s Simple Feature Access standards • OpenGIS Simple Feature Interface Standards (SFS) – A well-defined way for applications to store and access spatial data in relational databases – Described using vector elements; such as points, lines and polygons • Three ways to import geospatial data – Well-Known Text (WKT) – Well-Known Binary (WKB) – Geographic Markup Language (GML)

Well-Known Text (WKT)• WKT examples: • POINT(6 10) • POINT(-111.06687 45.01188) • LINESTRING(3 4,10 50,20 25) • POLYGON((1 1,5 1,5 5,1 5,1 1),(2 2, 3 2, 3 3, 2 3,2 2)) • POLYGON(( -75.17031 39.95601, -75.16786 39.95778, -75.17921 39.96874, -75.18441 39.96512, -75.17031 39.95601)) • MULTIPOINT(3.5 5.6,4.8 10.5) • MULTILINESTRING((3 4,10 50,20 25),(-5 -8,-10 -8,-15 -4)) • GEOMETRYCOLLECTION(POINT(4 6),LINESTRING(4 6,7 10)) • CIRCULARSTRING(1 5, 6 2, 7 3)

Geospatial Methods • STArea • STBuffer • STCentroid • STDifference • STDimension • STDistance • STEnvelope • STGeomFromText • STIntersection • STIntersects • STPointFromText, STLineFromText, STPolyFromText • STPointFromWKB, STLineFromWKB, STPolyFromWKB • STSymDifference • STUnion • GeomFromGml • Parse • ToString • and more (about 70 total)

Geo-Spatial Types

0,0 150,0 300,0 50,50 100,1000,150 150,150 300,15020,180 180,180 150,300 300,300

Spatial Improvements InSQL Server 2012 • Circular Arcs • CircularString • CompoundCurve • CurvePolygon • All existing methods work on circular objects • New spatial methods • BufferWithCurves • STNumCurves, STCurveN • STCurveToLine • CurveToLineWithTolerance • IsValidDetailed • HasZ, HasM, AsBinaryZM • ShortestLineTo • UnionAggregate, EnvelopeAggregate, CollectionAggregate, ConvexHullAggregate • MinDbCompatibilityLevel

Spatial Improvements InSQL Server 2012• Improved Precision • Constructions and relations use 48 bits of precision (previously 27 bits)• geography Enhancements • Support for objects larger than a logical hemisphere (“FullGlobe”) • Support for new and previous “geometry-only” methods• New SRID • Spatial reference ID 104001 (sphere of radius 1)• Performance Improvements • Better tuning and hints • Auto Grid indexing with 8 levels (previously 4 levels)• Other Improvements • New histogram stored procedures • Support for persisted computed columns

Resources • Workshop slides and code – http://sdrv.ms/VSLiveNY2012SQL • Aaron Bertrand’s Blog – http://sqlblog.com/blogs/aaron_bertrand • Bob Beauchemin’s Blog – http://sqlskills.com/BLOGS/BOBB • Itzik Ben-Gan’s Web Site – http://www.sql.co.il • James Serra’s Blog – http://www.jamesserra.com • simple-talk – Learn SQL Server – http://www.simple-talk.com/sql/learn-sql-server

Thank You! • Contact us – andrew.brust@bluebadgeinsights.com – lenni.lobel@sleektech.com • Visit our blogs – brustblog.com – lennilobel.wordpress.com • Follow us on Twitter – @andrewbrust – @lennilobel • Thanks for coming! 

SQL Server Workshop for Developers - Visual Studio Live! NY 2012

by Andrew Brust on May 29, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

SQL Server Workshop for Developers - Visual Studio Live! NY 2012 Presentation Transcript