Business Intelligence Technology Presentation


Published on

Tutorial style presentation of software components used in the SetFocus Business Intelligence Masters Program.

1 Comment
1 Like
  • Thanks for posting...

    Very good for BI beginners..
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Business Intelligence Technology Presentation

  1. 1. This portfolio showcases the work I performed during my participation in the Business Intelligence Masters program from SetFocus. The BI Masters program is an intensive ten week, project-oriented, mentored program with top industry experts. The program modules were designed to reach deeply into each component of the Microsoft BI product set. The slides that follow are presented in a tutorial spirit and constitute but a sampling of the work performed. John Paredes
  2. 2. Contents • Integration Services • Reporting Services • Analysis Services • SharePoint/PerformancePoint
  3. 3. Product Components Used • SQL Server Management Studio (SSMS) • Business Intelligence Development Studio (BIDS) – Integration Services (SSIS) – Reporting Services (SSRS) – Analysis Services (SSAS) • Microsoft Excel • PerformancePoint Server • SharePoint Server
  4. 4. Program Objectives Overview • Use SSMS/T-SQL to create a dimensional data warehouse • Create packages in SSIS to import data from disparate sources, perform ETL functions, load the data warehouse, and perform database maintenance functions • Define an OLAP database using SSAS and populate it with data from the data warehouse. Use MDX expressions to create calculated measures and Key Performance Indicators (KPI)
  5. 5. Program Objectives Overview 2 • Using SSRS, create reports from both relational and multidimensional data sources. Incorporate advanced report features such as data grouping, on-the-fly calculations, and data- driven formatting • Create reports using the other Microsoft BI platforms: Excel Pivot Tables, Performance Point, SharePoint • Create Dashboards (organized presentations of KPI’s and reports) and deploy them on SharePoint
  6. 6. Integration Services Microsoft SSIS
  7. 7. What is SSIS? Part of the Business Intelligence Development Studio • Microsoft’s tool for performing a broad range of data migration tasks including common ETL tasks such as data cleansing and data transformations • Can be used to load data from disparate sources into a SQL Server data warehouse or from a SQL Server database into an Analysis Services Database (OLAP) • SSIS applications are built as a series of pre-built packages – Wide assortment of pre-built packages – Graphical representation • SSIS applications have a block-diagram graphical representation making them “self-documenting”
  8. 8. Project Features • Loading Applicant data from disparate sources • Data cleansing using the fuzzy grouping transformation
  9. 9. An Illustration of Data Cleansing The dimensional data in the Applicant table had inconsistencies in the way the data was entered. Some types of inconsistencies could be corrected by a view that implemented simple fixes. In the case of the account manager field, a more sophisticated approach utilizing the fuzzy grouping transformation was used to get the required consistency. To illustrate its operation, we note that, for example, the data for “MMcConnaug” was entered in three different variations. Because the number of account managers was small and the differences between them big enough, it was easy for the Fuzzy Grouping transformation to identify the inconsistencies and choose a consistent representation. The account manager dimension table had a simple structure that is described via this sql statement: CREATE TABLE [dbo].[dimAcctMgr]( [AcctMgrPK] [int] IDENTITY(1,1) NOT NULL, [AcctMgr] [varchar](20) NOT NULL, CONSTRAINT [dimAcctMgr_PK] PRIMARY KEY CLUSTERED
  10. 10. An SSIS package was created to populate dimAcctMgr . The screenshot below shows the data flow task used by that package.
  11. 11. The Data Flow Task Used to Populate the Account Manager Dimension Table The OLE DB source transformation uses the SQL command data access mode to read from the view that removes nulls and blanks from the source table. The statement used is: SELECT distinct AcctMgr FROM DimAccount Order by AcctMgr The output of the OLE DB Source transformation is fed to the Fuzzy Grouping transformation. The figure to the right shows its output as seen from a data reader. SSIS lets you create data viewers to show what is happening inside a package. I created a data viewer to show what was going on inside of the Fuzzy Lookup transformation.
  12. 12. Output from the Fuzzy Grouping Transformation From the display you can see that the transformation has selected a representation for and has correctly associated the variations with it. Next, an Aggregate transformation is used to reduce the output data set to its unique values of account manager. Finally, the Look and Conditional Split transformations are used to ensure only new data is added to the table.
  13. 13. Loading the Fact Table This application has a single fact table, a “factless” fact table; all measures are counts. The process of loading the fact table consists of cycling through the Applicant table and populating a fact table with surrogate keys for each of the base dimensions. A degenerate key was also added. Each column of data from the data view is matched with its value in the corresponding dimension table. Using this match, the value of the integer key is read and is output as a column of the same name but prefixed with the word fact. Because the view does not perform the data cleansing for the account manager field, there will be some cases where there is no match for that field. Those data items will not have a PK. The conditional split transformation near the end of this procedure identifies such rows and directs them to the Fuzzy Lookup transform. This transform translates the text data into its standard representation after which an analogous process produces the key.
  14. 14. Loading the Fact table diagram
  15. 15. Reporting Services Microsoft SSRS 16
  16. 16. Reporting Services Overview • Included as an install option of SQL Server • Sophisticated, interactive reports can be created with Report Designer • Report definition information stored in RDL, an XML- based language • Standalone Report Server comes with SQL Server • Users can interact directly with Report Server or use Report Manager to view, subscribe to, and manage reports • Reports can be generated in a variety of formats 17
  17. 17. Steps for Creating a Report In SSRS 1. Create a Reporting Services project in the Business Intelligence Development Studio (BIDS) or Visual Studio 2. Create a Data Source 3. Design the Report 4. Deploy Report to a Report Server so it can be viewed from a web browser 18
  18. 18. 1. Open BIDS and create a new Report Server Project. Give its name and location. 19
  19. 19. 2. Create the Data Source A data source can be associated with an individual report or with the entire SSRS project. We will create a shared data source, which is associated with the project. You can choose from a variety of different sources including SQL Server, Analysis Services, Essbase, Oracle, and XML. In this example we will connect to a SQL Server database. 20
  20. 20. 3. Enter Connection Properties In this panel you specify the server, the method of authentication, and the database – everything the system needs to know to connect to the data source. 21
  21. 21. With the data source defined, you are ready to create the report… Begin by selecting Project/Add New Item… Then choose Report and give your report a name. This creates an XML file (it has a .rdl extension) that will save the report information you enter. 22
  22. 22. Designing a report consists of these steps: • Create a Data Set – To tell what data you need • Define the Layout – To tell where on the page to place the data elements • Preview the Report – To make sure it’s good 23
  23. 23. First Stop: the Data tab On the Data tab of the report designer, click on the Dataset drop- down list and select <New Dataset…>. This opens the Dataset dialog box. You are going to name the data set, then you need to create a query that retrieves the data that will be used in the report. There is a choice of two interfaces for describing the data you want: •Graphical Query Designer •Generic Query Designer This illustration is going to use the graphical query designer. 24
  24. 24. After clicking <New Dataset…> the dataset dialog box appears. Enter the dataset name and press OK. 25
  25. 25. Unpress the Generic Query Designer toggle button that is to the left of the exclamation mark. This will bring you to the graphical query designer (shown below.) 26
  26. 26. Right-click the diagram (top) pane of the query designer and select Add Table from the popup menu. Select the tables you want and press Add. 27
  27. 27. After you’ve added the tables, you’ll get a panel like this where you choose data columns and set the data orderings. 28
  28. 28. Next Stop: The Layout Tab With the dataset defined, the Datasets pane displays an item for each data column. Click the Layout tab. You can place Report Items such as data tables or charts on the page. Here we will place a table item. 29
  29. 29. Now place data items on the table. Below you see that all four data items have been dragged over to the detail row of the table region. 30
  30. 30. The design surface can have more than one report item on it. Here an image and text box has been dragged on to complete the report design. 31
  31. 31. Final Stop: Preview tab With the design complete push the Preview tab to see the report as it would appear to the end user. 32
  32. 32. Deploying A Report • Deploying means sending the report information to a web server so the report can be viewed by the user in a web browser • From the web browser the report can be viewed and also can be downloaded in various formats: pdf, XML,CSV, TIFF, Web Archive, and Excel. • Steps for Deploying a project – Select Solutions Configuration – Set TargetServerURL in ProjectName Properties – Deploy the project from the Build menu 33
  33. 33. Adding Calculations 34
  34. 34. SSRS Calculations • SSRS Can display information that is not explicitly represented in the data source via calculations • Calculations are expressions that are evaluated at report display time • Calculations can be used to provide new report information or to provide data driven formatting 35
  35. 35. I will be demonstrating these examples: • Truncate a long text expression to make it fit in the reporting space • Show how calculations can access global variable values and, using the concatenation operator, create a text expression to display • Create data driven formatting expressions. Data values will be used to determine font properties such as color and weight 36
  36. 36. To create a calculation, right click in the data field and select Expression… in the pop- up menu. A dialog box will appear into which a formula expression is typed. 37
  37. 37. The expression entered truncates the data of the pr_info field to its first 50 characters. 38
  38. 38. In the last example, the calculation was created from one of the report data fields. In this next example, we will create a calculation starting from a blank text box. The data that will be displayed will come from global variables. As with the previous example, start by right clicking in the text box and choose Expression... 39
  39. 39. The expression shown uses the concatenation operator and several global variables to build the text string that will be displayed. 40
  40. 40. Text formatting can be accomplished via static property values or with dynamic, data-driven values using expressions. 41
  41. 41. Let’s use an expression to define formatting. From the property sheet of the data field select a color and click <Expression…> 42
  42. 42. If the product of price and quantity is greater than 500, the data will appear in red, otherwise, it will be black. 43
  43. 43. I have also entered an expression to affect the font weight. After these two data driven fields have been defined the properties sheet looks as shown below. 44
  44. 44. With the formatting complete, click on the Preview tab to see the report. 45
  45. 45. Organizing Data by Groups 46
  46. 46. In SRSS, Groups… • Provide a structure for organizing the data presentation • Cluster data together based on a common attribute, for example, all orders from the same company • Provide a mechanism for giving hierarchical data presentations, each grouping variable corresponding to a hierarchy level 47
  47. 47. To create a group, right-click on the header and choose Insert Group… 48
  48. 48. In the Grouping and Sorting Properties dialog box that appears, insert an expression to define what constitutes a group. Here I use company name. With this, all purchases from a given company will appear together. 49
  49. 49. Here you can see the new rows introduced by the group, and the grouping variable, CompanyName.Value 50
  50. 50. There can be multiple levels of grouping. Click in the heading row of the first group to add another grouping variable. I will add a grouping based on time. 51
  51. 51. The grouping variable can be a data field or an expression you create. I will create one based on time. Start by choosing <Expression…> 52
  52. 52. The expression built uses two data fields and two functions. 53
  53. 53. Set grouping and sorting properties 54
  54. 54. This is what the report layout looks like with both groups defined. 55
  55. 55. Preview the final report to see it is grouped on company and month. 56
  56. 56. Analysis Services Microsoft SSAS
  57. 57. Analysis Services Overview Part of the Business Intelligence Development Studio • Included with the SQL Server License • Special version of Visual Studio • Microsoft’s application for creating multidimensional OLAP databases which are queried with the MDX language • Microsoft’s powerful data mining platform. Includes sophisticated algorithms that can operate on relational or OLAP data • This presentation demonstrates the creation of an OLAP database, also called a cube 58
  58. 58. Why Store Data in a Cube? Analysis services, like the SQL Server relational database, is a platform for storing data. But unlike the relational database, it does not store data in tables–it is stored in other types of structures comprising a cube. Why store data in cubes instead of tables? There are a number of reasons: • better query performance • fast, optimized aggregations calculations • more efficient storage of data through its simplified read-oriented design • richer calculation possibilities, supporting stored measures, calculated measures, and key performance indicators (KPI) • an easier to understand data model for the end user OLAP data is engineered to provide dimensional data views, hierarchical browsing, attribute-based breakouts and filtering. And OLAP data does not require or even use table joins! 59
  59. 59. Creating an Analysis Services (OLAP) Database Creating an analysis services database consists of creating the database structure, then populating those structures with data from external data sources. The structures comprising an OLAP cube include: • Dimensions and their associated elements – Hierarchies – Attributes • Measures and their associated elements – Stored Measures – Calculated Measures – Measure Groups – Key Performance Measures (KPI) – Measure Profiles 60
  60. 60. Development Steps 1. Create Analysis Services Project 2. Create Data Source 3. Create Data Source View 4. Create cube object definitions 5. Deploy definitions to OLAP server and load in data 6. Specify partitions and aggregations 61
  61. 61. Open BIDS and create a new Analysis Services Project… 62
  62. 62. Create a data source and a data source view In the data source you identify the relational data warehouse (DW) that hosts the source data and provide the connection information. In the data source view, you specify which the tables in the DW you want to use to supply data to the cube. If the DW has been designed using the classic star schema or snowflake approaches, the SSAS cube wizard can examine the DW’s structure and derive a cube definition from it, making your job easier. When the initial design from the wizard is complete you can go back and make modifications, for example, you can add or modify hierarchies and attribute relationships,. Once you have the cube design you want, you load the data in. 63
  63. 63. In this illustration, all tables from the star schema DW are used, only the sysdiagrams table (which holds the database diagram data) remains unselected. 64
  64. 64. Once you have defined the data source view, you can inspect the schema. By right clicking on a table, you can browse the data and define derived columns. But at this point, we still have no OLAP database. The creation of the OLAP database begins with creating a cube. 65
  65. 65. Creating the Cube Structure 1. With the data source view in place, you are now ready to create the cube. Right click on the cubes folder and select “New Cube.” This launches the cube wizard. 2. Make sure the wizard has correctly identified which are dimension and which are fact tables and tell it which table contains the data for the time dimension. 66
  66. 66. Setting the Time Dimension Parameters Time is a unique dimension with inherent assumptions about how it should work. You identify the time dimension as such so that the MDX functions (such as PrevMember and ParallelPeriod) specific to it will work. You also tell which of the source data columns map to well known time concepts such as years, quarters, and months. 67
  67. 67. After defining the time properties, the wizard displays the measures that it defined. Here you can select which ones you want to keep. I am keeping all of them in this illustration. The “Fact Units Count” measure counts the number of records in the source table. This information is used in optimizing the aggregation process. In the Review New Dimensions panel the dimensions that were created along with their hierarchies and attributes are displayed. 68
  68. 68. A cube has now been defined. This panel lets you review the data model (UDM) that has been created. 69
  69. 69. Completing the wizard, you give the cube a name. 70
  70. 70. The solution browser on the right side of the screen now shows the cube and dimensions that were defined. These objects can be modified by clicking on them. 71
  71. 71. Click on the Product dimension to observe that no hierarchy was by the wizard. I will create a hierarchy for it. 72
  72. 72. A new hierarchy is created by dragging the an aggregate attribute into the center panel. I use Category Code and Dim Product to form the hierarchy; Category and Item provide the labels the user will see (they are mapped to the name Property). Next I define attributes between these hierarchy levels. This is so the aggregation process adds the data from the level directly beneath it instead of always going to the leaf level, which would be a less efficient process entailing significantly more calculations. 73
  73. 73. In the case of the time dimension, there is an additional step. We want the months to display in chronological order, not alphabetical order, so we assign a value to the dimension’s OrderByAttribute property. The data source has a column called CHRON_ORDER that contains this ordering information. 74
  74. 74. The information entered to define the hierarchies and attributes is stored in XML files. The database doesn’t actually know what you have done yet. You must “process” the dimension. This brings that information into the OLAP database, which it then uses to create its internal structures. 75
  75. 75. Once a dimension has been processed, you can inspect it in the browser pane to verify the hierarchy has the expected structure. Note that the months show in chronological order. 76
  76. 76. After all the elements have been defined and the cube has been completely processed, you can inspect the data in the SSAS data browser. 77
  77. 77. Partitioning and Pre-Aggregating
  78. 78. The cube was and created. The leaf level data from the data warehouse was loaded. We inspected the data in the multidimensional data browser. We can tweak the physical design of the cube to improve scalability and query performance. Three primary mechanisms for doing this are: •Selecting ROLAP/HOLAP/MOLAP data storage options •Partitioning •Pre-aggregating MOLAP is probably the most commonly used data storage option and is the default. It means all the data will be stored in the multidimensional data cube. The illustration one the succeeding slides will used MOLAP. At the opposite end of the spectrum, is ROLAP where all data comes from relational data tables. With ROLAP, the SSAS database is only providing metadata structures for presenting information in the dimensional style. You can expect performance to be much slower. This mode is used in situations where the source data is not static, changes frequently and you want the reports to reflect those changes immediately. HOLAP is a hybrid approach where all the aggregate data is in the SSAS database except the leaf level data which resides in relational tables. When the volume of data is very large, it can helpful to chop the data store into pieces, or partitions. Storage mode is selectable per partition. This example uses a small amount of data and only one partition will be employed. 79
  79. 79. Aggregates are summary level data that are computed from the leaf level data that was loaded from the source. Often the aggregates are totals and subtotals, but other summary statistics such as averages or maximum values can also be used. Pre-calculating and storing the aggregate values normally improves query performance (at the cost of the storage space and time required to compute them.) The default is to do no pre- aggregates. You can see this from the partitions panel shown below. The data display shown earlier from the SSAS data browser included many aggregate data values. Those aggregates were all calculated on the fly. You can pre-calculate all aggregates or only some of them. If you are going to pre-calculate only some, there are different strategies that can be employed to determined which are chosen for calculation. You’ll see this ahead. 80
  80. 80. Let’s go through the aggregations process. Click the “Design Aggregations” hyperlink to bring up the wizard. In the first panel of the wizard, push the count button to compute the statistics that are used to drive the aggregation optimization process. 81
  81. 81. After a few seconds, the source data has been analyzed. It counts the number of records per partition. Once the statistics have been computed, you can ask the system to identify a set of aggregations to perform. You can direct the system to aggregate until A) a certain amount of storage has been used, B) a certain level of performance gain has been achieved, C) you click stop, or D) don’t do anything In this illustration, I am asking it to aggregate until it reaches a performance gain of 75%. The system will run an optimization algorithm to determine the best ones to use. 82
  82. 82. The system generates a chart telling what percent (of the total possible number of) aggregations it has identified and what level of performance gain would be achieved by computing them. At the completion of that step, the wizard has identified which aggregations to compute. You may elect to have it compute them now or you can defer the calculations till later. (They could take a while.) 83
  83. 83. Selecting “Deploy and Process now” and pushing Finish, you arrive at this screen. Push the RUN button to perform the calculations. When it finishes, you get a message heralding the successful completion of the deployment. The information under the Aggregations tab will be updated. 84
  84. 84. Creating Derived Measures and KPIs
  85. 85. Different Kinds of Reporting Data Thus far, all the measures that have been constructed have been displays of stored data or aggregates either stored or calculated on the fly. There are other kinds information that can be made available to an end user. • Calculated measures – Percents – Shares – Differences • Key Performance Indicators (KPIs) Calculated measures are calculated on the fly using MDX expressions. KPIs are measures with associated goals and graphics. I will show an example of both. 86
  86. 86. In this example, I create a calculated measure that gives difference between the data value at a given time and its value the previous time period. The calculation is defined from the Calculations tab. It is given a name and an MDX expression. In this example I make use of the PrevMember function. 87
  87. 87. Displaying the Units measure and the Units Increase measure side-by-side demonstrates that the calculated measure correctly computes the difference between the current value and the one a month ago. In the next series of slides I will use this calculated measure to construct a KPI. 88
  88. 88. What is a Key Performance Indicator? (KPI) Every KPI starts life off as a measure, presumably, a measure that is an indicator of company performance. With each KPI, we assume that the company has established a target value – goal – of what that indicator should be. For instance, sales revenue might be a performance indicator. The goal might be to sell at least $100,000 in a given quarter. The KPI will calculate the difference between the goal and the actual result. We assume the company can assess those differences declaring them as either good, so-so, or bad. For instance, the company may say, revenue > 100,000 is good, 90,000 to 100,000 is so-so, and revenue less than 90,000 is bad. This brings us to an essential distinguishing feature of the KPI: a graphical icon, known as an indicator that is displayed to communicate the status of the KPI to the end user. That graphic might be a happy face to show good, a neutral face to show so-so, and a frowning face to show bad. Traffic lights with green, amber, and red are often used. The choice of graphics is up to the client. Setting up a KPI in Analysis Services entails computing a value of status. The difference between the indicator and the goal is calculated, and the differences that are “good” are mapped to the number 1, so-so to 0 and bad to -1. That number is the KPI’s status. Optionally, you can define a trend for the KPI. The trend shows if, over time, the performance measure has been moving upwards or downwards. 89
  89. 89. KPI Summary • Begin with a measure indicating company performance • Have goals associated with that performance measure • Translate the difference between performance and goal into its status with values of 1, 0, -1 (corresponding to good, so-so, and bad.) • Display the status of the performance measure to the user as a graphic 90
  90. 90. You define KPIs from the KPI tab of the Cube browser. In this simple illustration, our calculated measure, “Units difference” is the performance indicator, and the goal is a constant value of 180. MDX expressions can be used to provide more complex goal statements. 91
  91. 91. Once you have defined the KPI, it may be inspected in the browser tab of the KPI tab. Here you see the performance metric has a value of 179, just under the target value. This is “so-so” and you see the neutral face showing. 92
  92. 92. SharePoint Server and PerformancePoint Server
  93. 93. Overview • SharePoint Server 2007 and Performance Point Server 2007 are both part of the Microsoft Office server suite of products, not Business Intelligence Development Studio • SharePoint server is a content management system with a multiplicity of goals including the ability to provision reports and analytic content created in Business Intelligence Development Studio and PerformancePoint • PerformancePoint includes monitoring, analytics, and planning functionality. Included in the product is Dashboard Designer, an application for formatting analytic content (including reports, charts, KPI’s and scorecards) into displays called dashboards
  94. 94. DashboardDesigner Content Development • Can import measures, KPIs, and dimensional elements from Analysis Services • Can Import reports and charts from Reporting Services and Excel • KPIs are organized and formatted into scorecards • Scorecards and reports, together with filters and inter-item links, are formatted into dashboards • Dashboards are what the end user will see • Dashboards can be displayed from Dashboard Server or SharePoint Server
  95. 95. Sample Reports from My SharePoint BI Site developed at SetFocus