Building a Data Warehouse


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Building a Data Warehouse

  1. 1. SALES DATA WAREHOUSE BY Kalyan Yadavalli A Master’s Project Report submitted in partial fulfillment of the requirements for the degree of M.S in Computer Science Old Dominion University 2006 Approved by ____________________________________________________ Chairperson of Supervisory Committee ___________________________________________________ ___________________________________________________ ___________________________________________________ Program Authorized to Offer Degree__________________________________________________ Date:
  2. 2. INDEX S.No TOPIC Page 1. Acknowledgement 3 2. Abstract 4 3. Background and Research 5 4. Data Warehouse Project Life Cycle 6 Life Cycle of Sales Data Warehouse – Kimball 6 Methodology 5. Sales Data Warehouse Project Cycle 8 Project Planning 8 Business Requirement Definition 8 Dimensional Modeling 9 Build a dimensional bus matrix 10 Design Fact tables 10 Dimension & Fact table detail 11 Dimension Model Example 13 6. Technical Architecture Design 14 7. Product Selection and Installation 15 8. Physical Design 15 9. Data Staging & Development 18 a.] Implement the ETL [Extraction-Transform-Load] process 18 ETL Architecture 18 Details: Extracting Dimension Data 19 ETL Code Sample 20 Managing a Dimension Table - Example 20 Managing a Fact Table - Example 27 b.] Build the data cube using SQL Analysis Services 31 10. End User Application Specification 32 11. End User Application Development 34 12. Deployment 36 13. Maintenance and Growth 37 14. References & Appendix 42 2
  3. 3. ACKNOWLEDGEMENTS I express my sincere thanks to my project advisor, Dr.Irwin Levinstein, Associate Professor in Computer Science, Old Dominion University, for his encouragement, support and understanding throughout the course of this project work. A special and sincere thanks to the IT department at The Virginian-Pilot, where I am currently employed, for allowing me use SQL Server 2005 suite of products and providing access to marketing database for building the data warehouse. Last but not the least, I express my sincere appreciation and thanks to my family and friends for their invaluable support, patience and encouragement throughout my Master’s course. 3
  4. 4. ABSTRACT Media companies especially the newspaper companies throughout the nation are experiencing a drop in revenues in their marketing and advertising departments. This loss of revenue can be attributed to poor planning and lack of transactional data on households and individuals in the market .The mission of Sales Data Warehouse project is to provide strategic and tactical support to all departments and divisions of a media company through the acquisition and analysis of data pertaining to their customers and markets. This project helps to identify areas of readership and marketing through creation of a Data Warehouse that will provide a company with a better understanding of its customers and markets. Overall, the project has identified three basic needs: Acquiring and maintaining core data about the households, individuals, and businesses within the market area. Acquiring and maintaining transactional data on the households, individuals, and businesses within the market area. Acquiring and implementing the tools needed to effectively manipulate and access the core and transactional data. This project provides a wide variety of benefits to a number of business units within a media company. These benefits are expected to help drive marketing and readership, as well as improve productivity and increase revenue. The benefits include: Marketing • Increased telemarketing close rates and increased direct mail response rates • Reduced cost and use of outside telemarketing services and reduced print and mailing costs • Identification of new product bundling and distribution opportunities • Increased acquisition and retention rates, and reduced cost of acquisitions Advertising • An increase in the annual rate of revenue growth. • Increase in new advertisers • Improved targeting capabilities The above requirements can be met by building a Sales Data Warehouse using Microsoft ® SQL Server™ 2005 suite [Database Engine, Analysis Services, Integration Services and Reporting Services] of products. 4
  5. 5. BACKGROUND AND RESEARCH A group of 6-7 key business users of the Marketing and Advertising departments comprising of the Acquisition managers, Retention managers and Marketing Analysts were interviewed for the Sales Data Warehouse project. The information collected from the interviews is summarized as follows. CRITICAL BUSINESS OBJECTIVES AND ISSUES: The overriding business objective is to increase net paid circulation. Some of the secondary objectives are reducing churn and increasing retention, profiling best customers so we can market to their non-subscribing look-alikes, improving efficiency in sales channels, monitoring sales and marketing activities, tracking subscriber behavior across multiple subscriptions and households, targeting sales and retention efforts, and focusing sales on high impact channels, prospects and offers. FINDINGS: From the interviews with the business users it was also found that there are two areas of the business that provide the answers to the above requirements: subscription sales and subscription tracking. Subscription sales address issues about sales performance, campaign sales results, and retention of new subscribers. Subscription tracking deals with issues that affect all subscribers, like best and loyal customers, long-term behavior, and renewal rates. Other important areas are stops, complaints, subscriber payments, and market profile. Subscription Sales The analysis of subscription sales has the highest potential business impact. New sales analysis helps to evaluate sales channels and retention, and it can cover recent and short time spans (e.g. weekly starts, current sales mix) and also longer time spans (retention, campaign performance). This analysis will help to tune offers, sales compensation, and manage sales performance. REQUIREMENTS: Acquiring and maintaining core (identification, demographic, psychographic) data about the households, individuals, and businesses within the market area. This data is needed to increase the marketing effectiveness through a better understanding of the profiles of existing and potential customers (who they are, where they are, what are their attributes, etc). Acquiring and maintaining transactional data on the households, individuals, and businesses within the market area. This data is needed to increase the marketing effectiveness through a better understanding of the history of existing and potential customers (what business have they done with the company, how and when have they been contacted by the company, etc). Acquiring and implementing the tools needed to effectively manipulate and access the core and transactional data. This includes tools for data importing {including historical data and nightly updates}, formatting, cleansing, filtering, and exporting. It also includes 5
  6. 6. tools for data analysis and data mining. This will enable the company to effectively maintain and utilize the data that is captured. A consummate knowledge of the marketplace will lead to better business performance. In marketing, that means acquiring and keeping the most subscribers efficiently. We think we can be efficient by delivering the best messages in the best way to each person or household in the marketplace. The best messages are based on what we know or can infer about preferences and lifestyles, and the best way is based on past responses to marketing. We want to forecast the return on proposed marketing efforts, select profitable proposals, measure actual performance, and improve our forecasting and selection of future proposals. Observed and reported information about the market comes from the marketing system, Claritas PRIZM clusters {household level segmentation system using demographic and lifestyle data}, and marketing promotions tracking. We need to get to where we track every way that we touch consumers and how they respond or don’t respond and we need to apply these competencies to how we market to businesses. Above all, decision makers will be able to use information to make quick decisions. DATA WAREHOUSE PROJECT LIFE CYCLE A data warehouse is a copy of data combined from different data sources specifically structured for querying and reporting. Life Cycle of Sales Data Warehouse – Kimball Methodology The following diagram depicts the sequence of high level tasks required for effective data warehouse design, development and deployment using the Kimball data warehousing life cycle – [Reference : The Data Warehouse Lifecycle Toolkit Tools and Techniques for Designing, Developing, and Deploying Data Warehouses] Maintenance & Technical Product Growth Architecture Selection & Project Design Installation Planning Business Requirement Definition Dimensional Physical Data Staging Deployment Modeling Design design & development End-User End-User Application Application Specification Development Kimball- Dimensional life cycle diagram Project Planning: 6
  7. 7. The lifecycle starts with project planning, which addresses the scope and definition of the marketing data warehouse and business justification. The two way arrow between project planning and business requirements indicates that planning is dependent on requirements. Business Requirements: Business requirements determine the data needed to address the business user’s analytical requirements. Business users and their requirements play an important role in the success of the data warehouse project .The requirements establish the foundation for the three parallel tracks focused on technology, data and end user applications. Data Track: Dimensional Modeling Dimension modeling is the name of the logical design technique often used for data warehouses. Every dimensional model is composed of one table with a multi part key, called the fact table, and a set of smaller tables called dimensional tables. Data Track: Physical Design This process defines the physical structures necessary to support the logical database design. It includes defining the naming standards and setting up database environment. Data Track: Data Staging Design and Development This process includes the three fundamental steps of a data warehouse namely extraction, transformation and load .The data is extracted from the source systems into a source staging database (initial data population) and then transformed in the staging database (incremental loads) and loaded into the production database. Technology Track: Technical Architecture Design Technical architecture tells us how to implement the data warehouse. It describes the flow of data from the source systems to the End users after a series of transformations. The tools and techniques that will be used in the implementation of the warehouse are specified in the architecture .The technical architecture design of the data warehouse depends on business requirements, current technical environment and planned strategic technical directions simultaneously. Technology Track: Product selection and installation Based on the architectural design the hardware platform, database management system, data staging tool are evaluated and selected. Once the products are selected, they are installed and thoroughly tested to ensure appropriate end-to-end integration within the data warehouse. Application Track: End User Specification Application specifications like the report template, user driven parameters and required calculations are collected from the business users at this stage. These specifications ensure that the developer and the business users have a common understanding of the applications to be delivered. Application Track: End User Application Development 7
  8. 8. This process involves building of specific set of reports based on the end user specifications. The reports are built in such a way that the business users can easily modify the report templates. Deployment: Deployment represents the convergence of technology, data and end user applications accessible from the business user’s desktop. Maintenance and Growth • Support the business users • Provide training to the business users • Manage the processes and procedures for ongoing operation of the data warehouse. Sales Data Warehouse Life Cycle Project Planning Scope: The scope of the Sales Data Warehouse is based on the business user requirements and involves the following. • Create a foundational model 1. Is a basis on which to build, grow, and extend 2. Has potential beyond what is implemented 3. Has inherent leverage for growth and extension • Create the processes that extract data from source systems; cleanse, transform, and match data according to specified business rules; import data into the data warehouse structure; use query, reporting, and mining tools to access the data and address key business questions • Cost-justify the initial investment in the project by answering the business users questions. The questions identified encompass business processes related to new subscriber acquisition, demographic household maintenance, subscriber account maintenance, subscription marketing, and customer list profiling. Business Requirement Definition This phase involves the following steps: • Collect some business questions the users want and answer for. • Gather details/requirements from the business users 8
  9. 9. • Get user sign off on the business questions. The following business questions and descriptions have been collected from the marketing business users as part of the requirements gathering phase of the data warehousing life cycle. 1. Can we profile our "best subscribers" (by segment, EZPAY, long-term, prepaid, etc?) to pull lists of "like" non-subscribers that we could touch in some way? Description: Best customers are those who subscribe to 7day, 52week, EZPay {automatic payment from credit card} subscriptions. 2. Who exists in the marketplace and have we touched them? Description: By building “complete” universe of households with extended demographics, and by attaching “touch” or contact histories to individuals, can determine what is working on what types of customers (and what isn’t working). We need to build touch history over time. Value increase as more history is accumulated. 3. Can we build a loyalty model based on a subscriber's payment history? Description: Loyalty can be defined as overall duration as subscriber, number of consecutive payments (with unbroken service), etc. We can then create a profile of a “loyal” subscriber and more effectively identify likely churners vs. likely loyal subscribers. Dimensional Modeling This is a very important step in the data warehousing project, as the foundation of the data warehousing system is the data model. A good data model will allow the data warehousing system to grow easily, as well as allowing for good performance. In data warehousing project, the logical data model is built based on user requirements, and then it is translated into the physical data model. This project uses Dimensional modeling, which is the name of the logical design technique often used for data warehouses. It is different from entity-relationship modeling. Entity relationship modeling is a logical design technique that seeks to eliminate data redundancy while Dimensional modeling seeks to present data in a standard framework that is intuitive and allows for high-performance access. Every dimensional model is composed of one table with a multi part key, called the fact table, and a set of smaller tables called dimensional tables. Each dimension table has a single part primary key that corresponds exactly to one of the components of the multi part key in the fact table. This characteristic star like structure is often called a star join. A fact table, because it has a multi part key made up of two or more foreign keys always expresses a many-to-many relationship. The most useful fact tables contain one or more numerical facts that occur for the combination of keys that define each record. The fundamental idea of dimensional modeling is that nearly every type of business data can be represented as a kind of cube of data, where the cells of the cube contain measured values and the edges of the cube define the natural dimensions of the data. 9
  10. 10. The following are the steps involved in building a dimensional model. Build a dimensional bus matrix The first step in designing a data warehouse is to build a dimensional matrix with the business processes as the rows of the matrix and the dimensions as the columns of the matrix. Example: Dimension Matrix Dimensions Date Demographics Rate- Salesperson Campaign Service- Term Business Processes Subscription X X X X X Sales ( starts) Subscription X X X X Tracking Complaints X X X Stops X X X Upgrades X X X X X And Downgrades If a dimension is reasonable enough to be listed under a business process a cross (X) entry is made at the intersection of the business process and the dimension .In the above matrix, the campaign dimension is linked to SubscriptionSales, UpgradesAnd Downgrades business processes and not with the other processes. We can find out the sales performance and the upgrade/downgrade [upgrade - moving from a weekend subscription to daily subscription, downgrade- moving from a daily subscription to weekend subscription] information as a result of a new campaign introduced in the market. Design Fact tables The dimension matrix information is used in the design of the fact tables. The following are the steps involved in the design of a fact table • Choose the Business Process as the fact table A business process from the dimension matrix is picked as a Fact table. Eg: SubscriptionSales (starts- starting a subscription) • Declare the grain Declaring a grain is equivalent to saying what an individual fact table record is. In case of SubscriptionSales fact table the grain is “Each Subscription sold”. • Choose the dimensions The corresponding dimensions of a fact table are chosen based on the dimension matrix built in the above step. 10
  11. 11. Eg: the dimensions date, demographics, rate-service-term, campaign, salesperson are selected for the SubscriptionSales fact table. • Choose the facts The last step in the design of a fact table is to add as many facts as possible within the context of the grain declared in the above step. The following facts were added to the SubscriptionSales fact table design. Eg: Units sold, Number of sales, Dollars sold, Discount Cost, Premium Cost. Fact Table Detail The following diagram shows the SubscriptionSales fact table details listing the dimension keys and facts along with the grain. SubscriptionSales Fact Table EffectiveDateKey CampaignKey SalespersonKey CustomerKey DempgraphicsKey SubscriptionKey ……………….. Grain: Each subscription sold Facts Units Sold Number of Sales (=1) Dollars Sold Discount Cost Premium Cost Dimension Table Detail The following dimension table detail diagram shows the individual attributes in a single dimension. Each dimension attribute has an Attribute Name, Attribute Description, Cardinality, Sample Data, and Slowly Changing Policy. Eg: Dimension: Subscriptions Slowly changing Attribute Attribute Cardinality dimension policy Name Description Sample Values Subscription Combined 500 Not Updated Pilot 7-day 13-week 50%, Name name of the Pilot Weekend 26-week pub, service, standard term, and rate 11
  12. 12. Rate Rate name 10 Overwritten 50%, 30%, Standard Rate Year The year the 6 Overwritten 2003, 2004 rate is valid Rate Area The region 5 Overwritten State, City, NDM, Outer that markets corresponds to the rate Rate Type Category of 4 Overwritten Mail, Standard, Discount, rates College Discount For 5 Overwritten 50%, 30%, 25% Category discounts, the category or grouping of discounts Service The name of 7 Overwritten Sunday Only, Weekend (2- the day), Weekend (3-day), 7- frequency of day delivery Frequency Grouping of 4 Overwritten Weekend, Daily, Mid-week Groups frequency of delivery Term The term 10 Overwritten 4-week, 8-week, 39-week, 52-week, 104 week Term length Grouping of 4 Overwritten 8-week, 13-week, 26-week, groups term lengths 52-week Short or Indicates 2 Overwritten Short Term, Long Term Long term length of term Publication Name of 5 Overwritten Pilot publication Business Business 3 Overwritten Pilot Media Group unit for grouping publications 12
  13. 13. Dimension Model [eg: SubscrptionSales logical design] Subscription Sales Payment Salesperson EffectiveDateKey CustomerKey PaymentKey SalespersonKey LoyaltyKey PaymentKey SalesConditionsKey CampaignKey SalesPersonKey SubscriptionsKey DurationKey DemographicsKey AuditKey AddressKey RouteKey AddressNumber SalesConditions SubscriptionNumber PBMAccount SalesConditionsKey Campaign NumberOfSales UnitsSold CampaignKey DollarsSold DiscountCost PremiumCost Route Subscriptions RouteKey SubscriptionsKey Address Loyalty Demographics AddressKey LoyaltyKey DemographicsKey 13
  14. 14. Technical Architecture Design While the user requirements tell us what needs to be done, the technical architecture tells us how to implement the data warehouse. It describes the flow of data from the source systems to the End users after a series of transformations. The tools and techniques that will be used in the implementation of the warehouse are specified in the architecture. The following diagram depicts the high level technical architecture of the Sales data warehouse. Source Systems Data Staging Area Presentation Area Data Access Tools Subscription SQL Services Sales Reporting Services Transform from Marke source to target. Dimensional. ting Extract Atomic and Excel Maintain Data conformed summary data. dimensions. Business Access process. Load Access Claritas Data Storage Flat files or Design Goals Demo Extract relational tables Ease-of-use. graphi Query cs Design Goals performance. Staging throughput. Name Integrity and Dimensional Phone Extract consistency. Bus: Data Conformed facts and dimensions Sales Data Warehouse High Level Technical Architecture The data from the source systems (Marketing database, External Demographics data, and Name Phone data from National Do Not Call list) is extracted to a data staging area into flat files or relational tables. The data is then transformed here and loaded into the fact tables and dimensional tables in the presentation area. All aggregations are calculated in the staging area. The transformed data in the presentation area is then accessed using data access tools like SQL Reporting Services, excel and Claritas. 14
  15. 15. Product Selection and Installation Data Warehousing database servers require good amount of CPU cycles to process data [Nightly processing to update the data warehouse] providing processed information to the end users. Hence the hardware requirements are very high. Considering the growth of the marketing database and the possibility of including new data sources in future, the following hardware specifications were selected for the Sales data warehouse server. Hardware Specifications: AMD Opteron Processor 252 2.6 GHz, 3.83 GB RAM Operating System: Windows Server 2003 Software Specifications: Microsoft ® SQL Server™ 2005 suite of products [Integration Services , Analysis Services and Reporting Services] and Cognos tools [ Impromptu and Powerplay] were considered for developing the data warehouse . Based on the ease of use and cost benefits, Microsoft ® SQL Server™ 2005 suite of products was selected and installed on both the test and production environments for the Sales data warehouse project. SQL Server Database Server: Holds the databases for Source data, Staging data and production data. SQL Server Integration Services [Used to automate the process of loading the data from the source systems to a staging area on the SQL Server] SQL Server Analysis Services [Used to create the data cube based on the transformed data in the data staging area] SQL Reporting Services [Used to create the front end reports for the business users] Internet Information Services 6.0 [IIS 6.0] - to host the front end reports and make them available to the end users through the web. Physical Design All the SQL code to create fact tables and their corresponding dimension is created in this phase. The physical design of a fact table clearly shows the filed names, data types, primary keys, foreign keys, source table information and a description of how the values are derived. Eg: Subscription Sales Fact table physical design. 15
  16. 16. Column Names Data N Type U L L Ke ? y? FK TO Dimension Description 16
  17. 17. EffectiveDateKe y int N FK DimDate Key of effective date EnteredDateKey Key of date entered in the int N FK DimDate system CustomerKey int N FK DimCustomer Key of customer LoyaltyKey int N FK DimLoyalty Key of loyalty score PaymentKey int N FK DimPaymentHistory Key of payment behavior SalesPersonKey Key of sales person for the int N FK DimSalesPerson change CampaignKey int N FK DimCampaign Key of campaign SalesConditions Key int N FK DimSalesConditions Key of sales conditions SubscriptionKey int N FK DimSubscription Key of subscription PersonKey Key of person on the int N FK dimPerson subscription UnbrokenServic Key of length of unbroken eKey int N FK DimUnbrokenService service DurationKey int N FK DimDuration Key of duration length DemographicsK ey int N FK DimDemographics Key of demographics RouteKey Key of route for this int N FK DimRoute address AddressKey int N FK DimAddress Key of address AuditKey int N FK DimAudit Key of audit records AddressNumber varchar( address number in CJ 10) N system SubscriptionNu varchar( subscriber number in CJ mber 2) N system PBMAccount varchar( pbm account number in CJ 10) Y system NumberOfSales int Y count of sales (always 1) UnitsSold number of units sold by this int Y transaction DollarsSold dollars resulting from this int Y sale DiscountCost cost of the discount over int Y the life of this sale PremiumCost cost of any premiums given int Y with this sale In addition to the field names, data types, primary keys, description the dimension table physical design specifies the slowly changing dimension type for the fields of the dimension. Eg: Customer Subscription dimension table physical design. Column Names Data N Slowly Type U changin L g L FK TO dimensi ? Key? Dimension Description on type 17
  18. 18. CustomerSubscriptio PK nKey int N ID Surrogate Primary Key Business key of the subscription AddressNum int Y BK summary record address Business key of the numbered SubscriptionNum int Y BK ` subscription at the address BusinessKey int Y BK Concatenated business key CustomerID int N Unique identifier for this customer 1 The method of delivery for the BillingMethod int customers bill 2 OriginalStartDateKe The earliest start date on record for this y int FK DimDate customer 1 The start date of this customer’s current StartDateKey int FK DimDate subscription 2 The most recent stop date for this StopDateKey int FK DimDate customer 2 The expiration date of this customer’s ExpireDateKey int FK DimDate current or most recent subscription 2 Exact status of this customer’s SubscriptionStatus int subscription 2 ActiveSubscription int Indicates Active or Inactive 2 StopReason int Reason for most recent stop 2 StopType int Type of stop reason 2 A formula-generated score representing CurrentLoyaltyScore int the subscriber’s current loyalty behavior 1 A formula-generated score representing CurrentPaymentBeh the subscriber’s current payment aviorScore int behavior 1 varc Key of the person record in the person har( dimension currently assigned to this CurrentPersonKey 10) FK DimPerson customer 1 Key of the household record that is varc currently assigned to this customer (for CurrentHouseholdKe har( DimHousehol finding other people in the customer y 2) FK d household) 1 varc Key of the address record in the har( address dimension currently assigned CurrentAddressKey 10) FK DimAddress to this customer 1 BusinessOrConsum Indicator about what kind of customer erIndicator int this is 1 AuditKey int N FK DimAudit What process loaded this row? The Slowly changing dimension type is defined as follows. Type 1: The new record replaces the original record. No trace of the old record exists. Type 2: A new record is added into the dimension table. Please refer the Appendix for more details. Once the physical designs were laid out, Kimball’s Data Modeling tool which was used to generate the SQL code and corresponding metadata. 18
  19. 19. Data Staging & Development This phase includes the following: a.] Implement the ETL [Extraction-Transform-Load] process. b.] Build the data cube using SQL Analysis Services. a.] Implement the ETL [Extraction-Transform-Load] process. ETL Architecture Kimball Data Source Schema Source Queries Modeling tool Creates the staging Creates the tables for Creates views and production the common source (queries) that feed database data to production database tables and metadata. Production Data Common Source Warehouse Database Staging Database Database Setup files on SQL server that is running the DTS packages. DTS Database Data Dates Configuration File Configuration Transformation File Services Package Database connection Important date info information for the for the ETL process ETL process 19
  20. 20. The source schema document, which has the SQL code to create source tables, is used to create the source tables in the source database and the Kimball Data warehouse tool, which creates the SQL code based on the physical design of the dimension and fact tables, is used to create the dimension and fact tables in the staging and production databases .Once the required tables are created in SQL database, SQL views are created in the Source database. These views are used in the DTS {Data Transformation Process} to feed the production database dimension and fact tables. The tables in the staging database allow the source data to be loaded and transformed without impacting the performance of the source system. The staging tables provide a mechanism for data validation and surrogate key generation before loading transformed data into the data warehouse. Dates.ini: Configuration file for the start and stop dates of a run, used by the DTS packages ETLConfig.ini: Configuration files for server names (source, staging, and target) and package locations, used by the DTS packages In short, the ETL (Extraction, Transformation, and Loading) process involves extracting the data from the source systems into a source DB; transform the data into the form the end users want to look at and then load the data into the production server .The final data in the production server will be used to create a data cube using the SQL Analysis Services. About 10-15 dimension packages and 5-6 fact packages have been developed as part of the ET L process using SQL’s Data Transformation Services [DTS]. Details: Extracting Dimension Data Whenever we extract dimension data, we always strive to minimize the data that we need to process. In other words, we strive to extract only new or changed dimension members. This is often not possible to do, as source systems are notoriously unreliable in flagging new, changed, or deleted rows. Thus, it’s common to have to extract today’s full image of dimension members, compare to the image current in the data warehouse dimension table, and infer changes. Sales Data Warehouse dimension template assumes that this comparison process is necessary. It’s designed to pull a full image of the dimensions and find the inserts, Type 1 updates, and Type 2 updates. The dimension packages perform the following work. 1. Extract the full sets of dimension rows 2. Most transformation logic occurs in the extract query, using SQL 3. Extracted rows are stored in a staging table until the package is re-run 4. There are steps for the staged rows to be fixed up, via SQL statements. There’s a statement for deleting bogus rows, and a separate statement for updating rows. 5. Find rows that are new; insert them into the target table. 6. Use a checksum to find rows that have seen a Type 1 change. Update the appropriate columns in the target table. 20
  21. 21. 7. Use a checksum to find rows that have seen a Type 2 change. Propagate a new dimension row. 8. Log the number of rows extracted, staged, deleted and updated from the staging tables, inserted into target, Type 1 and Type 2 rows updated in target. Details: Extracting Fact Data As with dimensions, we always strive to extract only new (or changed) facts. There is a stronger imperative with fact table extracts, as we typically see much greater data volumes on fact table processing. ETL Code Sample Managing a Dimension Table [ Eg: ROUTE dimension ] Route_Master - This table is used to store data from the source system. S_DimRoute – Stores the Route dimension data while the dimension is updated and later loaded into the route dimension. DimRoute- Stores the Route dimension data. Define Route Tables The following three Transact-SQL statements create the Route_Master,s_DimRoute and DimRoute tables which simulate the Route dimension table in the Sales data warehouse: /* Source table */ Create table Route_Master ( route_id varchar(6), rack char(2), route_type char(2), description varchar(30), route_status char(2), carrier_code varchar(8), city char(4), state char(2), zone char(2), district char(4), county_code char(4), zip varchar(10), abc_zone char(2), abc_class char(2), census_tract varchar(8), alert_count int, alert_message varchar(20), 21
  22. 22. route_switches varchar(26), marketing_zone1 char(4), marketing_zone2 char(4), marketing_zone3 char(4), marketing_zone4 char(4), marketing_zone5 char(4), marketing_zone6 char(4), marketing_zone7 char(4), marketing_zone8 char(4), marketing_zone9 char(4), marketing_zone10 char(4), smsa_code char(4), sub_carrier_1 varchar(8), sub_carrier_2 varchar(8), sub_carrier_3 varchar(8), sub_carrier_4 varchar(8), sub_carrier_5 varchar(8), household_count int, subscriber_count int, address_1 varchar(30), circ_division char(2), rate_code char(2), filler varchar(20) ) /* Staging table */ create table [s_dimroute] ( [bkrouteid] varchar(8) null, [abczone] varchar(25) null, [abcclass] varchar(25) null, [abccounty] varchar(25) null, [abczipcode] varchar(10) null, [routetype] varchar(25) null, [stategroup] varchar(25) null, [state] char(2) null, [town] varchar(45) null, [district] varchar(25) null, [zone] varchar(25) null, [metroregion] varchar(45) null, [truck] varchar(25) null, [microzone] varchar(45) null, [edition] varchar(25) null, 22
  23. 23. [carrier] varchar(45) null, [carriertype] varchar(25) null, [deliverytype] varchar(25) null, [mileage] varchar(25) null, [mileagebands] varchar(45) null, [deliverytime] varchar(25) null, [deliverytimebands] varchar(25) null, [penetrationlevel] varchar(25) null, [status] varchar(25) null, [carrierordinal] varchar(25) null, [auditkey] int null, [checksumtype1] as checksum(1), [checksumtype2] as checksum([abczone],[abcclass],[abccounty],[abczipcode],[routetype],[stategroup],[state], [town],[district],[zone],[metroregion ],[truck],[microzone],[edition],[carrier],[carriertype],[deliverytype],[mileage], [mileagebands],[deliverytime],[deliverytimeb ands],[penetrationlevel],[status],[carrierordinal]), [rowstartdate] datetime null, [rowstopdate] datetime null, [rowcurrentind] varchar(1) null ) on [primary] /* Dimension table - DimRoute */ create table [dimroute] ( [routekey] int identity not null, [bkrouteid] varchar(8) null, [abczone] varchar(25) null, [abcclass] varchar(25) null, [abccounty] varchar(25) null, [abczipcode] varchar(10) null, [routetype] varchar(25) null, [stategroup] varchar(25) null, [state] char(2) null, [town] varchar(45) null, [district] varchar(25) null, [zone] varchar(25) null, [metroregion] varchar(45) null, [truck] varchar(25) null, [microzone] varchar(45) null, [edition] varchar(25) null, 23
  24. 24. [carrier] varchar(45) null, [carriertype] varchar(25) null, [deliverytype] varchar(25) null, [mileage] varchar(25) null, [mileagebands] varchar(45) null, [deliverytime] varchar(25) null, [deliverytimebands] varchar(25) null, [penetrationlevel] varchar(25) null, [status] varchar(25) null, [carrierordinal] varchar(25) null, [auditkey] int not null, [checksumtype1] as checksum(1), [checksumtype2] as checksum([abczone],[abcclass],[abccounty],[abczipcode],[routetype],[stategroup],[state], [town],[district],[zone],[metroregion ],[truck],[microzone],[edition],[carrier],[carriertype],[deliverytype],[mileage], [mileagebands],[deliverytime],[deliverytimeb ands],[penetrationlevel],[status],[carrierordinal]), [rowstartdate] datetime null, [rowstopdate] datetime null, [rowcurrentind] varchar(1) null, constraint [pk_dimroute] primary key clustered ( [routekey] ) ) on [primary] Populate Route Tables The following three Transact-SQL statements populate the temporary, staging, and data warehouse sample tables by loading the route table records. /* populate the SQL source table route_master with data from the actual marketing source table ccisrout */ Insert into route_master Select * from ccisrout /* Populate the staging table S_DimRoute using the DimRoute_Source VIEW */ Insert into S_DimRoute select * , 8 as auditkey , convert(datetime,'1/1/2004') as rowstartdate 24
  25. 25. , convert(datetime,'12/31/2079') as rowstopdate ,'Y' as rowcurrentind From DimRoute_source . The view DimRoute_Source is defined as follows. Create view [dbo].[DimRoute_source] as select r.route_id+coalesce(r.rack,'') as bkrouteid, coalesce(abcz.description,'None') as abczone, coalesce(abcc.long_desc,'None') as abcclass, coalesce(cnty.description,'None') as abccounty, as zipcode, coalesce(rtyp.long_desc,'Type '+r.route_type,'None') as routetype, coalesce(ks.keystate,'Other') as stategroup, city.state_code as state, coalesce(city.description,'None') as town, coalesce(case when like '% '+rtrim(dist.district)+' %' then else rtrim(dist.district)+' ' end,'None') as district, coalesce(zone.description,'None') as zone, 'Undefined' as metroregion, coalesce(rsum.truck_run1,'None') as truck, 'Undefined' as microzone, coalesce(rsum.edition1,'None') as edition, coalesce(,'None') as carrier, coalesce(carr.carrier_type,'None') as carriertype, 'Undefined' as deliverytype, 'Undefined' as mileage, 'Undefined' as mileagebands, 'Undefined' as deliverytime, 'Undefined' as deliverytimebands, 'Undefined' as penetrationlevel, r.route_status as status, coalesce(right(r.carrier_code,2),'None') as carrierordinal from route_master r left outer join zone zone on = left outer join abc_class abcc on r.abc_class = abcc.type left outer join county cnty on r.county_code = cnty.code left outer join city city on = city.code left outer join district dist 25
  26. 26. on r.district = dist.district left outer join route_service_summary rsum on r.route_id = rsum.route_id and (r.rack = rsum.rack or r.rack is null and rsum.rack is null) and ('NFK' != 'RKE' or rsum.publication_code = 'AM') left outer join carrier_master carr on r.carrier_code = carr.carrier_code left outer join abc_class rtyp on r.route_type = rtyp.type left outer join abczone abcz on r.abc_zone = abcz.abc_zone and abcz.location = 'NFK' left outer join keystates ks on city.state_code = ks.state and ks.location = 'NFK' /* Populate the dimension table DimRoute from the staging table S_DimRoute */ Insert into DimRoute Select * from S_DimRoute Inserting New Dimension Records /* truncate the SQL source table and load the data from the marketing source table */ Truncate table route_master GO Insert into route_master Select * from ccisrout GO /* truncate the staging table and load the new data from the source using the DimRoute_source view */ Truncate table S_DimRoute GO Insert into S_DimRoute select * , 8 as auditkey , convert(datetime,'1/1/2004') as rowstartdate , convert(datetime,'12/31/2079') as rowstopdate ,'Y' as rowcurrentind From DimRoute_source . 26
  27. 27. /* Compare the staging table with the dimension table for any new records */ select L.* ,convert(datetime, '2004-01-01') as RowStartDate , convert(datetime, '2079-12-31') as RowStopDate , 'Y' as RowCurrentInd from S_DimRoute L left outer join DimRoute R on ( R.RowCurrentInd='Y' and L.BKRouteId = R.BKRouteId) where R.RouteKey IS NULL /* Managing Slowly Changing Dimensions */ The Sales data warehouse project implements the Type-1 & Type-2 slowly changing dimension types. Type-1 : Overwrite the dimension record /* Update the staging table S_DimRoute to simulate Type-1 changes .*/ update s_dimroute set truck='599' where truck='455A' and abcclass = 'CARRIER' /* Update the dimension table with type-1 changes */ update dimroute set truck = (select truck from staging..s_dimroute s where s.bkrouteid = dimroute.bkrouteid) where bkrouteid in (select L.bkrouteid from staging..s_dimRoute L left outer join poc..dimRoute R on (R.rowcurrentind = 'Y' and L.BKRouteId = R.BKRouteId) where L.[ChecksumType-1]<> R.[ChecksumType-1]) Type-2: Add a new dimension record. /* Update the staging table S_DimRoute to simulate Type-2 changes .*/ update s_dimroute set carrier = 'Tom Justice' where carrier = 'gerald harvey' /* Update the dimension table with Type-2 changes */ 27
  28. 28. Update dimroute SET rowstopdate = Convert(datetime, Convert (int, getdate() - 1)) , rowcurrentind = 'N' where bkrouteid IN (select L.bkrouteid from staging..s_dimRoute L left outer join poc..dimRoute R on (R.rowcurrentind = 'Y' and L.BKRouteId = R.BKRouteId) where L.[ChecksumType-2] <> R.[ChecksumType-2]) and rowcurrentind = 'Y' MANAGING A FACT TABLE [ Eg: FactSubscriptionSales ] S_FactSubscriptionSales - Staging table FactSubscriptionSales - Production Fact table Define the Staging & Production Fact tables CREATE TABLE [s_FactSubscriptionSales] ( [EffectiveDateKey] varchar(128) NULL, [EnteredDateKey] varchar(128) NULL, [PostedDateKey] varchar(128) NULL, [CustomerKey] varchar(128) NULL, [LoyaltyKey] varchar(128) NULL, [PaymentKey] varchar(128) NULL, [SalesPersonKey] varchar(128) NULL, [CampaignKey] varchar(128) NULL, [SalesConditionsKey] varchar(128) NULL, [SubscriptionKey] varchar(128) NULL, [PersonKey] varchar(128) NULL, [UnbrokenServiceKey] varchar(128) NULL, [DurationKey] varchar(128) NULL, [DemographicsKey] varchar(128) NULL, [RouteKey] varchar(128) NULL, [AddressKey] varchar(128) NULL, [AuditKey] varchar(128) NULL, [AddressNumber] varchar(10) NULL, [SubscriptionNumber] varchar(2) NULL, [PBMAccount] varchar(10) NULL, [NumberOfSales] int NULL, [UnitsSold] int NULL, [DollarsSold] money NULL, [DiscountCost] money NULL, [PremiumCost] money NULL ) ON [PRIMARY] GO /* Create table FactSubscriptionSales */ 28
  29. 29. CREATE TABLE [FactSubscriptionSales] ( [EffectiveDateKey] int NOT NULL, [EnteredDateKey] int NOT NULL, [PostedDateKey] int NOT NULL, [CustomerKey] int NOT NULL, [LoyaltyKey] int NOT NULL, [PaymentKey] int NOT NULL, [SalesPersonKey] int NOT NULL, [CampaignKey] int NOT NULL, [SalesConditionsKey] int NOT NULL, [SubscriptionKey] int NOT NULL, [PersonKey] int NOT NULL, [UnbrokenServiceKey] int NOT NULL, [DurationKey] int NOT NULL, [DemographicsKey] int NOT NULL, [RouteKey] int NOT NULL, [AddressKey] int NOT NULL, [AuditKey] int NOT NULL, [AddressNumber] varchar(10) NOT NULL, [SubscriptionNumber] varchar(2) NOT NULL, [PBMAccount] varchar(10) NULL, [NumberOfSales] int NULL, [UnitsSold] int NULL, [DollarsSold] money NULL, [DiscountCost] money NULL, [PremiumCost] money NULL ) ON [PRIMARY] GO Populate the Fact staging and Production tables /* populate the staging fact table using the FactSubscriptionSales_Source view */ Insert into S_FactSubscriptionSales Select *, 9 as auditkey from FactSubscriptionSales_Source. The FactSubscriptionSales_Source view is defines as follows: create view FactSubscriptionSales_source as select hist.effective_date as effective_date, hist.date_entered as date_entered, hist.posted_date as posted_date, hist.address_num + hist.subscriber_num + summ.date_started as customer, 0 as loyalty, 0 as payment, coalesce(hist.source_code,'') + coalesce(hist.solicitor_contest,'') as salesperson, coalesce(hist.campaign_code,'None') as campaign, 29
  30. 30. coalesce(hist.solicitor_contest, space(8)) + coalesce(hist.campaign_code, space(8)) as salesconditions, hist.publication_code + hist.service_code + coalesce(trs.rate+trs.term,pbm.original_rate_code + pbm.billing_period, summ.payment_type + '***') as subscription, coalesce(np.address_num, np2.address_num, space(10)) + convert(char(10), coalesce(np.subs_index, space(10))) + left(coalesce(space(10) + np.address_num, space(10) + np2.address_num, summ.pbm_account_key + space(10)),10) as person, pbm.billing_period + '~' + convert(varchar(10),case when summ.restart_date is null then datediff(dd,summ.date_started,hist.effective_date) when summ.restart_date > summ.date_started then datediff(dd,summ.restart_date,hist.effective_date) else datediff(dd,summ.date_started,hist.effective_date) end) as unbroken, pbm.billing_period + '~' + convert(varchar(10), datediff(dd, summ.date_started, hist.effective_date)) as duration, hist.address_num as demographics, case when hist.daily_route_id is null then hist.sunday_route_id+coalesce(hist.sunday_rack,'') else hist.daily_route_id + coalesce(hist.daily_rack,'') end as route_code, summ.address_num as address, summ.address_num as addressnumber, summ.subscriber_num as subscriptionnumber, summ.pbm_account_key as pbmaccount, 1 as sales, s.weeks * len(replace(days,' ','')) as unitssold, s.weekly_rate * convert(int,s.weeks) as dollarssold, s.weekly_discount * convert(int,s.weeks) as discountcost, convert(money,case right(coalesce(rtrim(hist.solicitor_contest),'B'),1) when 'B' then 0.00 else 5.00 end) as premiumcost from service_history hist join (select coalesce(max(fulldate),'1/1/1980') as maxdate from [POC]. [dbo].factsubscriptionsales f join [POC].[dbo].dimdate d on f.posteddatekey = d.datekey) xd on hist.posted_date > xd.maxdate join subscription_summary summ on hist.address_num = summ.address_num and hist.subscriber_num = summ.subscriber_num and hist.publication_code = summ.publication_code and hist.service_trans_code = 'S' and summ.date_started = hist.effective_date left outer join termratespan trs on hist.pbm_account_key = trs.pbm and hist.effective_date between trs.start and trs.stop left outer join pbm_account_master pbm on trs.pbm is null and hist.pbm_account_key = pbm.account_key left outer join all_subscriptions s on hist.publication_code = s.publ_cd and hist.service_code = s.service_code and isnull(trs.rate,pbm.original_rate_code) = s.rate_code and isnull(trs.term,pbm.billing_period) = s.period_cd 30
  31. 31. left outer join name_phone np on hist.address_num = np.address_num and summ.subs_index = np.subs_index left outer join name_phone np2 on np.address_num is null and np2.subs_index is null and summ.address_num = np2.address_num go /* populate the fact table FactSubscriptionSales */ Insert into FactSubscriptionSales Select * from FactSubscriptionSales GO Inserting New Records /* Insert into staging table */ Truncate table S_ FactSubscriptionSales Go Insert into S_ FactSubscriptionSales Select *, 9 as Auditkey from FactSubscriptionSales_Source /* Insert new rows into Production Table */ select coalesce(effectivedate.datekey,-1) as effectivedatekey ,coalesce(entereddate.datekey,-1) as entereddatekey ,coalesce(posteddate.datekey,-1) as posteddatekey ,coalesce(customer.customersubscriptionkey,-1) as customerkey ,coalesce(loyalty.loyaltykey,-1) as loyaltykey ,coalesce(payment.paymentbehaviorkey,-1) as paymentkey ,coalesce(salesperson.salespersonkey,-1) as salespersonkey ,coalesce(campaign.campaignkey,-1) as campaignkey ,coalesce(salesconditions.salesconditionskey,-1) as salesconditionskey ,coalesce(subscription.subscriptionskey,-1) as subscriptionkey ,coalesce(person.personkey,-1) as personkey ,coalesce(unbroken.durationkey,-1) as unbrokenservicekey ,coalesce(duration.durationkey,-1) as durationkey ,coalesce(demographics.demographicskey,-1) as demographicskey ,coalesce(route.routekey,-1) as routekey ,coalesce(address.addresskey,-1) as addresskey ,9 as auditkey ,addressnumber ,subscriptionnumber 31
  32. 32. ,pbmaccount ,numberofsales ,unitssold ,dollarssold ,discountcost ,premiumcost from s_FactSubscriptionSales fact left outer join poc..dimdate effectivedate on fact.effectivedatekey = effectivedate.fulldate left outer join poc..dimdate entereddate on fact.entereddatekey = entereddate.fulldate left outer join [poc].[dbo].dimdate posteddate on fact.posteddatekey = posteddate.fulldate left outer join poc..dimcustomersubscription customer on fact.customerkey = customer.businesskey and customer.rowcurrentind = 'Y' left outer join poc..dimloyalty loyalty on fact.loyaltykey = loyalty.loyaltyscore left outer join poc..dimpaymentbehavior payment on fact.paymentkey = payment.paymentbehaviorscore left outer join poc..dimsalesperson salesperson on fact.salespersonkey = salesperson.solicitorcode and salesperson.rowcurrentind = 'Y' left outer join poc..dimcampaign campaign on fact.campaignkey = campaign.campaigncode and campaign.rowcurrentind = 'Y' left outer join poc..dimsalesconditions salesconditions on fact.salesconditionskey = salesconditions.businesskey left outer join poc..dimsubscriptions subscription on fact.subscriptionkey = subscription.subscriptionname left outer join poc..dimperson person on fact.personkey = person.sourcekey and person.rowcurrentind = 'Y' left outer join poc..dimduration unbroken on fact.unbrokenservicekey = unbroken.termcodedays left outer join poc..dimduration duration on fact.durationkey = duration.termcodedays left outer join poc..dimAddress address on fact.addresskey = address.addressnum left outer join poc..dimdemographics demographics on address.addresskey = demographics.addresskey and demographics.rowcurrentind = 'Y' left outer join poc..dimRoute route on fact.routekey = route.bkrouteid and route.rowcurrentind = 'Y' where entereddate.fulldate > '10/31/2004' b.] Build the data cube using SQL Analysis Services. Based on the business user requirements the Sales data cube was built using SQL Server 2005 Analysis services. The following steps were followed to create a cube using SQL Analysis services and deploy the cube to the database. The Analysis Services project wizard was used to create the data source views and then to create the cube. • Create an analysis Services project in SQL Business Intelligence Studio 32
  33. 33. • Add the data source to the project [ Add the production database to the data source folder ] • Create data source view [ SubscriptionSales.dsv] • Create a cube [ be sure to include the above data source and the data source view ] • Deploy the cube to the SQL SERVER Analysis database. At this point, the cube is available for reporting purposes .Any changes to the cube will reflect in the database only after re-deploying the cube. All user defined named calculations, setting up of KPI [Key performance Indicator] are done in the cube section of the project .The cube has to be re-deployed for the changes to come into effect. These named calculations and KPI’s are then used in the front end reports. Eg: Named calculation: CalendarYear + ‘Qtr ‘+ Convert (Char (1), Calendar Quarter) As Quarter. The above expression will be evaluated when the cube is processed and will show up in the dimension as “Quarter’. Eg: KPI [Key performance Indicator] In SQL Server 2005 Analysis Services, a key performance indicator is a collection of calculations that are associated with a measure group in a cube that are used to evaluate business success. Typically, these calculations are a combination of Multidimensional Expressions (MDX) expressions or calculated members. KPIs also have additional metadata that provides information about how client applications should display the results of the KPI's calculations. Consider the following MDX expression. Cost Per Unit = ([Measures].[DiscountCost]+[Measures].[PremiumCost])/[Measures]. [Units Sold] DiscountCost, PremiumCost and UnitsSold are the facts or measures in the SubscriptioSales fact table .If the KPI goal for the above expression is set to .30 it means that the target for the above calculation is 0.3 . If the calculated value is less than 0.3 then the above calculated field ‘Cost per Unit’ will be shown with a status indicator downward arrow in the front end reports wherever the calculated field is used. End User Application Specification The following set of reports was specified by the users Retention and Acquisition departments of Marketing. Acquisition Reports Acquisition_Campaign SalesByClusterByChannelByTermByServiceByBillingMethod 33
  34. 34. Acquisition_CreditCardSalesByPrizmCluster Acquisition_EZPay by Lifestage Group Acquisition_EZPay Daily Service Addresses With Clusters Acquisition_EZPaySalesByPrizmCluster Acquisition_SalesByPrizmClusterByBillingMethod Acquisition_SalesByPrizmClusterBySalesChannel Acquisition_SalesByPrizmClusterByServiceByTerm Acquisition_SalesByZipCodeAndBillingMethod Retention Reports Retention_ActiveSubscriptionsWith2PlusComplaintsIn2005ByDistrict Retention_Can't Afford Stops July 04 to July 05 Retention_No Time To Read Stops by Lifestage Group Retention_No Time to Read Stops by Prizm Cluster Retention_No Time To Read Stops By Rate Retention_Number of Complaints Per HH on Stopped Subscriptions Retention_NumberofStopsByLastStopReason Retention_Stopped Metro Subscriptions With Missed Paper Complaint By Route Retention_Stopped Subscriptions By Cluster By Stop Reason Retention_Stopped Subscriptions by Complaint Type for Metro Area Retention_Subscriptions with 6 or More Complaints that expired 45 to 70 Days Ago Retention_TelemarketingStartsProfileByTermByRate Retention_TelemarketingStopsProfileByTermByRate General Reports Campaign Sales Campaign Sales by Service Channel Retention By Prepaid or Billed Channel Starts By Prepaid or Billed District Complaints drillthrough-ByNoPayments drillthrough-Sales-BySolicitor drillthrough-Stops-ByType Monthly Net Starts To Hard Stops Sales Channel Cost Per Unit Trend Sales Name and Address Drillthrough-test Starts-Stops Stops By Payment History End User Application Development SQL Reporting services was used to create the user specified front end reports .The following steps are involved in creating a reporting services project: • Open the Business Intelligence studio 34
  35. 35. • Create a new Reporting services project. • Add the cube [ developed in analysis services phase ] as a data source to the project • Create a report using the report wizard. The wizard provides all the dimensions corresponding to a fact table and the measures in the fact table to the user to drag them to the report. • Deploy the project to the reporting Services Database. The reports were made available to the Business users through the web after proper settings were done in the IIS. Eg: The following screenshots depict a main report [Solicitor Sales] and a drill through report for the parameter ‘Number Of sales =107’. Solicitor Sales Report grouped by Sales Type & Sales Channel This report gives summary details of solicited sales of the newspaper based on the Sales Type and Sales Channel .It also breaks down into Sales Agents who belong to a Sales Channel, the Number of Sales, Cost per unit, and Retention percentage in a Sales Channel. Cost per unit and Retention are calculated as follows Cost Per Unit = ([Measures].[DiscountCost]+[Measures].[PremiumCost])/[Measures]. [Units Sold] Retention = ([CustomerSubscription].[ActiveSubscription].&[Active],[Measures].[Numberof Sales])/ ([Customer Subscription].[Active Subscription].[All],[Measures].[Numberof Sales]) NumberOfSales gives the actual number of subscriptions sold via a Sales Channel. 35
  36. 36. Drill through report into Number of Sales per city & Zip The drill through report gives a breakdown of the Number of Sales per city and zip .It gives information about the Address to which the subscription was sold, the state,city and zip code information of the subscriber . The report also gives information about the number of units sold , dollars sold ,discount cost and premium cost . Units Sold : Each subscription sold x term { per week } [ weekend[3],daily[7],Sunday only[1] etc..] DiscountCost: the discount given to a subscription Dollars Sold: Revenue to the company as a result of selling subscriptions PremiumCost: Gift Certificates etc given to the subscribers for subscribing to the newspaper. 36
  37. 37. Deployment This phase of the Sales data warehouse project life cycle deals with the following issues. Deploying the Reports The reports developed using the SQL Reporting Services were deployed using the Reporting Services deployment wizard . The reports were deployed to an IIS webserver and were made available to the End users through the web.Security was also provided to the reports and only certain people were given access to view the reports.The reports can be viewed using an IE browser .[ version 6.0 or greater] Desktop Installation 37
  38. 38. Dot NET framework 2.0 was installed on the client machines to give access to the End Users to create their own reports through Report Builder feature of SQL Reporting Services based on the Sales cube data model. The users can create their own reports from the web and save them to the server. Maintenance and Growth This last phase of the Data Warehouse lifecycle deals wit the following issues Provide training to the business users The End Users were trained to view the Reports from the web and as well as to create their own reports using the Report Builder component of SQL Reporting Services 2005. NIGHTLY UPDATE AUTOMATION The nightly data update process has been automated using SSIS [SQL Server Integration Services]. The SSIS package shown below extracts the data from the marketing database source system into the SQL Source database on a nightly basis. The SSIS package then calls a DTS [ Data Transformation Services ] package which performs the job of Extract ,Transform and Load .Any new rows[type-2 changes] or updated rows [type-1 changes] will be tracked in the staging database and are updated within the production database . Get List Of Mappings: Gets a list of all the source files to be transferred from the source system and their corresponding map files to map to SQL tables. Transfer Al Files: {for loop container in SSIS} Transfers all the listed source files from the above step from the source system to the Data Warehouse server. Truncate Tables: Truncates all SQL source tables Process All File Mappings: Processes all source files and their corresponding map files and loads the source data to SQL source database. Get max Posted Date: Gets the max date from the data in the above step Write dates.ini file: Writes the date to a configuration file. Read package file: Reads the location of the DTS package on the data warehouse server. Old style DTS run call to 2000 package: Calls the DTS package to perform the ETL {Extract-Transform and Load} functionality. Fig : Depicts the high level automation process flow from the source systems to the source database and a call to DTS package to perform the ETL functionality 38
  39. 39. Once the data is loaded into the dimension and fact tables, another SSIS package calls the Analysis services project which processes the data warehouse cube and redeploys it to the Analysis Services Database. The reports developed using Reporting Services pick up the updated information from the data warehouse cube when they are called from the web. The following are screenshots of the SSIS package. 39
  40. 40. Fig 11.3: Depicts the automation of processing a data cube on a nightly basis .First, the dimension tables are processed followed by the fact tables and then deploy the cube to the SQL Analysis services database. CONCLUSION On the whole, five fact tables [Subscription Sales, Stops/Starts, Upgrades/Downgrades, Complaints, Payment History] have been developed and incorporated into the data cube for the Sales Data Warehouse project. 40
  41. 41. The reports generated from the data warehouse answered the following questions collected form the business users during the requirement gathering phase of the project. • Identify their best customers/loyal customers [ customer subscriptions /subscription sales] Best customers are those subscribers who subscribe to the longest service term [ > >= 26 week , 52 weeks ] or use the EZ pay feature where the subscriber registers his credit card information with the company and the subscription amount gets deducted from their account on a monthly basis without the mail costs involved . On the other hand ,loyal customers are those who never cancel their subscriptions and had been subscribing to the product for a very long period of time .For the Sales Data Warehouse project any subscriber who’s been subscribing to the product for a period of 2 yrs or more was considered a loyal customer. • Non-subscribers who can be reached The demographics information in the Data cube provided the business users with a list of all the people in a marketing area along with their addresses, phone numbers, age group information, and household information, education levels etc.This information can be used to find out the preferences of the non-subscribers and reach them through special campaigns. • Contact history of customers in market place [ Demographic data] The contact information of the current customers as well as the non-subscribers was provided to the business users through the Sales data warehouse project. Based on this information special discounts or gift certificates can be provided to the customers and make them upgrade their current subscription. [Upgrade from 13 week to 26 week or from Sunday only to a full week subscription.] Apart from answering the above questions, the data warehouse also provides general information to the business users on a daily basis without much calculation required . Eg: Number of Sales in a particular City, Revenue obtained as a result of a new campaign, Cost per unit value for a subscription, what campaigns are doing well and how many people responded to a campaign etc… The Report Builder component of the SQL Reporting Services provides the user an opportunity to create his/her own reports by simply dragging the required fields from the data cube onto the report screen. This custom report creation facility was highly appreciated by the end users. As a result of the Sales Data Warehouse, the following benefits have been achieved by the Marketing and Advertising departments of the company. 41
  42. 42. Benefits to Marketing • Increased telemarketing close rates and increased direct mail response rates Targeting the best customers , non-subscribers who resemble the profile of best customers , launching targeted campaigns can greatly increase the direct mail response rates from the customers and the telemarketing close rates.The Sales data warehouse provides all the above required information to the business . • Reduced cost and use of outside telemarketing services and reduced print and mailing costs Based on the information from the data warehouse, certain Routes in a city were identified for mailing out new campaigns to the customers in that Route thereby reducing the outside telemarketing services, print and mailing costs. • Identification of new product bundling and distribution opportunities The information about the current subscribers and the non-subscribers in a particular region helps to reconfigure the routes in that region creating new distribution points and minimize the costs of distribution and product bundling. • Increased acquisition and retention rates, and reduced cost of acquisitions There was certain cost involved in Acquiring new customers before the Data Warehouse was in place. The salespersons were paid to acquire a new customer and certain mailing costs were involved in promoting a particular campaign. The Sales data warehouse 1. will considerably bring down the bonuses paid to a salesperson , 2. bring down the cost involved in new acquisitions, 3. increase the retention rates by targeting a selective group of people in the marketplace and promoting new campaigns with good discounts and premiums {gift certificates} to those selective people . Benefits to Advertising • An increase in the annual rate of revenue growth. Promoting new campaigns targeting non-subscribers, increasing retention of the existing subscribers, upgrading current service terms [eg: from 13 week to 52 week} etc… will certainly provide a good yearly income to the company. • Increase in new advertisers New Campaigns with good discounts will attract new advertisers and additional revenue to the company. • Improved targeting capabilities The demographic information of non-subscribers coupled with the existing subscription information of the subscribers will improve the targeting capabilities of new campaigns thereby increasing the acquisition rate followed by the increased revenue growth. 42
  43. 43. REFERENCES The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, conforming and delivering data By Ralph Kimball & Joe Caserta. The Data Warehouse Lifecycle Toolkit Tools and Techniques for Designing, Developing, and Deploying Data Warehouses By Ralph Kimball, Laura Reeves, Margy Ross, Warren Thornthwaite Microsoft® SQL Server™ 2000 DTS Step by Step By Carl Rabeler Microsoft E-Learning - SQL Analysis Services 2005 [] Microsoft E-Learning - SQL Reporting Services 2005 [] Microsoft SQL Server 2000 resource kit APPENDIX: The fundamental idea of dimensional modeling is that nearly every type of business data can be represented as a kind of cube of data, where the cells of the cube contain measured values and the edges of the cube define the natural dimensions of the data. Terms commonly used in Dimension modeling: Dimension: A category of information. Attribute: A unique level within a dimension. For example, Month is an attribute in the Time Dimension. Hierarchy: The specification of levels that represents relationship between different attributes within a hierarchy. For example, one possible hierarchy in the Time dimension is Year --> Quarter --> Month --> Day. Slowly Changing Dimension: The "Slowly Changing Dimension" problem applies to cases where the attribute for a record varies over time. Consider the example below. 43
  44. 44. Christina is a customer with ABC Inc. She first lived in Chicago, Illinois. So, the original entry in the customer lookup table has the following record: Customer Key Name State 1001 Christina Illinois At a later date, she moved to Los Angeles, California on January, 2003. How should ABC Inc. now modify its customer table to reflect this change? This is the "Slowly Changing Dimension" problem. There are in general three ways to solve this type of problem, and they are categorized as follows: Type 1: The new record replaces the original record. No trace of the old record exists. Type 2: A new record is added into the customer dimension table. Therefore, the customer is treated essentially as two people. Type 3: The original record is modified to reflect the change. In the marketing data warehouse project, type-1 & type-2 SCD types have been implemented. 44