Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spreadmart To Data Mart BISIG Presentation


Published on

Presentation at the North Central BI Special Interest Group (BISIG) going over a case study of converting an Excel Spreadmart solution to a SSAS data mart solution

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Spreadmart To Data Mart BISIG Presentation

  1. 1. Spreadmart to Data Mart Conversion Joe Beeck – GfK Custom Research Dan English Principal Consultant Principal Developer/Team Lead
  2. 2. Who are we? – Dan and Joe Dan English Joe Beeck • • Developing with Microsoft technologies for over Principal Developer/Team Lead at GfK Custom 10 years Research North America. • • Over 5 years experience with Data Warehousing Has been working with Microsoft technologies for and Business Intelligence over 10 years • • Experienced in ETL and Analysis Services Current role primarily focuses on working with development, requirements gathering and data business users to identify requirements and modeling managing the project team • • Microsoft Certified IT Professional (MCITP) and Microsoft Certified Solution Developer (MCSD) Microsoft Certified Technology Specialist (MCTS)
  3. 3. Who is Magenic?  Founded in 1995, Magenic is a technical consulting firm focused exclusively on Microsoft technologies and has designed and delivered more than 500 Microsoft-based applications  Headquartered in Minneapolis, with offices in Chicago, Boston, Atlanta and San Francisco  2005 Microsoft Partner of the Year, Custom Development Solutions – Technical Innovation  2007 Microsoft Partner of the Year Finalist, Data Management  Microsoft Gold Certified Partner and National Systems Integrator  40 Enterprise Data Services (EDS) consultants
  4. 4. Who is GfK? Founded in1934 and headquartered in Nuremberg, Germany Size • $1.43B + in annual revenue • 9,300+ full-time employees (USA – 700+) • 2nd largest custom research company in North America • 2nd largest custom research company worldwide Full Service • Knowledge and resources to meet any client need • Global databases and custom research expertise • Qualitative and quantitative practices Global Coverage • 130 offices located in more than 70 countries
  5. 5. Today‟s Agenda • Market Research Overview • The Original Spreadmart Solution • What is the BI Maturity Model? • Spreadmarts vs. Data Marts • Case Study and Demo • Lessons Learned • Questions?
  6. 6. Market Research Overview
  7. 7. Why Do Market Research? To reduce the risk of decision making: • What hidden opportunities exist in the current market? • To whom should we target our advertising? • What product should we market next? • Should we change the formula of an existing product?
  8. 8. Case Study – Reversing Category Decline Industry: • Dairy Industry Business Problem: • How to stop and reverse declining dairy sales Background: • Dairy sales slipping • Negative publicity about dietary fats from dairy • Fewer servings per day recommended • The client, Dairy Trade Association, needed to understand consumer attitudes toward dairy products to direct strategy
  9. 9. Case Study – Reversing Category Decline Approach • Attitudinal segmentation • Identify how narrowly or broadly people view dairy • Understand/quantify the consumer perception that dairy is unhealthy • Measure consumers attitudes on: • Dairy category overall • Individual products • Health and lifestyle issues • Cross this attitudinal information with consumption patterns, lifestyle habits, and demographics • Combine and model results to create in-depth profiles of the respondent
  10. 10. Case Study – Reversing Category Decline Results • Major recommendation: It‟s about milk! Milk should be at the core of the communication message. • Results: Very successful campaign to reverse the trend and make milk cool again. • Milk sales rose • Public perception changed
  11. 11. Ways to Collect Data Type Situation • A moderate number of questions Telephone • A lot of people • No visual or sensory stimuli needed • A few questions – simple • A lot of people Mail • Few security concerns • Visual and/or sensory stimuli • Fewer questions • Simple to complex Online • A lot of people for relatively little money • Visual stimuli • More questions In-person • More complex • Visual and/or sensory stimuli
  12. 12. Types of Questions Closed End Open End Provides choice for the respondent. Respondent answers in own words; no responses for respondent to choose from. Good for “What do you do, Example: What, if anything, do where is it done, who uses it” you like about the product? type questions Please clarify. Should generally be used when Example: Why do you say that all (or most) of the possible you [respondent‟s answer to responses can be determined question 3]? beforehand.
  13. 13. Market Research Process Define Survey and Measures Conduct Survey Collect Data Process and Clean Data Report Results
  14. 14. The Original Spreadmart Solution
  15. 15. Business Requirements 300 000 survey responses per year – 25 000 per month 12 report templates 1500 reports generated per month Ability to generate historical reports 24-hour turnaround after receipt of data Perfect data
  16. 16. Speadmart Solution Run PERL Run VBA Pre- Use Adobe Manually script to Load and script to “stitch” aggregate Distiller to post files to validate data generate data and convert SharePoint together using 1,500 individual split into 17 everything to according to PostScript tabulation separate PDF a predefined reports software files (250 at Excel tabs documents file structure according to a time) the hierarchy 16 Hours 16 Hours 30 Hours 1 Hour 1 Hour 1 Hour
  17. 17. Spreadmart Issues Data had become decentralized over the course of 3+ years Excel became unusable due to increasing data volume and memory errors Unable to run historical reports without returning to saved versions of Excel documents Prone to error because of so many manual updates, lack of versioning control, and lack of integrity-checking software Custom updates increased reliance on individual developers. Transfer of knowledge became very difficult System/process became so slow that even small issues would cause delays in delivery to the client Solution had become fragmented and new report requests were no longer cost efficient Errors and delays were beginning to put contract in jeopardy
  18. 18. There must be a better solution…
  19. 19. BI Maturity Model – where are you at? STRUCTURE: Mgmt Reports Spreadsheets Data Marts Data Warehouses Enterprise DW BI Services System Individual Department Division Enterprise Inter-Enterprise SCOPE: By Wayne Eckerson, Director of Research, TDWI
  20. 20. Spreadmart BI – Infant (2nd) Stage Are the users What happens when Did they extract all How long does it extracting and the person responsible of the necessary take to extract reporting on the for the report goes on data to allow the data and how right data? vacation or is sick or management to ask clean is it once it leaves the company? further questions? is extracted? MS Access MS Excel MS PowerPoint Business Users Do they have enough What logic is Source Data data collected to being applied and perform yearly Is all of the data is this common comparisons or available in the logic within the trends over time? source system? organization?
  21. 21. Data Mart BI – Child (3rd) Stage OLAP Engine Data Mart Source Data Business Users
  22. 22. Spreadmart vs. Data Mart BI Spreadmart Data Mart • High end-user control • Shared/consistent view of data • Easy to create and use • Centralized logic • Can be pieced Pros • Highly interactive (slice-and- together Pros dice) • Highly customizable for • Secured the intended audience • Very Flexible • Low cost solution • Extremely Fast response time • Inconsistent view of the data • • No centralized logic Takes time to generate Cons • • Typically no security applied Less end-user control Cons • • Silos of data throughout Costs more to develop organization • Could potentially introduce new tools (training)
  23. 23. Spreadmart to Data Mart Case Study Spreadmart • Excel file report system • Lots of embedded business logic and conditional formatting • Generated over 1500+ files (most contained multiple reports) with macro • Process took approximately 30 hours to run • Initial Excel file was created and tested over a 6 month time period • If there were any data issues or report creation errors process had to be re-run • Not easy to implement additional change requests Data Mart • Star schema database engine designed • Analysis Service database created with centralized logic • Reporting Service reports created and data driven subscription setup • Generated same reports in approximately 30 minutes • Entire database along with reports was created and tested in 2 month time frame • Database and reporting structure extremely flexible to change requests
  24. 24. Data Mart Case Study
  25. 25. Reporting Services with SSAS data SSAS Designer within SSRS • Keep measures in the columns • Flattened hierarchy information • Very nice drag-n-drop feel and parameter setup MDX Query within SSAS data source • No drag-n-drop designer • Custom MDX scripting capability SSIS data source • OLE DB Source or DataReader (ADO.Net) • Ability to customize output • Join multiple datasets SQL Server Stored Procedure • Similar capabilities like SSIS • Custom formatting and data merging logic within stored procedures • OPENQUERY commands with linked server (SSAS)
  26. 26. Data Mart Conversion Steps 1. Received the business requirements for the deliverables 2. Reviewed the reporting deliverables, data files, and calculations required for the reports 3. Created the star schema database model 4. Created the ETL process to import the data file and load the star schema 5. Created the Analysis Service database 1. Setup the necessary dimensions, attributes, hierarchies 2. Produced the cube with necessary measures, measure groups, and calculations 6. Setup the linked server within SQL Server to access the SSAS database 7. Created the stored procedures to be used by Reporting Services 8. Created the Reporting Service reports 9. QA reports and all data associated with them 10.Setup data driven subscription to generate all of the reports to be delivered to the client
  27. 27. SSAS data to SSRS Demo DEMO
  28. 28. Lessons Learned The client needs to understand how their hierarchical data is applied ( re-casted each month or applied using type 2 dimension ) The benefits of the future BI solution need to be emphasized throughout the project Automate, Automate, Automate Stick to your process Business users are „key‟ – keep them involved throughout the process and use them for Q&A and validation Data is never as clean as you would expect – „trust but verify‟ Nothing is ever as „easy‟ as you think – even rounding can cause issues Document and comment on all processes with reasons why
  29. 29. Resources Microsoft BI Site SharePoint BI Features Introduction PerformancePoint Home Site PerformancePoint Developer Portal Channel9 MSDN BI Screencasts SQL Server 2008 Home Site Microsoft Virtual Labs (TechNet and MSDN) Magenic Blogs
  30. 30. Source Information BI Maturity Model – or Dan‟s Blog postings – Using Reporting Services (SSRS) with SSAS data and SSAS MDX Round = Banker‟s Rounding DateTool - and time-intelligence-implementation.aspx
  31. 31. Contact Information – Thank You! Dan English - Dan‟s BI Blog - Dan‟s Videos - or 3657354010876223112 Magenic - Joe Beeck -