Next gen bi and datawarehouse solutions ross lo forte
•Download as PPTX, PDF•
1 like•462 views
This document discusses Microsoft's investments in data warehousing technologies and solutions. It outlines Microsoft's focus on column store, data quality, real-time data warehousing, and advanced analytics. It also describes Microsoft's tier 1 offerings for data warehousing including integrated ETL and reporting tools, simplified management, and predictable performance. Finally, it introduces Fast Track data warehousing reference architectures that provide pre-tested hardware configurations optimized for data warehouse workloads.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Next gen bi and datawarehouse solutions ross lo forte
Similar to Next gen bi and datawarehouse solutions ross lo forte (20)
More from Microsoft Singapore
More from Microsoft Singapore (20)
Next gen bi and datawarehouse solutions ross lo forte
- 1. Next Generation BI and Data Warehouse Solutions Ross LoForte SQL Technology Architect Microsoft Technology Centers
- 2. Data Warehouse Industry Trends Microsoft has steadily invested in the most important data warehouse technologies Column Store Data Quality Real-Time DW and Streaming Data Advanced Analytics Key Trends MPP MDM Secure and Robust MPP(Parallel Data Warehouse) Master Data Services Database Security StreamInsight (Streaming Data) Data Quality (Zoomix) Column Store (Project Apollo)
- 3. Microsoft’s on-going investments in Data Warehousing Futures 2008 2005 10s of TB Warehouses Parallel partitioning Data compression New Reference Architectures Policy Based Admin. DB Resource Governance High Perf. Connectors(Oracle, Teradata, SAP BW) Data Profiling Policy based auditing Data Warehouse Scale Multi TB Warehouses Enterprise scalability DW Reference Architectures Unified manageability Enterprise class ETL tool Data Cleansing(Fuzzy lookup/matching) Data Protection & Tracing PB Warehouses >64 Core Processing Scale out through MPP Perf. Management Tools BI Resource Governance Improved Predictability Mixed workload support Continuous Loading Master Data Management(Stratature Integration) Integrated DQ Services (Zoomix) Rights Management Data Warehouse Management Heterogeneous Connectivity & Workloads Data Integrity & Quality Compliance & Security
- 4. SQL Server Top Achievements
- 5. Tier 1 offerings Tier 1 Services and Support Microsoft Data Warehousing solutions
- 6. Tier 1 offerings Microsoft Data Warehousing solutions Integrated ETL and Reporting tools Simplified management Predictable response Lower storage costs Integrated Master Data Management tool
- 7. Tier 1 offerings Microsoft Data Warehousing solutions All features and benefits of SQL Server 2008 R2 Enterprise Ability to scale up to 256 logical processors Ability to scale memory beyond 2TB Continuous loading using StreamInsight
- 8. Tier 1 offerings Microsoft Data Warehousing solutions Balanced solution for scan-centric workloads Best price-to-performance ratio Features 12 reference architectures validated by Microsoft Ability to scale up to 80 terabytes
- 9. Some Data Warehouses Today Big SAN Big SMP Server Connected together Server can consume 32 GB/Sec of IO, but SAN can only deliver 12 GB/Sec Queries are slow Despite significant investment in both Server and Storage
- 10. Challenges of traditionalData Warehouse IO Channel Sequential IO capacity of storage System CPU 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 010101010101010101011101101011101010110101010101010101010110111101010101101101011010110101 CPU Constraint CPU IO Channel Sequential IO capacity of storage System Storage System Constraint Sequential IO capacity of storage System CPU IO Channel IO Channel constraint
- 11. What is Fast Track Data Warehouse? A methodfor designing a cost-effective, balanced system for Data Warehouse workloads Reference hardware configurationsdeveloped in conjunction with hardware partners using this method Bestpracticesfor data layout, loading and management Relational Database Only – Not SSAS, IS, RS
- 13. Pretested Configurations Lowers TCO
- 14. Balanced CPU to I/O Channel Optimized for DW
- 15. Modular Building Block Approach
- 16. Scale Out or Up within limits of Server and SANOLTP Applications
- 17. Reference architectures boost performance and reduce risk HP Fast Track data warehouse configurations scale from SMB to Enterprise Prescriptive guidance and optimized methodology for data warehouse query workloads with large sequential data reads Balanced hardware approach ideal for data marts or small to mid-sized DW with scan-centric workloads Supports 1 to 80TB Data Warehouse at leading price/performance Configurations, tested performance guidance and best practices for deploying/operating/managing Packaged and custom support Entry1-5TB DL370 w/D2700 DAS Basic6 – 12TBDL38x w/MSA P2000 Mainstream12 – 24TBDL585 w/MSA P2000 Mainstream16 – 32 TB DL580 w/MSA P2000 Premium24 – 80 TBDL980 w/MSA P2000
- 18. Demo Fast Track Data Warehouse
- 19. Tier 1 offerings Microsoft Data Warehousing solutions Enterprise Data Warehouse Appliance offering High Scalability and performance Flexibility and choice Integrated with Microsoft BI
- 20. Parallel Data WarehouseAn appliance experience All hardware from a single vendor Orderable at the rack level Vendor will: Assemble appliances Image appliances with OS, SQL Server, and PDW software Appliance installed in 1 – 2 days Support: Microsoft provides first call support Hardware partner provides onsite break/fix support
- 21. Control Rack Data Rack Compute Nodes Storage Nodes SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL Control Nodes Active / Passive Management Node Dual Fiber Channel Dual Infiniband Landing Zone Backup node Spare Compute Node
- 22. Demo Parallel Data Warehouse
- 23. Admin Console – Home Page Menu options listed left to right by PDW activity and status.
- 24. Admin Console – Appliance State Appliance State tab lists the state of all active nodes within the appliance.
- 25. Admin Console – Dashboard Customizations Can optionally include up to 38 available performance counters. …
- 26. Admin Console – Dashboard The Dashboard tab provides near real-time performance counters.
- 27. Distributed Data Warehouse Architecture Each business unit has own Data Marts More responsive to business needs Fits budget realities Hub provides centralized data governance etc. Node-to-node data movement Parallel over Infiniband >500GB per min Parallel Database Export (PDE) 23
- 29. Data exploration & analysis
- 32. Contextual visualizationBUSINESS USER EXPERIENCE BUSINESS COLLABORATION PLATFORM DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
- 34. Excel Services
- 35. Web based forms & workflow
- 36. Collaboration
- 37. Search
- 40. PowerPivot for SharePointBUSINESS COLLABORATION PLATFORM DATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
- 45. Data Mining
- 46. Data WarehousingDATA INFRASTRUCTURE & BUSINESS INTELLIGENCE PLATFORM
- 47. Use Reports to Drive Decisions Create and share reports Maintain a single version of truth with your Excel Workbooks Drive decision based on facts
- 48. Use Dashboards to Drive Decisions Visual displays of information needed to achieve one or more objectives Single-Screen display of information Answer fundamental questions Alerts the user to issues or problems Span Operational, Performance, Personal Align strategies and organizational goals Measure and manage Key Performance Indicators (KPI) Modeled after the business, not the data
- 49. Use PowerPivot to Drive Self-Services PowerPivotfor Excel PowerPivotfor SharePoint 29
- 50. Microsoft Business Decision Appliance End-to-end, pre-configured stack quickly enables BI for Excel power users Rich insight: Empower users to easily create PowerPivot workbooks from real-time business data for faster, more accurate insights Reduced complexity: Overcome cost and complexity of BI; shift IT resources from running ad-hoc reports to innovation initiatives Easy manageability: Custom code for management dashboard and scripted data source integration ease deployment and simplify administration
- 51. Microsoft Data Warehouse Vision Make SQL Server the fastest and most affordable database for customers of all sizes Complete Data Warehouse Solution Flexibility and Choice Massive Scalability at a Low Cost Simplified Data Warehouse Management
- 52. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Editor's Notes
- This chart, taken from the TDWI Next Generation Data Warehouse Platforms report, shows data warehouse features and techniques plotted for delta (growth) and plan to use (commitment).As you’ll see in the rest of this presentation, Microsoft’s strategy for data warehousing aligns extremely well with the features and techniques that have broadcommitment and good growth. The ticks show specific technology areas Microsoft is strategically investing in.
- Introduction to Microsoft DW solutions training in resources
- Changed slide with animation
- The Fast Track Data Warehouse reduces time, cost, and unknowns of selecting and configuring hardware for SQL Server data warehouses based on HP’s success in SQL Server deployments. In fact, HP is the #1 platform for SQL Server Fast Track deployments.The reference architecture supports a continuum of performance and scalability, up to 48 TB,at industry leading price/performance. These pre-integrated, pre-optimized solutions eliminate the need for second-touch customization at a configuration center for faster deployment and lower costs.Components include:2-8P HP servers, either HP ProLiant DL38x, DL58x, or DL785 serversHP Storage Works P2000 MSA storageHP networkingMicrosoft SQL Server Enterprise EditionWe will soon be introducing a new “entry-level” configuration that dramatically simplifies deployment of small data warehouses on a cost-effective, compact form factor for lower TCO.And the entire solution is surrounded with expert installation and support services from HP and Microsoft to provide a seamless experience and low-risk installation and operation.
- Problems illustrated in the detail rows include a link to the Alerts page.
- Your customization selections persist until you log out (or close) the Admin Console session.
- Displays a very animated view of PDW activity.Silverlight is required
- Point of the Slide: Here is the solution, show some detail on how we think of the BI architecture in 3 main layers. Point out that self-service analysis takes place in Office and SharePoint, but the Business Intelligence Platform is there to take popular user-generated solutions and convert them into full organizational solutions.Prove that our better together (Microsoft BI) story is real! Flow of the Slide: A trusted BI platform is critical for a business intelligence solution to work today. If we’re going to achieve the promise of BI, we need to have the confidence and trust in the data, we need to know where it came from, and that it is both timely and reliable for us to use to make a decision. That’s where the power of an integrated BI platform like SQL Server comes into play.Once you have the data ready to use, the middle tier comes into play, where business users actually interact with the data and turn it into something that is useful to them to make the right decision. The numbers that we pull from other systems are just that—we need applications and content to turn them into actionable items—and that’s where the middle tier comes into play. Finally, we need the right tools and applications to ensure that we can use that data in the way we want to in order to make our decisions. Applications and tools that range from personal, to team, to organizational and corporate tools, all with a familiar look and feel, all integrated and working with my Operating system, email, Internet search function. And this is what the power of integration through Microsoft Office brings you. From the Office productivity tools like Excel, through to the team and collaboration tools of SharePoint. By integrating BI seamlessly into a broader business productivity suite that includes search, collaboration, unified communications, and content management, we offer the end user a much richer experience.
- Point of the Slide: Here is the solution, show some detail on how we think of the BI architecture in 3 main layers. Point out that self-service analysis takes place in Office and SharePoint, but the Business Intelligence Platform is there to take popular user-generated solutions and convert them into full organizational solutions.Prove that our better together (Microsoft BI) story is real! Flow of the Slide: A trusted BI platform is critical for a business intelligence solution to work today. If we’re going to achieve the promise of BI, we need to have the confidence and trust in the data, we need to know where it came from, and that it is both timely and reliable for us to use to make a decision. That’s where the power of an integrated BI platform like SQL Server comes into play.Once you have the data ready to use, the middle tier comes into play, where business users actually interact with the data and turn it into something that is useful to them to make the right decision. The numbers that we pull from other systems are just that—we need applications and content to turn them into actionable items—and that’s where the middle tier comes into play. Finally, we need the right tools and applications to ensure that we can use that data in the way we want to in order to make our decisions. Applications and tools that range from personal, to team, to organizational and corporate tools, all with a familiar look and feel, all integrated and working with my Operating system, email, Internet search function. And this is what the power of integration through Microsoft Office brings you. From the Office productivity tools like Excel, through to the team and collaboration tools of SharePoint. By integrating BI seamlessly into a broader business productivity suite that includes search, collaboration, unified communications, and content management, we offer the end user a much richer experience.
- Point of the Slide: Here is the solution, show some detail on how we think of the BI architecture in 3 main layers. Point out that self-service analysis takes place in Office and SharePoint, but the Business Intelligence Platform is there to take popular user-generated solutions and convert them into full organizational solutions.Prove that our better together (Microsoft BI) story is real! Flow of the Slide: A trusted BI platform is critical for a business intelligence solution to work today. If we’re going to achieve the promise of BI, we need to have the confidence and trust in the data, we need to know where it came from, and that it is both timely and reliable for us to use to make a decision. That’s where the power of an integrated BI platform like SQL Server comes into play.Once you have the data ready to use, the middle tier comes into play, where business users actually interact with the data and turn it into something that is useful to them to make the right decision. The numbers that we pull from other systems are just that—we need applications and content to turn them into actionable items—and that’s where the middle tier comes into play. Finally, we need the right tools and applications to ensure that we can use that data in the way we want to in order to make our decisions. Applications and tools that range from personal, to team, to organizational and corporate tools, all with a familiar look and feel, all integrated and working with my Operating system, email, Internet search function. And this is what the power of integration through Microsoft Office brings you. From the Office productivity tools like Excel, through to the team and collaboration tools of SharePoint. By integrating BI seamlessly into a broader business productivity suite that includes search, collaboration, unified communications, and content management, we offer the end user a much richer experience.
- Point of slide: To introduce PowerPivot for Excel and SharePoint.Script:The revolutionary PowerPivot technology includes both an Excel add-in and server-based technology that integrates with SharePoint. This allows your users to quickly manipulate very large data sets and share analysis with co-workers—with little or no IT assistance. PowerPivot supports the unmatched analytical power and ease-of-use of Excel while allowing users to connect to any data source—databases, reports, data feeds, Web pages, Excel files, and more. After performing analysis, you can publish it to SharePoint 2010, further enabling collaboration with others. These capabilities provide a single version of truth for all users. IT also benefits from the managed self-service capabilities within PowerPivot. IT can now create reports using SQL Server Reporting Services R2 and SharePoint 2010 andtrack the usage of PowerPivot applications and discover mission-critical Excel applications. Conclusion: PowerPivot is the key to managed self-service BI and empowering end users. PowerPivotenables seamless and secure sharing and collaboration on user-generated BI solutions, while helping IT organizations increase efficiency through the PowerPivot Management Dashboard.
- Here is the product specs, HW and SW configuration for your reference. I will not spend time on this. You can read the deck later.
- The Microsoft vision is to make SQL Server the fastest, simplest and most affordable database management system for data warehouse customers of all sizes. In 2008, Microsoft acquired DATAllegro and its MPP appliance that is now Parallel Data Warehouse. Its shared nothing architecture delivers enterprise-class performance and scales for the most demanding data warehouses. Microsoft investments in complementary technologies will significantly improve data warehouse performance and scalability. Microsoft aims to simplify IT for customers through innovation. Since its inception, SQL Server has proudly followed this tradition by simplifying the creation and deployment of databases. Microsoft is committed to building the most easily managed data warehouses through manageability improvements in the software and continuous investment in Reference Architectures and data warehouse appliances. Future appliances from Microsoft will dramatically simplify the deployment and management of a broader range data warehouses. Microsoft is committed to offering the lowest cost solutions for data warehouse customers of any size and achieves this through:The most competitive pricing across its data warehouse offerings (Fast Track Data Warehouse starts at under $11,000 per terabyte; Parallel Data Warehouse starts at under $20,000 per terabyte). Providing a complete Enterprise Data Warehouse and BI toolkit, including SQL Server Analysis Services, Reporting Services, Master Data Services, PowerPivot, and StreamInsight (in the R2 release, customers get all of these tools with the database at no extra cost).Hardware choice with a wide range of hardware platforms that helps customers avoid vendor lock-in and reduce their hardware acquisition and maintenance costs.The use of industry-standard hardware instead of proprietary components.