Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SQL Data Warehousing in SAP HANA (Sefan Linders)

394 views

Published on

SAP Inside Track Netherlands (sitNL) presentation on SQL Data Warehousing in SAP HANA by Sefan Linders (SAP)

Published in: Software
  • Be the first to comment

SQL Data Warehousing in SAP HANA (Sefan Linders)

  1. 1. PUBLIC Sefan Linders Data Warehouse Architect Customer Innovation & Enterprise Platform November 2017 SAP HANA SQL Data Warehousing Overview, Process, and Products
  2. 2. 2PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Disclaimer ▪ The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related document, or to develop or release any functionality mentioned therein. ▪ This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross negligence. ▪ All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
  3. 3. 3PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Why is data warehousing still necessary? The factors beside performance Characteristics ▪ Consolidates data across the enterprise ▪ Standardizes data model ▪ Supports decision making Main Tasks ▪ Define common semantics ▪ Harmonize data values ▪ Establish a ‘single version of truth’ ▪ Provide actuals and history Data Lake BI | Predictive | Planning Data Warehouse “Single Point of Truth” Analytics Hadoop Data Sources SAP | non-SAP | Cloud
  4. 4. 4PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ How does SAP approach Data Warehousing A closer look at SAP HANA Data Warehousing SAP HANA Platform Data Warehouse
  5. 5. 5PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Next Generation Data Warehousing Landscape BW/4HANA and SQL Data Warehouse on one platform SAP HANA Platform SAP Business Warehouse SAP BW/4HANA SAP HANA SQL Data Warehouse SAP HANA Application Services SAP HANA Integration Services SAP HANA Processing Services SAP HANA Database Services Data Warehouse
  6. 6. 6PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Application driven approach, SAP BW/4HANA as premium DW application with integrated services ▪ SAP BW/4HANA is an application offering. All data warehousing services via one integrated repository ▪ Optional integration of additional tools for modelling, monitoring and managing the data warehouse SQL driven approach, SAP HANA with loosely coupled tools and platform services, logically combined ▪ SQL approaches require several loosely coupled tools, usually having separate repositories ▪ “Best of breed” approach to build your own model BW/4HANA and SQL Data Warehouse Two ways to run, or get the best of both SAP HANA Platform SCHEDULING & MONITORING MODELING PLANNING OLAP LIFECYCLE MANAGEMENT ETL SAP BW/4HANA SAP HANA Platform SCHEDULING & MONITORING MODELING PLANNING OLAP LIFECYCLE MANAGEMENT ETL HANA SQL DW
  7. 7. 7PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ SQL Data Warehouse Data process perspective of SAP defined SQL DW Compute & Data Store Ingest Sources Consume Data Lake SQL Data Warehousing ETL Replication Streaming Virtual Access … 3rd-PartyAnalytics Sensor Machine … SAP Vora BI | Predictive | PlanningBusinessObjects™ CDS - NDSO Procedures Flowgraphs CalcViews Virtual Tables SQL SQL • WebIDE • DW Foundation • XS Advanced DW Scheduler Enterprise Architect EIM ->
  8. 8. 8PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ The SQL DW runs on XS Advanced Cloud and on-premise application platform Compute & Data Store Ingest Sources Consume Data Lake SQL Data Warehousing ETL Replication Streaming Virtual Access … 3rd-PartyAnalytics Sensor Machine … SAP Vora In- Memory BI | Predictive | PlanningBusinessObjects™ XS advanced runtime SAP Web IDE …HALM EA Designer HANA deployment infrastructure
  9. 9. 9PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Integrated Data Warehouse Process Introducing the SQL DW application toolset DESIGN RUNDEVELOP DEPLOY
  10. 10. 10PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Designing the SQL DW Modeling your processes and data SAP Enterprise Architect Designer Model across the enterprise Native HANA 2 application SAP PowerDesigner Model across the enterprise Desktop application DESIGN RUNDEVELOP DEPLOY
  11. 11. 11PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ SAP Enterprise Architect Designer Edition for SAP HANA Create and integrate enterprise, landscape, process, and data models to manage information and systems effectively ▪ Business process architecture ▪ Landscape and application architecture ▪ Requirements management ▪ Strategy architecture to document goals and projects ▪ Physical data modeling & data architecture ▪ Reverse engineering capabilities ▪ Lineage & Impact analysis Design Implementation Strategy TechnologyBusiness Process Data Landscape Requirements
  12. 12. 12PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
  13. 13. 13PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Demo screenshots EA Designer (the live presentation has a demo video instead)
  14. 14. 14PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Building the SQL DW One environment to build all artefacts SAP Web IDE for HANA Develop the entire DW from your browser Successor of HANA Studio Dev Major extensions for DW functions DESIGN RUNDEVELOP DEPLOY
  15. 15. 15PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
  16. 16. 16PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Demo screenshots SAP WebIDE (flowgraphs, calcviews, DB Explorer) (the live presentation has a demo video instead) ▪ SAP Web IDE for SAP HANA is the successor to SAP HANA web development workbench and the development perspectives of SAP HANA studio. ▪ It offers – Development of SAP HANA content and models – UI development with SAPUI5 – Node.js or XSJS business code – Git integration ▪ It is – Browser based – Installed as SAP HANA XSA application
  17. 17. 17PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Demo screenshots SAP WebIDE (flowgraphs, calcviews, DB Explorer) (the live presentation has a demo video instead)
  18. 18. DWH Innovations Native DSO
  19. 19. 19PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Native DataStoreObject (NDSO) Simplification of the Data Warehouse Classic DWH practise for request management and delta handling DB procedu re DB DB Metadata tables Batch ID Date Time User RunTime Batch 5 | Jan 17 | Batch 4 | Jan 16 | Batch 3 | Jan 15 | Batch 2 | Jan 14 | Batch 1 | Jan 13 | Native Data Store Object Custom design and development effort Out of the box NDSO Metadata tables Batch ID Date Time User RunTime Batch 5 | Jan 17 | Batch 4 | Jan 16 | Batch 3 | Jan 15 | Batch 2 | Jan 14 | Batch 1 | Jan 13 |
  20. 20. 20PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Native DataStoreObject (NDSO) Simplification of the Data Warehouse process Classic DWH practise for request management and delta handling - To be able to enable delta propagation, or roll-back of data loads, “Request” or “Batch” management is needed - Metadata on data loads needs to be stored in the target table load to (e.g. a batch ID), and a metadata framework is developed to record load date/time, execution user, number of records loaded - To allow for roll-back, additional table is needed to record all changes (before/after image), or all data changes need to be time-sliced in target table - Setting this up and keeping it running can take considerable effort, for example for design of metadata tables, roll-back database procedures, and monitoring functions. - Running these processes can be resource intensive and increase DWH load times DB procedu re DB DB Metadata tables Batch ID Date Time User RunTime Batch 5 | Jan 17 | Batch 4 | Jan 16 | Batch 3 | Jan 15 | Batch 2 | Jan 14 | Batch 1 | Jan 13 | Native Data Store Object - The NDSO provides request management and delta handling out of the box - The NDSO is delivered with a friendly user interface for load monitoring and request handling features such as roll-back - The NDSO integrates natively with EIM flowgraphs, and with 3rd party ETL - The NDSO supports the “delta language” of SAP data source extractors Design and development effort Out of the box NDSO Metadata tables Batch ID Date Time User RunTi me Batch 5 | Jan 17 | Batch 4 | Jan 16 | Batch 3 | Jan 15 | Batch 2 | Jan 14 | Batch 1 | Jan 13 |
  21. 21. 21PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Data set A Data set B The Native DataStoreObject (NDSO) adds a management layer to a “simple” table. The NDSO provides out of the box Delta and Request management. Request management (example) Data set C updated and deleted data from earlier loaded data set B. The NDSO “roll-back” function uses the changelog to restore to earlier state, in case of errors. Access as usual NDSO data can be accessed by CalcViews or any other tool like any other table Delta propagation The built-in changelog enables delta loads from SAP data sources, and to subsequent NDSO or BW-ADSO NDSO Any datasource E.g. SAP Extractor, SQL statement, SDI Flowgraph, Data Services, 3rd party ETL Data set C active table change log Native DataStoreObject (NDSO) Simplification of the Data Warehouse process
  22. 22. 22PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Deploying the SQL DW This is where DevOps comes in DESIGN RUNDEVELOP DEPLOY Open Source deployment Bring your own tools: Jenkins, Bamboo, XL release, etc. SAP HALM* Native HANA 2 application *Planned CTS+ XSA integrates with enhanced change and transport system (CTS+)
  23. 23. 25PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Classic DWH development All developers work in the same workspace and runtime, on the same version • In HANA XS Classic, or in a common best-of-breed data warehouse project, all developers work on the same repository and the same run- time environment. • Any change made by one developer and activated on the database, in the ETL tool, or other tooling, is immediately visible for all other developers. • This “shared workspace” and “shared runtime” make it hard to develop and test features or user stories isolated from other developers.
  24. 24. 27PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Developer and feature isolation Enabling parallel development and test • In HANA XS Advanced, all developers work in their isolated workspace. • Each developer also works with an isolated runtime. HANA XS Advanced automatically creates a runtime container for each developer. • All developed objects are stored in a shared repository: GIT, which keeps a full version history, and uses branching to support isolated feature development. GIT repository (continuous) Testing Deployment
  25. 25. 29PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Versioning and development with GIT Working in parallel on different repository versions User story 1 User story 2 Master Time
  26. 26. 30PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Deployment example Continuous… WebIDE Continuous Integration (CI) Server Daily Builds SIT/UAT Prod DeployDeploy Assemble & Deploy Regression Deploy Test++ Production Continuous Testing | Integration | Deployment
  27. 27. 31PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Deploying the SQL DW This is where DevOps comes in DESIGN RUNDEVELOP DEPLOY EIM & DWF Monitoring EIM, Scheduler & NDSO Monitor Build into Webide Data Lifecycle Manager Data Warehouse Foundation PowerDesigner & Enterprise Architect Designer Data Lineage
  28. 28. 33PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Demo screenshots (the live presentation has a demo video instead)
  29. 29. 34PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Data Lifecycle Manager Data Warehouse Foundation Data Lake (Cold Store) SQL Data Warehousing SAP Vora In-Memory (Hot Store) Dynamic Tiering (Warm Store) TBs - 10s of TBs 10s of TBs - PBs HADOOP SAP IQ DLM Generated Union & Pruning CalcViews Structured data for fast analytics Less frequently accessed, structured data Raw data: semi-structured, unstructured, streaming data etc. DLM DLM managed data placement Based on aging rules
  30. 30. 35PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Integrated Data Warehouse Process Introducing the SQL DW application toolset DESIGN RUNDEVELOP DEPLOY
  31. 31. 37PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Strengths ▪ Complete web approach with HANA XS Advanced platform. Still 100% open SQL approach. ▪ Strong and open repository versioning with Git ▪ Freedom to custom built data models and data management processes. Example: adopt Data Vault model. ▪ Leverage 3rd party tools and in-house standards, skills & knowledge ▪ DevOps enabler: Continuous Testing | Integration | Deployment Use Case ▪ Considerable share of non-SAP source systems and interfacing ▪ Specific data model requirements, for example for for auditability ▪ 3rd party DW replacement ▪ DevOps requirements Why should you choose HANA SQL DW SAP HANA Platform SCHEDULING & MONITORING MODELING PLANNING OLAP LIFECYCLE MANAGEMENT ETL HANA SQL DW
  32. 32. Thank you. Sefan Linders Data Warehouse Architect Customer Innovation & Enterprise Platform sefan.linders@sap.com SAP HANA SQL Data Warehousing Overview, Process, and Products

×