Prague data management meetup #31 2020-01-27
•
0 likes•27 views
The document discusses an upcoming meetup on data warehousing for beginners hosted by the Prague Data Management Meetup group. It provides context on the group and past meetup topics. It then covers various data management concepts like the data lifecycle, types of data architectures including data warehousing, differences between data lakes and data warehouses, and modern approaches to data integration.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Prague data management meetup #31 2020-01-27
Similar to Prague data management meetup #31 2020-01-27 (20)
More from Martin Bém
More from Martin Bém (19)
Recently uploaded
Recently uploaded (20)
Prague data management meetup #31 2020-01-27
- 1. #31 27.1.2020 PDM MEETUP DATA WAREHOUSING PRO ZAČÁTEČNÍKY
- 2. PRAGUE DATA MANAGEMENT MEETUP (PDM MEETUP) – Open professional group – Based on www.meetup.com – Everyone is welcomed – There are no bad topics, only bad speakers☺ – You can show anything to others – Operational since September 2015 – Sponsored by ADASTRA DATA MANAGEMENT DATA ACQUISITION DATA STORING DATA INTEGRATION DATA ANALYTICS DATA USAGE PDM MEETUP 2
- 3. MEETUP HISTORY # Date Topics 1 10. 9. 2015 Data Management 2 14. 10. 2015 Data Lake 3 23. 11. 2015 Dark Data (without Dark Energy and Dark Force) 4 12. 1. 2016 Data Lake 5 7. 3. 2016 Sad Stories About DW/BI Modeling (sad only) 6 23. 3. 2016 Self-service BI Street Battle 7 27. 4. 2016 Let's explore the new Microsoft PowerBI! 8 22. 9. 2016 Data Management pro začátečníky Data Management for Beginners 9 17. 10. 2016 Small Big Data 10 22. 11. 2016 Základy modelování DW/BI DW/BI Modeling Basics 11 23.1.2017 Komponenty datových skladů Data Warehouse Components 12 28.2.2017 Operational Data Store 13 28.3.2017 Metadata v DW/BI DW/BI Metadata # Date Topics 14 25.4.2017 Jak se stát DW/BI konzultantem Be a DW/BI Consultant 15 16.5.2017 SQL 16 29.5.2017 From IoT to AI: Applications of time series data 17 26.9.2017 Aktuální trendy v data managementu Actual trends in data management 18 24.10.2017 Datové platformy na technologiích Oracle Data platforms based on Oracle 19 21.11.2017 Big Data rychle a zběsile / Big Data Fast and Furious 20 30.1.2018 Jak se staví velké datové sklady How to build huge data warehouse 21 27.2.2018 Základy modelování DW/BI #2 DW/BI Modeling Basics 22 27.3.2018 Big Data: How to deal with sensorics (floating) data easily 23 17.4.2018 DW/BIaaS 24 22.5.2018 Be a Consultant / Jak se stát konzultantem 25 19.6.2018 Building AI-Powered Retail Store 26 17.9.2018 Information Management 101 27 23.10.2018 Blockchain 28 29.1.2019 DW & BI trendy v roce 2019 / DW & BI Trends in 2019 29 26.3.2019 Data Warehouse Automation 30 10.4.2019 Next Gen Data Integration Patterns With Jeff Pollock 31 26.1.2020 Data Warehousing pro začátečníky Data Warehousing for beginners
- 4. DATA MANAGEMENT ALWAYS & FOREVER PDM MEETUP 4
- 5. INFINITE DATA MANAGEMENT LOOP IS STILL SAME Collect Integrate Enrich Store Analyze Discover Use Curate
- 7. PDM MEETUP 7
- 8. Real Life Core Banking System
- 9. Our Report Samples 9ADASTRA Czech RepublicReferences
- 10. Schema Comparison (Same Data Domains) Operational database Data warehouse Customer CustNo CustFirstName CustLastName ... Order OrdNo OrdDate ... Places Employee EmpNo EmpFirstName EmpLastName ... Takes Manages Product ProdNo ProdName ProdQOH ... Contains Qty Customer CustId CustName CustPhone CustStreet CustCity CustState CustZip CustNation Store StoreId StoreManager StoreStreet StoreCity StoreState StoreZip StoreNation DivId DivName DivManager Sales SalesNo SalesUnits SalesDollar SalesCost Item ItemId ItemName ItemUnitPrice ItemBrand ItemCategory TimeDim TimeNo TimeDay TimeMonth TimeQuarter TimeYear TimeDayOfWeek TimeFiscalYear ItemSales CustSales TimeSales StoreSales Source: Coursera
- 11. BRIEF DATA MANAGEMENT HISTORY Modern Age Cloud Automation Logical Data Warehouse Extended Data Warehouse Data Lake Polyglot Architecture Kappa / Lambda Databus Data Pipeline Real-time Data Integration Big Data ETL Open Source Analytics Big Data Analytics Self-service BI & ETL Data Science Machine Learning & AI Hadoop without Hadoop Stream Analytics All data Analytics Data Management Platform Autonomous Technologies Decoupled Compute & Storage Serverless Prehistory Controlled Chaos Best Practice Awaking Manual Scripting Primeval Relational Analytics 1985 - 1995 Antiquity Titans: Kimball vs. Inmon Maturing Best practices Enterprise Data Warehouse ETL OLAP Reference Data Management Classic Relational Analytics 1995 – 2005 2005 - 2015 Middle Age Traditional Data Warehouse Hub-and-Spoke Architecture Data Governance Master Data Management Metadata-Driven Development ELT Data Vault Data Mining DW Appliance Columnar DB In-memory DB Hadoop Stack Dawn Unstructured Data Analytics 2015 - 2025 Future? 2025 - ∞
- 12. Data Landscape Core Backends Social Networks Web Data External Data Sensors Communication Master Data Data Analytics Devices Reporting Business Intelligence Data Visualization
- 13. Data Warehousing Dark Data Data Landscape Core Backends Social Networks Web Data External Data Sensors Communication Master Data Data Analytics Devices Reporting Business Intelligence Data Discovery Segmentation Data Visualization Star Schema Operational Data Snowflake Schema Unused Data OLAP Enterprise Core Data Planning
- 14. Data Landscape Big Data Data Warehousing Dark DataCore Backends Social Networks Web Data External Data Sensors Communication Master Data Data Analytics Devices Reporting Business Intelligence Data Discovery Data Science Machine Learning Segmentation Network Analytics Documents Voice Geo DataPredictive Analytics Graph Log Semi-structured Data Visualization Biometrics Image Automated Decisions Star Schema MessagesCold Data Operational Data DW Archive Snowflake Schema Unused Data OLAP Enterprise Core Data Planning Recommendations
- 15. Data Landscape Deep Data Fast Data Big Data Data Warehousing Dark DataCore Backends Social Networks Web Data External Data Sensors Communication Master Data Data Analytics Devices Reporting Business Intelligence Data Discovery Data Science Machine Learning Segmentation Network Analytics Documents Voice Geo DataPredictive Analytics Graph Log Semi-structured Voice Data Visualization Biometrics Biometrics Real-time Vision Stream Processing Sensor Processing Image Automated Decisions Events Star Schema Mined Data Messages MessagesCold Data Operational Data DW Archive Snowflake Schema Unused Data OLAP Enterprise Core Data Planning Recommendations
- 16. CLASSICAL DATA WAREHOUSE – Key data platform for decades but no more – Data system used for reporting and data analysis, and is considered a core component of business intelligence. DWs are central repositories of integrated data from one or more disparate sources. – A large amount of information from a company stored on a computer and used for making business decisions – Old mature concept – Core Features – Database (usually RDBMS) – Subject Orientation – Data Integration – History – Structure Stability – Batch processing & significant data latencies – DW, DWH, MIS, ADS, ADW, EDW, DP Data Warehouse Data Source Data Acquistion Data Integration Data Staging Data Repository Reporting & Other Data Usage Analytics Data Source
- 17. Data Staging Area Ralph Kimball Data Warehouse Bus (DW) Bottom-Up Conformed Data Marts (Kimball’s Data Warehouse) Conformed Dimensions Business Transformation CLASSICAL DATA WAREHOUSE ARCHITECTURES (HUB-AND-SPOKE) Data Sources Data Marts RDBMS RDBMS Reporting Data Apps Bill Inmon Enterprise Data Warehouse (EDW) Top-Down Dan Linstedt Data Vault (DV) Top-Down Technical Transformation Technical Transformation Technical Transformation Business Transformation Business Transformation Data Sources Data Sources Data Marts Data Staging Area Data Staging Area Data Warehouse Data Vault Business Vault Business Transformation RDBMS RDBMS Reporting Data Apps RDBMS RDBMS Reporting Data Apps Data Marts
- 18. DW Logical Layers L0: Stage Area L1: Relational Area L1: Consolidation Area L2: Data Mart Area – Data Mart Area – L2 – User Access Layer – Consolidation Area – Consolidated L1 – Common aggregates for L2 – Cleansed and consolidated data – Relational Area – Detailed L1 – Consistent, integrated, subject oriented data, universal data structure, historical data, maximal detail – System of record – Foundation Layer – Stage Area – Direct copy of source systems Extracts Reports Note: Consolidated and Detailed L1 can share same data structures General DWH Staging Area ODS Presentation Layer Datamart Area (Dependent Datamarts) Source systems Customer DB ETL Other...S4S3S2S1 Analytic tools (SPSS, SAS..) OLAP S1 S2 S3 S4 Other S1 Ostatní...S4S3S2 ETL Materialization OLAP? ETL ETL ETL ETL ETL ETL ETL ETL ETL ETL CDB ETL EAI ReportingReporting Reporting Reporting Relational Area ETL Application Application Materialization Application Application ETL
- 19. DATA INTEGRATION PATTERNS Mediator Load Extract Extract Load Transform Transform Load Extract Transform Source Target TEL ELT ETL API Call API LogicData API CDC Change Capture LoadExtract ReplicationTransport Pub/Sub SubscriptionPublisher Broker ETLT Extract Load Transform LoadTransform Data Pipeline Data Pipeline
- 20. Metadata-Driven Development Loop Release Deploy Operate Monitor Plan Build Analyze Test Design Generate Integrate Code Development Operations
- 21. Complexity Query Engine Modern Data Architectures Hub-and-Spoke Data Warehouse Polyglot Data Federation Data Virtualization Logical Data Warehouse Lambda Kappa Databus Speed Layer Pipeline Manager Batch Layer Object Storage Data Integration Data AcquistionData Sources Data Sources Data Sources Data Ingest Messaging CDC Bulk Copy Files Data Extractor Data Ingest Messaging CDC Bulk Copy Files Data Extractor Data Warehouse RDBMS RDBMS Reporting Data Apps Data Marts Analytics Serving Layer Data Lake REST SQL Pub/Sub Data Warehouse RDBMS Reporting RDBMS Data Apps Data Marts Analytics Serving Layer Data Lake REST SQL Pub/Sub Data Integration Data Warehouse RDBMS Reporting RDBMS Data Apps Data Marts Analytics Data Integration Data Warehouse RDBMS Reporting RDBMS Data Apps Data Marts Analytics Data Integration Speed Layer Pipeline Manager Data Acquistion Data Sources 21
- 22. 22 – DW vs. DL VS. XDW/DP Traditional Data Warehouse (DW) Data Lake (DL) Extended Data Warehouse (XDW) / Data Platform (DP) Data Structured Structured & Semi-Structured & Unstructured Structured & Semi-Structured & Unstructured Data Processing Processed Raw Processed & Raw Data Schema Schema-on-write Schema-on-read Schema-on-write & Schema-on-read Data Model Relational Object-based Relational & Object-based Data History Hierarchically archived No hierarchy Hierarchically archived & No hierarchy Agility Fixed configuration Reconfigured anytime as needed Fixed configuration Reconfigured anytime as needed Security Mature Maturing Mature Primary Users Data analysists & Business professionals Data Scientists Data analysists & Business professionals & Data scientists Technology RDBMS NoSQL DBMS Hadoop Other distributed storages RDBMS NoSQL DBMS Hadoop Other distributed storages Agility Low High Medium Added Value Medium Medium High Cost High Low Medium Operation After full release From start From start
- 23. DataOps vs. Adastra Information Management Data Ingest {} Data Integration Data Management Architecture Data Model Database Data Repository Deployment Data Usage E-R ModelHub & Spoke Kappa / Databus Graph Data Model Key Management Data Discovery Data Science On-premise Cloud Hybrid Cloud Multi-Cloud Data Warehouse Data Mart Sandbox Business Intelligence Reporting Machine Learning Data Lake RDBMS In-memory Document Store Multidimensional DB Graph DBMS Columnar DBMS Object Store NoSQL Multidimensional Model Data Archive Time Variance Data Latency Audit Date Tiering Data Retention Data SecurityAutomation Orchestration Aggregation Reconciliation ETL/ELT Cleansing Standardization Data Loading Data Replication Change Data Capture Manual Inputs Stream Processing Legacy Lambda Operational Data Store Snowflake Schema Big Data Fabric Star Schema Metadata Data Catalog Data Governance Data Adhoc Quering Data Literacy Reference & Master Data Management TCO Management Governance Polyglot Key-Value Column Family Data API File RepositoryDistributed File System Data Pipeline Master Data Repository DataOps Containers SLA Management
- 24. BUSINES PRIORITIES VS. CLASSICAL DATA WAREHOUES Grow revenue & profit Improve CX Improve products and services 360 degree view Digital transformation Accelerate responses to business and market changes Real-time data-driven decisions Faster predictive insights Smarter intelligent business Structured static data only Melting with data growth Business demand exceeds IT capacities & IT budgets Data siloed cross multiple platforms Growing operational overhead Missing real-time insights Unscalable Limited advanced analytics Really expensive TCO Outdated governance and security
- 25. CZ banka A Data Warehouse Data Warehouse Core Reporting & Other Data Usage Analytics Operational Data Store Data Source Data Acquistion Data Integration Data Staging ODS Data Repository Data Source ODS Data API Process Process Data Marts Data Synchro Data Quality Master Data Repository DQ Data API Data Quality Engine External Calculation Engines Process Process 25
- 26. CZ Banka B Data Warehouse Data Warehouse Core Reporting & Other Data Usage Analytics Operational Data Store Data Source Data Acquistion Data Integration Data Staging ODS Data Repository Data Source ODS API Process Process Data Marts Data Synchro Data Quality Engine Master Data Repository Reference Data Repostory Reference User Interface
- 27. Liberty Bank Data Warehouse Data Warehouse Core Reporting & Other Data Usage Analytics Data Source Data Acquistion Data Integration Data Staging Data Source Data Marts Data Quality Engine Master Data Repository Reference Data Repostory Reference User Interface
- 28. Data StoreApplication ServerWeb Server Pentaho Data Integration (Web Console) Adastra Workflow GUI Adastra Ref Books GUI Adastra Worflow Middleware Adastra Ref Books Middleware Pentaho Data Integration (Carte) Pentaho Data Integration (Repository) Adastra Worfklow for RDBMS Database Scheduler Adastra Ref Books Store Adastra ELT SAP PowerDesigner Adastra Code Generator External Components Adastra Data Model Runtime Design Time Liberty Bank
- 29. IKEA: Data Warehouse as a Managed Service Data volume & Processing Data from 3 countries: CZ/HU/SK 8 stores 2 830 000 customers Purchases from 2007 till now 105 000 000 transactions 620 000 000 transaction items 295 000 000 email events 1TB-total size of database Daily load takes about 5(2+3)hours DWH server Configuration Virtual Server 8vCPU, 32GB RAM, 1.5TB HDD Adastra ETL Framework & MS SSIS Cloud4Com VPN Cloud-IKEA MS SQL Server 2017 Standard Edition
- 30. Duo Bank of Canada: Data Warehouse as a Managed Service Data Warehose as a Managed Service in Cloud by Adastra CA Best-shoring and support by Adastra BG. Payment Card Industry Data Security Standard (PCI DSS) compliance „We created Duo Bank to do things differently. With a customer focused mindset, we’re committed to changing the way businesses connect with their customers by reimagining and recreating value-driven financial products and services. At the heart of everything we do is our commitment to innovation, customer experience, efficiency and delivering exceptional value.
- 31. Data LakeOn-premise Data Sources Landing & Staging Area Raw Data Area Data Mart Area Business Intelligence Microsoft Power BIData Loader Azure Data Lake Storage Azure Data Factory Azure SQL Database Azure Data Catalog Data LakeOn-premise Data Sources Landing & Staging Area Raw Data Area Data Mart Area Business Intelligence Amazon Insight Data Loader Amazon S3 Amazon Data Pipeline Amazon RDS Amazon Glue Amazon Athena Integrace On-premise řešení s analytikou v Cloudu
- 33. FK_FXRXFACT__FXRX FK_ACC__GLACC FK_ACCPROVFACT__ACC FK_GLACCTRN__GLACC FK_GLACC__GLACCTP FK_GLACC__CCY FK_GLACC__ACCSTAT FK_FXRX_FX_CCY FX FK_FXRX__FXRXTP FK_FXRX__CCY FK_ACCTRN_MERCH_PT MERCH FK_ACCTRN_CNTPTT_ACC CNTPTT FK_ACCTRN_CNTPT_PTBANKCONT CNTPT FK_ACCTRN__TRNPURP FK_ACCTRN__PTBANKCONT FK_ACCTRN__POS FK_ACCTRN__CRDB FK_ACCTRN__CNL FK_ACCTRN__CCY FK_ACCTRN__CARD FK_ACCTRN__ACCTRNTP FK_ACCTRN__ACCTRNSTAT FK_ACCTRN__ACC FK_ACCRWAFACT__RWATP FK_ACCRWAFACT__ACCSTDTP FK_ACCPROVFACT__ACCSTDTP FK_ACCINTRS__PERFRQ FK_ACCINTRS__INTRSRXTP FK_ACCINTRS__INTRSBASRX FK_ACCINTRS__ACCINTRSTP FK_ACCFTRTRN_MERCH_PT MERCH FK_ACCFTRTRN__POS FK_ACCFTRTRN__CNL FK_ACCFTRTRN__BLOCTP FK_ACCFTRTRN__BLOCSTAT FK_ACCFTRTRN__AUTHSTAT FK_ACCFTRTRN__ACC FK_ACCBALFACT__ACCSTDTP FK_ACC__CCY FK_ACC__ACCTP FK_ACC__ACCSTAT <<Ref Table>> Account Status (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Status Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Account <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Key Account Type Key Account Status Key Currency Key GL Account Key POS Key Account Number Account Name IBAN Open Date Activation Date Close Date Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) DATE DATE DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk2> <fk1> <fk3> <fk4> <ak> <pk,ak,fk4> <<Ref Table>> Account Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Currency (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Currency Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <<Ref Table>> Accounting Standard Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Accounting Standard Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Account Balance Fact <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Snap Date Account Key Accounting Standard Type Key Balance Overdraft Balance Reserve Balance Planned Balance Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime DATE INTEGER INTEGER NUMBER(19,3) NUMBER(19,3) NUMBER(19,3) NUMBER(19,3) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk,ak> <pk,fk2> <fk1> <ak> <pk,ak,fk2> <<ADS Table>> Account Future Transaction <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<DW Column>> Account Future Transaction Key Transaction Date Account key Blocation Type Key Blocking Status Key Authorization Status Key Merchant Party Key POS Key Channel Key Blocking Reference Number Blocking Amount Expiry Blocking Date Blocking Description Transaction Value Date Transaction Entry Date Transaction Entry DateTime Source Identifier Source System Identifier Update Datetime Update Process Identifier Delete Flag Insert Process Identifier Insert Datetime Update Effective Date Source Update DateTime INTEGER DATE INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) NUMBER(19,3) DATE VARCHAR2(255 CHAR) DATE DATE DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) DATE VARCHAR2(255) INTEGER VARCHAR2(255) DATE DATE DATE <pk> <pk,ak> <fk1> <fk4> <fk3> <fk2> <fk7> <fk6> <fk5> <ak> <pk,ak,fk1,fk6,fk7> <<Ref Table>> Authorization Status (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Authorization Status Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Blocking Status (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Blocking Status Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Blocation Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Blocation Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Channel (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Channel Key Identifier Description Local Description Source Identifier Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> POS (<ABDM_DWH_CLIENT_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> POS Key Party Key POS Type Key Organisation Unit Key POS Identifier POS Description Opening Hours Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Source Update DateTime Update Effective Date INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk3> <fk2> <fk1> <ak> <pk,ak,fk1,fk3> <<ADS Table>> Party (<ABDM_DWH_CLIENT_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Party Key Unified Party Key Party Type Key Party Status Key Business Sector Key Legal Form Key Country Key Language Key Housing Type Key Gender Key Personal Identifier Company Identifier P Code First Name First Name Latin Family Name Family Name Latin Middle Name Business Name Business Name Latin Short Name Short Name Latin Salutation Birth Date Resident Flag Bankruptcy Flag Start Date End Date Source System Identifier Source Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) DATE INTEGER INTEGER DATE DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk8> <fk7> <fk6> <fk1> <fk5> <fk2> <fk4> <fk3> <fk9> <pk,ak,fk8> <ak> <<Ref Table>> Account Interest Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Interest Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Account Interest <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Interest Key Account Key Account Interest Type Key Interest Rate Type Key Interest Base Rate Key Period Frequency Key Interest Rate Interest Limit amount Interest Start Date Interest End Date Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER NUMBER(10,6) NUMBER(19,3) DATE DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk5> <fk1> <fk3> <fk2> <fk4> <ak> <pk,ak,fk5> <<Ref Table>> Interest Base Rate (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Interest Base Rate Key Period Frequency Key Identifier Description Local Description Market Flag Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <fk> <ak> <ak> <<Ref Table>> Interest Rate Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Interest Rate Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Period Frequency (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Period Frequency Key Period Code Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Account Provision Fact <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Snap Date Account Key Accounting Standard Type Key Provision Total Balance Provision Principal Balance Provision Interest Balance Provision Fee Balance Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime DATE INTEGER INTEGER NUMBER(19,3) NUMBER(19,3) NUMBER(19,3) NUMBER(19,3) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk,ak> <pk,fk2> <fk1> <ak> <pk,ak,fk2> <<ADS Table>> Account RWA Fact <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Snap Date Account Key Accounting Standard Type Key RWA Type Key RWA Exposure RWA Rate RWA Balance Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime DATE INTEGER INTEGER INTEGER NUMBER(19,3) NUMBER(10,6) NUMBER(19,3) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk,ak> <pk,fk3> <fk1> <fk2> <ak> <pk,ak,fk3> <<Ref Table>> RWA Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> RWA Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Account Transaction <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<DW Column>> Account Transaction Key Transaction Date Account Key Card Transaction Location Key Card Key Party Bank Contact Key Counterparty Account Key Counterparty Bank Contact Key GL Account Key Credit/Debit Key Account Transaction Type Key Account Transaction Status Key Transaction Purpose Key Currency Key Channel Key POS Key Merchant Party Key Transaction Reference Number Transaction Batch Identifier Transaction Amount Transaction Amount Local Currency Transaction Amount Account Currency Transaction Account FX Rate Transaction Value Date Transaction Entry Date Transaction Entry DateTime Client Internal Transaction Flag Cancel Flag Reversal Flag Message For Recipient Message For Sender Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER DATE INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) NUMBER(19,3) NUMBER(19,3) NUMBER(19,3) VARCHAR2(255 CHAR) DATE DATE DATE INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <pk,ak> <fk1> <fk14> <fk4> <fk9> <fk12> <fk11> <fk7> <fk3> <fk2> <fk10> <fk5> <fk6> <fk8> <fk13> <ak> <pk,ak,fk1,fk4,f...> <<Ref Table>> Account Transaction Status (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Transaction Status Key Account Transaction Business Status Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<Ref Table>> Account Transaction Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Account Transaction Type Key Account Transaction Category Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <fk> <ak> <ak> <<ADS Table>> Card (<ABDM_DWH_PRODUCT_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Card Key Product Key Card Type Key Card Status Key View Card Number Card Identifier Card Name Activation Date Expired Date Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) DATE DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <ak> <pk,ak> <<Ref Table>> Credit/Debit (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Credit/Debit Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> Party Bank Contact (<ABDM_DWH_CLIENT_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Party Bank Contact Key Party Key Bank Contact Type Key Institution Party Key Bank Contact Number Specific Symbol Variable Symbol Constant Symbol IBAN Bank Account Name Swift Fee Bank Identification Code Valid Flag Source System Identifier Source Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk3> <fk1> <fk2> <pk,ak,fk2,fk3> <ak> <<Ref Table>> Transaction Purpose (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Transaction Purpose Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> FX Rate : 1 <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> FX Rate Key Currency Key FX Currency Key FX Rate Type Key FX Scale Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk1> <fk3> <fk2> <ak> <pk,ak> <<Ref Table>> FX Rate Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> FX Rate Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> GL Account <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> GL Account Key Account Status Key Currency Key GL Account Type Key GL Account Number GL Account Group Description Party Account Flag Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk1> <fk2> <fk3> <ak> <pk,ak> <<Ref Table>> GL Account Type (<ABDM_DWH_REF_TAB_ADS>) <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> GL Account Type Key Identifier Description Local Description Source ID Source System ID Delete Flag Insert Datetime Insert Process Identifier Update Datetime Update Effective Date Update Process Identifier INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE VARCHAR2(255) DATE DATE VARCHAR2(255) <pk> <ak> <ak> <<ADS Table>> GL Account Transaction <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> GL Account Transaction Key Transaction Date GL Account Key Cost Code Key Cost Centre Key Cost Project Key Invoice Transaction ID Invoice number Invoice Document Identifier Debit/Credit Key GL Transaction Date GL Transaction Amount GL Transaction Amount Local Currency Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER DATE INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER DATE NUMBER(19,3) NUMBER(19,3) VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <pk,ak> <fk> <ak> <pk,ak,fk> <<ADS Table>> FX Rate : 2 <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> FX Rate Key Currency Key FX Currency Key FX Rate Type Key FX Scale Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime INTEGER INTEGER INTEGER INTEGER INTEGER VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk> <fk1> <fk3> <fk2> <ak> <pk,ak> <<ADS Table>> FX Rate Fact <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<DW Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> <<Audit Column>> Snap Date FX Rate Key Rate Buy Rate Rate Sell Value Date Source Identifier Source System Identifier Delete Flag Insert Process Identifier Insert Datetime Update Process Identifier Update Datetime Update Effective Date Source Update DateTime DATE INTEGER NUMBER(10,6) NUMBER(10,6) NUMBER(10,6) DATE VARCHAR2(255 CHAR) VARCHAR2(255 CHAR) INTEGER VARCHAR2(255) DATE VARCHAR2(255) DATE DATE DATE <pk,ak> <pk,fk> <ak> <pk,ak,fk>
- 34. Star Schema vs. Snowflake Schema Source: Wikipedia
- 35. Records Primary groups Candidate groups John Smith null John Smith null Jane Smith 420347213 Jane Watson 420347213 J Smith 420347213 J Smith null Jane Watson 420347213 John Smith 095252433 John Smith 095252433 John Smith 095242434 John Smith 095242434 Janette Smith null Secondary groups ? Unique Kandidátské skupiny (ilustrace)
- 36. Velké řešení => Komplexní Governance nutná Concepts Vision & Mission Guiding Principles Organization & Roles Business Rules Activities Scope Benefits & Goals Components Data Architecture Data Quality Data Integration Operations Security RDM & MDM Metadata Data Platform & BI Tools CASE Enteprise Metadata Repository Data Quality Tools QA Framework Workflow & Orchestration IDE Audit Log Resource Management RDBMS NoSQL Hadoop Integration tools Monitoring Source Code Repository Testing Tools Others Why What How
- 37. Nesprávně implementovaný Data Lake = Data Swamp 37 Jezero je jen centrální uložiště s otevřeným modelem
- 38. Shrnutí a doporučení pro zavedení DataOps DataOps je naprostá nutnost, protože stále selhávají 2 ze 3 analytických projektů (Data Kitchen) DataOps je data management framework zaměřený na zlepšení a zrychlení komunikace, datové integrace a automizace datových toků Reálná zavedení DataOps potvrzují smysluplný přínos ve více než 80% případů (výzkum Research 451) Bez DataOps nelze uvažovat o efektivní daty řízené kultuře (data-driven culture) DataOps nejde koupit (ikdyž existují „DataOps nástroje“), ale musí se vybudovat jako integrální součást organizace (třeba pomocí „DataOps nástrojů“) DataOps klade velký důraz na kontinuální dodávku hodnoty pomocí datová analytiky (Value Pipeline) a její průběžné rychle inovaci (Inovation Pipeline) sanboxing a self-service DevOps Agile Data Management Lean Manufacturing DataOps Innovation Pipeline Value Pipeline Value Prototyping Verification Standardization Analytics Domain Quality Data Datové a logické testy Version Control System Branch & Merge Více prostředí Parametrizace zpracování Práce beze strachu a hrdinství Datová architektura DataOps Metriky Komunikace
- 39. 39 Zrychlování a zkvalitňování datových skladů pomocí samočinných nástrojů a procesů Soustředění se více na data místo rutinních věcí nějakých souvisejícími s daty Automatizace vývoje (Development) Vyšší produktivita vývojářů => rychlejší dodávky Konzistence postupů a standardů => lépe udržovatelná řešení Automatizace usnadňuje použití agilních přístupů Standardizovaný testovací proces zajišťuje kontinuální Quality Assurance Snadnější vývoj a prototypování umožnují snadnější reakce na změny Snadná impact analýza změn datového skladu díky metadatům Základní typy Model Driven Data Driven Automatizace provozu (Operations) Nasazovací proces je zjednodušený a postavený na balíčcích omezující ruční práci Dokumentace se generuje automaticky a je konzistentní s aktuálním releasem Snadná impact analýza dopadů provozních změn na datový sklad a koncové uživatele Enterprise rozšíření zajišťující delší životnost řešení Robustnější standardizované procesy zajištující stabilnější a kvalitnější provoz Lepší bezpečnost díky Quality Assurance, standardům a postupům data Warehousing Automatizace (DWA) Adastra Ajilius AnalytiX DS Attunity Compose (Biready) BI Builder BI builders biGenius Birst Centennium Automation Tool Datavault Builder DDM Studio Dimodelo Effektor Gamma Systems Halo BI Insource Data Academy Instant Business Intelligence (SeETL) Kalido LeapFrogBI Optimal ODE Quipu TimeXtender Varigence WhereScape
- 45. Současné výzvy Data Managementu Zdroje: 451 Research, DataOps: the foundation for agility, security and transformational change, March 2019 Data Kitchen, Washington DC DataOps Meetup, 2019 87% of data science projects never get to production. Data analytics investment up, but “data driven” organizations down 37% to 31% 60% of all data analytic projects fail 79% of data projects have too many errors
- 46. Metadata Glue 46
- 47. 2005 & 2019 Side by Side Business Intelligence Data Sources ERP CRM External Systems Internal Systems Analytics Reporting OLAP Data Mining Data Integration ETL EAI Industry Know-How Database Data Warehouse Data Mart Operational Data Store Staging Area End User Access Intranet EIS & Monitoring Analytics Tools Others Management Technical Expertise Data Quality Metadata Analytics Department Customer Care Others Enrichment & Consolidation & Event Processing MDM DQ Reference Data Management Complex Event Proccesing Message Requeueing DMP Data Acquistion & Data Ingest Speed Processing Batch Processing Change Data Capture Direct Data Extractor Bulk Copy Publisher/Subscriber Data Sources Relational Data Semi-Structured Data Unstructured Data Streams Events Signals User Files Analytics Statistics OLAP Advanced Analytics Artificial Intelligence Machine Learning Stream Analytics Geospatial Analytics Data Integration ETL ELT Big Data ELT Data API Microservices Self-service ETL Real-time Integration Governance Data Model Data Strategy Data Delivery Architecture Methodology Standards Metadata Management Data Catalogue Data Lineage Business Glossary Documentation Information Lifecycle Testing Strategy BICC Data Store Data Warehouse Data Mart Data Lake ODS NoSQL Sandbox Event Hub Big Data Platform In-memory Columnar Data Access Data Connector Query Engine Data API Web GUI Application Integration Mobile Applications Indexing & Search Business Intelligence Reports Ad-hoc Query Dashboard Data Visualization Data Discovery Self-Service BI Mobile BI Data Science GUI Business Users & Applications Development & Operations Monitoring Alerts & Notification Scheduling Workflow Security Resource Management Release Management High Availbility Backup & Restore Data Purge Automation Metadata Driven Development
- 48. Truth in data Primary data Primary data (another system) Secondary data Consolidated data …Noise generator Truth Independent truth in data does not exist Truth depends on Business and Data architect definition
- 49. Other Topics – DW vs. Business Intelligence – DW vs. Operational Data Store – DW vs. Master Data Management – DW vs. Big Data – Metadata – Data Lineage – Data Governance – Implementation – Data Modelling – Mapping – Parellel Processing – Metadata Driven Development – Information Delivery – KPIs, Metrics, Dimensions – Data Analytics – Semantic Data Layer – Self-service BI – Data Virtualization / Data Federation – Operations – Automation – Workflow – Disaster Recovery – Technologies 49
- 50. Data Warehousing & Business Intelligence Data Platform A Data Warehouse, a Data Lake, a Big Data Platform or Data anything for storing and managing data for analytics. Data Integration Processes combining and transforming data from different sources and providing consolidated structures of data in motion and data at rest. Data Analytics Processes of inspecting, transforming, modeling data in motion and data at rest. Data Science is included. Data Governance A framework to ensure the appropriate behavior in the valuation, creation, integration, storing, consumption and control of data and analytics. DataOps An automated methodology to improve the quality and reduce the cycle time of data analytics based on Agile, DevOps and Lean Manufacturing Reporting & Business Intelligence Presenting data to end-users in a way that is understandable and actionable. Technical Solutions Business Solutions General Augmented Analytics Data Discovery Data Storytelling Data for Planning External Data Enrichment Self-service BI Finance Budgeting & Planning Business Performance Reporting Profitability Analytics Risk Management Fraud Detection Loan Classification Portfolio Reporting Risk Based Pricing Risk Modeling CRM & Marketing Campaign Monitoring Churn Prevention Customer Lifetime Value Customer Segmentation Geolocation Analytics Know Your Customer Network Analytics Omnichannel Communication Sentiment Analytics Sales Product Propensity Sales Network Performance Up-sell & X-sell Others HR Attrition Predictive Maintenance Quality Assurance
- 51. Data Warehousing řeší LEGO