The document discusses the components of a data warehouse, including: - Data stores such as the data warehouse itself, data marts, operational data stores, and big data platforms. - Data integration tools for extracting, transforming, and loading data from various sources. - Access tools for querying, reporting, visualization, and advanced analytics. - Metadata for technical, business, and transformation documentation. - Administration and management functions like operations, security, and quality assurance. - Development tools for modeling, ETL design, and testing.

1. Komponenty datových skladů
#11
23. 1. 2017
Prague Data Management Meetup

2. Agenda
• Prague Data Management Meetup
• Komponenty datových skladů

3. Prague Data Management Meetup
Data Management
Získávaní dat
Ukládání dat
Zpracování dat
Interpretace dat
Použití dat
• Otevřená profesionální zájmová
skupina
• Každý je vítán (ať už v pasivní
nebo aktivní roli)
• Témat není nikdy dost
• Snaha o pravidelné měsíční
setkávání
• Fungujeme od září 2015

4. Historie
Datum Téma
10. 9. 2015 Data Management
14. 10. 2015 Data Lake
23. 11. 2015 Dark Data (without Dark Energy and Dark Force)
12. 1. 2016 Data Lake (znova)
7. 3. 2016 Sad Stories About DW Modeling (sad stories only)
23. 3. 2016 Self-service BI Street Battle
27. 4. 2016 Let's explore the new Microsoft PowerBI!
22. 9. 2016 Data Management pro začátečníky
17. 10. 2016 Small Big Data
22. 11. 2016 Základy modelování DW
23.1.2017 Komponenty datových skladů

5. Data Management

6. Big Data

7. Data Warehouse
• Data integration from various data
sources in requested quality and
time
• Publish and share consistent
information for purposes and users
• Flexible and effective ad-hoc
reporting a analysis
• Features
• Subject Orientation
• Data Integration
• Low variability
• History
• Main perspectives
• Data Integration
• Data Storage
• Data Access
• New oppurtunities
• Complex Event Processing in real time
• Application Integration
• Real-time decision support
• Operational Data Store
• Integration with Big Data Platform
EDW, DW, DSS, ADS, ADW, DP…

8. Operational Database vs. Data Warehouse
Characteristic Operational Database Data Warehouse
Currency Current Historical
Details level Individual Individual and summary
Orientation Process Subject
Records per request Few Thousands
Normalization level Mostly normalized Normalization relaxed
Update level Highly volatile Mostly refreshed (non volatile)
Data model Relational Relational (star schemas) and multidimensional (data cubes)
Source: Coursera

9. Adastra Business Intelligence Reference Architecture
9
ODS
Operational
reporting
Enterprise DWH Big Data
Platform
Data Lake
Event
Processing
Semantic
Models
Advanced Analytics
Perceptual / cognitive intelligence
Information Management
Relational / Structured data Unstructured data Streaming
Data Workflow
Orchestration
Data Transformation /
Processing
Data
Management
Event Ingestion
Complex Event
Processing
Notifications
BI / Application
Integration
Machine Learning
In-database Data Mining, R
Recognition of human
interaction and intent
SMP and MPP
In-memory technologies
In-memory Columnar
In-memory technologies Hadoop, NoSQL
Business Intelligence / Data Delivery
Real-time DashboardsDashboards and visualizationsReports Self-service BIMobile BI
IoT Network
Field Gateway
Big data
OLAP

11. DW Logical Layers
L0: Stage Area
L1: Relational Area
L1: Consolidation
Area
L2: Data Mart Area
• Data Mart Area
• L2
• User Access Layer
• Consolidation Area
• Consolidated L1
• Common aggregates for L2
• Cleansed and consolidated data
• Relational Area
• Detailed L1
• Consistent, integrated, subject oriented data,
universal data structure, historical data, maximal
detail
• System of record
• Foundation Layer
• Stage Area
• Direct copy of source systems
Extracts
Reports
Note: Consolidated and Detailed L1 can
share same data structures
General DWH
Staging Area ODS
Presentation Layer
Datamart Area (Dependent Datamarts)
Source systems
Customer
DB
ETL
Other...S4S3S2S1
Analytic tools
(SPSS, SAS..)
OLAP
S1 S2 S3 S4 Other
S1 Ostatní...S4S3S2
ETL
Materialization
OLAP?
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
CDB
ETL
EAI
ReportingReporting Reporting Reporting
Relational Area
ETL
Application Application
Materialization
Application Application
ETL

12. Data Warehouse Components
Data Stores Access Tools Metadata
Data
Integration
Tools
Administration
and
Management
Development
Tools
Not only technology!

13. Data Stores
Logical Stores
• Data Warehouse
• Data Mart
• Operational Data Store
• Customer Data Integration
• Product Data Integration
• Data Hub
• Data Lake
• Data Archive
• Big Data Platform
Physical Stores
• RDBMS
• OLAP
• HDFS
• NoSQL
• SMP
• MPP
• Cluster
• Appliance

14. Data Integration Tools
Custom Scripts
ELT vs. ETL
Real-time
Change Data Capture
Logical Mapping
Physical Mapping
Workflow
Dependencies
Restartability
Error Handling
Scheduling
Events
Monitoring
QA
Testing
Design Patterns
Design Standards
Kappa Architecture
Lambda Architecture
Data Quality
Matching
Data Cleanup
Data Profiling
Data Checks

15. Data Sources and Targets (ODI Examples)
Apache Derby Apache HDFS Apache Hive Apache HBase Cloudera CDH dBase
HyperSQL
Database Engine
IBM DB2
IBM DB2 for
Linux Unix and
Windows
IBM DB2 for i IBM DB2 for z/OS IBM Informix
IBM Informix
Dynamic Server
(DS)
IBM Informix
Extended Parallel
Server (XPS)
IBM Netezza NPS
IBM Websphere
MQ
Ingres InterBase
ISO Database
Language SQL
(generic SQL-92
database)
Java Message
Service (JMS)
Microsoft Access Microsoft Excel
Microsoft SQL
Server
MySQL Server
Oracle Database Oracle Essbase
Oracle Hyperion
Planning
Oracle Service
Bus
Oracle TimesTen
In-Memory
Database
Paradox Pervasive PSQL PostgreSQL
SAP BW SAP ERP ECC
Teradata
Database
Textové soubory XML soubory JSON soubory

16. Batch Data Transformation: ETL vs. ELT
Extract Load Transformation
Extract Transformation Load

17. Data Transformation Model Pattern
Source table
Target table
table
Target table
Filter
SRC
Filter
TRG
Differential
member
(minus or
outer join or
Merge)
Filter
OUT
Lookup tables
Join SRC

18. Lambda Architecture Kappa Architecture

19. Oracle Data Integrator

20. Pentaho Data Integration

21. Metadata
Technical
metadata
Business
metadata
Static metadata
Dynamic
metadata
Context
metadata
Data model
Transformation
model
Methodology

23. Design Standards
• Naming conventions
• Models, nodes, layers, schemas
• Entities
• Attributes
• Keys
• Relationships
• Indexes, constraints
• Level of normalization
• Level of convergence
• Key strategy
• Standard attributes
Supplier standards
Customer standards
Third party product
standards

24. Typical Metadata Backbone
Primary generic
principles of the data
warehouse solution
(goals and objectives)
Logical architecture
document
Physical architecture
document
Environment map
Logical data model
name and design
standard
Data history and
retention strategy
Physical data model
design standard
including partitioning
List of acronyms Business rule document KPI catalogue
Transformation design
standard
Process definition for
analysis, design,
development, testing,
release management
and bug fixing
Templates for analysis,
design, development,
testing, release
management and bug
fixing
Test catalog Developer guideline
Operation guideline
including DR strategy
Quality assurance SLA templates
Information delivery
strategy
Data Mart strategy Data Quality strategy
Business entity mapping
to data model with
examples for end users
Business data domain
document
Security architecture

25. Access Tools
• Microsoft Excel rulez
• SQL Query Tools
• Enteprise Reporting Tools
• Self-service BI
• Data Discovery
• Data Mining Tools (Weka, Azure
ML..)
• Statistics Tools (SPSS, SAS, R…)
• Information Delivery
• Real-time decision
• Application integration

27. Oracle Business Intelligence
• On-premise i cloud varianta
• Podpora pro pokročilou analytiku, self-service
vizualizace i Mobile BI
Oracle Big Data Discovery
• Nativní self-service analytika pro Big Data řešení
27

28. Tableau Desktop Qlik Sense Desktop
Microsoft Power BI Desktop SAS Visual Analytics

29. Administration and Management
Daily operations Environments Data quality checks
Managing and
updating metadata
Auditing and
reporting data
warehouse usage
and status
Purging data
Replicating, sub-
setting and
distributing data
Backup and
recovery
Data warehouse
storage
management.
Bug tracking and
fixing

30. DW Stacks
Stack Others
RDBMS
Oracle Database
MySQL
Microsoft SQL Server
Microsoft SQL Server APS
Azure SQL Data Warehouse
Amazon Redshift
HP Vertica
IBM dashDB
IBM DB2
PostgreSQL
SAP HANA
SAP IQ
SAP SQL Anywhere
Teradata Database
ETL/ELT
Oracle Data Integrator
Oracle Golden Gate
MS Integration Services
Azure Data Factory
Clover ETL
IBM InfoSphere DataStage
Informatica PowerCenter
Pentaho Data Integration
SAP Data Services
SAS Data Integration
Talend Data Integration
BI & Analytics
Oracle Big Data Discovery
Oracle Business Intelligence
Oracle Endeca Data Discovery
Oracle Essbase
Oracle R Enterprise
Azure Machine Learning
MS Analysis Services
MS Datazen
MS Excel BI
MS Power BI
MS Reporting Services
Revolution R
Amazon QuickSight
GoodData
IBM Cognos Reporting
IBM Watson Analytics
Microstrategy Analytics
Qlik Sense
Qlikview
SAP Business Objects
SAS Visual Analytics
Tableau
Teradata Aster Discovery Platform
Appliances
Oracle Exadata
Oracle SuperCluster
MS Analytic Platform System
IBM Netezza Twinfin
SAP HANA
Teradata Data Warehouse Appliance
HP Vertica Analytics System

31. Microsoft Stack
31
Excel + Power BI add-ins
Query, Pivot, View, Map
SharePoint
Power Pivot Gallery, Power View
Excel
Data Mining
Power BI Desktop Power BI Portal
Azure ML
End-to-End DW & Big Data Platform, Driving Analytics on any Data
Power BI Mobile App
Analytics Platform System
(APS)

32. Oracle Stack

33. Data Warehouse Components
Data Stores Access Tools Metadata
Data
Integration
Tools
Administration
and
Management
Development
Tools
Which ones are missing?
People? BICC?