Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

A company of Daimler AG
LECTURE @DHBW: DATA WAREHOUSE
PART IV: FRONTEND, METADATA,
PROJECTS, ADVANCED TOPICS
ANDREAS BUCKENHOFER, DAIMLER TSS

ABOUT ME
https://de.linkedin.com/in/buckenhofer
https://twitter.com/ABuckenhofer
https://www.doag.org/de/themen/datenbank/in-memory/
http://wwwlehre.dhbw-stuttgart.de/~buckenhofer/
https://www.xing.com/profile/Andreas_Buckenhofer2
Andreas Buckenhofer
Senior DB Professional
andreas.buckenhofer@daimler.com
Since 2009 at Daimler TSS
Department: Big Data
Business Unit: Analytics

DAIMLER TSS. IT EXCELLENCE: COMPREHENSIVE, INNOVATIVE, CLOSE.
We're a specialist and strategic business partner for innovative IT Solutions within Daimler –
not just another supplier!
As a 100% subsidiary of Daimler, we live the culture of excellence and aspire to take an
innovative and technological lead.
With our outstanding technological and methodical competence we are a competent provider of
services that help those who benefit from them to stand out from the competition. When it
comes to demanding IT questions we create impetus, especially in the core fields car IT and
mobility, information security, analytics, shared services and digital customer
experience.
Data Warehouse / DHBWDaimler TSS GmbH 3
TSS 2 0 2 0 ALWAYS ON THE MOVE.

Daimler TSS GmbH 4
LOCATIONS
Data Warehouse / DHBW
Daimler TSS China
Hub Beijing
6 Employees
Daimler TSS Malaysia
Hub Kuala Lumpur
38 Employees
Daimler TSS India
Hub Bangalore
16 Employees
Daimler TSS Germany
6 Locations
More than1000 Employees
Ulm (Headquarters)
Stuttgart Area
Böblingen, Echterdingen,
Leinfelden, Möhringen
Berlin

• After the end of this lecture you will be able to
• Understand function of Frontend Tools and Information Design
• Understand the necessity for metadata
• Understand lifecycle of DWH projects
• Advanced topics like Operational BI, DWH Appliances, Cloud BI
WHAT YOU WILL LEARN TODAY
Data Warehouse / DHBWDaimler TSS 5

LOGICAL STANDARD DATA WAREHOUSE ARCHITECTURE
Data Warehouse
FrontendBackend
External data sources
Internal data sources
Staging
Layer
(Input
Layer)
OLTP
OLTP
Core
Warehouse
Layer
(Storage
Layer)
Mart Layer
(Output
Layer)
(Reporting
Layer)
Integration
Layer
(Cleansing
Layer)
Aggregation
Layer
Metadata Management
Security
DWH Manager incl. Monitor

• Reporting (Standard, ad-hoc)
• OLAP
• Dashboards, Scorecards
• Advanced Analytics / Data Mining / Text Mining
• Search & Discovery
INTERFACE TO THE END USER

Standard Reports
• Prepared static reports that can be executed at request by end users
• Are executed at the end of an ETL process and e.g. send by email to end users
• Normally based on fact tables and its dimensions
• Reports are often lists similar to Excel-Sheets but can also contain graphics (e.g. line
charts)
Ad-hoc Reports
• End users create their own reports („Self service“)
REPORTING (STANDARD, AD-HOC)

ROLAP / MOLAP Client Frontend
• Prepared cubes (multidimensional or relational fact tables)
• User can perform interactive analysis of data
• Rollup / drill-down
• Pivot
• Slicing
• Dicing
OLAP

„Progress reports“
Provide an overall view of KPIs (Key Performance Indicators)
Combination of several elements from Reporting and/or OLAP (e.g. line
charts) into an overall view (like a „cockpit“)
Dashboard is more focused on operational goals
• High-level overview what is happening
Scorecard is more focused on strategic goals
• Plan a strategy and identify why something happens
DASHBOARDS, SCORECARDS

See Mr. Bollinger‘s lecture
ADVANCED ANALYTICS / DATA MINING / TEXT MINING

Not just numerical data
Analysis of new data types gets more and more important
• Text
• GPS coordinates
• Pictures
• Videos
Data can be available in RDBMS (e.g. text modules/indexes available),
Hadoop or SQL DBs
SEARCH & DISCOVERY

MANY GRAPHICAL ELEMENTS TO USE IN REPORTS
Source: https://github.com/d3/d3/wiki/Gallery

MANY GRAPHICAL ELEMENTS … CHAMBER OF HORROR
Source: Hichert / Faisst, http://www.backup-page.hichert.com/

Some remarks about previous slide
• 3D elements introduce clutter and give not more information
• Pie chart most often does not make sense
• Line chart barely readable
• Labels are placed outside of the graphic
• Tachometer costs a lot of space and show
• Too much color in general
• Color without meaning, e.g. red should be used for alarms / errors
MANY GRAPHICAL ELEMENTS … CHAMBER OF HORROR

STORY TELLING WITH APPROPRIATE VISUALIZATION
Famous example by Hans Rosling (watch 3:08 onwards)
https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen?language=de

Information design is the practice of presenting information in a way that
fosters efficient and effective understanding of it.
(source: Wikipedia, https://en.wikipedia.org/wiki/Information_design )
Some authors are well known for their criticism of many graphical
representations - they provide rules for good information design
• Edward Tufte
• Stephen Few
• Rolf Hichert
INFORMATION DESIGN

Define standards, e.g.
• use always the same colors and with care (red = negative, green = positive)
• pie charts are rarely useful and should be avoided (better use bar chart or line chart)
• No 3D elements as these elements don’t enhance information but introduce clutter
INFORMATION DESIGN

TABLE WITH INTEGRATED BAR CHARTS
Source: Hichert, http://www.hichert.com/de/resource/table-template-02/

Consumers / BI Users
• use reports, OLAP and dashboards to obtain information
Power Users
• Use reports , OLAP and dashboards to obtain information
• Create new reports and dashboards
Data Scientists
• Statistical / mathematical geeks
• Analyze / explore data
• Need to analyze raw (non-cleansed, non-transformed) data
BI END USER ROLES

WHAT IS METADATA?
Data
about
other data

Business Metadata
• Definition of business vocabulary and relationships
• Definition of the value range
• Linkage to physical representation
TYPES OF METADATA

Report metadata
• Report definitions
• Data sources
• Column definitions
• Computations
Logical and physical metadata of data model
• Table structure
• Definition of columns
• Relationships between tables and columns
• Dimension hierarchy
TYPES OF METADATA

THE AREAS OF METADATA

THE AREAS OF METADATA CONNECTED

Components of a data warehouse system are interconnected
• BI report user has to know
• the meaning, definitions of the shown measures, „KPIs“ (key performance indicators)
• BI report designer has to know
• the table definitions
• the meaning of the column values
• ETL job designer has to know
• the table definitions or the exact definition of the measures
• Database administrator has to know
• Which tables are used by ETL jobs, reports
WHY A COMMON METADATA REPOSITORY?

Metadata driven ETL development
• Generate parts of ETL code
• increasing interest for Data Vault development projects
• Tools e.g. MID Innovator, Quipu, AnalytiX DS, Talend, Pentaho, Wherescape, and others
Common metadata repository ensures consistency across all components
• Many tools involved (DB, ETL, Frontend, …)
Enables cross component metadata analysis
• Data Lineage
• Impact Analysis

“Data lineage”
• Import & Browse Full BI Report Metadata
• Navigate through report attributes
• Visually navigate through data lineage across tools
• Combines
operational &
design viewpoint

“Impact Analysis”
• Show complete change impact in graphical or list form
• Includes impact on reports in BI tools
• Visually navigate through impacted objects across tools
• Allows impact analysis on any object type
WHAT HAPPENS IF I CHANGE THIS COLUMN?

• Show relationships between business terms, data model entities, and
technical and report fields
• Requires cross-tool mapping of business terms
• Allows field meaning to be understood
• Allows business term relationships to be understood
WHAT DOES THIS FIELD MEAN?

• Shows objects that user manages
• Shows stewardship relationships on business terms
• Shows user group associations
WHAT OBJECTS DOES THIS USER OWN?

• Navigation through
complete job details
• Navigation of complete
operational metadata
WHAT HAPPENED ON THE LAST JOB RUN?

LOGICAL STANDARD DATA WAREHOUSE ARCHITECTURE
Data Warehouse
FrontendBackend
External data sources
Internal data sources
Staging
Layer
(Input
Layer)
OLTP
OLTP
Core
Warehouse
Layer
(Storage
Layer)
Mart Layer
(Output
Layer)
(Reporting
Layer)
Integration
Layer
(Cleansing
Layer)
Aggregation
Layer
Metadata Management
Security
DWH Manager incl. Monitor
Top Down (Inmon)
Bottom Up (Kimball)

Top-Down (Inmon)
• Comprehensive approach regarding available data
• Design Core Warehouse Layer = integrated data model first considering all
requirements
• Design data marts afterwards
Bottom-Up (Kimball)
• Approach focusing on fast delivery of first results
• Design one data mart first
• Next Marts are modeled afterwards usually using Kimball architecture
• conformed dimensions to integrate different data marts / fact tables
TOP-DOWN VS BOTTOM-UP APPROACH

TOP-DOWN VS BOTTOM-UP APPROACH
ADVANTAGES AND DISADVANTAGES
Top-Down (Inmon) Bottom-Up (Kimball)
☺ Core Warehouse Layer is designed optimal ☺ Early involvement of end users
☺ Data from Core Warehouse Layer is reused in many
Marts
☺ Fast results
Time-consuming approach with high preparatory effort Focus on single Marts leads to risk that overall view is
lost, esp. properly designed Core Warehouse Layer
High risk with changing requirements Data often not reused but inconsistently copied across
Marts

Both approaches have their down-sides
• Top-Down takes enormous initial effort to build data model for Core Warehouse Layer
• Bottom-Up is risky as central / integrated focus is lost
Think big, start small
• Think Big: Design conceptual data model for Core Warehouse Layer
covering whole enterprise
• Start small: Implement physical data model for Core and Mart Layer in
iterations by each business department
THINK BIG, START LOCAL

• Classical
• Waterfall model
• Incremental model
• Agile
• Scrum
• Kanban
PROCESS MODEL

• DWH is not a system or product
• DWH databases are more complex with different layers and data models
• Data first, code is secondary
• Data quality is a major concern
• Data integration is a challenging objective
• Business need difficult to justify quantitatively
WHAT’S DIFFERENT IN DWH PROJECTS?

WHY DO DWH PROJECTS FAIL?

Feasibility study Analysis Design Implementation Test
Operations and
maintenance
PROJECT PHASES

• Benefits of DWH
• Cost-effectiveness
• SW selection
• HW selection
• Staff requirement including external Know-How
• Data protection and data security agreement, data classification
• Proof of Concept (PoC) to challenge different possible solutions
• Architectural concept
FEASIBILITY STUDY

• Documentation of user requirements: specification sheet
• Backend including ETL
• Frontend
• Security
• Metadata, Business glossary
• Non-functional requirements
• Analysis of data sources
• Data quality
• Data models
• Data security
ANALYSIS

• Technical description how to implement specifications
• Data model for different DWH layers
• Data integration design
• Frontend design
• Security concept
• Capacity planning
DESIGN

• Installation of development, test, integration, production, maintenance
environment
• Usage of Metadata repository for implementation of data model, etl,
frontend, security
• Launch of DWH
• Release Management
IMPLEMENTATION

• Functional
• Data quality / data validation
• Usability
• Performance
• Operational
• Security
TEST

• Deployment of new features, changes or bug fixes
• End user training
• Monitoring
• Production concept
• Initial load and future delta loads
• Keep the system running
OPERATIONS AND MAINTENANCE

Organizational team that coordinate and standardize DWH activities within an
(end user) organization
• Define standards and create BI portfolio (e.g. which tools/products to use)
• Create DWH architecture and govern BI activities
• Establish processes for business and IT interaction
• Monitor DWH/BI market for new trends
• Determine skills and experience of Business users
BICC: BI CENTER OF EXCELLENCE

Define 3-5 criteria for the evaluation of an ETL tool
How does a relational DBMS (like Oracle, DB2, MS SQL Server) meet these
requirements?
EXERCISE

• Supplier profile
• Support
• HW/SW requirements
• License / maintenance Costs
• Usability
• Reliability
• Performance and scalability
• Multi-tenant
• Interfaces
• Scheduling
EXERCISE - DEFINE 5 CRITERIA FOR THE EVALUATION OF
AN ETL TOOL

• RDBMS provide many of the functionalities but additional programming required
• RDBMS are often used for ETL/ELT by programming with SQL, PL/SQL, SQLT, etc
EXERCISE - HOW DOES A RELATIONAL DBMS MEET
THESE REQUIREMENTS?
ETL Tool Manual ETL
Informatica, Talend, Oracle ODI, etc. SQL, PL/SQL, SQLT, etc.
Separate license No additional license
Workflow, error handling, and restart/recovery
functionality included
Workflow, error handling, and restart/recovery
functionality must be implemented manually
Impact analysis and where-used (lineage) functionality
available
Impact analysis and where-used (lineage) functionality
difficult
Faster development, easier maintenance Slower development, more difficult maintenance
Additional (Tool-) Know How required Know How often available

NEWER / ADVANCED TOPICS
• OPERATIONAL DATA WAREHOUSING
• DATA WAREHOUSE APPLIANCES
• CLOUD BI

Classical“ Data Warehouses
• Information in the warehouse used to support strategic business decisions
• Kept separate from operational systems
• Load of new data only in larger intervals (mostly weekly or monthly)
• Shorter intervals not required by users
• Huge system resources of the ETL process
Near Real Time Operational Data Warehousing
• Information in the warehouse used for tactical business decisions as well
• Low latency of information in data warehouse therefore needed
• Not only mathematical aggregations
OPERATIONAL DATA WAREHOUSING

With classical data warehouses users have to access two types of systems to
get a complete image of a customer (for instance for CRM applications or in
call centers)
• the data warehouse to see what happened in the past
• the OLTP systems to get the most current information
With an operational data warehouse
• all this information is in one system
• tighter integration with operational systems is easier
• for instance personalized offers „closing the loop“
WHY OPERATIONAL DATA WAREHOUSING?

SMARTFACTORY: OPERATIONAL BI SERVICE PLATFORM
Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 279
Workers getting
alarms on their
watch
Containing and
displaying
complex manuals,
e.g. during repair
New data
sources
sending lots
of data with
high speed
(sensor data,
logs, etc.)
Right-Time
data required
for
automated
actions, e.g.
cordless
screwdriver
knows and
adjusts
torque

Near Real time /Right time ETL
• Executed asynchronously; triggered by business transactions in the OLTP
• Incremental real-time load
• Tighter integration of operational and data warehouse systems
DWHs become „mission critical“
• Higher requirements on availability and performance
Higher „transactional“ system load on data warehouse system
• DWH DB has to deal with typical DWH system load and transactional load
Data Quality mandatory
• Data is used for automated decisions
CHALLENGES FOR OPERATIONAL DATA WAREHOUSING

COMPARISON CLASSICAL DWH – OPERATIONAL DWH
Classical DWH Operational DWH
Strategic
• Passive
• Historical trends
Tactical
• Prediction
• Automatic execution of decisions
Batch
• E.g. daily batch
Near Real-Time / Right-Time
• Up-to-date view
Lower Availability
• System can be down for
maintenance and longer response
times for some reports are accepted
Availability
• System becomes critical and must
fulfill high availability and
performance requirements

Setting up and configuring a data warehouse system is a complex task
• Hardware
• Servers + Storage + Network
• Connectivity to source systems
• Software
• Database management system
• ETL software
• Reporting and analytics software
• ...
An optimal performance of the whole system is difficult to achieve
DATA WAREHOUSE APPLIANCES

Data Warehouse Appliances are
• Pre-configured and pre-tested hard- and software configurations
developed for running a data warehouse
• Optimized for data warehousing workload / Only suited for running
OLAP
• In contrast one size fits all: RDBMS are suited for OLTP, OLAP and mixed workloads
• Ready to be used after they are delivered to the customer
• Products, e.g. Teradata, HP Vertica, Exasol, Oracle Exadata, IBM Netezza
(IBM PureData System for Analytics), MS Analytic Platform System
DATA WAREHOUSE APPLIANCES

APPLIANCE SIMPLICITY (E.G. ORACLE EXADATA)

• Move as many operations as possible to storage cell instead of moving
data to the DB server
• E.g. filter data already at storage cell and not at DB server
• Avoid transferring unnecessary data
• Column-oriented In-memory storage with high compression
• Many appliances are based on shared nothing architecture
• Each node is independent
• Each node has its own storage or memory
• Parallel processing simpler and faster as no overhead due to contention
TYPICAL ENHANCEMENTS

• BI applications (database, ETL tools, Frontend) are hosted in a public
cloud, e.g.
• AWS (Amazon Web Services)
• Microsoft Azure
• …
• Many tools nowadays are available in the cloud first
• Vendors try to force customers to use clouds
• Or even available in the cloud only
• E.g. Microsoft Power BI
CLOUD BI

• Security concerns for sensitive data
• But new data source coming from Internet. Storing the data in a (public)
cloud can make sense, e.g.
• Connected Cars, IOT in general
CLOUD BI

CLOUD BI ARCHITECTURE
Source: Lang: Business Intelligence erfolgreich umsetzen, 5.Aufl., p. 185

• Analytics as a service
• Provide complete BI (Analytics) SW stack including
• data storage
• data integration (ETL)
• data visualization and/or data modeling (Frontend)
• Meta data management
• Data as a service
• Provide quality data for further usage
• Data marketplace
CLOUD BI ARCHITECTURE

CLOUD BI – DATA WAREHOUSING SERVICES
Source: http://db-engines.com/en/system/Amazon+Redshift%3BSnowflake

SNOWFLAKE ARCHITECTURE
Don‘t confuse
Snowflake
product with
Snowflake
dimensional
model from
session 2

Snowflake Storage
• Snowflake loads data into its internal optimized, compressed, columnar format
• Snowflake itself uses (!) Amazon Web Service’s S3 (Simple Storage Service) cloud
storage
Query Processing
• Each virtual warehouse is an MPP (Multi Parallel Processing) compute cluster
composed of multiple compute nodes allocated by Snowflake from Amazon EC2
• Each virtual warehouse is an independent compute cluster that does not share
compute resources with other virtual warehouses

Cloud Services
• Authentication and access control
• Infrastructure management
• Metadata management
• Query parsing and optimization
• Security

• Kimball et al: The Data Warehouse Lifecycle Toolkit, Wiley 2008
• Bauer / Günzel: Data-Warehouse-Systeme, dpunkt, 2013
• Köppen et al: Data Warehouse Technologien, mitp, 2016
END OF LECTURE
GOOD TEXT BOOKS
Source: http://dilbert.com/strip/2014-05-07

Daimler TSS GmbH
Wilhelm-Runge-Straße 11, 89081 Ulm / Telefon +49 731 505-06 / Fax +49 731 505-65 99
tss@daimler.com / Internet: www.daimler-tss.com/ Intranet-Portal-Code: @TSS
Domicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle
THANK YOU

How to document / identify requirements?
• Must be easy to understand from non-technical users during Analysis/Technical
concept phase
• Must provide sufficient information for System Design phase
The following slides provide some example work products that are produced
during Analysis/Technical concept phase and may be refine during System
Design phase
POSSIBLE DWH ANALYSIS AND DESIGN WORK
PRODUCTS

• Answer most important questions of participating business units
• Provide high-quality data
• Introduction in time
• Usage of modern technology
• Business orientation
• Easy to use
• Executive sponsor
• Patience – user acceptance evolves over time
CRITICAL SUCCESS FACTORS FOR BUILDING A DATA
WAREHOUSE

• New applications and data sources
• Increase demand for an
• Operational DWH, e.g.
• Industry 4.0 / Smart Factory
• Internet Of Things
• Internet of medical things
• Connected Cars
EXAMPLES OF OPERATIONAL DATA WAREHOUSING
Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 277
Replace pen &
paper with
electronic
workflows
Decision support for each end user
and not only management
Increasing demand to publish same
content on different devices

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Similar to Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW) (20)

Recently uploaded

Recently uploaded (20)

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Editor's Notes