What is Business Intelligence? Business intelligence is a broad category of applications and technologies for gathering, providing access to, and analyzing data for the purpose of helping enterprise users make better business decisions. Ekta Pardhi

What is OLTP ? Traditional RDBMS are used for OLTP On-Line Transaction Processing used for daily processing detailed, up to date data read/update a few records isolation, recovery and integrity are critical

What is OLTP ?

Traditional RDBMS are used for OLTP

On-Line Transaction Processing

used for daily processing

detailed, up to date data

read/update a few records

isolation, recovery and integrity are critical

Data, Data everywhere yet ... I can’t find the data I need data is scattered over the network many versions, subtle differences I can’t get the data I need need an expert to get the data I can’t understand the data I found available data poorly documented I can’t use the data I found results are unexpected data needs to be transformed from one form to other

I can’t find the data I need

data is scattered over the network

many versions, subtle differences

I can’t get the data I need

need an expert to get the data

I can’t understand the data I found

available data poorly documented

I can’t use the data I found

results are unexpected

data needs to be transformed from one form to other

What is OLAP(On-line Analytical Processing) ? Online Analytical Processing dynamic synthesis, analysis and consolidation of large volumes of multi-dimensional data normally implemented using specialized multi-dimensional DBMS a method of visualizing and manipulating data with many inter-relationships Support common analytical operations such as consolidation drill-down slicing and dicing

What is OLAP(On-line Analytical Processing) ?

Online Analytical Processing

dynamic synthesis, analysis and consolidation of large volumes of multi-dimensional data

normally implemented using specialized multi-dimensional DBMS

a method of visualizing and manipulating data with many inter-relationships

Support common analytical operations such as

consolidation

drill-down

slicing and dicing

Online Analytical Processing (OLAP ) It enables analysts, managers and executives to gain insight into data through fast, consistent, interactive access to a wide variety of possible views of information that has been transformed from raw data to reflect the real dimensionality of the enterprise as understood by the user. Time Product Region SQL 2000 Oracle Access SQL 2005 1KEY OLAP

OLAP Applications - multi-dimensional views of data Core requirement of building a ‘realistic’ business model . Provides basis for analytical processing through flexible access to corporate data. The underlying database design that provides the multi-dimensional view of data should treat all dimensions equally.

Core requirement of building a ‘realistic’ business model .

Provides basis for analytical processing through flexible access to corporate data.

The underlying database design that provides the multi-dimensional view of data should treat all dimensions equally.

Dimensions: Product, Region, Time Hierarchical summarization paths Product Region Time Industry Country Year Category Region Quarter Product City Month Week Office Day Conceptual Data Model Multi-dimensional view of Data Each dimension is described by a set of attributes; the attributes of a dimension may be related via hierarchy of relationships. Month 1 2 3 4 7 6 5 Product Toothpaste Juice Cola Milk Cream Soap Region W S N

Each dimension is described by a set of attributes; the attributes of a dimension may be related via hierarchy of relationships.

OLAP Applications - support for complex calculations Must provide a range of powerful computational methods such as that required by sales forecasting, which uses trend algorithms such as moving averages and percentage growth. Mechanisms for implementing computational methods should be clear and non-procedural .

OLAP Applications - support for complex calculations

Must provide a range of powerful computational methods such as that required by sales forecasting, which uses trend algorithms such as moving averages and percentage growth.

Mechanisms for implementing computational methods should be clear and non-procedural .

OLAP Applications – time intelligence Key feature of almost any analytical application as performance is almost always judged over time. Time hierarchy is not always used in same manner as other hierarchies. Concepts such as year-to-date and period-over-period comparisons should be easily defined.

OLAP Applications – time intelligence

Key feature of almost any analytical application as performance is almost always judged over time.

Time hierarchy is not always used in same manner as other hierarchies.

Concepts such as year-to-date and period-over-period comparisons should be easily defined.

What is Data warehouse? A DW is a subject-oriented, integrated, time-varying, non-volatile collection of data that is used primarily in organizational decision making Ekta Pardhi

What is Data warehouse?

A DW is a subject-oriented, integrated, time-varying, non-volatile collection of data that is used primarily in organizational decision making

Graphically Data Warehouse Subject Oriented Credit Risk Interest Rate Risk Forex Risk ALM Subject Area OLTP Systems Integrated Credit Risk Code: XXXX Interest Rate Risk Code: CXXYY Forex Risk Code: XX/XX.XX Common Code for various source codes ALM Subject Area Non Volatile OLTP U S E R S Data Warehouse U S E R S Read Write Read Time variant Dec 98 1995 1996 1997 1998 OLTP OLAP

Data Warehouse is not … a product, an environment a system, an architecture an end to itself but is at the heart of the Business Intelligence (BI) infrastructure of the organization

Data Warehouse is not …

a product, an environment

a system, an architecture

an end to itself but is at the heart of the Business Intelligence (BI) infrastructure of the organization

OLTP/OLAP OLTP (on-line transaction processing) Major task of traditional relational DBMS Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc. OLAP (on-line analytical processing) Major task of data warehouse system Data analysis and decision making Distinct features (OLTP vs. OLAP) User and system orientation: customer vs. market Data contents: current, detailed vs. historical, consolidated Database design: ER + application vs. star + subject View: current, local vs. evolutionary, integrated Access patterns: update vs. read-only but complex queries

OLTP (on-line transaction processing)

Major task of traditional relational DBMS

Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc.

OLAP (on-line analytical processing)

Major task of data warehouse system

Data analysis and decision making

Distinct features (OLTP vs. OLAP)

User and system orientation: customer vs. market

Data contents: current, detailed vs. historical, consolidated

Database design: ER + application vs. star + subject

View: current, local vs. evolutionary, integrated

Access patterns: update vs. read-only but complex queries

OLTP vs. OLAP

Examples of OLAP applications in various functional areas

Nature of OLAP Analysis Aggregation -- (total sales, percent-to-total) Comparison -- Budget vs. Expenses Ranking -- Top 10, quartile analysis Access to detailed and aggregate data Complex criteria specification Visualization

Aggregation -- (total sales, percent-to-total)

Comparison -- Budget vs. Expenses

Ranking -- Top 10, quartile analysis

Access to detailed and aggregate data

Complex criteria specification

Visualization

1KEY OLAP Related Terms Dimension Attribute Hierarchy Fact table Measures Model(cube) Calculated member

1KEY OLAP Related Terms

Dimension

Attribute

Hierarchy

Fact table

Measures

Model(cube)

Calculated member

Dimension A dimension table contains the specific name of each member of the dimension and Dimensions determine the contextual background for the facts

Dimension

A dimension table contains the specific name of each member of the dimension and Dimensions determine the contextual background for the facts

Dimension

Attribute Information about a specific dimension member

Attribute

Information about a specific dimension member

Attribute

Fact table Fact table contains values for one or more measures at the lowest level of detail for one or more dimensions

Fact table

Fact table contains values for one or more measures at the lowest level of detail for one or more dimensions

Fact Table

Measures A summarizable numerical value used to monitor business activity

Measures

A summarizable numerical value used to monitor business activity

Measure

Calculated member A mechanism for aggregating measures using formulas more complex than those stored in a cube

Calculated member

A mechanism for aggregating measures using formulas more complex than those stored in a cube

Hierarchies Think of the points of the data cube as partitioned along each dimension (at some level of granularity) E.g., time dimension: partition according to days, weeks, months,… E.g., cars dimension: partition by model, by color, both model and color, … E.g., dealers dimension: partition by dealer, by city, by state, ...

Hierarchies

Think of the points of the data cube as partitioned along each dimension (at some level of granularity)

E.g., time dimension: partition according to days, weeks, months,…

E.g., cars dimension: partition by model, by color, both model and color, …

E.g., dealers dimension: partition by dealer, by city, by state, ...

Aggregation Hierarchies

Model(cube) A collection of one or more related measure groups and their associated dimensions

Model(cube)

A collection of one or more related measure groups and their associated dimensions

T i m e p r o d c u s t p r o m o f a c t date, custno, prodno, cityname, sales

Region Scheme Time East West North South 1997 1998 1999 2000 Housing Car Edu. Industry Cell 200 Measure Dimension Components of a Data Cube (3D)

1KEY OLAP console

DW Modeling Modeling data warehouses: dimensions & measures Star schema : A fact table in the middle connected to a set of dimension tables A star schema is a set of tables comprised of a single, central fact table surrounded by de-normalized dimensions. Each dimension is represented in a single table. Star schema implement dimensional data structures with de- normalized dimensions. Snowflake schema : A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake A snowflake schema is a set of tables comprised of a single, central fact table surrounded by normalized dimension hierarchies. Each dimension level is represented in a table. Snowflake schema implement dimensional data structures with fully normalized dimensions.

DW Modeling

Modeling data warehouses: dimensions & measures

Star schema : A fact table in the middle connected to a set of dimension tables

A star schema is a set of tables comprised of a single, central fact table surrounded by de-normalized dimensions. Each dimension is represented in a single table. Star schema implement dimensional data structures with de- normalized dimensions.

Snowflake schema : A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake

A snowflake schema is a set of tables comprised of a single, central fact table surrounded by normalized dimension hierarchies. Each dimension level is represented in a table. Snowflake schema implement dimensional data structures with fully normalized dimensions.

Star Schema A single fact table and for each dimension one dimension table Does not capture hierarchies directly f a c t T i m e p r o d c u s t p r o m o f a c t date, custno, prodno, cityname, sales

A single fact table and for each dimension one dimension table

Does not capture hierarchies directly

Snowflake Schema Represent dimensional hierarchy directly by normalizing tables. Easy to maintain and saves storage f a c t T i m e p r o d c u s t p r o m o f a c t date, custno, prodno, cityname, sales r e g i o n

Represent dimensional hierarchy directly by normalizing tables.

Easy to maintain and saves storage

Components of a star schema Excellent for ad-hoc queries, but bad for online transaction processing

Star schema example

Star Schema – Example 1 OrderNo SalespersonID CustomerNo ProdNo DateKey CityName Quantity TotalPrice Fact Table CityName State Country City DateKey Date Month Year Date ProdNo ProdName ProdDescr Category CategoryDescr UnitPrice QOH Product OrderNo OrderDate Order SalespersonID SalespersonName City Quota Salesperson CustomerNo CustomerName CustomerAddress City Customer

Snowflake – Example 1

Star Schema – Example 2 Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures time_key day day_of_the_week month quarter year time location_key street city province_or_street country location item_key item_name brand type supplier_type item branch_key branch_name branch_type branch

Snowflake Schema –Example Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales Measures time_key day day_of_the_week month quarter year time location_key street city_key location item_key item_name brand type supplier_key item branch_key branch_name branch_type branch supplier_key supplier_type supplier city_key city province_or_street country city

Features of a OLAP Reporting Solution Must connect with an OLAP data source Must allow the user to drill down into the data with flexibility and ease Must provide a user-friendly visual interface that includes charts, because reporting is most effective when it is visual. Must allow the user to save and retrieve custom reports.

Features of a OLAP Reporting Solution

Must connect with an OLAP data source

Must allow the user to drill down into the data with flexibility and ease

Must provide a user-friendly visual interface that includes charts, because reporting is most effective when it is visual.

Must allow the user to save and retrieve custom reports.

Some variants

Metadata Metadata explains what data exists, where it is located and how to access it. The metadata is a core of a data logistics system, the infrastructure for DW and ultimately the intelligence system.

Metadata

Metadata explains what data exists, where it is located and how to access it. The metadata is a core of a data logistics system, the infrastructure for DW and ultimately the intelligence system.

Data Mart A Data mart is a collection of data specifically designed for the use of a department. Data mart is nothing but a smaller, more focused data warehouse. In many organizations it is useful to create data marts for specific business units that have very specific data analysis needs. The data mart reflects the peculiar needs of the department. A typical example of a departmental data mart in a Bank would be that of a Human Resource Department.

Data Mart

A Data mart is a collection of data specifically designed for the use of a department.

Data mart is nothing but a smaller, more focused data warehouse.

In many organizations it is useful to create data marts for specific business units that have very specific data analysis needs.

The data mart reflects the peculiar needs of the department.

A typical example of a departmental data mart in a Bank would be that of a Human Resource Department.

Enterprise Data Warehouse vs. Data Mart

DW and Data Mart: some other issues Characteristics Data Mart Enterprise DW Time to build Months Years Complexity to build Low to medium High Shared Shared in Dept. Shared across Update Frequently Less Frequently Type of user Subject focused Corporate and top executives

Implementation and Success Criteria Do not try to implement the entire data warehouse at once The project should break up the functionality to be delivered in different phases You will not only deliver something tangible for your users, but you may also flush out issues that can be quickly corrected Users are constantly knocking on your door The buzz in the hallways mentions the data warehouse, or meetings make reference to it as the source of data The data warehouse and OLAP becomes the heartbeat of the business, where decisions are made from the data intelligence it provides

Implementation and Success Criteria

Do not try to implement the entire data warehouse at once

The project should break up the functionality to be delivered in different phases

You will not only deliver something tangible for your users, but you may also flush out issues that can be quickly corrected

Users are constantly knocking on your door

The buzz in the hallways mentions the data warehouse, or meetings make reference to it as the source of data

The data warehouse and OLAP becomes the heartbeat of the business, where decisions are made from the data intelligence it provides

Strengths of OLAP It is a powerful visualization tool It provides fast, interactive response times It is good for analyzing time series It can be useful to find some clusters and outliners

It is a powerful visualization tool

It provides fast, interactive response times

It is good for analyzing time series

It can be useful to find some clusters and outliners

Data Warehousing and OLAP Key Benefits Centralized Data Repository Single Version of the Truth Clean, Accurate Data Organized, Timely Availability Historical Detail Shared and Accessible Ad hoc Discovery & Analysis

Data Warehousing and OLAP Key Benefits

Centralized Data Repository

Single Version of the Truth

Clean, Accurate Data

Organized, Timely Availability

Historical Detail

Shared and Accessible

Ad hoc Discovery & Analysis

Walkthrough MAIA OLAP

Walkthrough MAIA OLAP

1KEYOLAP Project Management Data warehouse projects are significantly more dynamic and complicated than traditional systems projects – more change to: Scope Tool evaluation and performance Cross-departmental issues, requirements Cross-departmental data integration and usage Staffing resources Cultural issues DW Project Manager must be more actively involved in all facets of project, understand all facets and issues – not all project managers will succeed as DW project managers

1KEYOLAP Project Management

Data warehouse projects are significantly more dynamic and complicated than traditional systems projects – more change to:

Scope

Tool evaluation and performance

Cross-departmental issues, requirements

Cross-departmental data integration and usage

Staffing resources

Cultural issues

DW Project Manager must be more actively involved in all facets of project, understand all facets and issues – not all project managers will succeed as DW project managers

Thank You! Ekta Pardhi [email_address]

Thank You!

Ekta Pardhi

[email_address]