When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema
 

When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema

on

  • 322 views

What is Reverse Star Schema? ...

What is Reverse Star Schema?

Why and when would I use a Reverse Star Schema?

How would I implement a Reverse Star Schema?

What about data integrity?

In this slideshare, we'll walk through a real life implementation within the healthcare industry.

Statistics

Views

Total Views
322
Views on SlideShare
320
Embed Views
2

Actions

Likes
0
Downloads
7
Comments
0

1 Embed 2

http://www.slideee.com 2

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema Presentation Transcript

  • When Facts and Dimensions Alone Aren't the Answer: Logically Reversing the Star Schema Kathryn Watson
  • Solution Architect at Perficient Economist & Financial Analyst by education, BI Professional by passion Email: kathryn.watson@perficient.com LinkedIn: www.linkedin.com/pub/kathryn-watson/a/431/354/ Kathryn Watson
  •  What is a Reverse Star Schema?  Why and when would I use a Reverse Star Schema?  How would I implement a Reverse Star Schema?  What about data integrity? While addressing these questions, we’ll walk through a real life implementation within the healthcare industry. The Questions We’ll Answer Today:
  • The Star Schema has long been the foundation of effective BI applications ● Optimized for the speedy return of business facts and their corresponding descriptors ● Supports the manner in which businesses ask typical business questions Let’s Start With the Basics
  • Show me A, filtered by B and aggregated by C & D Let’s Start With the Basics (cntd.) A B C D
  • Show me Z, filtered and aggregated by W, X & Y Let’s Start With the Basics (cntd.) Z W X Y
  •  Data warehouses commonly include multiple facts  These multiple facts often share dimensions, called conformed dimensions Let’s Start With the Basics (cntd.)
  •  Conformed dimensions facilitate answering multi- faceted business questions.  Show me A & Z, aggregated and/or filtered by N Let’s Start With the Basics (cntd.) A B C D Z W X Y N
  •  As BI implementations mature, the questions your users are asking often do as well.  Show me Z, filtered by B & C and aggregated by Y And Continue Along the BI Maturity Model A B C D Z W X Y N
  • In this scenario: ● B and C are not dimensions of Z ● B and C are not subsets of Y ● B and C are not directly related to the Fact (Z) they are required to affect And Continue Along the BI Maturity Model (cntd.) A B C D Z W X Y N
  • We’ve now encountered a very practical use case for the Reverse Star Schema. And Continue Along the BI Maturity Model (cntd.)
  • A typical Reverse Star Schema flips the concept of what information is being returned to the user. Rather than factual information being the star of the show, the dimensional information is the main event. In all Reverse Star Schemas, the relationships between facts and dimensions are flipped; rather than a one-to- many dimension to fact relationship, the Reverse Star Schema introduces a one-to-many or many-to-many fact to dimension relationship. So, What is a Reverse Star Schema?
  •  Sometimes, the return of the intersection of dimensional data is truly all the user needs  More often, the Reverse Star Schema is the avenue to establish cause-and-effect relationships amongst indirectly related data concepts When and Why Would I Use This?
  • Let’s return to the scenario we showcased previously: show me Z, filtered by B & C and aggregated by Y When and Why Would I Use This? (cntd.) A B C D Z W X Y N
  • B & C’s relationship to Z is established through fact A and conformed dimension N When and Why Would I Use This? (cntd.) A B C D Z W X Y N
  •  The values of A & N aren’t of interest to the user  A & N are simply avenues through which the needed attributes are accessed When and Why Would I Use This? (cntd.) A B C D Z W X Y N
  • Let’s apply this concept to the real world: When and Why Would I Use This? (cntd.) One of the nation’s top cancer care centers was seeking an on- demand means to identify patient cohorts. Physicians should be able to refine cohorts by basic tumor information and various disconnected factors of cancer care: Tumors – initial presentation date and type Histologies – cancer type and structure over time Stagings – cancer severity Statuses – active, in remission, etc. Treatments – chemotherapy, radiation, other A filter applied to any of these concepts should affect the results of all concepts, even though the concepts are independent of each other.
  • Let’s apply this concept to the real world (cntd.): During the analysis phase, it was confirmed that everything was directly related to the patient’s instance of cancer (Tumor). All other concepts, though factual concepts in their own right, for this purpose were informational. It’s all about the Tumor. Everything else is informational. When and Why Would I Use This? (cntd.)
  • Much like our use case scenario, a multi-fact Star Schema emerged. ● 5 Facts with Independent Dimensions ● Tumors/Documents (F_CLNCL_DOC) ● Histologies (F_TMR_HISTOL) ● Stagings (F_TMR_STG) ● Statuses (F_TMR_STAT) ● Treatments (F_TMR_TRTMT) ● 1 Conformed Dimension shared by all facts ● Tumors (D_PT_TMR) Let’s apply this concept to the real world (cntd.): When and Why Would I Use This? (cntd.)
  • Let’s apply this concept to the real world (cntd.): When and Why Would I Use This? (cntd.)
  • Implemented in OBIEE, the Dimensional Model = OBIEE Physical Layer, except for: ● EIW Dimensions ● Date dimension aliased for each of its purposes (17 aliases) ● Provider dimension aliased for each of its purposes (3 aliases) ● 4 Additional Facts ● Users stated the requirement to analyze treatments as separate entities for some use- cases, and as consolidated activities in other use-cases ● This is achieved by placing database views on top of F_TMR_TRTMT, with one view for each treatment type Let’s apply this concept to the real world (cntd.): When and Why Would I Use This? (cntd.)
  • Let’s apply this concept to the real world (cntd.): When and Why Would I Use This? (cntd.)
  • Let’s apply this concept to the real world (cntd.): When and Why Would I Use This? (cntd.) But the classic Star Schema did not support our main requirement: A filter applied to any of these concepts should affect the results of all concepts, even though the concepts are independent of each other. It’s all about the Tumor. Everything else is informational. In the classic Star Schema design, each fact and its dimensions are independent of the other facts. Filters applied to the dimensions of one fact do not affect the results returned in another fact.
  • Ok, How Would I implement This? To fill our requirement that filters applied to the dimension of one fact should affect the results returned in another fact: ● “Sub-Facts” (Statuses, Stagings, Histologies, Treatments) should be treated as dimensions of the central fact (Tumors/Documents) through the conformed dimension (Tumors). ● This would create a reverse snowflake schema ● One fact: F_CLNCL_DOC (Tumors/Documents) ● Central Dimension: D_PT_TMR (Tumors) ● All other tables are dimensions that flake off of D_PT_TMR and their “Sub-Facts”. (One to many join between D_PT_TMR and other facts reversed, becoming many to one) Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) So, should our Dimensional Model/RPD Physical Layer be re- designed to support this? No. Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) The Dimensional Model/RPD Physical Layer should remain As Is (a classic Star Schema) ● The “Sub-Facts” truly are facts/transactions. They can be analyzed independently ● Current requirements dictate that the “Sub-Facts” are simply avenues to descriptive attributes ● However, future requirements may dictate the need to analyze these “Sub-Facts” individually; we don’t want to lose this capability through a redesign Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) The Reverse Star Schema is built logically in the RPD BMM layer: ● Individual Dimension Tables are brought over from the Physical Layer ● The Logical Table Source (LTS) for each Dimension Table is edited to include its “Sub-Fact” table as well as D_PT_TMR Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) The Reverse Star Schema is built logically in the RPD BMM layer (cntd.): ● The dimension table is then logically joined directly to the central fact (FACT_ALS_F_CLNCL_DOC) Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) The Reverse Star Schema is built logically in the RPD BMM layer (cntd.): ● The joins in the Physical Layer remain the same. Continuing with our real world example: Dimension DimensionFact Fact
  • Ok, How Would I implement This? (cntd.) This process is completed until all dimension tables join directly to the central fact (FACT_ALS_F_CLNCL_DOC) in the BMM “Sub-Facts” were not brought in as logical tables in the BMM for this implementation, as there were no requirements for them. ● These “Sub-Facts” can be brought in as independent facts later if needed, creating a multi-facted Star Schema ● If a “Sub-Fact” is brought in at a later time, it need simply be joined to its relevant Dimensions; OBIEE will recognize that only one table in the LTS need be used for the Join Continuing with our real world example:
  • Ok, How Would I implement This? (cntd.) Continuing with our real world example:
  • What about data integrity? Continuing with our real world example: Aggregation is fine so long as your aggregations are all count distinct. But what about when your aggregation is something other than a count distinct? Ay, there’s the rub! Luckily, OBIEE has a quick and easy solution: level based measures and/or measures based upon dimensions
  • What about data integrity? Continuing with our real world example: We had to set our non-count distinct aggregations to be Based on Dimensions in order to achieve correct aggregation of our measures ● For the conformed dimension (DIM_ALS_D_PT_TMR or Tumors), take the Distinct Average of the measure ● For all Other Dimensions, simply Average the Measure
  • What about data integrity? Continuing with our real world example: Why does this work? ● OBIEE will first average distinct the measure at the Tumor level. Since the Tumor level is the lowest granularity, this will equate to the original measure ● OBIEE will then average the population of these distinct averages by whatever other dimensions are brought into the request
  • What about data integrity? Continuing with our real world example: Why is this necessary? ● In our data set, our “Sub-Facts” can have multiple instances per tumor. ● We have transitioned our “Sub-Facts” to act as avenues between their own dimensions and the central fact (FACT_F_CLNCL_DOC) for the formation of the Star Schema in the BMM ● As a result, we can (and do) experience “ballooning”
  • What about data integrity? Continuing with our real world example: Why is this necessary (cntd.)? Two Tumors, each with one row of data: vs. Same two Tumors, with Staging Date added to the query. The 1st Tumor now has two rows of data, one for each of its Staging Dates. The 2nd Tumor still has only one row of data, as it has only one Staging Date.
  • What about data integrity? Continuing with our real world example: Why is this necessary? (cntd.) ● With the inflated data, we can’t simply average: tumors that have ballooned out would be weighted more heavily in our average calculation than those that have not 349.67 If the Aggregation Rule in our RPD was simply set to “Average,” OBIEE would calculate the average of the above as 349.67 [ (493 + 493 + 63) / 3 ]
  • What about data integrity? Continuing with our real world example: Why is this necessary? (cntd.) ● To remove the “weight” given to a measure when its tumor appears in the results more than once, we have to tell OBIEE to base the aggregation on Dimensions
  • What about data integrity? Continuing with our real world example: Why is this necessary? (cntd.) ● OBIEE would first calculate the Average at the Tumor Level AVG(DISTINCT "SCD-Tumor Management"."FACT_ALS_F_CLNCL_DOC"."Follow Up Time")
  • What about data integrity? Continuing with our real world example: Why is this necessary? (cntd.) ● OBIEE would then average this average AVG("SCD-Tumor Management"."FACT_ALS_F_CLNCL_DOC"."Follow Up Time") ● Our average calculates correctly, at 278 [ (493 + 63) / 2 ]
  • Questions?
  • Additional Implementation Notes Non Star Schema Structures: ● Non-Star Schema structures can work and do not necessarily cause incorrect results ● However, they are not ideal and can greatly complicate implementation, expansion, and maintenance
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● To demonstrate, let’s start with a simple snowflake structure
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● And then create a Dimensional Hierarchy on DIM_PROVIDERS
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● OBIEE names your hierarchy DIM_PATIENTSDim, but you created it on DIM_PROVIDERS. Why? ● OBIEE considers the dimension that is joined directly to the fact to be the “true”, as it’s the lowest level dimension ● DIM_PROVIDERS is connected to DIM_PATIENTS at a higher level, and OBIEE has consolidating both tables into one dimensional hierarchy, with DIM_PROVIDERS the parent level of DIM_PATIENTS
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● What would have happened if DIM_PATIENTS had 2 flakes? Dimension Dimension Dimension Dimension Fact
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● You’d end up with 1 dimensional hierarchy with 2 separate trees
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● So why does this complicate things? ● Level-based measures are based upon Dimensional Hierarchies ● In a Snowflake Schema, constructing many types of level-based measures becomes impossible, or at the least very complex.
  • Additional Implementation Notes (cntd.) Non Star Schema Structures (cntd.): ● Let’s calculate the average Follow-Up Days for a Provider by Age Bucket ● Because Providers and Age Buckets are in the same Dimensional Hierarchy, we can’t do this calculation. ● OBIEE limits you to only one level per Dimensional Hierarchy
  • Additional Implementation Notes (cntd.) RPD Consistency Check Warnings: ● Because you have mapped multiple tables in the LTS’s of your Dimensions, OBIEE expects you to use those tables in the Column Mappings ● If you do not make use of a mapped table, the Consistency Checker will give you a warning:
  • Additional Implementation Notes (cntd.) RPD Consistency Check Warnings (cntd.): ● To resolve this warning, create a new Logical Column in your table and map it to a column in the unused table
  • Additional Implementation Notes (cntd.) Fabricated Data: Missing Data ● In our data set we were faced with the fact that a Tumor may not be present in all of the major subject areas (Statuses, Stagings, Histologies, Treatments) ● In the instances where a Tumor was not present in a subject area, that Tumor would fall out of the data population if a dimension from that subject area was brought in Users did not want this fallout to occur and they emphasized that it was just as important to see where a Tumor did not have information in a given subject area as it was to see what information a tumor did have in a given subject area
  • Additional Implementation Notes (cntd.) Fabricated Data: Missing Data (cntd.) ● RPD Resolution Possibility ● This fallout could have been prevented through Outer Joins in the OBIEE RPD ● The Logical Mappings of all dimension attributes could be changed to include an IFNULL clause [It was required that a NULL/missing object state its status (Missing)] ● With hundreds of attributes this is neither sustainable nor realistic
  • Additional Implementation Notes (cntd.) Fabricated Data: Missing Data (cntd.) ● Data Creation in Database Tables Resolution Possibility ● Each and every Dimension Table in our Dimensional Model has an entry for a key of -2 and attribute codes, names, descriptions, etc. of ‘!Record Does Not Exist in Source’ ● At the end of the ETL process, the central conformed dimension (D_PT_TMR) was LEFT OUTER JOINed to each of the dimensional database “Sub-Facts” ● F_TMR_STAT ● F_TMR_STG ● F_TMR_HISTOL ● F_TMR_TRTMT
  • Additional Implementation Notes (cntd.) Fabricated Data: Missing Data (cntd.) ● Data Creation in Database Tables Resolution Possibility (cntd.) ● Where a match was not found in a Fact table, a row was inserted in that Fact table for the missing Tumor ID. All Dimensional Keys were set to -2, !Record Does Not Exist in Source ● Users then gained the ability to pull tumors with missing information.