Real-Time DW with OBIEE and Oracle Database    Stewart Bryson, Managing Director, Rittman Mead America    Collaborate, Orl...
Who am I?          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 ...
• Stewart Bryson, Managing Director, Rittman                                                 Mead America                 ...
Who is Rittman Mead?          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comTh...
• Oracle BI&DW Experts                                             • Consulting, Training and Support                     ...
Rittman Mead BI Forum, May 2011          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittma...
Registrations open for the Rittman Mead BI Forum, May 2011    • Brighton, UK, May 18-20th 2011    • Atlanta, GA, May 25-27...
What will this session answer?          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittman...
• What do I mean by “real-time data warehousing?”                                              ‣ Zero latency, or just red...
What do I mean by “real-time”?          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittman...
• Traditional data warehouses                                                    ‣ Daily to weekly batch-load cycles      ...
An interval will always exist between the occurrence of a measurable                event and our ability to process that ...
L1&-6)&/(&)7#.@@)&)2-#-"0)*#(@#.2$(/.23#%-%#$%2#6%?)#.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%                        ...
Business Case          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday,...
• We want to be able to segment our reporting                                             between “real-time” needs and st...
Latency versus Query Performance          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittm...
Latency versus Query Performance          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittm...
Latency versus Query Performance                                                   Latency                                ...
Latency versus Query Performance                                                   Latency                                ...
Oracle Next-Generation Reference Architecture          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com...
Requirement: Identifying and                               Capturing Change          T : (888) 631 1410 or +44 (0) 8446 69...
Requirement: Change Data Capture          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittm...
Requirement: Change Data Capture          • Oracle CDC          • OWB Trickle-feed            mappings          • Trigger-...
Requirement: Change Data Capture          • Oracle CDC                        • Oracle CDC          • OWB Trickle-feed    ...
Requirement: Change Data Capture                                              • Oracle CDC                       • OWB Map...
Requirement: Change Data Capture                                              • Oracle CDC                       • OWB Map...
Oracle Asynchronous Change Data Capture          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: ww...
Oracle Asynchronous Change Data Capture                                       Hot Log                                     ...
Oracle Asynchronous Change Data Capture                                       Hot Log                                     ...
Requirement: Change Data Capture                                              • Oracle CDC                       • OWB Map...
Requirement: Change Data Capture                                              • Oracle CDC                       • OWB Map...
Requirement: Change Data Capture          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittm...
Requirement: Change Data Capture              • Trickle-feed mappings allow the development of Streams                Capt...
Federated OLTP/EDW Reporting          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanme...
• Layering in OLTP data to provide the real-time                                             component                    ...
Classic Batch Load Star Schema: Process Flow          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Classic Batch Load Star Schema: Fact Table Load          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.c...
Federation with OBIEE          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comT...
OBIEE Metadata and Data Layers          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittman...
OBIEE Metadata and Data Layers    • Physical Data Sources, such      as relational databases,                             ...
OBIEE Metadata and Data Layers    • A Semantic Model made up      of Metrics, dimensions,                                 ...
OBIEE Metadata and Data Layers    • The Web Catalog contains      definitions of reports, alerts                          ...
Flow of Data Through the Three-Layer Semantic Model          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanme...
Flow of Data Through the Three-Layer Semantic Model                                                 Simplification of the ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Map to Physical Models          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com ...
Logical Table Sources Respect the Physical Model          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead....
Logical Table Sources Respect the Physical Model          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead....
Logical Table Sources Respect the Physical Model          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead....
WITH      SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,           T43901.CUST_LAST_NAME as c3,           T43971.SAL_AM...
WITH      SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,           T43901.CUST_LAST_NAME as c3,           T43971.SAL_AM...
WITH      SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,           T43901.CUST_LAST_NAME as c3,           T43971.SAL_AM...
WITH      SAWITH0 AS (select sum(T44105.AMOUNT) as c1,           T44042.CUSTOMER_FIRST_NAME as c2,           T44042.CUSTOM...
WITH      SAWITH0 AS (select sum(T44105.AMOUNT) as c1,           T44042.CUSTOMER_FIRST_NAME as c2,           T44042.CUSTOM...
Demonstration    Fragmentation with OLTP and EDW          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead....
• Strengths                                               ‣ Low latency                                               ‣ No...
EDW with a Real-Time Component          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittman...
• Modeling complimentary real-time structures in an                                             EDW                       ...
The Real-Time Partition                                Fact Table          T : (888) 631 1410 or +44 (0) 8446 697 995 E : ...
The Real-Time Partition                                                                           Fact Real-Time          ...
The Real-Time Partition                                                                           Fact Real-Time          ...
The Real-Time Partition                          • All ETL processing of fact records are directed at                     ...
The Real-Time Partition                          • All ETL processing of fact records are directed at                     ...
The Real-Time Partition                          • All ETL processing of fact records are directed at                     ...
The Real-Time Partition                          • All ETL processing of fact records are directed at                     ...
Early-Arriving Facts          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comTh...
What if Type 2 dimension changes haven’t arrived yet?                                           • Grab the most recent dim...
Real-Time Integration: Fact Table RT Load          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: ...
Real-Time Integration: Fact Table RT Load          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: ...
Real-Time Integration: Correct Dimensions          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: ...
Real-Time Integration: “Closing the Books” on the Day          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittman...
Plugging it together with OBIEE (Non-Fragmentation)      • Using standard database        functionality (UNION ALL)      •...
Plugging it together with OBIEE (Non-Fragmentation)      • Using standard database        functionality (UNION ALL)      •...
Plugging it together with OBIEE (Non-Fragmentation)      • Using standard database        functionality (UNION ALL)      •...
Demonstration    EDW with a Real-Time Component          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.c...
Summary          T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 Ap...
• Latency versus performance and scalability                                           • Latency and performance versus de...
Real-Time DW with OBIEE and Oracle Database    Stewart Bryson, Managing Director, Rittman Mead America    Collaborate, Orl...
Upcoming SlideShare
Loading in...5
×

Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11

2,933

Published on

Presentation by Stewart Bryson, Rittman Mead, on real-time data warehousing techniques using the Oracle Database and Oracle Business Intelligence Enterprise Edition. As delivered at Collaborate'11, Orlando, April 2011.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,933
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
278
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11

  1. 1. Real-Time DW with OBIEE and Oracle Database Stewart Bryson, Managing Director, Rittman Mead America Collaborate, Orlando, April 2011 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 1
  2. 2. Who am I? T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 2
  3. 3. • Stewart Bryson, Managing Director, Rittman Mead America • Oracle BI/DW Architect and Delivery Specialist • Oracle Database and BI Stack since 1996 • Regular Community Speaker • Writer for Rittman Mead Blog: http://www.rittmanmead.com/blog • Developer of Transcend Framework • Email : stewart.bryson@rittmanmead.com • Twitter : @stewartbryson T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 3
  4. 4. Who is Rittman Mead? T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 4
  5. 5. • Oracle BI&DW Experts • Consulting, Training and Support • Global Company, Offices in the US, UK and India • Clients in the UK, USA, Europe, Middle-East • Voted UKOUG BI Partner of the Year 2008 • 16 Sessions at Oracle Open World 2010 • Twitter : @rittmanmead T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 5
  6. 6. Rittman Mead BI Forum, May 2011 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 6
  7. 7. Registrations open for the Rittman Mead BI Forum, May 2011 • Brighton, UK, May 18-20th 2011 • Atlanta, GA, May 25-27th 2011 • Covers OBIEE, DW, EPM • Guest speakers including: ‣ Jeff McQuigg ‣ Kevin McGinley ‣ Jean-Pierre Dijcks ‣ Mike Durran ‣ Shyam Varan Nath ‣ Philippe Lions ‣ Emiel van Bockel • OBIEE 11g Masterclass with Mark Rittman and Tony Heljula • Full details at http://www.rittmanmead.com/biforum2011 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 7
  8. 8. What will this session answer? T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 8
  9. 9. • What do I mean by “real-time data warehousing?” ‣ Zero latency, or just reduced latency? ‣ Do I really mean “right-time data warehousing?” • What can I use to reduce latency? ‣ What does OBIEE offer? ‣ What does the Oracle Database (and OWB) offer? • Can I blend these different offerings? ‣ Of course you can! ‣ It’s the name of the presentation T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 9
  10. 10. What do I mean by “real-time”? T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 10
  11. 11. • Traditional data warehouses ‣ Daily to weekly batch-load cycles ‣ Key-metrics are always delayed ‣ Information may not be “actionable” • Real-time data warehouses ‣ Up to the minute or hour reporting ‣ Key-metrics are moving ‣ Information is actionable T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 11
  12. 12. An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact” T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 12
  13. 13. L1&-6)&/(&)7#.@@)&)2-#-"0)*#(@#.2$(/.23#%-%#$%2#6%?)#.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*% +%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))#-(#>)#0&(0%3%-)#-6&(136#/%2 %33&)3%-.(2#-%>+)*#4*))#L.31&)#M<= K6)#.?.*.(2#>)-B))2#%-%#%$E1.*.-.(2#%2#%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0% $(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)#.2#0&(?..237#%*#&%0.+"#%*#0(* An interval will always exist between the occurrence of a measurable (2#%+&)%"#%$E1.&)#%-%7#(&#(#"(1#%+*(#2))#-(#&)0(&-#(2#2)B+"#%)#@%$-1%+#.2@(&/%-.(2C event and our ability to process that event as a “reportable fact” L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+) FIGURE 1: BI latency $)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2# Event Report $1*-(/)&#.*P#%2#-6)#+.D)+.6((#-6% $1*-(/)&P*#)/(3&%06.$#B(1+#*B *100+.)&= K6.*#.2@(&/%-.(2#2))*#-(#>)#0&)*)2 %&)#*0)%D.23#-(#-6)/<#%2#0&)@)&%> +.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/% .2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6 Recording Propagation Aggregation Query >"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/) 0&).$-)#$61&2#B(1+#>)#.2@+1)2$) 6(1&*= S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23# Real-time query latency -(#%+)&-#-(#0(-)2-.%+#@&%1#4*-($D#/%& $(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1* D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.? ***+,-./0.12345,416,+537 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 12
  14. 14. Business Case T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 13
  15. 15. • We want to be able to segment our reporting between “real-time” needs and static reports ‣ Most users will want real-time numbers delivered with dashboards ‣ Some users (financial-minded) don’t want numbers to change each time they run a report • Use the standard Oracle BI Stack ‣ Oracle Database Enterprise Edition ‣ Oracle Business Intelligence Enterprise Edition T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 14
  16. 16. Latency versus Query Performance T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 15
  17. 17. Latency versus Query Performance T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 15
  18. 18. Latency versus Query Performance Latency Query Performance T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 15
  19. 19. Latency versus Query Performance Latency Query Performance T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 15
  20. 20. Oracle Next-Generation Reference Architecture T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 16
  21. 21. Requirement: Identifying and Capturing Change T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 17
  22. 22. Requirement: Change Data Capture T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 18
  23. 23. Requirement: Change Data Capture • Oracle CDC • OWB Trickle-feed mappings • Trigger-based capture • Recording modification dates • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 18
  24. 24. Requirement: Change Data Capture • Oracle CDC • Oracle CDC • OWB Trickle-feed • OWB Trickle-feed mappings mappings • Trigger-based • Message Queues capture • Views over • Recording database links modification dates • File movements • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 18
  25. 25. Requirement: Change Data Capture • Oracle CDC • OWB Mappings • Oracle CDC and Process Flows • OWB Trickle-feed • OWB Trickle-feed mappings • Other ETL Tools mappings • Message Queues • Custom ETL code • Trigger-based capture • Views over • Custom transaction database links code • Recording modification dates • File movements • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 18
  26. 26. Requirement: Change Data Capture • Oracle CDC • OWB Mappings • Oracle CDC and Process Flows • OWB Trickle-feed • OWB Trickle-feed mappings • Other ETL Tools mappings • Message Queues • Custom ETL code • Trigger-based capture • Views over • Custom transaction database links code • Recording modification dates • File movements • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 18
  27. 27. Oracle Asynchronous Change Data Capture T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 19
  28. 28. Oracle Asynchronous Change Data Capture Hot Log Online redo logs mined locally Change tables are updated and stored locally Hot Log Distributed Online redo logs are mined locally Change tables are updated and stored remotely Autolog Online Online redo logs are mined remotely Change tables are updated and stored remotely Autolog Archive Archived redo logs are mined remotely Change tables are updated and stored remotely T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 19
  29. 29. Oracle Asynchronous Change Data Capture Hot Log Online redo logs mined locally Change tables are updated and stored locally Hot Log Distributed Online redo logs are mined locally Latency Change tables are updated and stored remotely Autolog Online Online redo logs are mined remotely Change tables are updated and stored remotely Autolog Archive Archived redo logs are mined remotely Change tables are updated and stored remotely T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 19
  30. 30. Requirement: Change Data Capture • Oracle CDC • OWB Mappings • Oracle CDC and Process Flows • OWB Trickle-feed • OWB Trickle-feed mappings • Other ETL Tools mappings • Message Queues • Custom ETL code • Trigger-based capture • Views over • Custom transaction database links code • Recording modification dates • File movements • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 20
  31. 31. Requirement: Change Data Capture • Oracle CDC • OWB Mappings • Oracle CDC and Process Flows • OWB Trickle-feed • OWB Trickle-feed mappings • Other ETL Tools mappings • Message Queues • Custom ETL code • Trigger-based capture • Views over • Custom transaction database links code • Recording modification dates • File movements • MINUS Command T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 20
  32. 32. Requirement: Change Data Capture T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 21
  33. 33. Requirement: Change Data Capture • Trickle-feed mappings allow the development of Streams Capture and Apply processes as continuously running mappings. T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 21
  34. 34. Federated OLTP/EDW Reporting T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 22
  35. 35. • Layering in OLTP data to provide the real-time component • General approach is to use a BI tool such as OBIEE • No real database or modeling involvement T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 23
  36. 36. Classic Batch Load Star Schema: Process Flow T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 24
  37. 37. Classic Batch Load Star Schema: Fact Table Load T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 25
  38. 38. Federation with OBIEE T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 26
  39. 39. OBIEE Metadata and Data Layers T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 27
  40. 40. OBIEE Metadata and Data Layers • Physical Data Sources, such as relational databases, Relational, OLAP, File and Application Data Sources OLAP Cubes and application source adapters T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 27
  41. 41. OBIEE Metadata and Data Layers • A Semantic Model made up of Metrics, dimensions, Enterprise Semantic Model hierarchies and calculations, derived from... • Physical Data Sources, such as relational databases, Relational, OLAP, File and Application Data Sources OLAP Cubes and application source adapters T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 27
  42. 42. OBIEE Metadata and Data Layers • The Web Catalog contains definitions of reports, alerts Web Catalog dashboards, KPIs, based on... • A Semantic Model made up of Metrics, dimensions, Enterprise Semantic Model hierarchies and calculations, derived from... • Physical Data Sources, such as relational databases, Relational, OLAP, File and Application Data Sources OLAP Cubes and application source adapters T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 27
  43. 43. Flow of Data Through the Three-Layer Semantic Model T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 28
  44. 44. Flow of Data Through the Three-Layer Semantic Model Simplification of the Data Model Integration of Disparate DataSources Addition of Business Logic and Calculations T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 28
  45. 45. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 29
  46. 46. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 29
  47. 47. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 29
  48. 48. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 29
  49. 49. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 30
  50. 50. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 30
  51. 51. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 30
  52. 52. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 31
  53. 53. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 31
  54. 54. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 31
  55. 55. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 31
  56. 56. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 32
  57. 57. Logical Table Sources Map to Physical Models T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 32
  58. 58. Logical Table Sources Respect the Physical Model T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 33
  59. 59. Logical Table Sources Respect the Physical Model T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 33
  60. 60. Logical Table Sources Respect the Physical Model T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 33
  61. 61. WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4 from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978 where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE (2010-09-18 00:00:00 , YYYY-MM-DD HH24:MI:SS) < T43978.TRANS_DATE ) union all select T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4 from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105 where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )), SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3 from SAWITH0 D3 group by D3.c2, D3.c3) select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4 from SAWITH1 D2 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 34
  62. 62. WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4 from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978 where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE (2010-09-18 00:00:00 , YYYY-MM-DD HH24:MI:SS) < T43978.TRANS_DATE ) union all select T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4 from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105 where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )), SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3 from SAWITH0 D3 group by D3.c2, D3.c3) select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4 from SAWITH1 D2 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 34
  63. 63. WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4 from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978 where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE (2010-09-18 00:00:00 , YYYY-MM-DD HH24:MI:SS) < T43978.TRANS_DATE ) union all select T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4 from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105 where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )), SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3 from SAWITH0 D3 group by D3.c2, D3.c3) select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4 from SAWITH1 D2 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 34
  64. 64. WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4 from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105 where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE(2003-01-09 00:00:00 , YYYY-MM-DD HH24:MI:SS) ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR) select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5 from SAWITH0 D1 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 35
  65. 65. WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4 from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105 where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE(2003-01-09 00:00:00 , YYYY-MM-DD HH24:MI:SS) ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR) select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5 from SAWITH0 D1 • The BI Server understands from the Fragmentation content how to “optimize” the query • Only one LTS is needed to provide the answer T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 35
  66. 66. Demonstration Fragmentation with OLTP and EDW T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 36
  67. 67. • Strengths ‣ Low latency ‣ No complex modeling or ETL ‣ Incorporated into any standard dimensional EDW • Weaknesses ‣ Performance ‣ Difficult to conform multiple systems T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 37
  68. 68. EDW with a Real-Time Component T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 38
  69. 69. • Modeling complimentary real-time structures in an EDW • Use functionality of the Oracle Database and OWB • Complex ETL and Database interaction T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 39
  70. 70. The Real-Time Partition Fact Table T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 40
  71. 71. The Real-Time Partition Fact Real-Time Fact Table Partition T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 40
  72. 72. The Real-Time Partition Fact Real-Time Fact Table Partition T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 40
  73. 73. The Real-Time Partition • All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch • No indexes, small data set • Partition Exchange at the end of the day T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 41
  74. 74. The Real-Time Partition • All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch • No indexes, small data set • Partition Exchange at the end of the day T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 41
  75. 75. The Real-Time Partition • All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch • No indexes, small data set • Partition Exchange at the end of the day T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 41
  76. 76. The Real-Time Partition • All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch • No indexes, small data set • Partition Exchange at the end of the day T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 41
  77. 77. Early-Arriving Facts T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 42
  78. 78. What if Type 2 dimension changes haven’t arrived yet? • Grab the most recent dimension record when that entity already exists • Use a default “Not Known Yet” record when they don’t • Prior to the partition exchange, we run our late- arriving dimension loads • We update the Real-Time Partition with the correct surrogate keys ‣ OWB generates an efficient MERGE ‣ Small data sets in the Real-Time Partition T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 43
  79. 79. Real-Time Integration: Fact Table RT Load T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 44
  80. 80. Real-Time Integration: Fact Table RT Load T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 44
  81. 81. Real-Time Integration: Correct Dimensions T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 45
  82. 82. Real-Time Integration: “Closing the Books” on the Day T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 46
  83. 83. Plugging it together with OBIEE (Non-Fragmentation) • Using standard database functionality (UNION ALL) • Could be done with a view • Using Surrogate Keys in the RT Partition • OBIEE doesn’t have to manage different LTS’s T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 47
  84. 84. Plugging it together with OBIEE (Non-Fragmentation) • Using standard database functionality (UNION ALL) • Could be done with a view • Using Surrogate Keys in the RT Partition • OBIEE doesn’t have to manage different LTS’s T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 47
  85. 85. Plugging it together with OBIEE (Non-Fragmentation) • Using standard database functionality (UNION ALL) • Could be done with a view • Using Surrogate Keys in the RT Partition • OBIEE doesn’t have to manage different LTS’s T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 47
  86. 86. Demonstration EDW with a Real-Time Component T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 48
  87. 87. Summary T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 49
  88. 88. • Latency versus performance and scalability • Latency and performance versus degree of difficulty • Different variations on the two test cases T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 50
  89. 89. Real-Time DW with OBIEE and Oracle Database Stewart Bryson, Managing Director, Rittman Mead America Collaborate, Orlando, April 2011 T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.comThursday, 21 April 2011 51
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×