Gürcan Orhan (Principal Datawarehouse Consultant, Global Maksimum)
Jérôme Françoisse (Consultant, Rittman Mead Consulting)...
Gürcan Orhan (Principal Datawarehouse Consultant, Global Maksimum)
Jérôme Françoisse (Consultant, Rittman Mead Consulting)...
This session is brought to you by…
23 September 2013
23 SEPTEMBER / ORACLE OPEN WORLD
WHO AM I?
+19 years of IT experience.
+10 years of DWH experience.
+7 years of Oracle Dat...
Big & Fast Data
Processing
Data Visualization
Data
Science
Global Maksimum
Data & Information Technologies
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Jérôme Françoisse
• ...
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
• Oracle Gold partne...
23 SEPTEMBER / ORACLE OPEN WORLD
ODI (Oracle Data Integrator) is a tool,
that can talk,
or learn how to talk,
with any dat...
23 SEPTEMBER / ORACLE OPEN WORLD
INGREDIENTS
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
EDQ : THE SALAMI
23 SEPTEMBER / ORACLE OPEN WORLD
EDQ : INTEGRATED SOLUTION FOR ALL DQ
Broadest DQ offering
• Best of breed capabilities fo...
23 SEPTEMBER / ORACLE OPEN WORLD
EDQ : INTEGRATED SOLUTION FOR ALL DQ
• Process metrics
• Remediation
• Case Management
• ...
23 SEPTEMBER / ORACLE OPEN WORLD
EDQ : OVERVIEW
23 SEPTEMBER / ORACLE OPEN WORLD
EDQ : KEY ATTRIBUTES FOR SUCCESS
RELEVANCE
ACCURACY
INTEGRITY
CONSISTENCY
COMPLETENESS
VA...
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
OGG : THE SAUCE
• Oracle’s standard tool for data replication
• Sub-second speed
23 SEPTEMBER / ORACLE OPEN WORLD
OGG : ARCHITECTURE
• Oracle’s standard tool for data replication and real-time
• Capture ...
23 SEPTEMBER / ORACLE OPEN WORLD
OGG : STRENGTHS
• Sub-second speed
• Reliability
• Low impact on source and target system...
23 SEPTEMBER / ORACLE OPEN WORLD
OGG : USES
• Real-time data integration
• Zero-downtime migration
• Disaster Recovery - D...
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : A DATABASE MACHINE
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : A DATABASE MACHINE
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : MIGRATION STEPS
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : MORE TO DO
Beware of
Partitioned tables’ related data blocks remains uncompress...
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : MORE TO DO
(NOT) EXISTS instead of (NOT) IN
Search code, modify related code ➛ ...
23 SEPTEMBER / ORACLE OPEN WORLD
EXADATA : MORE TO DO
Don’t drop your indexes
If you are not using them for performance is...
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
ODI : THE CHEESE
• Oracle’s standard tool for data integration
• Can read and write from ...
23 SEPTEMBER / ORACLE OPEN WORLD
ODI : E-L-T APPROACH
• Reduce network transfer
• Reduce hardware costs
• Set-based operat...
23 SEPTEMBER / ORACLE OPEN WORLD
ODI : HOW DOES IT WORK?
• Declarative design, separation of business logic and technical
...
23 SEPTEMBER / ORACLE OPEN WORLD
ODI : OVERVIEW
23 SEPTEMBER / ORACLE OPEN WORLD
• In ODI Studio
• In the repositories : at design-time…
ODI : LINEAGE
and at run-time
23 SEPTEMBER / ORACLE OPEN WORLD
ODI : EASE OF CHANGE
interactiveHelper.performAction(new
InterfaceActionSetKM(myIkm, odiI...
23 SEPTEMBER / ORACLE OPEN WORLD
RECIPE
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : EASE OF ARCHITECTURE
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : EASE OF USAGE
Performance
Oracle premier/platinum support
Pre-installed, fas...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : EASE OF CONTROL
Control replication
in ETL process
Control data
quality in E...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : EASE OF WORKLOAD
Start loading
DWH immediately.
No more
extraction phase
Ale...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : ODI TOOLS
EDQ
• OdiEnterpriseDataQuality
CDC
• OdiWaitForData
• OdiWaitForLo...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : ODI AND OGG ➛ ARCHITECTURE
• Knowledge Module support for OGG
• JKM Oracle t...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : ODI AND GG ➛ SET UP
1. Model configuration 2. Datastore(s) selection
3. Mode...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : ODI AND GG ➛ INTEGRATION
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : ODI AND EXADATA
No extra hardware
• ODI Agent on Exadata JVM
• ODI Repositor...
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME !
23 SEPTEMBER / ORACLE OPEN WORLD
PIZZA TIME : DELICIOUS
23 SEPTEMBER / ORACLE OPEN WORLD
http://gurcanorhan.wordpress.com
gurcan_orhan
http://tr.linkedin.com/in/gurcanorhan
http:...
Upcoming SlideShare
Loading in...5
×

EDQ&OGG&ODI on Exadata for Perfection

585

Published on

Oracle Open World 2013;
Session ID : CON4067
Session Name : Oracle Data Quality Solutions, Oracle GoldenGate, and Oracle Data Integrator on Oracle Exadata
Session Date : 23rd September 2013
Session Time : 16:45 - 17:45

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
585
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "EDQ&OGG&ODI on Exadata for Perfection"

  1. 1. Gürcan Orhan (Principal Datawarehouse Consultant, Global Maksimum) Jérôme Françoisse (Consultant, Rittman Mead Consulting) CON4067 ORACLE DATA QUALITY SOLUTIONS, ORACLE GOLDENGATE, AND ORACLE DATA INTEGRATOR ON ORACLE EXADATA 23 September 2013
  2. 2. Gürcan Orhan (Principal Datawarehouse Consultant, Global Maksimum) Jérôme Françoisse (Consultant, Rittman Mead Consulting) CON4067 DELICIOUS : ENTERPRISE DATA QUALITY, GOLDEN GATE AND DATA INTEGRATOR OVER EXADATA FOR PERFECTION 23 September 2013
  3. 3. This session is brought to you by… 23 September 2013
  4. 4. 23 SEPTEMBER / ORACLE OPEN WORLD WHO AM I? +19 years of IT experience. +10 years of DWH experience. +7 years of Oracle Data Integrator experience, +5 years of Oracle Warehouse Builder experience. Sybase Power Designer, CA ERwin Data Modeler OBIEE, Cognos, Microstrategy, Business Objects, Tableau Joined Turkcell October 2008, Turkcell Technology March 2010, Global Maksimum June 2013 Oracle Excellence Awards - Technologist of the Year 2011 : Enterprise Architect (Oracle Magazine Editors’ Choice of Awards, Enterprise Architect) DWH & BI Chair : TROUG (Turkish Oracle User Group) Published Customer Snapshot for NODI @Oracle.com Published video about ODI @Oracle.com (Oracle Media Network) Published OTN Podcast about “Data Warehousing and ODI” Presenter in Oracle Open World since 2010 Presenter in ODTUG Kscope’11-13 Presenter in UKOUG, other OUGs Presenter in various universities in Turkey http://gurcanorhan.wordpress.com gurcan_orhan http://tr.linkedin.com/in/gurcanorhan
  5. 5. Big & Fast Data Processing Data Visualization Data Science Global Maksimum Data & Information Technologies
  6. 6. T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com Jérôme Françoisse • Consultant for Rittman Mead ‣ Oracle BI/DW Architect/Analyst/Developer • ODI Trainer • Providing ODI support on OTN Forums • Email : jerome.francoisse@rittmanmead.com • Twitter : @JeromeFr
  7. 7. T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com • Oracle Gold partner with offices in US (Atlanta), Europe, Australia and India • World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI • Provide consulting, training, global managed services for customers around the world • 120+ consultants including 1 Oracle ACE Director and 2 Oracle ACEs • All expert in Oracle BI, DW, EPM and analytics tech. Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE, Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca • 16 Sessions at Oracle Open World 2013 • Blog : http://www.rittmanmead.com/blog/ • Twitter : @rittmanmead Rittman Mead
  8. 8. 23 SEPTEMBER / ORACLE OPEN WORLD ODI (Oracle Data Integrator) is a tool, that can talk, or learn how to talk, with any database system, or any operating system, in its own language. This is the power of ODI. MY DEFINITION OF ODI
  9. 9. 23 SEPTEMBER / ORACLE OPEN WORLD INGREDIENTS
  10. 10. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  11. 11. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  12. 12. 23 SEPTEMBER / ORACLE OPEN WORLD EDQ : THE SALAMI
  13. 13. 23 SEPTEMBER / ORACLE OPEN WORLD EDQ : INTEGRATED SOLUTION FOR ALL DQ Broadest DQ offering • Best of breed capabilities for both Party Data and Product Data • Profiling, standardization, matching, case management, governance Most usable DQ offering • Completely integrated offering – designed to work together • Designed for business and technical users • Transparent operation and results – no black boxes Pervasive operation for enterprise quality governance • Within legacy systems and MDM Hubs • As part of migration/system load • On data entry/capture • As part of data movement/transfer
  14. 14. 23 SEPTEMBER / ORACLE OPEN WORLD EDQ : INTEGRATED SOLUTION FOR ALL DQ • Process metrics • Remediation • Case Management • Quality metrics • Party (individuals, households) match • Entity match • Semantic (category) match • Statistical match • Match review • Merge/survivorship • Global parse • Category parse • Extract • Transform • Address verification & geocoding • Substitute • Enrich • Classify • Statistics • Patterns • Phrases • Duplicates • Completeness • Max/min values
  15. 15. 23 SEPTEMBER / ORACLE OPEN WORLD EDQ : OVERVIEW
  16. 16. 23 SEPTEMBER / ORACLE OPEN WORLD EDQ : KEY ATTRIBUTES FOR SUCCESS RELEVANCE ACCURACY INTEGRITY CONSISTENCY COMPLETENESS VALIDITY TIMELINESS ACCESSIBILITY COMPLIANCE Is it relevant to its intended purpose? Is it correct and objective, and can it be validated? Does it have a coherent, logical structure? Is it consistent and easily to understand? Does it provide all the information required? Is it within acceptable parameters for the business? Is it up to date and available whenever required? Can it be easily accessed and exported to the target application? Does it comply with regulatory standards?
  17. 17. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  18. 18. 23 SEPTEMBER / ORACLE OPEN WORLD OGG : THE SAUCE • Oracle’s standard tool for data replication • Sub-second speed
  19. 19. 23 SEPTEMBER / ORACLE OPEN WORLD OGG : ARCHITECTURE • Oracle’s standard tool for data replication and real-time • Capture data based on transaction log • Handle the distribution and the delivery of committed data in real-time • Heterogeneous systems • Uni-directional or bi-directional
  20. 20. 23 SEPTEMBER / ORACLE OPEN WORLD OGG : STRENGTHS • Sub-second speed • Reliability • Low impact on source and target systems • Platform independent universal data format • Easy to deploy - simple configuration of parameter files • Conflict detection and resolution • Data encryption
  21. 21. 23 SEPTEMBER / ORACLE OPEN WORLD OGG : USES • Real-time data integration • Zero-downtime migration • Disaster Recovery - Data Protection • Data Distribution
  22. 22. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  23. 23. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : A DATABASE MACHINE
  24. 24. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : A DATABASE MACHINE
  25. 25. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : MIGRATION STEPS
  26. 26. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : MORE TO DO Beware of Partitioned tables’ related data blocks remains uncompressed on HCC after updating rows Use CTAS, IAS and Exchange Partition methods for performance
  27. 27. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : MORE TO DO (NOT) EXISTS instead of (NOT) IN Search code, modify related code ➛ execute faster
  28. 28. 23 SEPTEMBER / ORACLE OPEN WORLD EXADATA : MORE TO DO Don’t drop your indexes If you are not using them for performance issues
  29. 29. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  30. 30. 23 SEPTEMBER / ORACLE OPEN WORLD ODI : THE CHEESE • Oracle’s standard tool for data integration • Can read and write from various source / target technologies • Talks the language of the technology • Extensible through drivers, Knowledge Modules and ODI Tools
  31. 31. 23 SEPTEMBER / ORACLE OPEN WORLD ODI : E-L-T APPROACH • Reduce network transfer • Reduce hardware costs • Set-based operations • Leverage target platform capabilities • Referential Integrity Check
  32. 32. 23 SEPTEMBER / ORACLE OPEN WORLD ODI : HOW DOES IT WORK? • Declarative design, separation of business logic and technical implementation • Reusability • Data Quality
  33. 33. 23 SEPTEMBER / ORACLE OPEN WORLD ODI : OVERVIEW
  34. 34. 23 SEPTEMBER / ORACLE OPEN WORLD • In ODI Studio • In the repositories : at design-time… ODI : LINEAGE and at run-time
  35. 35. 23 SEPTEMBER / ORACLE OPEN WORLD ODI : EASE OF CHANGE interactiveHelper.performAction(new InterfaceActionSetKM(myIkm, odiInterface.getTargetDataStore(), KMType.IKM, ne w KMOptionRetainerLazy())); KMOptionRetainerCopy KMOptionRetainerHomonym y interactiveHelper.performAction(new InterfaceActionSetKMOptionValue ( odiInterface.getTargetDataStore(), KMType.IKM, “DETECTION_STRATEGY”, “NO T_EXISTS”));
  36. 36. 23 SEPTEMBER / ORACLE OPEN WORLD RECIPE
  37. 37. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : EASE OF ARCHITECTURE
  38. 38. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : EASE OF USAGE Performance Oracle premier/platinum support Pre-installed, fast time2market Mission critical high-availability
  39. 39. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : EASE OF CONTROL Control replication in ETL process Control data quality in ETL process Control session properties in ETL process Control database maintenance in ETL process
  40. 40. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : EASE OF WORKLOAD Start loading DWH immediately. No more extraction phase Alerts for data type, column size changes No ETL crash because of Data Quality errors Let Exadata finish ETL process as fast as can be…
  41. 41. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : ODI TOOLS EDQ • OdiEnterpriseDataQuality CDC • OdiWaitForData • OdiWaitForLogData • OdiWaitForTable • OdiRefreshJournalCount • OdiRetrieveJournalData
  42. 42. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : ODI AND OGG ➛ ARCHITECTURE • Knowledge Module support for OGG • JKM Oracle to Oracle Consistent (OGG) • Automatic deployment of OGG configuration files • Integration with ODI CDC framework • Uses same design and deployment approach as other ODI CDC KMs
  43. 43. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : ODI AND GG ➛ SET UP 1. Model configuration 2. Datastore(s) selection 3. Model journalization 4. Deploy GG files
  44. 44. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : ODI AND GG ➛ INTEGRATION
  45. 45. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : ODI AND EXADATA No extra hardware • ODI Agent on Exadata JVM • ODI Repository on Exadata DB Generic or Specific IKMs • IKM SQL Control Append_BIG_DATA • IKM Oracle PEL • IKM Oracle Incremental Update (MERGE) For Exadata • NOT EXISTS • MERGE (no staging) http://www.oracle.com/technetwork/indexes/samplecode/dataintegration-sample-522116.html https://blogs.oracle.com/warehousebuilder/entry/odi_11g_how_to_load
  46. 46. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME !
  47. 47. 23 SEPTEMBER / ORACLE OPEN WORLD PIZZA TIME : DELICIOUS
  48. 48. 23 SEPTEMBER / ORACLE OPEN WORLD http://gurcanorhan.wordpress.com gurcan_orhan http://tr.linkedin.com/in/gurcanorhan http://www.rittmanmead.com/blog JeromeFr http://www.linkedin.com/in/jeromefrancoisse Session ID : CON4070 Session Title : How to Handle DEV&TEST&PROD for Oracle Data Integrator Venue / Room : Moscone West – 2012 Date / Time : 9/26/13, 2:00 pm - 3:00 pm

×