Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DELICIOUS : ORACLE ENTERPRISE DATA QUALITY,
ORACLE GOLDEN GATE AND ORACLE DATA INTEGRATOR ON
ORACLE EXADATA IN THE CLOUD
6...
@gurcan_orhan #OGhTech17 6 JUNE 2017
WHO AM I?+20 years of IT experience.
+14 years of DWH experience.
+10 years of Oracle...
Ekol Germany
Warehousing
Solutions
begin with the
Kardelen
Facility
1996 2003 2010 2012 2014 2016
201520132011200820021990...
Headquarters
Ekol Locations
Current
countries
15 80
locations
201723
countries
92
locations
201831
countries
104
locations...
@gurcan_orhan #OGhTech17 6 JUNE 2017
Overview
@gurcan_orhan #OGhTech17 6 JUNE 2017
MY DEFINITION OF ODI
ODI (Oracle Data Integrator) is a tool,
that can talk,
or learn ...
@gurcan_orhan #OGhTech17 6 JUNE 2017
INGREDIENTS
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
EDQ : THE SALAMI
@gurcan_orhan #OGhTech17 6 JUNE 2017
EDQ : INTEGRATED SOLUTION FOR ALL DQ
Broadest DQ offering
• Best of breed capabilitie...
@gurcan_orhan #OGhTech17 6 JUNE 2017
EDQ : INTEGRATED SOLUTION FOR ALL DQ
• Process metrics
• Remediation
• Case Managemen...
@gurcan_orhan #OGhTech17 6 JUNE 2017
EDQ : OVERVIEW
@gurcan_orhan #OGhTech17 6 JUNE 2017
EDQ : KEY ATTRIBUTES FOR SUCCESS
RELEVANCE
ACCURACY
INTEGRITY
CONSISTENCY
COMPLETENES...
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
OGG : THE SAUCE
• Oracle’s standard tool for data replication
• Sub-second speed
http...
@gurcan_orhan #OGhTech17 6 JUNE 2017
OGG : ARCHITECTURE
• Oracle’s strategic tool for data replication and real-time
• Cap...
@gurcan_orhan #OGhTech17 6 JUNE 2017
OGG : STRENGTHS
• Sub-second speed
• Reliability
• Low impact on source and target sy...
@gurcan_orhan #OGhTech17 6 JUNE 2017
OGG : USES
• Real-time data integration
• Zero-downtime migration
• Disaster Recovery...
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : A DATABASE MACHINE
https://cloud.oracle.com/en_US/database
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : A DATABASE MACHINE
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : MIGRATION STEPS
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : MORE TO DO
Beware of
Partitioned tables’ related data blocks remains uncomp...
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : MORE TO DO
Search code, modify related code ➛ execute faster
@gurcan_orhan #OGhTech17 6 JUNE 2017
EXADATA : MORE TO DO
Don’t drop your indexes
If you are not using them for performanc...
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
ODI : THE CHEESE
• Oracle’s standard tool for data integration
• Can read and write f...
@gurcan_orhan #OGhTech17 6 JUNE 2017
ODI : E-L-T APPROACH
• Reduce network transfer
• Reduce hardware costs
• Set-based op...
@gurcan_orhan #OGhTech17 6 JUNE 2017
ODI : HOW DOES IT WORK?
• Declarative design, separation of business logic and techni...
@gurcan_orhan #OGhTech17 6 JUNE 2017
ODI : OVERVIEW
@gurcan_orhan #OGhTech17 6 JUNE 2017
• In ODI Studio
• In the repositories : at design-time…
ODI : LINEAGE
and at run-time
@gurcan_orhan #OGhTech17 6 JUNE 2017
ODI : EASE OF CHANGE
interactiveHelper.performAction(new InterfaceActionSetKM(myIkm,
...
@gurcan_orhan #OGhTech17 6 JUNE 2017
RECIPE
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : EASE OF USAGE
Performance
Oracle premier/platinum support
Pre-installed,...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : EASE OF CONTROL
Control replication
in ETL process
Control data
quality ...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : EASE OF WORKLOAD
Start loading
DWH immediately.
No more
extraction phase...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : ODI TOOLS
EDQ
• OdiEnterpriseDataQuality
CDC
• OdiWaitForData
• OdiWaitF...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : ODI AND OGG ➛ ARCHITECTURE
• Knowledge Module support for OGG
• JKM Orac...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : ODI AND GG ➛ SET UP
1. Model configuration 2. Datastore(s) selection
3. ...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : ODI AND GG ➛ INTEGRATION
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : ODI AND EXADATA
No extra hardware
• ODI Agent on Exadata JVM
• ODI Repos...
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME !
@gurcan_orhan #OGhTech17 6 JUNE 2017
PIZZA TIME : DELICIOUS
@gurcan_orhan #OGhTech17 6 JUNE 2017
http://gurcanorhan.wordpress.com
@gurcan_orhan
http://tr.linkedin.com/in/gurcanorhan
Upcoming SlideShare
Loading in …5
×

Delicious : EDQ, OGG and ODI over Exadata for Perfection

1,432 views

Published on

Oracle Golden Gate (OGG) handles extraction phase for operational reporting, Enterprise Data Quality (EDQ) leverage source system based data quality problems (misspells/duplications in defined or undefined conditions) and Oracle Data Integrator (ODI) handles and controls both of those tools and moreover loading your data into data warehouse.
These 3 tools are integrated to work together while you have control of OGG and EDQ via ODI over Exadata machine in a faster manner with minimum effort of development and tools already exist in ODI.
At the same time use Exadata's extensive features to decrease ETL jobs' duration and obtain high availibility.
In this presentation, see how this 3 tools are merged to be used together.

Published in: Technology, Business
  • Be the first to comment

Delicious : EDQ, OGG and ODI over Exadata for Perfection

  1. 1. DELICIOUS : ORACLE ENTERPRISE DATA QUALITY, ORACLE GOLDEN GATE AND ORACLE DATA INTEGRATOR ON ORACLE EXADATA IN THE CLOUD 6 June 2017 Gürcan Orhan Enterprise Data Warehouse Architect Ekol Logistics - Turkey http://gurcanorhan.wordpress.com @gurcan_orhan http://tr.linkedin.com/in/gurcanorhan
  2. 2. @gurcan_orhan #OGhTech17 6 JUNE 2017 WHO AM I?+20 years of IT experience. +14 years of DWH experience. +10 years of Oracle Data Integrator experience. +8 years of Oracle Warehouse Builder experience. Sybase Power Designer, ERwin Data Modeler, SDDM OBIEE, Cognos, Microstrategy, Business Objects, Qlikview, Tableau IBM Data Stage, SAP Data Services, Informatica, etc… Oracle Excellence Awards - Technologist of the Year 2011 : Enterprise Architect DWH & BI Chair : TROUG (Turkish Oracle User Group) Published Customer Snapshot for NODI @Oracle.com Published videos about ODI @Oracle.com Published OTN Podcasts about “Data Warehousing and ODI” “ODI and the Evolution of Data Integration” Lots of “2MTT”s Articles in OTech Magazine, SearchSoftwareQuality.com Annual panelist for ODTUG “Ask the Experts Panel : ODI” Presenter in OOW since 2010 (7 times in a row ⭐ ) Presenter in many OUG conferences in globe Presenter in various universities in Turkey
  3. 3. Ekol Germany Warehousing Solutions begin with the Kardelen Facility 1996 2003 2010 2012 2014 2016 201520132011200820021990 Acquire STS Int. Transport Ekol Bosnia Ekol France Ekol Greece Ekol Ukraine Ekol Spain Ekol Bulgaria Ekol Czech Rep. Ekol Iran Ekol PolandEkol Italy Ekol Romania Ekol HungaryAcquire Unok/Unatsa n Rainbow Replaced by Quadro (software) Intermodal operations Ro-Ro operations Established Ekol Milestones
  4. 4. Headquarters Ekol Locations Current countries 15 80 locations 201723 countries 92 locations 201831 countries 104 locations 201942 countries 122 locations USA CN
  5. 5. @gurcan_orhan #OGhTech17 6 JUNE 2017 Overview
  6. 6. @gurcan_orhan #OGhTech17 6 JUNE 2017 MY DEFINITION OF ODI ODI (Oracle Data Integrator) is a tool, that can talk, or learn how to talk, with any database system, or any operating system, in its own language. This is the power of ODI.
  7. 7. @gurcan_orhan #OGhTech17 6 JUNE 2017 INGREDIENTS
  8. 8. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  9. 9. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  10. 10. @gurcan_orhan #OGhTech17 6 JUNE 2017 EDQ : THE SALAMI
  11. 11. @gurcan_orhan #OGhTech17 6 JUNE 2017 EDQ : INTEGRATED SOLUTION FOR ALL DQ Broadest DQ offering • Best of breed capabilities for both Party Data and Product Data • Profiling, standardization, matching, case management, governance Most usable DQ offering • Completely integrated offering – designed to work together • Designed for business and technical users • Transparent operation and results – no black boxes Pervasive operation for enterprise quality governance • Within legacy systems and MDM Hubs • As part of migration/system load • On data entry/capture • As part of data movement/transfer
  12. 12. @gurcan_orhan #OGhTech17 6 JUNE 2017 EDQ : INTEGRATED SOLUTION FOR ALL DQ • Process metrics • Remediation • Case Management • Quality metrics • Party (individuals, households) match • Entity match • Semantic (category) match • Statistical match • Match review • Merge/survivorship • Global parse • Category parse • Extract • Transform • Address verification & geocoding • Substitute • Enrich • Classify • Statistics • Patterns • Phrases • Duplicates • Completeness • Max/min values
  13. 13. @gurcan_orhan #OGhTech17 6 JUNE 2017 EDQ : OVERVIEW
  14. 14. @gurcan_orhan #OGhTech17 6 JUNE 2017 EDQ : KEY ATTRIBUTES FOR SUCCESS RELEVANCE ACCURACY INTEGRITY CONSISTENCY COMPLETENESS VALIDITY TIMELINESS ACCESSIBILITY COMPLIANCE Is it relevant to its intended purpose? Is it correct and objective, and can it be validated? Does it have a coherent, logical structure? Is it consistent and easily to understand? Does it provide all the information required? Is it within acceptable parameters for the business? Is it up to date and available whenever required? Can it be easily accessed and exported to the target application? Does it comply with regulatory standards?
  15. 15. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  16. 16. @gurcan_orhan #OGhTech17 6 JUNE 2017 OGG : THE SAUCE • Oracle’s standard tool for data replication • Sub-second speed https://cloud.oracle.com/goldengate
  17. 17. @gurcan_orhan #OGhTech17 6 JUNE 2017 OGG : ARCHITECTURE • Oracle’s strategic tool for data replication and real-time • Capture data based on transaction log • Handle the distribution and the delivery of committed data in real-time • Heterogeneous systems (including to/from Big Data and to/from the Cloud) • Uni-directional or bi-directional
  18. 18. @gurcan_orhan #OGhTech17 6 JUNE 2017 OGG : STRENGTHS • Sub-second speed • Reliability • Low impact on source and target systems • Platform independent universal data format • Easy to deploy - simple configuration of parameter files • Conflict detection and resolution • Data encryption • Secure and Flexible
  19. 19. @gurcan_orhan #OGhTech17 6 JUNE 2017 OGG : USES • Real-time data integration • Zero-downtime migration • Disaster Recovery - Data Protection • Data Distribution
  20. 20. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  21. 21. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : A DATABASE MACHINE https://cloud.oracle.com/en_US/database
  22. 22. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : A DATABASE MACHINE
  23. 23. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : MIGRATION STEPS
  24. 24. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : MORE TO DO Beware of Partitioned tables’ related data blocks remains uncompressed on HCC after updating rows Use CTAS, IAS and Exchange Partition methods for performance
  25. 25. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : MORE TO DO Search code, modify related code ➛ execute faster
  26. 26. @gurcan_orhan #OGhTech17 6 JUNE 2017 EXADATA : MORE TO DO Don’t drop your indexes If you are not using them for performance issues
  27. 27. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  28. 28. @gurcan_orhan #OGhTech17 6 JUNE 2017 ODI : THE CHEESE • Oracle’s standard tool for data integration • Can read and write from various source / target technologies • Talks the language of the technology • Extensible through drivers, Knowledge Modules and ODI Tools https://cloud.oracle.com/en_US/data-integrator
  29. 29. @gurcan_orhan #OGhTech17 6 JUNE 2017 ODI : E-L-T APPROACH • Reduce network transfer • Reduce hardware costs • Set-based operations • Leverage target platform capabilities • Referential Integrity Check
  30. 30. @gurcan_orhan #OGhTech17 6 JUNE 2017 ODI : HOW DOES IT WORK? • Declarative design, separation of business logic and technical implementation • Reusability • Data Quality
  31. 31. @gurcan_orhan #OGhTech17 6 JUNE 2017 ODI : OVERVIEW
  32. 32. @gurcan_orhan #OGhTech17 6 JUNE 2017 • In ODI Studio • In the repositories : at design-time… ODI : LINEAGE and at run-time
  33. 33. @gurcan_orhan #OGhTech17 6 JUNE 2017 ODI : EASE OF CHANGE interactiveHelper.performAction(new InterfaceActionSetKM(myIkm, odiInterface.getTargetDataStore(), KMType.IKM, new KMOptionRetainerLazy())); KMOptionRetainerCopy KMOptionRetainerHomonym y interactiveHelper.performAction(new InterfaceActionSetKMOptionValue ( odiInterface.getTargetDataStore(), KMType.IKM, “DETECTION_STRATEGY”, “NOT_EXISTS”));
  34. 34. @gurcan_orhan #OGhTech17 6 JUNE 2017 RECIPE
  35. 35. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : EASE OF USAGE Performance Oracle premier/platinum support Pre-installed, fast time2market Mission critical high-availability
  36. 36. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : EASE OF CONTROL Control replication in ETL process Control data quality in ETL process Control session properties in ETL process Control database maintenance in ETL process
  37. 37. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : EASE OF WORKLOAD Start loading DWH immediately. No more extraction phase Alerts for data type, column size changes No ETL crash because of Data Quality errors Let Exadata finish ETL process as fast as can be…
  38. 38. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : ODI TOOLS EDQ • OdiEnterpriseDataQuality CDC • OdiWaitForData • OdiWaitForLogData • OdiWaitForTable • OdiRefreshJournalCount • OdiRetrieveJournalData
  39. 39. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : ODI AND OGG ➛ ARCHITECTURE • Knowledge Module support for OGG • JKM Oracle to Oracle Consistent (OGG) • Automatic deployment of OGG configuration files • Integration with ODI CDC framework • Uses same design and deployment approach as other ODI CDC KMs
  40. 40. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : ODI AND GG ➛ SET UP 1. Model configuration 2. Datastore(s) selection 3. Model journalization 4. Deploy GG files
  41. 41. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : ODI AND GG ➛ INTEGRATION
  42. 42. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : ODI AND EXADATA No extra hardware • ODI Agent on Exadata JVM • ODI Repository on Exadata DB Generic or Specific IKMs • IKM SQL Control Append_BIG_DATA • IKM Oracle PEL • IKM Oracle Incremental Update (MERGE) For Exadata • NOT EXISTS • MERGE (no staging) http://www.oracle.com/technetwork/indexes/samplecode/dataintegration-sample-522116.html https://blogs.oracle.com/warehousebuilder/entry/odi_11g_how_to_load
  43. 43. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME !
  44. 44. @gurcan_orhan #OGhTech17 6 JUNE 2017 PIZZA TIME : DELICIOUS
  45. 45. @gurcan_orhan #OGhTech17 6 JUNE 2017 http://gurcanorhan.wordpress.com @gurcan_orhan http://tr.linkedin.com/in/gurcanorhan

×