- The document discusses upgrading Oracle GoldenGate 11g and Oracle Data Integrator 11g to their 12c versions. It provides an overview of the steps to upgrade each product including preparing the source and target systems, installing 12c, updating supplemental logging, and finalizing the upgrade by altering processes to write a new sequence number. It also discusses using the convprm tool to convert GoldenGate parameter files during the upgrade process.
GoldenGate and Oracle Data Integrator - A Perfect Match...Michael Rainey
Oracle Data Integrator and Oracle GoldenGate excel as standalone products, but paired together they are the perfect match for real-time data warehousing. Following Oracle’s Next Generation Reference Data Warehouse Architecture, this discussion will provide best practices on how to configure, implement, and process data in real-time using ODI and GoldenGate. Attendees will see common real-time challenges solved, including parent-child relationships within micro-batch ETL.
Presented at Rittman Mead BI Forum 2013 Masterclass.
GoldenGate and ODI - A Perfect Match for Real-Time Data WarehousingMichael Rainey
Oracle Data Integrator and Oracle GoldenGate excel as standalone products, but paired together they are the perfect match for real-time data warehousing. Following Oracle’s Next Generation Reference Data Warehouse Architecture, this discussion will provide best practices on how to configure, implement, and process data in real-time using ODI and GoldenGate. Attendees will see common real-time challenges solved, including parent-child relationships within micro-batch ETL.
Presented at RMOUG Training Days 2013 & KScope13.
Tame Big Data with Oracle Data IntegrationMichael Rainey
In this session, Oracle Product Management covers how Oracle Data Integrator and Oracle GoldenGate are vital to big data initiatives across the enterprise, providing the movement, translation, and transformation of information and data not only heterogeneously but also in big data environments. Through a metadata-focused approach for cataloging, defining, and reusing big data technologies such as Hive, Hadoop Distributed File System (HDFS), HBase, Sqoop, Pig, Oracle Loader for Hadoop, Oracle SQL Connector for Hadoop Distributed File System, and additional big data projects, Oracle Data Integrator bridges the gap in the ability to unify data across these systems and helps deliver timely and trusted data to analytic and decision support platforms.
Co-presented with Alex Kotopoulis at Oracle OpenWorld 2014.
Real-Time Data Replication to Hadoop using GoldenGate 12c AdaptorsMichael Rainey
Oracle GoldenGate 12c is well known for its highly performant data replication between relational databases. With the GoldenGate Adaptors, the tool can now apply the source transactions to a Big Data target, such as HDFS. In this session, we'll explore the different options for utilizing Oracle GoldenGate 12c to perform real-time data replication from a relational source database into HDFS. The GoldenGate Adaptors will be used to load movie data from the source to HDFS for use by Hive. Next, we'll take the demo a step further and publish the source transactions to a Flume agent, allowing Flume to handle the final load into the targets.
Presented at the Oracle Technology Network Virtual Technology Summit February/March 2015.
Oracle GoldenGate, Streams, & Oracle Data Integrator
GoldenGate is Oracle’s strategic replication solution while Streams continues as a supported Oracle Database feature. GoldenGate has advantages over Streams such as better ROI, being an enterprise-wide solution, and having higher reliability, flexibility and performance. Oracle Data Integrator is an ETL tool that can be used with GoldenGate for periodic data loads from operational systems to data warehouses with lower latency compared to batch-oriented tools.
Practical Tips for Oracle Business Intelligence Applications 11g ImplementationsMichael Rainey
The document provides practical tips for Oracle Business Intelligence Applications 11g implementations. It discusses scripting installations and configurations, LDAP integration challenges, implementing high availability, different methods for data extracts, and simplifying disaster recovery. Specific tips include scripting all processes, configuring the ODI agent JVM and connection pools for performance, understanding external LDAP authentication in ODI, implementing active-active high availability for ODI agents, choosing the right data extract method based on latency and volume, and using DataGuard and CNAMEs to simplify failover for disaster recovery.
Real-time Data Warehouse Upgrade – Success StoriesMichael Rainey
Providing a real-time BI solution for its global customers and operations department is a necessity for IFPI, the International Federation of the Phonographic Industry, whose primary objective is to safeguard the rights of record producers through various anti-piracy strategies.
For the data warehousing team at IFPI, using Oracle Streams and Oracle Warehouse Builder (OWB) for real-time data replication and integration was becoming a challenge. The solution was difficult to maintain and overall throughput was degrading as data volumes increased. The need for greater stability and performance led IFPI to implement Oracle GoldenGate and Oracle Data Integrator.
Co-presented with Nick Hurt at Rittman Mead BI Forum 2014 and KScope14.
Integrating Oracle Data Integrator with Oracle GoldenGate 12cEdelweiss Kammermann
The document discusses integrating Oracle Data Integrator (ODI) with Oracle GoldenGate (OGG) for real-time data integration. It describes how OGG captures change data from source systems and delivers it to ODI. Key steps include configuring OGG installations and JAgents, defining OGG data servers in ODI, applying journalizing to ODI models, and creating and starting ODI processes that integrate with the OGG capture and delivery processes. The integration provides benefits like low impact on sources, great performance for real-time integration, and support for heterogeneous databases.
GoldenGate and Oracle Data Integrator - A Perfect Match...Michael Rainey
Oracle Data Integrator and Oracle GoldenGate excel as standalone products, but paired together they are the perfect match for real-time data warehousing. Following Oracle’s Next Generation Reference Data Warehouse Architecture, this discussion will provide best practices on how to configure, implement, and process data in real-time using ODI and GoldenGate. Attendees will see common real-time challenges solved, including parent-child relationships within micro-batch ETL.
Presented at Rittman Mead BI Forum 2013 Masterclass.
GoldenGate and ODI - A Perfect Match for Real-Time Data WarehousingMichael Rainey
Oracle Data Integrator and Oracle GoldenGate excel as standalone products, but paired together they are the perfect match for real-time data warehousing. Following Oracle’s Next Generation Reference Data Warehouse Architecture, this discussion will provide best practices on how to configure, implement, and process data in real-time using ODI and GoldenGate. Attendees will see common real-time challenges solved, including parent-child relationships within micro-batch ETL.
Presented at RMOUG Training Days 2013 & KScope13.
Tame Big Data with Oracle Data IntegrationMichael Rainey
In this session, Oracle Product Management covers how Oracle Data Integrator and Oracle GoldenGate are vital to big data initiatives across the enterprise, providing the movement, translation, and transformation of information and data not only heterogeneously but also in big data environments. Through a metadata-focused approach for cataloging, defining, and reusing big data technologies such as Hive, Hadoop Distributed File System (HDFS), HBase, Sqoop, Pig, Oracle Loader for Hadoop, Oracle SQL Connector for Hadoop Distributed File System, and additional big data projects, Oracle Data Integrator bridges the gap in the ability to unify data across these systems and helps deliver timely and trusted data to analytic and decision support platforms.
Co-presented with Alex Kotopoulis at Oracle OpenWorld 2014.
Real-Time Data Replication to Hadoop using GoldenGate 12c AdaptorsMichael Rainey
Oracle GoldenGate 12c is well known for its highly performant data replication between relational databases. With the GoldenGate Adaptors, the tool can now apply the source transactions to a Big Data target, such as HDFS. In this session, we'll explore the different options for utilizing Oracle GoldenGate 12c to perform real-time data replication from a relational source database into HDFS. The GoldenGate Adaptors will be used to load movie data from the source to HDFS for use by Hive. Next, we'll take the demo a step further and publish the source transactions to a Flume agent, allowing Flume to handle the final load into the targets.
Presented at the Oracle Technology Network Virtual Technology Summit February/March 2015.
Oracle GoldenGate, Streams, & Oracle Data Integrator
GoldenGate is Oracle’s strategic replication solution while Streams continues as a supported Oracle Database feature. GoldenGate has advantages over Streams such as better ROI, being an enterprise-wide solution, and having higher reliability, flexibility and performance. Oracle Data Integrator is an ETL tool that can be used with GoldenGate for periodic data loads from operational systems to data warehouses with lower latency compared to batch-oriented tools.
Practical Tips for Oracle Business Intelligence Applications 11g ImplementationsMichael Rainey
The document provides practical tips for Oracle Business Intelligence Applications 11g implementations. It discusses scripting installations and configurations, LDAP integration challenges, implementing high availability, different methods for data extracts, and simplifying disaster recovery. Specific tips include scripting all processes, configuring the ODI agent JVM and connection pools for performance, understanding external LDAP authentication in ODI, implementing active-active high availability for ODI agents, choosing the right data extract method based on latency and volume, and using DataGuard and CNAMEs to simplify failover for disaster recovery.
Real-time Data Warehouse Upgrade – Success StoriesMichael Rainey
Providing a real-time BI solution for its global customers and operations department is a necessity for IFPI, the International Federation of the Phonographic Industry, whose primary objective is to safeguard the rights of record producers through various anti-piracy strategies.
For the data warehousing team at IFPI, using Oracle Streams and Oracle Warehouse Builder (OWB) for real-time data replication and integration was becoming a challenge. The solution was difficult to maintain and overall throughput was degrading as data volumes increased. The need for greater stability and performance led IFPI to implement Oracle GoldenGate and Oracle Data Integrator.
Co-presented with Nick Hurt at Rittman Mead BI Forum 2014 and KScope14.
Integrating Oracle Data Integrator with Oracle GoldenGate 12cEdelweiss Kammermann
The document discusses integrating Oracle Data Integrator (ODI) with Oracle GoldenGate (OGG) for real-time data integration. It describes how OGG captures change data from source systems and delivers it to ODI. Key steps include configuring OGG installations and JAgents, defining OGG data servers in ODI, applying journalizing to ODI models, and creating and starting ODI processes that integrate with the OGG capture and delivery processes. The integration provides benefits like low impact on sources, great performance for real-time integration, and support for heterogeneous databases.
Offload, Transform, and Present - the New World of Data IntegrationMichael Rainey
How much time and effort (and budget) do organizations spend moving data around the enterprise? Unfortunately, quite a lot. These days, ETL developers are tasked with performing the Extract (E) and Load (L), and spending less time on their craft, building Transformations (T). This changes in the new world of data integration. By offloading data from the RDBMS to Hadoop, with the ability to present it back to the relational database, data can be seamlessly integrated between different source and target systems. Transformations occur on data offloaded to Hadoop, using the latest ETL technologies, or in the target database, with a standard ETL-on-RDBMS tool. In this session, we’ll discuss how the new world of data integration will provide focus on transforming data into insightful information by simplifying the data movement process.
Presented at Enkitec E4 2017.
This document provides an overview of Oracle GoldenGate 12c, a heterogeneous replication tool. It describes GoldenGate's key features like real-time data integration and query offloading. The document outlines GoldenGate's topologies, architecture, supported databases, and data types. It compares GoldenGate to Oracle Streams and details new features in 12c like optimized capture methods and improved high availability. Basic concepts are explained, such as classic and integrated capture, downstream and bi-directional replication. Restrictions on data types and database features are also noted.
ODI12c as your Big Data Integration HubMark Rittman
Presentation from the recent Oracle OTN Virtual Technology Summit, on using Oracle Data Integrator 12c to ingest, transform and process data on a Hadoop cluster.
Oracle GoldenGate is the leading real-time data integration software provider in the industry - customers include 3 of the top 5 commercial banks, 3 of the top 3 busiest ATM networks, and 4 of the top 5 telecommunications providers.
Oracle GoldenGate moves transactional data in real-time across heterogeneous database, hardware and operating systems with minimal impact. The software platform captures, routes, and delivers data in real time, enabling organizations to maintain continuous uptime for critical applications during planned and unplanned outages.
Additionally, it moves data from transaction processing environments to read-only reporting databases and analytical applications for accurate, timely reporting and improved business intelligence for the enterprise.
Enable GoldenGate Monitoring with OEM 12c/JAgentBobby Curtis
Oracle GoldenGate monitoring can be performed using Oracle Enterprise Manager 12c. The Oracle GoldenGate plug-in for Oracle Enterprise Manager 12c provides monitoring capabilities such as viewing process status and lag times, accessing log files, starting and stopping processes, and viewing statistics. In addition to the plug-in, credentials must be configured and the GoldenGate processes must be running in order to monitor GoldenGate using Oracle Enterprise Manager 12c.
Presentación de Oracle Database Cloud Service como servicio en la nube, tema de interés puntero puesto que actualmente la dirección de las empresas va en ese punto de llevar sus bases de datos y aplicaciones a la nube.
End to-end hadoop development using OBIEE, ODI, Oracle Big Data SQL and Oracl...Mark Rittman
This document discusses an end-to-end example of using Hadoop, OBIEE, ODI and Oracle Big Data Discovery to analyze big data from various sources. It describes ingesting website log data and Twitter data into a Hadoop cluster, processing and transforming the data using tools like Hive and Spark, and using the results for reporting in OBIEE and data discovery in Oracle Big Data Discovery. ODI is used to automate the data integration process.
Part 1 - Introduction to Hadoop and Big Data Technologies for Oracle BI & DW ...Mark Rittman
Delivered as a one-day seminar at the SIOUG and HROUG Oracle User Group Conferences, October 2014
In this presentation we cover some key Hadoop concepts including HDFS, MapReduce, Hive and NoSQL/HBase, with the focus on Oracle Big Data Appliance and Cloudera Distribution including Hadoop. We explain how data is stored on a Hadoop system and the high-level ways it is accessed and analysed, and outline Oracle’s products in this area including the Big Data Connectors, Oracle Big Data SQL, and Oracle Business Intelligence (OBI) and Oracle Data Integrator (ODI).
What is Big Data Discovery, and how it complements traditional business anal...Mark Rittman
Data Discovery is an analysis technique that complements traditional business analytics, and enables users to combine, explore and analyse disparate datasets to spot opportunities and patterns that lie hidden within your data. Oracle Big Data discovery takes this idea and applies it to your unstructured and big data datasets, giving users a way to catalogue, join and then analyse all types of data across your organization.
In this session we'll look at Oracle Big Data Discovery and how it provides a "visual face" to your big data initatives, and how it complements and extends the work that you currently do using business analytics tools.
Oracle GoldenGate Demo and Data Integration ConceptsFumiko Yamashita
The document discusses an Oracle GoldenGate demo and data integration concepts. It provides an agenda that includes an Oracle GoldenGate UI demo of migrating data from a Siebel 7 and Database 10g source system to a Siebel 8 and Database 11g target system. It also discusses a technical demo of data transformation. Additionally, it covers data integration concepts such as operational reporting vs data warehousing, master data management, and change data capture.
BIWA2015 - Bringing Oracle Big Data SQL to OBIEE and ODIMark Rittman
The document discusses Oracle's Big Data SQL, which brings Oracle SQL capabilities to Hadoop data stored in Hive tables. It allows querying Hive data using standard SQL from Oracle Database and viewing Hive metadata in Oracle data dictionary tables. Big Data SQL leverages the Hive metastore and uses direct reads and SmartScan to optimize queries against HDFS and Hive data. This provides a unified SQL interface and optimized query processing for both Oracle and Hadoop data.
Deploying Full Oracle BI Platforms to Oracle Cloud - OOW2015Mark Rittman
- Mark Rittman presented on deploying full OBIEE systems to Oracle Cloud. This involves migrating the data warehouse to Oracle Database Cloud Service, updating the RPD to connect to the cloud database, and uploading the RPD to Oracle BI Cloud Service. Using the wider Oracle PaaS ecosystem allows hosting a full BI platform in the cloud.
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Rittman Analytics
Oracle Data Integration Platform is a cornerstone for big data solutions that provides five core capabilities: business continuity, data movement, data transformation, data governance, and streaming data handling. It includes eight core products that can operate in the cloud or on-premise, and is considered the most innovative in areas like real-time/streaming integration and extract-load-transform capabilities with big data technologies. The platform offers a comprehensive architecture covering key areas like data ingestion, preparation, streaming integration, parallel connectivity, and governance.
Presentación sobre la futura base de datos 18c, en la cual se incorpora todo lo mejor de las tecnologías Oracle, perfilando así una base de datos autónoma.
Presentation delivered at Oracle Open World 2019 (San Francisco, California)
Abstract: This session walks through everything you need to know to move your databases to Oracle Cloud, based on the speakers’ own experience in doing it, focused on Oracle Exadata cloud services. The speakers review the configuration required, common problems they had to resolve in the process of having several TB databases running in Oracle Cloud, and operational tasks after they were running there.
The document discusses transforming data management to the cloud. It describes how Oracle's database cloud services provide complete data management across multiple data types at any scale both on-premises and in the cloud. It highlights how the cloud offers lower costs through pay-as-you-go pricing and lower operational expenses, as well as increased agility through rapid provisioning and elastic scaling. Oracle 12c and new features in 12c Release 2 provide database consolidation, isolation at scale, and online operations for pluggable databases in the cloud.
The document discusses tuning Oracle GoldenGate performance, including available tools for monitoring replication lag and throughput. It presents a case study examining lag times of over 1 hour 30 minutes for a replication configuration and uses tools like the Streams Performance Advisor and lag reports to identify potential bottlenecks. Recommendations are provided for configuration changes and monitoring to improve replication performance.
Best Practices – Extreme Performance with Data Warehousing on Oracle DatabaseEdgar Alejandro Villegas
The document discusses best practices for data warehousing performance on Oracle Database. It covers using Oracle Exadata Database Machine for mixed workloads including data warehousing. Key strategies discussed are partitioning tables for pruning and parallelism, using hybrid columnar compression for storage savings and faster scans, and enabling auto parallelism and queuing for optimal parallel query processing.
This document discusses optimizing testing using data virtualization. It describes how data is often the constraint in software development and testing processes. Traditional attempts to solve this problem, like copying subsets of production data or taking snapshots, are inefficient and don't provide developers and testers access to fresh, full production data. The document introduces data virtualization as a solution, allowing instant provisioning of full production databases on demand for various testing environments.
The course presents the debate on the methodology of social research trying to highlight the potentiality, the limitations and the rules of the different social research approaches. The focus will be on the relationship between information and decision for highlight the centrality of the reliability concept. Will be presented the general rules of qualitative and quantitative research and of the concensensus method.
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Coll...Michael Rainey
Big Data integration is an excellent feature in the Oracle Data Integration product suite (Oracle Data Integrator, GoldenGate, & Enterprise Data Quality). But not all analytics require big data technologies, such as labor cost, revenue, or expense reporting. Ralph Kimball, an original architect of the dimensional model in data warehousing, spent much of his career working to build an enterprise data warehouse methodology that can meet these reporting needs. His book, "The Data Warehouse ETL Toolkit", is a guide for many ETL developers. This session will walk you through his ETL Subsystem categories; Extracting, Cleaning & Conforming, Delivering, and Managing, describing how the Oracle Data Integration products are perfectly suited for the Kimball approach.
Presented at Collaborate16 in Las Vegas.
Offload, Transform, and Present - the New World of Data IntegrationMichael Rainey
How much time and effort (and budget) do organizations spend moving data around the enterprise? Unfortunately, quite a lot. These days, ETL developers are tasked with performing the Extract (E) and Load (L), and spending less time on their craft, building Transformations (T). This changes in the new world of data integration. By offloading data from the RDBMS to Hadoop, with the ability to present it back to the relational database, data can be seamlessly integrated between different source and target systems. Transformations occur on data offloaded to Hadoop, using the latest ETL technologies, or in the target database, with a standard ETL-on-RDBMS tool. In this session, we’ll discuss how the new world of data integration will provide focus on transforming data into insightful information by simplifying the data movement process.
Presented at Enkitec E4 2017.
This document provides an overview of Oracle GoldenGate 12c, a heterogeneous replication tool. It describes GoldenGate's key features like real-time data integration and query offloading. The document outlines GoldenGate's topologies, architecture, supported databases, and data types. It compares GoldenGate to Oracle Streams and details new features in 12c like optimized capture methods and improved high availability. Basic concepts are explained, such as classic and integrated capture, downstream and bi-directional replication. Restrictions on data types and database features are also noted.
ODI12c as your Big Data Integration HubMark Rittman
Presentation from the recent Oracle OTN Virtual Technology Summit, on using Oracle Data Integrator 12c to ingest, transform and process data on a Hadoop cluster.
Oracle GoldenGate is the leading real-time data integration software provider in the industry - customers include 3 of the top 5 commercial banks, 3 of the top 3 busiest ATM networks, and 4 of the top 5 telecommunications providers.
Oracle GoldenGate moves transactional data in real-time across heterogeneous database, hardware and operating systems with minimal impact. The software platform captures, routes, and delivers data in real time, enabling organizations to maintain continuous uptime for critical applications during planned and unplanned outages.
Additionally, it moves data from transaction processing environments to read-only reporting databases and analytical applications for accurate, timely reporting and improved business intelligence for the enterprise.
Enable GoldenGate Monitoring with OEM 12c/JAgentBobby Curtis
Oracle GoldenGate monitoring can be performed using Oracle Enterprise Manager 12c. The Oracle GoldenGate plug-in for Oracle Enterprise Manager 12c provides monitoring capabilities such as viewing process status and lag times, accessing log files, starting and stopping processes, and viewing statistics. In addition to the plug-in, credentials must be configured and the GoldenGate processes must be running in order to monitor GoldenGate using Oracle Enterprise Manager 12c.
Presentación de Oracle Database Cloud Service como servicio en la nube, tema de interés puntero puesto que actualmente la dirección de las empresas va en ese punto de llevar sus bases de datos y aplicaciones a la nube.
End to-end hadoop development using OBIEE, ODI, Oracle Big Data SQL and Oracl...Mark Rittman
This document discusses an end-to-end example of using Hadoop, OBIEE, ODI and Oracle Big Data Discovery to analyze big data from various sources. It describes ingesting website log data and Twitter data into a Hadoop cluster, processing and transforming the data using tools like Hive and Spark, and using the results for reporting in OBIEE and data discovery in Oracle Big Data Discovery. ODI is used to automate the data integration process.
Part 1 - Introduction to Hadoop and Big Data Technologies for Oracle BI & DW ...Mark Rittman
Delivered as a one-day seminar at the SIOUG and HROUG Oracle User Group Conferences, October 2014
In this presentation we cover some key Hadoop concepts including HDFS, MapReduce, Hive and NoSQL/HBase, with the focus on Oracle Big Data Appliance and Cloudera Distribution including Hadoop. We explain how data is stored on a Hadoop system and the high-level ways it is accessed and analysed, and outline Oracle’s products in this area including the Big Data Connectors, Oracle Big Data SQL, and Oracle Business Intelligence (OBI) and Oracle Data Integrator (ODI).
What is Big Data Discovery, and how it complements traditional business anal...Mark Rittman
Data Discovery is an analysis technique that complements traditional business analytics, and enables users to combine, explore and analyse disparate datasets to spot opportunities and patterns that lie hidden within your data. Oracle Big Data discovery takes this idea and applies it to your unstructured and big data datasets, giving users a way to catalogue, join and then analyse all types of data across your organization.
In this session we'll look at Oracle Big Data Discovery and how it provides a "visual face" to your big data initatives, and how it complements and extends the work that you currently do using business analytics tools.
Oracle GoldenGate Demo and Data Integration ConceptsFumiko Yamashita
The document discusses an Oracle GoldenGate demo and data integration concepts. It provides an agenda that includes an Oracle GoldenGate UI demo of migrating data from a Siebel 7 and Database 10g source system to a Siebel 8 and Database 11g target system. It also discusses a technical demo of data transformation. Additionally, it covers data integration concepts such as operational reporting vs data warehousing, master data management, and change data capture.
BIWA2015 - Bringing Oracle Big Data SQL to OBIEE and ODIMark Rittman
The document discusses Oracle's Big Data SQL, which brings Oracle SQL capabilities to Hadoop data stored in Hive tables. It allows querying Hive data using standard SQL from Oracle Database and viewing Hive metadata in Oracle data dictionary tables. Big Data SQL leverages the Hive metastore and uses direct reads and SmartScan to optimize queries against HDFS and Hive data. This provides a unified SQL interface and optimized query processing for both Oracle and Hadoop data.
Deploying Full Oracle BI Platforms to Oracle Cloud - OOW2015Mark Rittman
- Mark Rittman presented on deploying full OBIEE systems to Oracle Cloud. This involves migrating the data warehouse to Oracle Database Cloud Service, updating the RPD to connect to the cloud database, and uploading the RPD to Oracle BI Cloud Service. Using the wider Oracle PaaS ecosystem allows hosting a full BI platform in the cloud.
Data Integration for Big Data (OOW 2016, Co-Presented With Oracle)Rittman Analytics
Oracle Data Integration Platform is a cornerstone for big data solutions that provides five core capabilities: business continuity, data movement, data transformation, data governance, and streaming data handling. It includes eight core products that can operate in the cloud or on-premise, and is considered the most innovative in areas like real-time/streaming integration and extract-load-transform capabilities with big data technologies. The platform offers a comprehensive architecture covering key areas like data ingestion, preparation, streaming integration, parallel connectivity, and governance.
Presentación sobre la futura base de datos 18c, en la cual se incorpora todo lo mejor de las tecnologías Oracle, perfilando así una base de datos autónoma.
Presentation delivered at Oracle Open World 2019 (San Francisco, California)
Abstract: This session walks through everything you need to know to move your databases to Oracle Cloud, based on the speakers’ own experience in doing it, focused on Oracle Exadata cloud services. The speakers review the configuration required, common problems they had to resolve in the process of having several TB databases running in Oracle Cloud, and operational tasks after they were running there.
The document discusses transforming data management to the cloud. It describes how Oracle's database cloud services provide complete data management across multiple data types at any scale both on-premises and in the cloud. It highlights how the cloud offers lower costs through pay-as-you-go pricing and lower operational expenses, as well as increased agility through rapid provisioning and elastic scaling. Oracle 12c and new features in 12c Release 2 provide database consolidation, isolation at scale, and online operations for pluggable databases in the cloud.
The document discusses tuning Oracle GoldenGate performance, including available tools for monitoring replication lag and throughput. It presents a case study examining lag times of over 1 hour 30 minutes for a replication configuration and uses tools like the Streams Performance Advisor and lag reports to identify potential bottlenecks. Recommendations are provided for configuration changes and monitoring to improve replication performance.
Best Practices – Extreme Performance with Data Warehousing on Oracle DatabaseEdgar Alejandro Villegas
The document discusses best practices for data warehousing performance on Oracle Database. It covers using Oracle Exadata Database Machine for mixed workloads including data warehousing. Key strategies discussed are partitioning tables for pruning and parallelism, using hybrid columnar compression for storage savings and faster scans, and enabling auto parallelism and queuing for optimal parallel query processing.
This document discusses optimizing testing using data virtualization. It describes how data is often the constraint in software development and testing processes. Traditional attempts to solve this problem, like copying subsets of production data or taking snapshots, are inefficient and don't provide developers and testers access to fresh, full production data. The document introduces data virtualization as a solution, allowing instant provisioning of full production databases on demand for various testing environments.
The course presents the debate on the methodology of social research trying to highlight the potentiality, the limitations and the rules of the different social research approaches. The focus will be on the relationship between information and decision for highlight the centrality of the reliability concept. Will be presented the general rules of qualitative and quantitative research and of the concensensus method.
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Coll...Michael Rainey
Big Data integration is an excellent feature in the Oracle Data Integration product suite (Oracle Data Integrator, GoldenGate, & Enterprise Data Quality). But not all analytics require big data technologies, such as labor cost, revenue, or expense reporting. Ralph Kimball, an original architect of the dimensional model in data warehousing, spent much of his career working to build an enterprise data warehouse methodology that can meet these reporting needs. His book, "The Data Warehouse ETL Toolkit", is a guide for many ETL developers. This session will walk you through his ETL Subsystem categories; Extracting, Cleaning & Conforming, Delivering, and Managing, describing how the Oracle Data Integration products are perfectly suited for the Kimball approach.
Presented at Collaborate16 in Las Vegas.
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationMichael Rainey
The document discusses various aspects of ETL subsystems based on Ralph Kimball's model. It begins with an introduction and background on the presenter and Rittman Mead. It then covers the major subsystems involved in extracting, cleaning and conforming data when using Oracle Data Integration and Oracle GoldenGate. It includes descriptions of change data capture, data profiling with Oracle Enterprise Data Quality, data cleansing with ODI constraints and EDQ, building an error event schema in ODI, and deduplicating data with EDQ. Screenshots and diagrams are provided to illustrate many of these concepts.
El comunicado anuncia la suspensión de clases en todas las escuelas públicas y privadas del distrito de San Antonio de Huarochirí hasta el 20 de marzo debido al aumento de riesgos por las fuertes lluvias y crecimiento de los ríos, lo que dificulta que los estudiantes y maestros lleguen a los colegios de manera segura.
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingMichael Rainey
This document provides an overview and summary of a presentation on integrating Oracle GoldenGate and Apache Kafka for real-time data streaming. It introduces the speaker, describes Rittman Mead as a specialist in Oracle data integration and analytics, and outlines the challenges of integrating new data sources. The bulk of the document then dives into a step-by-step example of using GoldenGate to replicate transactional data from an Oracle database to Kafka in real-time via Kafka's publish-subscribe capabilities.
The document discusses the scripting capabilities of Groovy and how it simplifies scripting compared to Java. Groovy allows separating code into simple scripts without unnecessary class and method definitions. It supports various approaches for running scripts, including via a GroovyShell, GroovyScriptEngine, or by implementing the JSR-223 scripting API. Groovy also allows predefining variables and methods in a script's binding to make domain-specific languages more natural to use.
The document discusses outcome-based education (OBE) principles and processes. Key points include:
- OBE focuses on defining clear learning outcomes that students should be able to demonstrate by the end of a course, rather than focusing on curriculum content.
- Core OBE principles are clarity of focus on intended outcomes, high expectations for students, designing the curriculum backwards from outcomes, and expanding opportunities for all students.
- The OBE process involves constructive alignment where teaching methods and assessments are aligned with learning activities to achieve intended outcomes.
Oracle GoldenGate and Apache Kafka A Deep Dive Into Real-Time Data StreamingMichael Rainey
We produce quite a lot of data. Some of this data comes in the form of business transactions and is stored in a relational database. This relational data is often combined with other non-structured, high volume and rapidly changing datasets known in the industry as Big Data. The challenge for us as data integration professionals is to then combine this data and transform it into something useful. Not just that, but we must also do it in near real-time and using a big data target system such as Hadoop. The topic of this session, real-time data streaming, provides us a great solution for that challenging task. By combining GoldenGate, Oracle’s premier data replication technology, and Apache Kafka, the latest open-source streaming and messaging system for big data, we can implement a fast, durable, and scalable solution. This session will walk through the implementation of GoldenGate and Kafka.
Presented at Collaborate16 in Las Vegas.
Data warehouse migration to oracle data integrator 11gMichael Rainey
Pacific Northwest National Laboratory migrated their data warehouse from SQL Server and Visual Basic to Oracle Data Integrator. They developed a SQL parsing tool and used the ODI SDK to programmatically build ODI objects from the SQL metadata, automating the migration of over 4,900 packages. This minimized implementation risks and allowed them to complete the migration in a fraction of the originally estimated 2-3 years. The automated approach reduced human errors and is now used for ongoing operations.
Oracle Data Integrator 12c - Getting StartedMichael Rainey
I think it’s time for a fresh look at Oracle Data Integrator 12c. What is ODI? How has it evolved over the years and where is it going? And, of course, how do you get started with Oracle Data Integrator? I plan to share what I love about ODI, how to get started building your first ODI project, and what makes Oracle Data Integrator 12c the premier ETL and data warehousing tool on the market. It’s time to get back to the basics!
Presented at UTOUG Training Days 2017.
Application High Availability and Upgrades Using Oracle GoldenGateShane Borden
This presentation will discuss the techniques and methods used to deploy a High Availability Active / Active configuration using Oracle GoldenGate. Discussion will surround deploying GoldenGate utilizing the built in Conflict Detection and Resolution (CDR) functionality as well as the other configuration items needed for a true active / active system. Focus will also be given to the other IT resources that must be involved in order to achieve a successful deployment.
Srimanta Maji has over 2 years of experience as an Oracle Database Administrator and SQL Server DBA. He has expertise in installing, configuring, and managing Oracle and SQL Server databases, including backup/recovery, performance tuning, and user/role management. He currently works as a Systems Engineer at Tata Consultancy Services on projects for clients like State Bank of India and Australia Gas and Light.
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data StreamingMichael Rainey
We produce quite a lot of data! Much of the data are business transactions stored in a relational database. More frequently, the data are non-structured, high volume and rapidly changing datasets known in the industry as Big Data. The challenge for data integration professionals is to combine and transform the data into useful information. Not just that, but it must also be done in near real-time and using a target system such as Hadoop. The topic of this session, real-time data streaming, provides a great solution for this challenging task. By integrating GoldenGate, Oracle’s premier data replication technology, and Apache Kafka, the latest open-source streaming and messaging system, we can implement a fast, durable, and scalable solution.
Presented at Oracle OpenWorld 2016
The document describes a migration from an Oracle database topology to a PostgreSQL database topology at ACI. It discusses the starting Oracle topology with issues around operational complexity and non-ACID compliance. It then describes the target PostgreSQL topology with improved performance, availability and lower costs. The document outlines decisions around tools, extensions, code changes and testing approaches needed for the migration. It also discusses options for migrating the data and cutting over to the new PostgreSQL environment.
Golden Gate - How to start such a project?Trivadis
This document provides information about starting a GoldenGate replication project. It discusses establishing a project plan, running a proof of concept, designing a topology, defining rules and processes, and preparing documentation and scripts. It emphasizes keeping the setup simple, configuring databases correctly to avoid unnecessary overhead, implementing critical components like patching, a repository to generate scripts, heartbeat monitoring, and multiple types of monitoring. It also stresses being prepared to verify replicated data between source and destination.
Yaseen Edappil is seeking a challenging position as an Oracle DBA. He has over 6 years of experience as an Oracle DBA providing production support on Oracle databases. He is Oracle certified with OCP 12c and 10g certifications. At his previous role at Capgemini, he provided Oracle DBA support to Warner Bros, managing critical databases and performing administration, upgrades, automation, backup/recovery and performance tuning. He has expertise in Oracle 12c, 11g, and 10g databases as well as Oracle EBS 11i/R12.
Oracle Goldengate first acquaintance
The Oracle's strategic solution for real time data integration. Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in real time
The document provides guidance on migrating to Oracle Autonomous Database Cloud using various methods. It discusses using DBMS_CLOUD to load data directly from object storage, using Oracle Data Pump to export and import data between databases, and using SQL*Loader to load data from flat files. It also covers using Oracle SQL Developer which provides wizards and tools to simplify the migration process. The document highlights important considerations for each method such as the source database version and file format and location. It provides examples for executing each migration method.
Extreme replication at IOUG Collaborate 15Bobby Curtis
This document summarizes a session on tuning Oracle GoldenGate performance between an Oracle source and target database. It discusses tools for monitoring GoldenGate performance such as lag reports, process statistics, and database views. It also provides a case study example configuration and recommendations for tuning integrated extract and replicat parameters such as parallelism settings.
Oracle ACE Director Dan Morgan presented those slides about migrating to database 12c and how to get it right. For more information, visit www.perftuning.com
Between 2015 and 2017 a large percentage of Oracle's existing customer base will be upgrading their existing databases to the new version 12cR1. Most of the time when upgrades happen the only benefits organizations receive are the satisfaction of having survived the upgrade unscathed. In general, the new database, other than having a new version number, provides little in the way of tangible benefits.
With the re-architecture that can come with a 12cR1 upgrade it is, for the first time, possible to plan for and receive substantial measurable benefits, and possible to make costly mistakes that could create substantial liabilities that are both business and financial.
Oracle ACE Director and industry veteran Dan Morgan, in a presentation targeted to IT/IS management explores both the benefits and the risks and provide a guideline for "getting it right."
This Performance Tuning's Lunch & Learn event focuses on management, planning, and budgeting, not features and technology, and provides you and your management teams the information they need to perform the next database upgrade or migration cycle.
Ranganadha Gupta has over 15 years of experience as an Oracle Database Administrator, currently serving as a Delivery Lead at Hewlett Packard Enterprises. He has expertise in Oracle databases from 9i to 12c, along with technologies like RAC, RMAN, Exadata, and GoldenGate. The document provides an overview of his skills, qualifications, and work history supporting Oracle databases for various organizations.
KScope14 - Real-Time Data Warehouse Upgrade - Success StoriesMichael Rainey
Providing real-time data to its global customers is a necessity for IFPI (International Federation of the Phonographic Industry), a not-for-profit organization with a mission to safeguard the rights of record producers and promote the value of recorded music. Using Oracle Streams and Oracle Warehouse Builder (OWB) for real-time data replication and integration, meeting this goal was becoming a challenge. The solution was difficult to maintain and overall throughput was degrading as data volume increased. The need for greater stability and performance led IFPI to implement Oracle GoldenGate and Oracle Data Integrator. This session will describe the innovative approach taken to complete the migration from a Streams and OWB implementation to a more robust, maintainable, and performant GoldenGate and ODI integrated solution.
Mohammad Abdus Samad has over 7 years of experience as an Oracle Database Administrator with expertise in Oracle 11g RAC, ASM, GoldenGate replication, Data Guard, and performance tuning. He has extensive experience implementing and maintaining Oracle databases for large financial institutions in Bangladesh and the UK. Currently, he is pursuing a Master's Degree in Computer Science while available for full-time employment.
An AMIS Overview of Oracle database 12c (12.1)Marco Gralike
Presentation used by Lucas Jellema and Marco Gralike during the AMIS Oracle Database 12c Launch event on Monday the 15th of July 2013 (much thanks to Tom Kyte, Oracle, for being allowed to use some of his material)
M.
Oracle ACE and Performance Tuning Corporation (PTC) founder, Ed Whalen discussed data migrations or upgrades, the choices ahead for your organization and how they relate to 12c. For more information about this topic, visit Performance Tuning Corporation website at www.perftuning.com .
The currently released version of Oracle Database 12c is the one most organizations will begin to migrate to this year and yet few organizations have the experience necessary to understand the issues that must be addressed to avoid pitfalls such as downtime, delays, or over-spending.
Oracle ACE Ed Whalen, a 20+ year industry veteran, Performance Tuning Corporation founder and distinguished author will provide advanced best practices for upgrading or migrating Oracle databases. Depending on specific system configurations, requirements and business needs, there are numerous methods for performing this vital activity and choosing the right one is critical.
This webinar abswers questions such as:
- When to upgrade or migrate an Oracle database?
- What are the options for migration or upgrade?
- What is the best method?
Performance Tuning's Lunch & Learn is focused on management, planning, and budgeting, not features and technology, and will provide you and your management teams the information they need to perform the next database upgrade or migration cycle.
AMIS organiseerde op maandagavond 15 juli het seminar ‘Oracle database 12c revealed’. Deze avond bood AMIS Oracle professionals de eerste mogelijkheid om de vernieuwingen in Oracle database 12c in actie te zien! De AMIS specialisten die meer dan een jaar bèta testen hebben uitgevoerd lieten zien wat er nieuw is en hoe we dat de komende jaren gaan inzetten!
Deze presentatie is deze avond gegeven als een plenaire sessie!
Developing Kafka Streams Applications with Upgradability in Mind with Neil Bu...HostedbyConfluent
Does your organization struggle with updating of its Kafka Streams application? Releasing a new version of a Kafka Streams application can be challenging, especially if its state has to be preserved between releases. Consider these best-practices and architectural ideas to make this process smoother and improve your release process.
Having experienced accidental removal of change-log topics and needing to expand partitions, it is much easier to handle with some planning. With the proper planning, you can achieve easier application upgrades.
Key take-aways from the session include:
* How do minimize the rebuilding of the state-stores.
* How to change stream topologies without affecting the existing state stores.
* What you can do when you absolutely need to increase the number of partitions within your application.
* How to leveraging schemas for application releases.
* Measures to prevent data corruption, especially if Kafka is not only your system of record but also your source of truth.
* Techniques to support rolling back an application.
* The advantages of splitting apart a Kafka Streams application into multiple applications.
Similar to GoldenGate and Oracle Data Integrator - A Perfect Match- Upgrade to 12c (20)
Data Warehouse - Incremental Migration to the CloudMichael Rainey
A data warehouse (DW) migration is no small undertaking, especially when moving from on-premises to the cloud. A typical data warehouse has numerous data sources connecting and loading data into the DW, ETL tools and data integration scripts performing transformations, and reporting, advanced analytics, or ad-hoc query tools accessing the data for insights and analysis. That’s a lot to coordinate and the data warehouse cannot be migrated all at once. Using a data replication technology such as Oracle GoldenGate, the data warehouse migration can be performed incrementally by keeping the data in-sync between the original DW and the new, cloud DW. This session will dive into the steps necessary for this incremental migration approach and walk through a customer use case scenario, leaving attendees with an understanding of how to perform a data warehouse migration to the cloud.
Presented at RMOUG Training Days 2019
Continuous Data Replication into Cloud Storage with Oracle GoldenGateMichael Rainey
Continuous flow. Streaming. Near real-time. These are all terms used to identify the business’s need for quick access to data. It’s a common request, even if the data must flow from on-premises to the cloud. Oracle GoldenGate is the data replication solution built for fast data. In this session, we’ll look at how GoldenGate can be configured to extract transactions from the Oracle database and load them into a cloud object store, such as Amazon S3. There are many different use cases for this type of continuous load of data into the cloud. We’ll explore these solutions and the various tools that can be used to access and analyze the data from the cloud object store, leaving attendees with ideas for implementing a full source-to-cloud data replication solution.
Presented at ITOUG Tech Days 2019
There is a fundamental shift underway in IT to include open, software defined, distributed systems like Hadoop. As a result, every Oracle professional should strive to learn these new technologies or risk being left behind. This session is designed specifically for Oracle database professionals so they can better understand SQL on Hadoop and the benefits it brings to the enterprise. Attendees will see how SQL on Hadoop compares to Oracle in areas such as data storage, data ingestion, and SQL processing. Various live demos will provide attendees with a first-hand look at these new world technologies. Presented at Collaborate 18.
Going Serverless - an Introduction to AWS GlueMichael Rainey
Going "serverless" is the latest technology trend for enterprises moving their processing to the cloud, including data integration and ETL tools. But what does that mean and when should I use serverless ETL? In this session, we'll dive into the world of Amazon's fully managed data processing service called AWS Glue. With no server to provision or resources to allocate, and an easy to populate metadata catalog, AWS Glue allows the data engineer to focus on his or her craft; building data transformations and pipelines. Gaining an understanding of the similarities and differences between traditional ETL tools, such as Oracle Data Integrator, and Glue will prepare attendees for the new world of data integration. Presented at Collaborate 18.
As a data integration professional, it’s almost a guarantee that you’ve heard of real-time stream processing of Big Data. The usual players in the open source world are Apache Kafka, used to move data in real-time, and Spark Streaming, built for in-flight transformations. But what about relational data? Quite often we forget that products incubated in the Apache Foundation can also serve a purpose for “standard” relational databases as well. But how? Well, let’s introduce Oracle GoldenGate and Oracle Data Integrator for Big Data. GoldenGate can extract relational data in real time and produce Kafka messages, ensuring relational data is a part of the enterprise data bus. These messages can then be ingested via ODI through a Spark Streaming process, integrating with additional data sources, such as other relational tables, flat files, etc, as needed. Finally, the output can be sent to multiple locations: on through to a data warehouse for analytical reporting, back to Kafka for additional targets to consume, or any number of targets. Attendees will walk away with a framework on which they can build their data streaming projects, combining relational data with big data and using a common, structured approach via the Oracle Data Integration product stack.
Presented at BIWA Summit 2017.
Oracle data integrator 12c - getting startedMichael Rainey
Oracle Data Integrator (ODI) is a data integration tool that can extract, load, and transform heterogeneous data sources. It is flexible and uses a flow-based mapping approach. The presentation provided an overview of ODI and guidance on installation, configuration, and getting started with the developer quickstart to create models, schemas, and mappings between data stores. Key components like knowledge modules generate integration code, while packages and load plans orchestrate the data integration processes.
We all have business user requirements that drive our data warehouse and business intelligence system implementation. Delivered in various formats, these documents, emails, chats, or meeting minutes can be interpreted many different ways, depending on the reader. An effective approach to end the miscommunication...draw a picture. When properly constructed, an architecture diagram, data model, or process model will have only a single meaning. That is why a picture can replace a thousand words.
Presented as a TED style talk at Rittman Mead BI Forum 2014.
A Walk Through the Kimball ETL Subsystems with Oracle Data IntegrationMichael Rainey
Big Data integration is an excellent feature in the Oracle Data Integration product suite (Oracle Data Integrator, GoldenGate, & Enterprise Data Quality). But not all analytics require big data technologies, such as labor cost, revenue, or expense reporting. Ralph Kimball, an original architect of the dimensional model in data warehousing, spent much of his career working to build an enterprise data warehouse methodology that can meet these reporting needs. His book, "The Data Warehouse ETL Toolkit", is a guide for many ETL developers. This session will walk you through his ETL Subsystem categories; Extracting, Cleaning & Conforming, Delivering, and Managing, describing how the Oracle Data Integration products are perfectly suited for the Kimball approach.
Presented at Oracle OpenWorld 2015 & BIWA Summit 2016.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
4. info@rittmanmead.com www.rittmanmead.com @rittmanmead
About Rittman Mead
• Oracle Gold partner
- World leading specialist partner for technical excellence, solutions delivery
and innovation in Oracle BI
- Provide consulting, training, managed services for customers worldwide
• 100+ consultants including 1 Oracle ACE Director and 3 Oracle ACEs
- All experts in Oracle BI, DW, EPM and Analytics tech
- Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE,
Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca
• Blog: www.rittmanmead.com/blog Twitter: @rittmanmead
4
5. info@rittmanmead.com www.rittmanmead.com @rittmanmead 5
Visual Redesign Business User Training
Ongoing SupportEngagement Toolkit
Average user adoption for BI
platforms is below 25%
Rittman Mead’s User Engagement Service can help
6. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Today’s Topic
• Upgrade Oracle Data Integrator 11g and Oracle
GoldenGate 11g to 12c
- Overview of ODI & GoldenGate
- What are the steps for upgrade of each product
- How to cleanup after the conversion to 12c
- What to watch out for when using ODI & GoldenGate
together
6
7. info@rittmanmead.com www.rittmanmead.com @rittmanmead
GoldenGate - Data Replication
• High-performance data replication
- Heterogeneous sources and targets
- Minimal impact on source and target systems
- JMS integration
- Conflict detection and resolution
- Password and data encryption for enhanced security
• Enhance decision-making with real-time data
7
9. info@rittmanmead.com www.rittmanmead.com @rittmanmead
• Integrated Apply
• Big Data integration
- GoldenGate Adapters - load HDFS, Flume, etc
• Capture from and deliver to Oracle Exadata Storage Server
• Support for multi-tenant databases - Oracle 12c
• Enhanced performance
- coordinated mode
What’s new in GoldenGate 12c?
9
10. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Oracle Data Integrator Concepts
• ODI is a complete and unified data integration solution
- Supports batch-based, real-time and service-based integration
• “Design Once, Run Anywhere”
- Provides reusability, flexibility and extensibility using Knowledge
Modules (KM)
- Separates the logical from the physical implementation
• Based in an ELT architecture to leverage target platform capabilities
- Higher performance
- Platform Independent
• Combines data movement, data synchronization, data transformation, data
quality, data management and data services
10
14. info@rittmanmead.com www.rittmanmead.com @rittmanmead
What’s new in ODI 12c?
• Flow-based Mappings
- Deployment specifications
- Component Knowledge Modules
• Debugger
• WebLogic Management Framework
• Improved performance
- Parallel target table load
- Session blueprints
• Enhanced KM and Procedure editors
13
15. info@rittmanmead.com www.rittmanmead.com @rittmanmead
GoldenGate and ODI - A Perfect Match
• JKM Oracle to Oracle Consistent (OGG) Knowledge Module
- Oracle Data Integrator Metadata used to generate GoldenGate parameter files
(extract, pump, replicate) and
configuration files
- Delivered with ODI
• ODI CDC Framework generated
- Staging table - replicate of
source
- J$ (journal) table - change rows
• Journalized data used in
transformations (via JV$ views)
14
16. info@rittmanmead.com www.rittmanmead.com @rittmanmead
GoldenGate and ODI - A Perfect Match
• JKM Oracle to Oracle Consistent (OGG) Knowledge Module
- Oracle Data Integrator Metadata used to generate GoldenGate parameter files
(extract, pump, replicate) and
configuration files
- Delivered with ODI
• ODI CDC Framework generated
- Staging table - replicate of
source
- J$ (journal) table - change rows
• Journalized data used in
transformations (via JV$ views)
14
17. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Oracle Data Integrator - Journalizing (CDC)
• Oracle Data Integrator Change Data Capture (CDC) delivered via Journalizing
- Identify, capture, and deliver changes made to source data
- Journalizing Knowledge Module (JKM) performs setup and
creates infrastructure
• ODI CDC Framework
- Capture Process - mechanism for capturing changed data from the
source database (Ex. Oracle GoldenGate)
- Journals - tables (J$) hold references to changed records and the
change type (insert / update / delete)
- Journalizing Views - (JV$, JV$D) provides access to changed data,
used by IKM / LKM in mappings
- Subscribers - used to allow consumption of changed data at different intervals, for multiple
applications, etc.
15
19. info@rittmanmead.com www.rittmanmead.com @rittmanmead
GoldenGate Upgrade Considerations
• GoldenGate 12c is supported by the 64-bit OS only
(Oracle installations)
• Start with simple upgrade, then implement new
features (integrated replicat, integrated extract, etc)
• Parameter file updates
- Quoted object names - use the convprm tool
- Upgrade extract (or replicat) only
- Character-Set conversion can be handled
17
20. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrading GoldenGate 11g to 12c - convprm
• Run prior to performing the upgrade on a backup of the files
18
Usage: convprm [options] <list of file names>
-h, --help: displays usage.
-v, --version: displays version information.
-i, --follow-include: also converts files included through an OBEY or INCLUDE directive.
-n, --no-backup: does not create a copy of the original file.
-s, --silent: does not display status messages.
-q, --quotes: performs quote conversion (active by default).
-b, --before-function: converts from BEFORE.column syntax to @BEFORE(column) (active by default).
-d, --dry-run: does not change the input or create a backup file. It only prints out what would happen.
21. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrading GoldenGate 11g to 12c - convprm
•Start with the dry run
19
[oracle@ODIGettingStarted test]$ ./convprm -d ../sogg/dirprm/*
*** Conversion started for file ../sogg/dirprm/EXTC.prm.
extract EXTC
userid ggsrc, password oracle
exttrail /home/oracle/sogg/dirdat/EXToc/oc
TABLE SUGARCRM.ACCOUNTS;
TABLE SUGARCRM.ACCOUNTS_CONTACTS;
TABLE SUGARCRM.ACCOUNTS_OPPORTUNITIES;
TABLE SUGARCRM.MEETINGS_LEADS;
TABLE SUGARCRM.MEETINGS;
TABLE SUGARCRM.MEETINGS_CONTACTS;
--- Conversion complete for file ../sogg/dirprm/EXTC.prm.
22. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrading GoldenGate 11g to 12c - convprm
•Run the full parameter file conversion
20
[oracle@ODIGettingStarted togg]$ ./convprm ./dirprm/*
*** Conversion started for file ./dirprm/jagent.prm.
--- Conversion complete for file ./dirprm/jagent.prm.
*** Conversion started for file ./dirprm/mgr.prm.
--- Conversion complete for file ./dirprm/mgr.prm.
*** Conversion started for file ./dirprm/REPSA1.prm.
+ string literal to ANSI SQL quotes at 26,29; length = 8.
+ string literal to ANSI SQL quotes at 26,39; length = 11.
+ string literal to ANSI SQL quotes at 26,69; length = 8.
...
+ string literal to ANSI SQL quotes at 62,79; length = 9.
--- Conversion complete for file ./dirprm/REPSA1.prm.
*** 3 files have been converted.
23. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrading GoldenGate 11g to 12c - convprm
•Run the full parameter file conversion
20
[oracle@ODIGettingStarted togg]$ ./convprm ./dirprm/*
*** Conversion started for file ./dirprm/jagent.prm.
--- Conversion complete for file ./dirprm/jagent.prm.
*** Conversion started for file ./dirprm/mgr.prm.
--- Conversion complete for file ./dirprm/mgr.prm.
*** Conversion started for file ./dirprm/REPSA1.prm.
+ string literal to ANSI SQL quotes at 26,29; length = 8.
+ string literal to ANSI SQL quotes at 26,39; length = 11.
+ string literal to ANSI SQL quotes at 26,69; length = 8.
...
+ string literal to ANSI SQL quotes at 62,79; length = 9.
--- Conversion complete for file ./dirprm/REPSA1.prm.
*** 3 files have been converted.
[oracle@ODIGettingStarted dirprm]$ diff REPSA1.prm REPSA1.prm.bck
26c26
< WINDOW_ID = @STRCAT(@GETENV('RECORD', 'FILESEQNO'),
@STRNUM(@GETENV('RECORD', 'FILERBA'), RIGHTZERO, 10))
---
> WINDOW_ID = @STRCAT(@GETENV("RECORD", "FILESEQNO"),
@STRNUM(@GETENV("RECORD", "FILERBA"), RIGHTZERO, 10))
...
24. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c Steps
• Ensure all transactions have completed on source database and extract
- Source system outage is necessary only when SQL or DDL operations are
performed
• Record current SCN
• Stop all process groups and managers on source & target
• Install GoldenGate 12c
• Upgrade the supplemental logging on the source (ulg.sql)
• Set the extract, pump, and replicat to start at the recorded SCN
• Start the processes!
21
25. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Source Preparation
• Check for long-running transactions
INFO EXTRACT group, SHOWCH
• If needed, restore archive logs to checkpoint…
• Or…”clear” them using FORCETRANS or SKIPTRANS or wait for completion
22
26. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Source Preparation
• Check for long-running transactions
INFO EXTRACT group, SHOWCH
• If needed, restore archive logs to checkpoint…
• Or…”clear” them using FORCETRANS or SKIPTRANS or wait for completion
22
27. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Source Preparation
• Check for long-running transactions
INFO EXTRACT group, SHOWCH
• If needed, restore archive logs to checkpoint…
• Or…”clear” them using FORCETRANS or SKIPTRANS or wait for completion
22
28. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Source Preparation
• Record the current SCN
SQL> select current_scn from v$database;
• Check the extract to determine if it has
completed processing transactions
23
35. info@rittmanmead.com www.rittmanmead.com @rittmanmead
• If the source or target is Oracle 11.2.0.4 or 12.1.0.2+, set the ENABLE_GOLDENGATE_REPLICATION
initialization parameter to TRUE
• Run the ulg.sql (update log group) script on the source system to update the supplemental logging
- Supplemental log groups must have been created via Add Trandata
Upgrade GoldenGate 11g to 12c - Update Logging
27
36. info@rittmanmead.com www.rittmanmead.com @rittmanmead
• If the source or target is Oracle 11.2.0.4 or 12.1.0.2+, set the ENABLE_GOLDENGATE_REPLICATION
initialization parameter to TRUE
• Run the
- Supplemental log groups must have been created via Add Trandata
Upgrade GoldenGate 11g to 12c - Update Logging
27
37. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Finalize Upgrade
• Alter the extract (and pump) processes to write a new sequence number
• Issue the command info extract groupname, detail to get the new sequence number
28
GGSCI (ODIGettingStarted) 3> alter extract EXTC ETROLLOVER
2014-12-03 19:13:52 INFO OGG-01520 Rollover performed. For each affected output trail
of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for
that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new
trail file; it will not happen automatically.
EXTRACT altered.
GGSCI (ODIGettingStarted) 4> alter extract REPSP ETROLLOVER
2014-12-03 19:14:07 INFO OGG-01520 Rollover performed. For each affected output trail
of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for
that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new
trail file; it will not happen automatically.
EXTRACT altered.
38. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade GoldenGate 11g to 12c - Finalize Upgrade
• Alter the extract (pump) and replicat processes to begin at the new sequence number
• Start the processes on source and target
29
GGSCI (ODIGettingStarted) 7> alter extract REPSP, extseqno 14, extrba 0
EXTRACT altered.
GGSCI (ODIGettingStarted) 1> alter replicat REPSA1, extseqno 14, extrba 0
REPLICAT altered.
39. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Oracle Data Integrator Upgrade Considerations
• Can be completed before or after GoldenGate upgrade
• Interfaces will be converted to Mappings using the
DataSet component
- Further work necessary to fully utilize flow-based
Mappings
• Agent upgrade
- If possible, install new agent. (WLS must be upgraded to 12c anyway).
- If not…follow upgrade procedure.
• Upgrade key - used to migrate additional objects later
30
41. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Upgrade ODI 11g to 12c - Preparation
• Backup the existing environment and clone
the repositories for upgrade
- Ensure the table schema_version_registry has an
ODI entry (Support Doc ID 1622194.1)
- Set the work repository schema JDBC URL to the
cloned version
32
70. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Move to ODI 12c Journalizing Knowledge Module
• Import the JKM
• Configure and start
JAgent on source/target
• Setup the GoldenGate
Data Server/Physical
Schemas (Online)
• Change the JKM in the
Model and set the
Options
46
71. info@rittmanmead.com www.rittmanmead.com @rittmanmead
Summary
• Upgrade both GoldenGate and ODI at the same time or individually
- Backup and clone environment
- Use tricks, such as “Convert to Flow”, to finalize migration to 12c
- Journalizing integration will still work after upgrade of ODI
• Minimize interruption of source and target
- Capture source SCN before upgrade and restart GoldenGate at
that point in the transaction logs
• Oracle documentation is good!
47