Guido Schmutz, profile picture
Uploaded byGuido Schmutz
PDF, PPTX599 views

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

The document discusses bi-directional integration solutions between Oracle RDBMS and Apache Kafka, outlining key agendas such as blueprints for both Oracle to Kafka and Kafka to Oracle data flows. It provides detailed explanations of various integration methods including polling, change data capture, and use of APIs. Furthermore, it highlights the role of microservices in modern applications and the importance of asynchronous communication in data processing.

Embed presentation

Download as PDF, PPTX
BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Guido Schmutz (guido.schmutz@trivadis.com)
Agenda 1. Introduction 2. Blueprints Oracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Summary
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 170th edition
Introduction
Microservices / Modern Applications • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! M3M2 ACL Event Hub M1
Microservices / Modern Applications Integrate with Traditional System M3M2 ACL Event Hub M1 ACL • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! Traditional App
Use Case Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense “Modern Apps”Traditional Apps (Legacy)
MessageMessageMessageMessage MessageMessage Properties - Message Message Message A1 A2 A3 Message B1 B2 B3 B4 A1 A2 A3 B B1 B2 B3 B4 Table A A1 A2 A3 Table B B1 B2 B3 B4 FlatDB Model Aggregate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Properties - Latency Traditional System Event Hub Data Flow RDBMS latency latency Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Properties – Anti-Corruption Layer (ACL) Traditional System Event Hub Data Flow RDBMS Traditional System Event Hub Data Flow RDBMS ACL ACL Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Examples: • View Layer • Storage Procedure • JSON Support in DB • … Examples: • StreamSets • Kafka Connect • Kafka Streams / KSQL • … Database Dataflow
Blueprints Oracle RDBMS => Apache Kafka
Blueprints Oracle RDBMS => Apache Kafka (DB-K) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Que with bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5
DB-K_1: Polling of RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_1: Polling of RDBMS table/view Kafka Connect with JDBC Source Connector
Kafka Connect & JDBC Connector • Many connectors available • Single Message Transforms (SMT) • declarative style, simple data flows • framework is part of Apache Kafka
DB-K_1 – Configure JDBC Connector #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
DB-K_2: Change Data Capture (CDC) on RDBMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy DB-K_2: Change Data Capture (CDC) on RDBMS Using Oracle GoldenGate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Alternatives: StreamSets Data Collector Attunity Debezium …
DB-K_3: Polling of RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow DB-K_3: Polling of RDBMS API StreamSets invokes Oracle Rest Data Service Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Oracle REST Data Services (ORDS) • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
DB-K_3 – Setup ORDS (I) ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
DB-K_3 – Setup ORDS (II) ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name ''emailAddress'' VALUE cu.email), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''id'' VALUE li.product_id, ''name'' VALUE li.product_name, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
StreamSets Data Collector • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing • custom sources, sinks, processors • Monitoring and Error Detection
DB-K_4: Produce to Event Hub from RDBMS Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Native Kafka Producer using Java in DB Does not feel right!
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Invoke REST Proxy from PL/SQL Invoking a REST Service from DB not well-supported
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon … DB-K_4: Produce to Event Hub from RDBMS Oracle Big Data SQL integrates with Kafka Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
DB-K_5: RDBMS Queue with bridge to Event Hub Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
DB-K_5: RDBMS Queue with bridge to Event Hub Oracle Advanced Queuing & Kafka Connect JMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
BEGIN dbms_aqadm.create_queue_table ( queue_table => 'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.princ ipal=order_processing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonS impleConverter`" } }' DB-K_5 – Setup AQ and Kafka Connect (I)
DB-K_5: RDBMS Queue with bridge to Event Hub Oracle AQ with Kafka API & MirrorMaker Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense AQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing
Blueprints Apache Kafka => Oracle RDBMS
Blueprints Apache Kafka => Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4
K-DB_1: Write to RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_1: Write to RDBMS table/view Kafka Connect and JDBC Sink Connector
K-DB_2: Write over RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_2: Write over RDBMS API Kafka Connect invokes Oracle Rest Data Service
DB-K_2 – Setup ORDS (I) ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
K-DB_3: Consume from Event Hub Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon… K-DB_3: Consume from Event Hub Oracle Big Data SQL exposes topic as table Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
K-DB_4: Event Hub with bridge to RDBMS queue Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ K-DB_4: Event Hub with bridge to RDBMS queue Oracle Advanced Queuing & Kafka Connect JMS Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing K-DB_4: Event Hub with bridge to RDBMS queue Oracle AQ with Kafka API & MirrorMaker Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Summary
Summary Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 https://github.com/gschmutz/various-demos/tree/master/bidirectional-integration-oracle-kafka DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Queue with bridge to Event Hub
Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service sEvent Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

More Related Content

PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
PDF
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 
Spark (Structured) Streaming vs. Kafka Streams
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Building Event-Driven (Micro) Services with Apache Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
byGuido Schmutz
 

What's hot

PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
PDF
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
byGuido Schmutz
 
PDF
Kafka as an event store - is it good enough?
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
PDF
Streaming Visualization
byGuido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PDF
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
PDF
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
byGuido Schmutz
 
PDF
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
PDF
Building Event-Driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
PDF
Ingesting streaming data into Graph Database
byGuido Schmutz
 
PDF
Streaming Visualisation
byGuido Schmutz
 
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
PDF
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Building event-driven Microservices with Kafka Ecosystem
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
byGuido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
byGuido Schmutz
 
Ingesting and Processing IoT Data - using MQTT, Kafka Connect and KSQL
byGuido Schmutz
 
Kafka as an event store - is it good enough?
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
byGuido Schmutz
 
Streaming Visualization
byGuido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning
byGuido Schmutz
 
What is Apache Kafka? Why is it so popular? Should I use it?
byGuido Schmutz
 
Building Event-Driven (Micro)Services with Apache Kafka
byGuido Schmutz
 
Ingesting streaming data into Graph Database
byGuido Schmutz
 
Streaming Visualisation
byGuido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
byGuido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
byGuido Schmutz
 
Building event-driven Microservices with Kafka Ecosystem
byGuido Schmutz
 

Similar to Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

PPTX
Event streaming webinar feb 2020
byMaheedhar Gunturu
 
PDF
Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL
byScyllaDB
 
PPTX
Oracle database 12c_and_DevOps
byMaria Colgan
 
PDF
Building data pipelines at Shopee with DEC
byRim Zaidullin
 
PDF
APAC ksqlDB Workshop
byconfluent
 
PDF
Let your DBAs get some REST(api)
byLudovico Caldara
 
PDF
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
byconfluent
 
PDF
Query Your Streaming Data on Kafka using SQL: Why, How, and What
byHostedbyConfluent
 
PDF
Concepts and Patterns for Streaming Services with Kafka
byQAware GmbH
 
PDF
KSQL - Stream Processing simplified!
byGuido Schmutz
 
PDF
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
byconfluent
 
PDF
Event Driven Microservices
byFabrizio Fortino
 
PDF
10 essentials steps for kafka streaming services
byinovia
 
PPTX
Capture the Streams of Database Changes
byconfluent
 
PDF
Real-time Data Streaming from Oracle to Apache Kafka
byconfluent
 
PDF
KSQL – The Open Source SQL Streaming Engine for Apache Kafka (Big Data Spain ...
byKai Wähner
 
PDF
All Streams Ahead! ksqlDB Workshop ANZ
byconfluent
 
PDF
M|18 Real-time Analytics with the New Streaming Data Adapters
byMariaDB plc
 
PDF
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
byconfluent
 
PDF
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 
Event streaming webinar feb 2020
byMaheedhar Gunturu
 
Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL
byScyllaDB
 
Oracle database 12c_and_DevOps
byMaria Colgan
 
Building data pipelines at Shopee with DEC
byRim Zaidullin
 
APAC ksqlDB Workshop
byconfluent
 
Let your DBAs get some REST(api)
byLudovico Caldara
 
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
byconfluent
 
Query Your Streaming Data on Kafka using SQL: Why, How, and What
byHostedbyConfluent
 
Concepts and Patterns for Streaming Services with Kafka
byQAware GmbH
 
KSQL - Stream Processing simplified!
byGuido Schmutz
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
byconfluent
 
Event Driven Microservices
byFabrizio Fortino
 
10 essentials steps for kafka streaming services
byinovia
 
Capture the Streams of Database Changes
byconfluent
 
Real-time Data Streaming from Oracle to Apache Kafka
byconfluent
 
KSQL – The Open Source SQL Streaming Engine for Apache Kafka (Big Data Spain ...
byKai Wähner
 
All Streams Ahead! ksqlDB Workshop ANZ
byconfluent
 
M|18 Real-time Analytics with the New Streaming Data Adapters
byMariaDB plc
 
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
byconfluent
 
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 

More from Guido Schmutz

PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
PDF
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
PDF
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
PDF
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
PDF
Location Analytics - Real Time Geofencing using Apache Kafka
byGuido Schmutz
 
PDF
Stream Processing – Concepts and Frameworks
byGuido Schmutz
 
PDF
Kafka as an Event Store - is it Good Enough?
byGuido Schmutz
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
30 Minutes to the Analytics Platform with Infrastructure as Code
byGuido Schmutz
 
ksqlDB - Stream Processing simplified!
byGuido Schmutz
 
Kafka as your Data Lake - is it Feasible?
byGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
byGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
byGuido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
byGuido Schmutz
 
Fundamentals Big Data and AI Architecture
byGuido Schmutz
 
Location Analytics - Real Time Geofencing using Apache Kafka
byGuido Schmutz
 
Stream Processing – Concepts and Frameworks
byGuido Schmutz
 
Kafka as an Event Store - is it Good Enough?
byGuido Schmutz
 
Introduction to Stream Processing
byGuido Schmutz
 

Recently uploaded

PDF
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
PPTX
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
PDF
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 
PDF
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
PDF
WORLD-LITERATURE-2ND-SEMESTER2025-2026.pdf
byCHERIEMAYCAMO
 
PDF
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
DOCX
Data Analyst Training Course | Advanced Data Analytics Certification | GVT Ac...
byGVT Academy
 
PDF
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
PDF
chapter one introduction to Probability theory.pdf
bySeid42
 
PDF
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
PPT
Microsoft_SQL_ServerConsolidationTWP.ppt
byssuser8c3398
 
PDF
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
PPT
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
PDF
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
PDF
Unlocking the Future of Logistics & Supply Chain
byKunal Suri
 
PPTX
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
PPTX
Top Institute for Data Analyst Course in Navi Mumbai
byPatil sagar
 
PPTX
Lecture 14- Introduction to Hypothesis Testing.pptx
bygbadamosiayomide
 
PPTX
The Singularity Sprint (2026): Neuralink + the AI Arms Race — Ridwan Sassman
byridwansassman
 
PPTX
Organizational Structure and Design Topic 4
bydavenieralayaay
 
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
WORLD-LITERATURE-2ND-SEMESTER2025-2026.pdf
byCHERIEMAYCAMO
 
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
Data Analyst Training Course | Advanced Data Analytics Certification | GVT Ac...
byGVT Academy
 
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
chapter one introduction to Probability theory.pdf
bySeid42
 
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
Microsoft_SQL_ServerConsolidationTWP.ppt
byssuser8c3398
 
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
Unlocking the Future of Logistics & Supply Chain
byKunal Suri
 
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
Top Institute for Data Analyst Course in Navi Mumbai
byPatil sagar
 
Lecture 14- Introduction to Hypothesis Testing.pptx
bygbadamosiayomide
 
The Singularity Sprint (2026): Neuralink + the AI Arms Race — Ridwan Sassman
byridwansassman
 
Organizational Structure and Design Topic 4
bydavenieralayaay
 

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

  • 1.
    BASEL | BERN| BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Guido Schmutz (guido.schmutz@trivadis.com)
  • 2.
    Agenda 1. Introduction 2. BlueprintsOracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Summary
  • 3.
    BASEL | BERN| BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 170th edition
  • 4.
  • 5.
    Microservices / ModernApplications • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! M3M2 ACL Event Hub M1
  • 6.
    Microservices / ModernApplications Integrate with Traditional System M3M2 ACL Event Hub M1 ACL • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! Traditional App
  • 7.
    Use Case Customer Microservice {} Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense “Modern Apps”Traditional Apps (Legacy)
  • 8.
    MessageMessageMessageMessage MessageMessage Properties - Message MessageMessage A1 A2 A3 Message B1 B2 B3 B4 A1 A2 A3 B B1 B2 B3 B4 Table A A1 A2 A3 Table B B1 B2 B3 B4 FlatDB Model Aggregate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 9.
    Properties - Latency TraditionalSystem Event Hub Data Flow RDBMS latency latency Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 10.
    Properties – Anti-CorruptionLayer (ACL) Traditional System Event Hub Data Flow RDBMS Traditional System Event Hub Data Flow RDBMS ACL ACL Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Examples: • View Layer • Storage Procedure • JSON Support in DB • … Examples: • StreamSets • Kafka Connect • Kafka Streams / KSQL • … Database Dataflow
  • 11.
    Blueprints Oracle RDBMS=> Apache Kafka
  • 12.
    Blueprints Oracle RDBMS=> Apache Kafka (DB-K) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Que with bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5
  • 13.
    DB-K_1: Polling ofRDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 14.
    Event Hub Stream Data Integration API Applications /Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_1: Polling of RDBMS table/view Kafka Connect with JDBC Source Connector
  • 15.
    Kafka Connect &JDBC Connector • Many connectors available • Single Message Transforms (SMT) • declarative style, simple data flows • framework is part of Apache Kafka
  • 16.
    DB-K_1 – ConfigureJDBC Connector #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
  • 17.
    DB-K_2: Change DataCapture (CDC) on RDBMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 18.
    Stream Data Integration &Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy DB-K_2: Change Data Capture (CDC) on RDBMS Using Oracle GoldenGate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Alternatives: StreamSets Data Collector Attunity Debezium …
  • 19.
    DB-K_3: Polling ofRDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 20.
    Event Hub Stream Data Integration API Applications /Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow DB-K_3: Polling of RDBMS API StreamSets invokes Oracle Rest Data Service Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 21.
    Oracle REST DataServices (ORDS) • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
  • 22.
    DB-K_3 – SetupORDS (I) ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
  • 23.
    DB-K_3 – SetupORDS (II) ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name ''emailAddress'' VALUE cu.email), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''id'' VALUE li.product_id, ''name'' VALUE li.product_name, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
  • 24.
    StreamSets Data Collector •GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing • custom sources, sinks, processors • Monitoring and Error Detection
  • 25.
    DB-K_4: Produce toEvent Hub from RDBMS Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 26.
    Event Hub Stream Data Integration API Applications /Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Native Kafka Producer using Java in DB Does not feel right!
  • 27.
    Event Hub Stream Data Integration API Applications /Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Invoke REST Proxy from PL/SQL Invoking a REST Service from DB not well-supported
  • 28.
    Event Hub Stream Data Integration API Applications /Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon … DB-K_4: Produce to Event Hub from RDBMS Oracle Big Data SQL integrates with Kafka Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 29.
    DB-K_5: RDBMS Queuewith bridge to Event Hub Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 30.
    DB-K_5: RDBMS Queuewith bridge to Event Hub Oracle Advanced Queuing & Kafka Connect JMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 31.
    BEGIN dbms_aqadm.create_queue_table ( queue_table =>'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
  • 32.
    curl -X "POST""$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.princ ipal=order_processing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonS impleConverter`" } }' DB-K_5 – Setup AQ and Kafka Connect (I)
  • 33.
    DB-K_5: RDBMS Queuewith bridge to Event Hub Oracle AQ with Kafka API & MirrorMaker Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense AQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing
  • 34.
    Blueprints Apache Kafka=> Oracle RDBMS
  • 35.
    Blueprints Apache Kafka=> Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4
  • 36.
    K-DB_1: Write toRDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 37.
    Event Hub Stream Data Integration API Applications /Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_1: Write to RDBMS table/view Kafka Connect and JDBC Sink Connector
  • 38.
    K-DB_2: Write overRDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 39.
    Event Hub Stream Data Integration API Applications /Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_2: Write over RDBMS API Kafka Connect invokes Oracle Rest Data Service
  • 40.
    DB-K_2 – SetupORDS (I) ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
  • 41.
    K-DB_3: Consume fromEvent Hub Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 42.
    Event Hub Stream Data Integration API Applications /Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon… K-DB_3: Consume from Event Hub Oracle Big Data SQL exposes topic as table Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 43.
    K-DB_4: Event Hubwith bridge to RDBMS queue Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 44.
    Stream Data Integration &Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ K-DB_4: Event Hub with bridge to RDBMS queue Oracle Advanced Queuing & Kafka Connect JMS Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 45.
    Stream Data Integration &Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing K-DB_4: Event Hub with bridge to RDBMS queue Oracle AQ with Kafka API & MirrorMaker Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 46.
  • 47.
    Summary Customer Microservice { } CustomerAPI CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 https://github.com/gschmutz/various-demos/tree/master/bidirectional-integration-oracle-kafka DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Queue with bridge to Event Hub
  • 48.
    Bulk Source Ref Architecture DataPlatform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
  • 49.
    Bulk Source Ref Architecture DataPlatform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service sEvent Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream