DWX23 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMPDWX23-A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

A High-Speed Data Ingestion Service in
Java Using MQTT, AMQP, and STOMP
Developer Week ’23
Juarez Barbosa Junior - @juarezjunior
Sr. Principal Java Developer Evangelist
ORACLE
June 2023
Copyright © 2023, Oracle and/or its affiliates

About me
• Juarez Barbosa Junior - @juarezjunior
• Senior Principal Java Developer Evangelist
• Over 20 years of experience in IT
• SW Engineering, Developer Relations
• Microsoft, Oracle, IBM, Nokia, Unisys, Accenture, and a
few startups
• Microsoft Azure Developer Relations Lead
• IBM Watson Tech Evangelist & Cloud Rockstar
• IBM Mobile Tech Evangelist & Global Thought Leader
• Nokia Developers Global Champion
• Lead Software/DevOps Architect
• Expertise
• Java, Cloud, DevOps, Cloud-native, Blockchain

Java App Dev with Oracle Database

Overview of Oracle DB Access with Java
User
Java
Code
JDBC
Reactive
Extension Standard
JDBC API
R2DBC
+
3rd party
Reactive
Streams
Libraries
Async call with non-blocking
backpressure
operators (map, reduce, filters),
concurrency modeling,
monitoring, tracing
Implements Java SE
reactive stream
interface (Flow)
Full Reactive
Streams
Sync/blocking JDBC calls
Java
Business
Logic
User Java code
Oracle
Database
Oracle JDBC
driver
VTs/lightweight JDBC calls

Reactive JDBC - Sync vs Async JDBC
Setup
Blocking
Handle Result
Setup
Non-Blocking
Handle Result
Synchronous JDBC Setup
Blocking
Handle Result
Setup
Non-Blocking
Handle Result
Setup
Non-Blocking
Handle Result
Reactive JDBC
Database
Setup
Blocking
Handle Result
Database

Support for the Latest Java Versions
• Java 11 - native support, compiled with it
• Java 17 - certified
• JDBC Standards - 4.2 and 4.3
• GraalVM - native image instrumentation
• Reactive Streams - Java Flow API support
• Project Loom - Virtual Threads support
• Data access is critical in mission-critical apps

Classic Java Platform Threads
⚫ Database access with blocking threads
⚫ A JDBC call blocks a thread for 100’s of milliseconds (thread-per-request style)
⚫ Thread count increases linearly with the number of JDBC calls
⚫ Performance degrades as the number of threads increases
⚫ 1 MB of stack memory per thread
⚫ Scheduling many platform threads is expensive
⚫ Preemptive scheduling vs time-slicing

Virtual Threads - JEPs 425, 436, 444
⚫ Virtual Threads – JEPs (JDK Enhancement Proposals)
⚫ Lightweight threads that dramatically reduce the effort of writing, maintaining, and
observing high-throughput concurrent applications
⚫ Enable applications written in the simple thread-per-request style to scale with near-
optimal hardware utilization
⚫ Enable easy troubleshooting, debugging, and profiling of virtual threads with existing
JDK tools
⚫ Do not remove the traditional implementation of threads
⚫ Do not alter the basic concurrency model of Java
⚫ Enable existing code that uses Java threads (java.lang.Thread, etc) to adopt virtual
threads with minimal change

Virtual Threads - JEPs 425, 436, 444
⚫ java.lang.Thread
⚫ final Boolean - isVirtual()
⚫ static Thread.Builder.OfVirtual - ofVirtual()
⚫ static Thread.Builder.OfPlatform - ofPlatform()
⚫ static Thread - startVirtualThread(Runnable task)
⚫ java.util.concurrent.Executors
⚫ static ExecutorService - newVirtualThreadPerTaskExecutor()

Demo # 1: Virtual vs Platform Threads
⚫ Virtual Threads versus Platform Threads
⚫ JEP 425 Virtual Threads (Preview)
⚫ JEP 436 (Second Preview)
⚫ JEP 444 (Target -> JDK 21)
⚫ Runs on Java 19, 20, 21
⚫ javac --release 19 --enable-preview
⚫ java --enable-preview
⚫ Oracle JDBC Driver 21c instrumented to support Virtual Threads
⚫ Verifiable comparison of OS/HW resources consumption (Platform Threads x Virtual
Threads)

Reactive Streams Ingestion (RSI)
⚫ Java Library for Reactive Streams Ingestion
⚫ Streaming capability: Ingest data in an unblocking, and
reactive way from a large group of clients
⚫ Group records through RAC (Real App Clusters),
and Shard affinity using native UCP (Universal Connection
Pool)
⚫ Optimize CPU allocation while decoupling record
Processing from I/O
⚫ Fastest insert method for the Oracle Database through
Direct Path Insert, bypassing SQL and writing directly into the DB files

Demo # 2: Reactive Streams Ingestion (RSI)
⚫ PushPublisher API
// Push Publisher for Simple Usage
PushPublisher<Customer> pushPublisher =
ReactiveStreamsIngestion.pushPublisher();
pushPublisher.subscribe(rsi.subscriber());
// Ad-hoc usage
pushPublisher.accept(
new Customer(1, "John Doe", "North"));
// As a Consumer of a Stream
Customer[] customers = …
Stream
.of(customers)
.filter(c -> c.region.equals("NORTH"))
.forEach(pushPublisher::accept);
// As a Consumer for 3rd party Reactive Stream
// Libraries eg: Reactor https://projectreactor.io/
Flux
.just(
new Customer(1, "John Doe", "North"),
new Customer(2, "Jane Smith", "South"))
.concatWithValues(new Customer(3, "Bob", "South"))
.doOnComplete(() -> {System.out.println("Done!");})
.doOnCancel(() -> {System.out.println("Canceled!");})
.subscribe(pushPublisher::accept);

From Sync to Reactive JDBC: Oracle R2DBC
static String queryJdbc(java.sql.Connection connection) throws
SQLException {
try (java.sql.Statement statement = connection.createStatement()) {
ResultSet resultSet =
statement.executeQuery("SELECT * FROM CUSTOMERS");
if (resultSet.next())
return resultSet.getString(1);
else
throw new NoSuchElementException("Query returned zero rows");
}
}
static Publisher<String> queryR2dbc(io.r2dbc.spi.Connection
connection) {
return Flux.from(connection.createStatement(
"SELECT * FROM CUSTOMERS")
.execute())
.flatMap(result ->
result.map(row -> row.get(0, String.class)))
.switchIfEmpty(Flux.error(
new NoSuchElementException("Query returned zero rows")));
}

Demo # 3: Oracle R2DBC
⚫ Oracle Reactive Relational Database Connectivity (R2DBC)
⚫ Oracle R2DBC Driver is a Java library that supports reactive
Programming with Oracle Database
⚫ It implements the R2DBC Service Provider Interface (SPI) as
specified by the Reactive Relational Database Connectivity (R2DBC)
⚫ The R2DBC SPI exposes Reactive Streams as an abstraction for
Remote database operations
⚫ The sample code uses Project Reactor. It could use RxJava, Akka,
or any RS library
⚫ Runs on Java 11+

Virtual Threads or
Reactive?
• Oracle JDBC supports both!
• Want Virtual Threads?
• Oracle JDBC has been “Virtual
Thread Compatible” since 21.1.0.0
• Want Reactive?
• Oracle R2DBC 1.0.0 is available now
• Consume Flow interfaces directly
from Oracle JDBC’s Reactive
Extensions

Virtual Threads or Reactive?
• Benefits of Virtual Threads:
• Easier to read and write
• Easier to debug
• Integration with JDK tools
• Do not alter the basic concurrency model of Java
• Limitations of Virtual Threads:
• Some libraries are not compatible
• Still not in GA –> JDK 21
• JEP 425: Virtual Threads (Preview)
• JEP 436: Virtual Threads (Second Preview)
• JEP 444: Virtual Threads
Benefits of Reactive:
• Reactive Libraries (Reactor, RxJava, Akka, Vert.x)
• Stream-like API with a functional style
• Low-level concurrency is handled for you
(locks, atomics, queues).
Limitations of Reactive:
• Steep learning curve
• Harder to read and write
• Harder to debug

MOM - Message-Oriented Middleware
• A middleware component that allows
communication and exchanges the data
(messages).
• It involves passing data between applications
using a communication channel that carries self-
contained units of information (messages).
• In a MOM-based communication environment,
messages are sent and received asynchronously.
• MOM -> Apache ActiveMQ

AMQP – Advanced Message
Queuing Protocol
• The Advanced Message Queuing Protocol
(AMQP) is an open standard for passing
business messages between applications or
organizations.
• Realize the savings commoditization
brings; remove vendor lock-in
• Connect applications on different
platforms; choose the right platform for
the job
• Connect to business partners using a full-
featured open standard; remove
technical barriers to trade
• Position for innovations built upon the
foundation of AMQP
• Security, Reliability, Interoperability,
Standard, Openness

MQTT – Message Queuing
Telemetry Transport
• An OASIS standard messaging protocol for
the Internet of Things (IoT). It is designed as
an extremely lightweight publish/subscribe
messaging transport that is ideal for
connecting remote devices with a small code
footprint and minimal network bandwidth.

STOMP – Simple (or
Streaming) Text Oriented
Messaging Protocol
• STOMP provides an interoperable wire
format so that STOMP clients can
communicate with any STOMP message
broker to provide easy and widespread
messaging interoperability among many
languages, platforms and brokers..

ActiveMQ & JMS Message Consumer
⚫ ActiveMQ
⚫ ActiveMQ is a Java-based open-source message broker which supports REST API and
various wire-level protocols, such as MQTT, AMQP, and STOMP.
⚫ JMS Message Consumer
⚫ Like with any messaging-based application, you need to create a receiver that will
handle the messages that have been sent. The sample code snippet creates an AMQP
connection that connects to ActiveMQ using the given username, password,
hostname, and port number.
⚫ Port number 5672 is the default port that ActiveMQ is listening to over the AMQP
protocol when it starts up. It also creates a topic subscriber (consumer) to receive
messages that have been published to the "event" topic.

ActiveMQ & JMS Message Consumer

Demo #4: Architecture

Demo #4: Project Structure

References
• JDK 19 / Project Loom
• JDK 19 - https://openjdk.org/projects/jdk/19/
• Loom - https://openjdk.org/projects/loom/
• JEP 425 Virtual Threads (Preview) - https://bugs.openjdk.org/browse/JDK-8277131
• Reactive Streams Ingestion Library
• Getting Started with the Java library for Reactive Streams Ingestion (RSI) -
https://bit.ly/3rEiRnC
• High-throughput stream processing with the Java Library for Reactive Streams Ingestion (RSI),
Virtual Threads, and the Oracle ATP Database - https://bit.ly/3rATCTd
• RSI - https://docs.oracle.com/en/database/oracle/
• R2DBC
• Oracle R2DBC Driver – https://github.com/oracle/oracle-r2dbc
• Develop Java applications with Oracle Database
• JDBC – https://www.oracle.com/database/technologies/appdev/jdbc.html

Oracle LiveLabs
Showcasing how Oracle’s solutions can
solve your business problems
500+
free workshops,
available or in
development
3.5 million
people have already visited
LiveLabs
developer.oracle.com/livelabs
learn something new …at your pace!
600+
events run
using LiveLabs
workshops

Create your FREE
Cloud Account
• Go to
https://signup.cloud.oracle.c
om/

3 membership tiers
Connect: @oracleace facebook.com/OracleACEs
aceprogram_ww@oracle.com
500+ technical experts &
community leaders helping peers globally
The Oracle ACE Program recognizes & rewards individuals for
their technical & community contributions to the Oracle community
Nominate
yourself or a candidate:
ace.oracle.com/nominate
Learn more - ace.oracle.com
blogs.oracle.com/ace

DWX23 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMPDWX23-A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

Recommended

Recommended

More Related Content

Similar to DWX23 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMPDWX23-A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP

Similar to DWX23 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMPDWX23-A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP (20)

More from Juarez Junior

More from Juarez Junior (15)

Recently uploaded

Recently uploaded (20)

DWX23 - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMPDWX23-A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP