PhillyJug_ Getting Started With Real-time Cloud Native Streaming With Java Live at Judge Group in King of Prussia Pennsylvania hosted by the Philadelphia Java Users Group (PhillyJUG)

1. © 2023 Cloudera, Inc. All rights reserved.
Getting Started With Real-time
Cloud Native Streaming With Java
Apache Pulsar Development 101 with Java
Tim Spann
Principal Developer Advocate
15-March-2023

2. © 2023 Cloudera, Inc. All rights reserved.
TOPICS

3. © 2023 Cloudera, Inc. All rights reserved. 3
Topics
● Introduction to Streaming
● Introduction to Apache Pulsar
● Introduction to Apache Kafka
● FLaNK Stack
● Demos

4. © 2023 Cloudera, Inc. All rights reserved.

5. © 2023 Cloudera, Inc. All rights reserved.
FLiPN-FLaNK Stack
Tim Spann
@PaasDev // Blog: www.datainmotion.dev
Principal Developer Advocate.
Princeton Future of Data Meetup.
ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC
https://github.com/tspannhw/EverythingApacheNiFi
https://medium.com/@tspann
Apache NiFi x Apache Kafka x Apache Flink x Java

6. © 2023 Cloudera, Inc. All rights reserved. 6
FLiP Stack Weekly
This week in Apache Flink, Apache Pulsar, Apache
NiFi, Apache Spark, Java and Open Source friends.
https://bit.ly/32dAJft

7. © 2023 Cloudera, Inc. All rights reserved. 7
Largest Java Conference in the US!
12 tracks on Java, Cloud, Frameworks, Streaming, etc…
Devnexus.com
Join me! Save with SEEMESPEAK
https://devnexus.com/presentations/apache-pulsar-development-101-with-java

8. © 2023 Cloudera, Inc. All rights reserved.
STREAMING

9. © 2023 Cloudera, Inc. All rights reserved. 9
STREAMING FROM … TO .. WHILE ..
Data distribution as a first class citizen
IOT
Devices
LOG DATA
SOURCES
ON-PREM
DATA SOURCES
BIG DATA CLOUD
SERVICES
CLOUD BUSINESS
PROCESS SERVICES *
CLOUD DATA*
ANALYTICS /SERVICE
(Cloudera DW)
App
Logs
Laptops
/Servers Mobile
Apps
Security
Agents
CLOUD
WAREHOUSE
UNIVERSAL
DATA DISTRIBUTION
(Ingest, Transform, Deliver)
Ingest
Processors
Ingest
Gateway
Router, Filter &
Transform
Processors
Destination
Processors

10. © 2023 Cloudera, Inc. All rights reserved. 10
End to End Streaming Pipeline Example
Enterprise
sources
Weather
Errors
Aggregates
Alerts
Stocks
ETL
Analytics
Clickstream Market data
Machine logs Social
SQL

11. © 2023 Cloudera, Inc. All rights reserved. 11

12. Streaming for Java Developers
Multiple users, frameworks, languages, devices, data sources & clusters
• Expert in ETL (Eating, Ties
and Laziness)
• Deep SME in Buzzwords
• No Coding Skills
• R&D into Lasers
CAT AI
• Will Drive your Car?
• Will Fix Your Code?
• Will Beat You At Q-Bert
• Will Write my Next Talk
STREAMING ENGINEER
• Coding skills in Python,
Java
• Experience with Apache
Kafka
• Knowledge of database
query languages such as
SQL
• Knowledge of tools such
as Apache Flink, Apache
Spark and Apache NiFi
JAVA DEVELOPER
• Frameworks like Spring,
Quarkus and micronaut
• Relational Databases, SQL
• Cloud
• Dev and Build Tools

13. © 2023 Cloudera, Inc. All rights reserved.
APACHE PULSAR

15. Run a Local Standalone Bare Metal
wget
https://archive.apache.org/dist/pulsar/pulsar-2.11.0/apache-pulsar-2.11.0-bin.
tar.gz
tar xvfz apache-pulsar-2.11.0-bin.tar.gz
cd apache-pulsar-2.11.0
bin/pulsar standalone
https://pulsar.apache.org/docs/en/standalone/

16. <or> Run in Docker
docker run -it
-p 6650:6650
-p 8080:8080
--mount source=pulsardata,target=/pulsar/data
--mount source=pulsarconf,target=/pulsar/conf
apachepulsar/pulsar:2.11.0
bin/pulsar standalone
https://pulsar.apache.org/docs/en/standalone-docker/

17. Building Tenant, Namespace, Topics
bin/pulsar-admin tenants create meetup
bin/pulsar-admin namespaces create meetup/philly
bin/pulsar-admin tenants list
bin/pulsar-admin namespaces list meetup
bin/pulsar-admin topics create persistent://meetup/philly/first
bin/pulsar-admin topics list meetup/philly
https://github.com/tspannhw/Meetup-YourFirstEventDrivenApp

18. CLI Message Producing & Consuming
bin/pulsar-client produce
"persistent://meetup/philly/first" --messages 'Hello
Pulsar!'
bin/pulsar-client consume
"persistent://meetup/philly/first" -s first-reader -n 0

19. Monitoring and Metrics Check
curl http://localhost:8080/admin/v2/persistent/meetup/philly/first/stats |
python3 -m json.tool
bin/pulsar-admin topics stats-internal persistent://meetup/philly/first
curl http://localhost:8080/metrics/
bin/pulsar-admin topics peek-messages --count 5 --subscription first-reader
persistent://meetup/philly/first
bin/pulsar-admin topics subscriptions persistent://meetup/philly/first

20. Cleanup
bin/pulsar-admin topics delete persistent://meetup/philly/first
bin/pulsar-admin namespaces delete meetup/philly
bin/pulsar-admin tenants delete meetup

21. 101
Unified
Messaging
Platform
Guaranteed
Message
Delivery
Resiliency Infinite
Scalability

22. Streaming
Consumer
Consumer
Consumer
Subscription
Shared
Failover
Consumer
Consumer
Subscription
In case of failure in
Consumer B-0
Consumer
Consumer
Subscription
Exclusive
X
Consumer
Consumer
Key-Shared
Subscription
Pulsar
Topic/Partition
Messaging

23. Tenants / Namespaces / Topics
Tenants
(Compliance)
Tenants
(Data Services)
Namespace
(Microservices)
Topic-1
(Cust Auth)
Topic-1
(Location Resolution)
Topic-2
(Demographics)
Topic-1
(Budgeted Spend)
Topic-1
(Acct History)
Topic-1
(Risk Detection)
Namespace
(ETL)
Namespace
(Campaigns)
Namespace
(ETL)
Tenants
(Marketing)
Namespace
(Risk Assessment)
Pulsar Cluster

24. Messages - The Basic Unit of Pulsar
Component Description
Value / data payload The data carried by the message. All Pulsar messages contain raw bytes, although message data
can also conform to data schemas.
Key Messages are optionally tagged with keys, used in partitioning and also is useful for things like
topic compaction.
Properties An optional key/value map of user-defined properties.
Producer name The name of the producer who produces the message. If you do not specify a producer name, the
default name is used.
Sequence ID Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the
message is its order in that sequence.

25. Pulsar Cluster
● “Bookies”
● Stores messages and cursors
● Messages are grouped in
segments/ledgers
● A group of bookies form an
“ensemble” to store a ledger
● “Brokers”
● Handles message routing and
connections
● Stateless, but with caches
● Automatic load-balancing
● Topics are composed of
multiple segments
●
● Stores metadata for
both Pulsar and
BookKeeper
● Service discovery
Store
Messages
Metadata &
Service Discovery
Metadata &
Service Discovery
Metadata
Storage

26. Producer-Consumer
Producer Consumer
Publisher sends data and
doesn't know about the
subscribers or their status.
All interactions go through
Pulsar and it handles all
communication.
Subscriber receives data
from publisher and never
directly interacts with it
Topic
Topic

27. Apache Pulsar: Messaging vs Streaming
Message Queueing - Queueing
systems are ideal for work queues
that do not require tasks to be
performed in a particular order.
Streaming - Streaming works
best in situations where the
order of messages is important.

28. Pulsar Subscription Modes
Different subscription modes
have different semantics:
Exclusive/Failover -
guaranteed order, single active
consumer
Shared - multiple active
consumers, no order
Key_Shared - multiple active
consumers, order for given key
Producer 1
Producer 2
Pulsar Topic
Subscription D
Consumer D-1
Consumer D-2
Key-Shared
<
K
1,
V
10
>
<
K
1,
V
11
>
<
K
1,
V
12
>
<
K
2
,V
2
0
>
<
K
2
,V
2
1>
<
K
2
,V
2
2
>
Subscription C
Consumer C-1
Consumer C-2
Shared
<
K
1,
V
10
>
<
K
2,
V
21
>
<
K
1,
V
12
>
<
K
2
,V
2
0
>
<
K
1,
V
11
>
<
K
2
,V
2
2
>
Subscription A Consumer A
Exclusive
Subscription B
Consumer B-1
Consumer B-2
In case of failure in
Consumer B-1
Failover

29. Flexible Pub/Sub API for Pulsar - Shared
Consumer consumer =
client.newConsumer()
.topic("my-topic")
.subscriptionName("work-q-1")
.subscriptionType(SubType.Shared)
.subscribe();

30. Flexible Pub/Sub API for Pulsar - Failover
Consumer consumer = client.newConsumer()
.topic("my-topic")
.subscriptionName("stream-1")
.subscriptionType(SubType.Failover)
.subscribe();

31. Data Offloaders
(Tiered Storage)
Client Libraries
StreamNative Pulsar Ecosystem
hub.streamnative.io
Connectors
(Sources & Sinks)
Protocol Handlers
Pulsar Functions
(Lightweight Stream
Processing)
Processing Engines
… and more!
… and more!

32. Kafka
On Pulsar
(KoP)

33. MQTT
On Pulsar
(MoP)

34. AMQP
On Pulsar
(AoP)

35. Schema Registry
Schema Registry
schema-1 (value=Avro/Protobuf/JSON) schema-2 (value=Avro/Protobuf/JSON) schema-3
(value=Avro/Protobuf/JSON)
Schema
Data
ID
Local Cache
for Schemas
+
Schema
Data
ID +
Local Cache
for Schemas
Send schema-1
(value=Avro/Protobuf/JSON) data
serialized per schema ID
Send (register)
schema (if not in
local cache)
Read schema-1
(value=Avro/Protobuf/JSON) data
deserialized per schema ID
Get schema by ID (if
not in local cache)
Producers Consumers

36. Building Real-Time Requires a Team

37. Pulsar - Spring
https://github.com/spring-projects-experimental/spring-pulsar

38. Pulsar - Spring - Code
@Autowired
private PulsarTemplate<Observation> pulsarTemplate;
this.pulsarTemplate.setSchema(Schema.
JSON(Observation.class));
MessageId msgid = pulsarTemplate.newMessage(observation)
.withMessageCustomizer((mb) -> mb.key(uuidKey.toString()))
.send();
@PulsarListener(subscriptionName = "aq-spring-reader", subscriptionType = Shared,
schemaType = SchemaType.
JSON, topics = "persistent://public/default/aq-pm25")
void echoObservation(Observation message) {
this.log.info("PM2.5 Message received: {}", message);
}

39. Pulsar - Spring - Configuration
spring:
pulsar:
client:
service-url: pulsar+ssl://sn-academy.sndevadvocate.snio.cloud:6651
auth-plugin-class-name: org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
authentication:
issuer-url: https://auth.streamnative.cloud/
private-key: file:///scr/sndevadvocate-tspann.json
audience: urn:sn:pulsar:sndevadvocate:my-instance
producer:
batching-enabled: false
send-timeout-ms: 90000
producer-name: airqualityjava
topic-name: persistent://public/default/airquality

40. Spring - Pulsar as Kafka
https://www.baeldung.com/spring-kafka
@Bean
public KafkaTemplate<String, Observation> kafkaTemplate() {
KafkaTemplate<String, Observation> kafkaTemplate =
new KafkaTemplate<String, Observation>(producerFactory());
return kafkaTemplate;
}
ProducerRecord<String, Observation> producerRecord = new ProducerRecord<>(topicName,
uuidKey.toString(),
message);
kafkaTemplate.send(producerRecord);

41. Spring - MQTT - Pulsar
https://roytuts.com/publish-subscribe-message-onto-mqtt-using-spring/
@Bean
public IMqttClient mqttClient(
@Value("${mqtt.clientId}") String clientId,
@Value("${mqtt.hostname}") String hostname,
@Value("${mqtt.port}") int port)
throws MqttException {
IMqttClient mqttClient = new MqttClient(
"tcp://" + hostname + ":" + port, clientId);
mqttClient.connect(mqttConnectOptions());
return mqttClient;
}
MqttMessage mqttMessage = new MqttMessage();
mqttMessage.setPayload(DataUtility.serialize(payload));
mqttMessage.setQos(0);
mqttMessage.setRetained(true);
mqttClient.publish(topicName, mqttMessage);

42. Spring - AMQP - Pulsar
https://www.baeldung.com/spring-amqp
rabbitTemplate.convertAndSend(topicName,
DataUtility.serializeToJSON(observation));
@Bean
public CachingConnectionFactory
connectionFactory() {
CachingConnectionFactory ccf =
new CachingConnectionFactory();
ccf.setAddresses(serverName);
return ccf;
}

43. Reactive Spring - Pulsar

44. Reactive Spring - Pulsar

45. The FLiPN kitten crosses the stream 4 ways with Apache Pulsar

46. Demo

47. © 2023 Cloudera, Inc. All rights reserved. 47
REST + Spring Boot + Pulsar + Friends

48. © 2023 Cloudera, Inc. All rights reserved.
APACHE PULSAR JAVA FUNCTION

49. https://github.com/tspannhw/pulsar-airquality-function
public class AirQualityFunction implements Function<byte[], Void> {
@Override
public Void process(byte[] input, Context context) {
if ( input == null || context == null ) {
return null;
}
//context.getInputTopics().toString()
if ( context.getLogger() != null && context.getLogger().isDebugEnabled() ) {
context.getLogger().debug("LOG:" + input.toString());
}
context.newOutputMessage(“NewTopicName”, JSONSchema.of(Observation.class))
.key(UUID.randomUUID().toString())
.property(“Language”, “java”)
.value(observation)
.send();
}
}

50. © 2023 Cloudera, Inc. All rights reserved.
APACHE KAFKA

51. © 2023 Cloudera, Inc. All rights reserved. 51
Apache Kafka
• Highly reliable distributed
messaging system
• Decouple applications, enables
many-to-many patterns
• Publish-Subscribe semantics
• Horizontal scalability
• Efficient implementation to
operate at speed with big data
volumes
• Organized by topic to support
several use cases
Source
System
Source
System
Source
System
Kafka
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Source
System
Source
System
Source
System
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Many-To-Many
Publish-Subscribe
Point-To-Point
Request-Response

52. © 2023 Cloudera, Inc. All rights reserved.
STREAM TEAM

53. © 2023 Cloudera, Inc. All rights reserved. 53
Cloudera’s Data-In-Motion Services
Cloudera Offers Two Core Data-In-Motion Services: DataFlow & Stream Processing
CLOUDERA SDX — Secure, Monitor and Govern your
Streaming workloads with the same tooling using Apache
Ranger & Apache Atlas.
STREAM PROCESSING — Powered by Apache Flink and
Kafka, it provides a complete, enterprise-grade stream
management and stateful processing solution. With
support for industry standard interfaces like SQL,
developers, data analysts, and data scientist can easily
build a wide variety of hybrid real-time applications.
DATAFLOW — Powered by Apache NiFi, it enables
developers to connect to any data source anywhere with
any structure, process it, and deliver to any destination
using a low-code authoring experience.

54. © 2023 Cloudera, Inc. All rights reserved. 54
CSP Community
Edition
• Kafka, KConnect, SMM, SR,
Flink, and SSB in Docker
• Runs in Docker
• Try new features quickly
• Develop applications locally
● Docker compose file of CSP to run from command line w/o any
dependencies, including Flink, SQL Stream Builder, Kafka, Kafka
Connect, Streams Messaging Manager and Schema Registry
○ $>docker compose up
● Licensed under the Cloudera Community License
● Unsupported
● Community Group Hub for CSP
● Find it on docs.cloudera.com under Applications

55. © 2023 Cloudera, Inc. All rights reserved. 55
STREAMING DATA WITH CLOUDERA DATA FLOW (CDF)
01 03
04
05
Cloudera Flow
Management
Cloudera Stream
Processing
Streams
Replication
Manager
Cloudera Flow
Management
Cloudera
Machine
Learning
02
Collect
Buffer
Replicate
Distribute
Score
Regional Data Centers Global Public Cloud
Edge

56. © 2023 Cloudera, Inc. All rights reserved. 56
ENABLING ANALYTICS AND INSIGHTS ANYWHERE
Driving enterprise business value
REAL-TIME
STREAMING
ENGINE
ANALYTICS &
DATA WAREHOUSE
DATA SCIENCE/
MACHINE LEARNING
CENTRALIZED DATA
PLATFORM
STORAGE & PROCESSING
ANALYTICS & INSIGHTS
Stream
Ingest
Ingest – Data
at Rest
Deploy
Models
BI
Solutions
SQL Predictive
Analytics
• Model Building
• Model Training
• Model Scoring
Actions &
Alerts
[SQL]
Real-Time
Apps
STREAMING DATA
SOURCES
Clickstream Market data
Machine logs Social
ENTERPRISE DATA
SOURCES
CRM
Customer
history
Research
Compliance
Data
Risk Data
Lending

57. © 2023 Cloudera, Inc. All rights reserved.
© 2019 Cloudera, Inc. All rights reserved. 57
EVENT-DRIVEN ORGANIZATION
Modernize your data and applications
CDF Event Streaming Platform
Integration - Processing - Management - Cloud
Stream
ETL
Cloud
Storage
Application
Data Lake Data Stores
Make
Payment
µServices
Streams
Edge - IoT Dashboard

58. © 2023 Cloudera, Inc. All rights reserved.
DATAFLOW
APACHE NIFI

59. © 2023 Cloudera, Inc. All rights reserved. 59
CLOUDERA FLOW AND EDGE MANAGEMENT
Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data
center) to any downstream system with built in end-to-end security and provenance
Advanced tooling to industrialize
flow development (Flow Development
Life Cycle)
ACQUIRE
• Over 300 Prebuilt Processors
• Easy to build your own
• Parse, Enrich & Apply Schema
• Filter, Split, Merger & Route
• Throttle & Backpressure
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG
PROCESS
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ENCRYPT
TALL
EVALUATE
EXECUTE
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
ROUTE RATE
DISTRIBUTE LOAD
DELIVER
• Guaranteed Delivery
• Full data provenance from
acquisition to delivery
• Diverse, Non-Traditional Sources
• Eco-system integration
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLOG

60. © 2023 Cloudera, Inc. All rights reserved. 60
Cloudera DataFlow: Universal Data Distribution Service
Process
Route
Filter
Enrich
Transform
Distribute
Connectors
Any
destination
Deliver
Ingest
Active
Passive
Connectors
Gateway
Endpoint
Connect & Pull
Send
Data born in
the cloud
Data born
outside the
cloud
UNIVERSAL DATA DISTRIBUTION WITH CLOUDERA DATAFLOW (CDF)
Connect to Any Data Source Anywhere then Process and Deliver to Any Destination

61. 61
© 2023 Cloudera, Inc. All rights reserved.
WHAT IS APACHE NIFI?
Apache NiFi is a scalable, real-time streaming data
platform that collects, curates, and analyzes data so
customers gain key insights for immediate
actionable intelligence.

62. © 2023 Cloudera, Inc. All rights reserved. 62
APACHE NIFI
Enable easy ingestion, routing, management and delivery of
any data anywhere (Edge, cloud, data center) to any
downstream system with built in end-to-end security and
provenance
ACQUIRE PROCESS DELIVER
• Over 300 Prebuilt Processors
• Easy to build your own
• Parse, Enrich & Apply Schema
• Filter, Split, Merger & Route
• Throttle & Backpressure
• Guaranteed Delivery
• Full data provenance from acquisition to
delivery
• Diverse, Non-Traditional Sources
• Eco-system integration
Advanced tooling to industrialize flow development
(Flow Development Life Cycle)
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLO
G
FTP
SFTP
HL7
UDP
XML
HTTP
EMAIL
HTML
IMAGE
SYSLO
G
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
CONTROL RATE
DISTRIBUTE LOAD
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ENCRYPT
TALL
EVALUATE
EXECUTE

63. © 2023 Cloudera, Inc. All rights reserved. 63
https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
● Reduce, Reuse, Recycle. Use Parameters to reuse
common modules.
● Put flows, reusable chunks into separate Process
Groups.
● Write custom processors if you need new or
specialized features
● Use Cloudera supported NiFi Processors
● Use Record Processors everywhere
No More Spaghetti Flows

64. © 2023 Cloudera, Inc. All rights reserved.
JAVA DEV FOR NIFI

65. © 2023 Cloudera, Inc. All rights reserved. 65
https://github.com/tspannhw/nifi-tensorflow-processor
public class TensorFlowProcessor extends AbstractProcessor {
public static final PropertyDescriptor MODEL_DIR = new
PropertyDescriptor.Builder().name(MODEL_DIR_NAME)
.description("Model Directory").required(true).expressionLanguageSupported(true)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws
ProcessException {
FlowFile flowFile = session.get();
if (flowFile == null) {
flowFile = session.create();
}
try {
flowFile.getAttributes();
}

66. © 2023 Cloudera, Inc. All rights reserved.
APACHE FLINK

67. © 2023 Cloudera, Inc. All rights reserved. 67
Flink SQL
https://www.datainmotion.dev/2021/04/cloudera-sql-stream-builder-ssb-updated.html
● Streaming Analytics
● Continuous SQL
● Continuous ETL
● Complex Event Processing
● Standard SQL Powered by Apache Calcite

68. © 2023 Cloudera, Inc. All rights reserved. 68
Flink SQL
-- specify Kafka partition key on output
SELECT foo AS _eventKey FROM sensors
-- use event time timestamp from kafka
-- exactly once compatible
SELECT eventTimestamp FROM sensors
-- nested structures access
SELECT foo.’bar’ FROM table; -- must quote nested
column
-- timestamps
SELECT * FROM payments
WHERE eventTimestamp > CURRENT_TIMESTAMP-interval
'10' second;
-- unnest
SELECT b.*, u.*
FROM bgp_avro b,
UNNEST(b.path) AS u(pathitem)
-- aggregations and windows
SELECT card,
MAX(amount) as theamount,
TUMBLE_END(eventTimestamp, interval '5' minute) as
ts
FROM payments
WHERE lat IS NOT NULL
AND lon IS NOT NULL
GROUP BY card,
TUMBLE(eventTimestamp, interval '5' minute)
HAVING COUNT(*) > 4 -- >4==fraud
-- try to do this ksql!
SELECT us_west.user_score+ap_south.user_score
FROM kafka_in_zone_us_west us_west
FULL OUTER JOIN kafka_in_zone_ap_south ap_south
ON us_west.user_id = ap_south.user_id;
Key Takeaway: Rich SQL grammar with advanced time and aggregation tools

69. 69
© 2022 Cloudera, Inc. All rights reserved.
SQL STREAM BUILDER (SSB)
SQL STREAM BUILDER allows
developers, analysts, and data
scientists to write streaming
applications with industry
standard SQL.
No Java or Scala code
development required.
Simplifies access to data in Kafka
& Flink. Connectors to batch data in
HDFS, Kudu, Hive, S3, JDBC, CDC
and more
Enrich streaming data with batch
data in a single tool
Democratize access to real-time data with just SQL

70. © 2023 Cloudera, Inc. All rights reserved.
DEMO

71. © 2023 Cloudera, Inc. All rights reserved. 71
End to End Streaming Demo Pipeline
Enterprise
sources
Weather
Errors
Aggregates
Alerts
Stocks
ETL
Analytics
Streaming SQL
Clickstream Market data
Machine logs Social
https://github.com/tspannhw/CloudDemo2021

72. AI BASED ENHANCEMENTS
SERVE
SOURCES
Data Warehouse
Report
Sensorid
Sensor conditions
Machine Learning
Predict, Automate
Control System
REPORT
Visualize
CLOUD
Collect
COLLECT
Message Broker
Data Flow
Distribute
Sensor id
Temperature
COLLECT and DISTRIBUTE DATA
1. Data is collected from sensors that use mqtt protocol via
CEM and sent to CDP Public Cloud
2. Two CDF flows run in the cloud to accomplish our two
goals: streaming analytics and batch analytics
ENRICH, REPORT
Report, Automate
Real time alerting
SQL
Stream Builder
Data Visualization
Edge
Management
Humidity
Timestamp
Visualize
Data Visualization
USE DATA
3. Streaming Use Case: some conditions of our greenhouse
must be avoided and have to be controlled in real time.
Some warnings have been defined to alert us in case
alerting conditions are met, control system is
automatically activated to adjust environmental
variables.
4. Batch analytics: to ensure the optimal growth of our
plants the ideal conditions have to be met for each
plant. Each 6 hours the plant conditions are monitored
and in case some control adjustment is required a ML
model gives a suggestion about getting the optimal
point minimizing the cost.
EDGE
Control System

73. © 2023 Cloudera, Inc. All rights reserved. 73

74. © 2023 Cloudera, Inc. All rights reserved.
RESOURCES AND WRAP-UP

75. © 2023 Cloudera, Inc. All rights reserved. 75
● https://streamnative.io/blog/engineering/2022-11-29-spring-into-pulsar-part-2-spri
ng-based-microservices-for-multiple-protocols-with-apache-pulsar/
● https://streamnative.io/blog/release/2022-09-21-announcing-spring-for-apache-pu
lsar/
● https://docs.spring.io/spring-pulsar/docs/current-SNAPSHOT/reference/html/
● https://spring.io/blog/2022/08/16/introducing-experimental-spring-support-for-apa
che-pulsar
● https://medium.com/@tspann/using-the-new-spring-boot-apache-pulsar-integratio
n-8a38447dce7b
Spring + Pulsar References

76. © 2023 Cloudera, Inc. All rights reserved. 76
● https://spring.io/guides/gs/spring-boot/
● https://spring.io/projects/spring-amqp/
● https://spring.io/projects/spring-kafka/
● https://github.com/spring-projects/spring-integration-kafka
● https://github.com/spring-projects/spring-integration
● https://github.com/spring-projects/spring-data-relational
● https://github.com/spring-projects/spring-kafka
● https://github.com/spring-projects/spring-amqp
Spring Things

77. © 2023 Cloudera, Inc. All rights reserved. 77
STREAMING RESOURCES
• https://dzone.com/articles/real-time-stream-processing-with-hazelcast-an
d-streamnative
• https://flipstackweekly.com/
• https://www.datainmotion.dev/
• https://www.flankstack.dev/
• https://github.com/tspannhw
• https://medium.com/@tspann
• https://medium.com/@tspann/predictions-for-streaming-in-2023-ad4d739
5d714
• https://www.apachecon.com/acna2022/slides/04_Spann_Tim_Citizen_Str
eaming_Engineer.pdf

78. © 2023 Cloudera, Inc. All rights reserved. 78
78
Apache
Pulsar
in Action
Please enjoy David’s complete
book which is the ultimate
guide to Pulsar.

79. © 2021 Cloudera, Inc. All rights reserved. 79
https://github.com/tspannhw https://www.datainmotion.dev/

80. © 2021 Cloudera, Inc. All rights reserved. 80
March 16 - Virtual
March 17 - Trenton
April 4 - Atlanta
April 24 - San Francisco
Upcoming Events

81. © 2023 Cloudera, Inc. All rights reserved. 81
Resources

82. 82
TH N Y U