1. FLiPN Awesome Streaming with
Open Source
2022.07.29
Tim Spann | Developer Advocate

2. Tim Spann
Developer Advocate
● FLiP(N) Stack = Flink, Pulsar and NiFi Stack
● Streaming Systems/ Data Architect
● Experience:
○ 15+ years of experience with batch and streaming
technologies including Pulsar, Flink, Spark, NiFi, Spring,
Java, Big Data, Cloud, MXNet, Hadoop, Datalakes, IoT
and more.

3. ● NiFi, Flink, Spark, Pulsar
● Ingest Data
● IoT Data Example
● MQTT
● Spark SQL
● Flink SQL
● WebSocket Dashboard
● What, How and Why?

4. Demo

6. Apache Pulsar
Apache Flink
Apache NiFi
Apache Spark
+ Open Source Data
Stores
https://streamnative.io/blog/engineering/2022-04-14-what-the-flip-is-the-flip-stack/

7. ➔ Perform in Real-Time
➔ Process Events as They Happen
➔ Joining Streams with SQL
➔ Find Anomalies Immediately
➔ Ordering and Arrival Semantics
➔ Continuous Streams of Data
DATA STREAMING

8. • Drag and Drop
• Hundreds of Sources
• Hundreds of Sinks
• Support Custom JVM Apps
• Scalable
Apache NiFi

9. • Continuous SQL
• Real-time Event Processing
• Native Connector for Pulsar
• Scalable
• Batch and Stream
Apache Flink

10. • Standard for ETL and ML
• Python, Scala, Java
• Scalable
• Streaming To/From Pulsar
• Lakehouse
Apache Spark

11. • Unified Messaging & Streaming
• Multiple Protocol Support
• Compute and Storage Separation
• Scalable
• Multi-Tenant
Apache Pulsar

12. A Unified Messaging Platform
Message Queuing
Data Streaming

13. Ingest

14. Ingest

15. Ingest

16. Local Weather Sensors
IoT
https://github.com/tspannhw/FLiP-Pi-Weather
pressure | real||["null","float"]
temperature | real||["null","float"]
humidity | real||["null","float"]
devicetemperature | real||["null","float"]
dewpoint | real||["null","float"]
lux | real||["null","float"]

17. MQTT
On Pulsar
(MoP)

18. MQTT from Python
pip3 install paho-mqtt
import paho.mqtt.client as mqtt
client = mqtt.Client("mqtt-iot")
row = { }
row['gasKO'] = str(readings)
json_string = json.dumps(row)
json_string = json_string.strip()
client.connect("pulsar-server", 1883, 180)
client.publish("persistent://public/default/topic1", payload=json_string,
qos=0, retain=True)

19. streamnative.io

20. Spark SQL
https://github.com/tspannhw/FLiP-Pi-BreakoutGarden

21. streamnative.io
SQL
select aqi, parameterName, dateObserved, hourObserved, latitude,
longitude, localTimeZone, stateCode, reportingArea from
airquality
select max(aqi) as MaxAQI, parameterName, reportingArea from
airquality group by parameterName, reportingArea
select max(aqi) as MaxAQI, min(aqi) as MinAQI, avg(aqi) as
AvgAQI, count(aqi) as RowCount, parameterName, reportingArea
from airquality group by parameterName, reportingArea

22. WebSocket Dashboard

23. WebSocket Dashboard

24. var wsUri =
"ws://pulsar1:8080/ws/v2/consumer/persistent/public/default/pm25/a2
5-rdr?subscriptionType=Shared&receiverQueueSize=500";
websocket = new WebSocket(wsUri);
websocket.onopen = function(evt) {
console.log('open');
};
websocket.onerror = function(evt) {console.log('ERR', evt)};
websocket.onmessage = function(evt) {
var dataPoints = JSON.parse(evt.data);
WebSocket Receive

25. if ( dataPoints === undefined || dataPoints == null ||
dataPoints.payload === undefined || dataPoints.payload == null
) { return; }
if (IsJsonString(atob(dataPoints.payload))) {
var pulsarMessage = JSON.parse(atob(dataPoints.payload));
if ( pulsarMessage === undefined || pulsarMessage == null ) {
return;
}
var publishTime = " ";
if ( !isEmpty(dataPoints.publishTime) ) {
publishTime = dataPoints.publishTime;
}
WebSocket Data Parse

26. val dfPulsar =
spark.readStream.format("pulsar").option("se
rvice.url",
"pulsar://pulsar1:6650").option("admin.url",
"http://pulsar1:8080").option("topic",
"persistent://public/default/aq1").load()
Apache Spark for SQL ETL

27. Devices

28. What, How and Why?
EVENTS
MESSAGES

29. More and Faster
● https://github.com/tspannhw/FLiP-Py-Pi-GasThermal
● https://github.com/tspannhw/FLiPC-FastStreamingIntoClickhouseWithApachePulsar
● https://github.com/tspannhw/FLiPN-DEVNEXUS-2022
● https://github.com/tspannhw/airquality-mqtt-consumer
● https://github.com/tspannhw/FLiP-Pi-Weather

30. FLiP Stack Weekly
This week in Apache Flink, Apache Pulsar, Apache
NiFi, Apache Spark, Java and Open Source friends.
https://bit.ly/32dAJft

31. Pulsar
Resources Page
Learn More
https://streamnative.io/blog/engineering/2022-04-14-w
hat-the-flip-is-the-flip-stack/
https://streamnative.io/cloudforkafka/
https://github.com/tspannhw/airquality
https://github.com/tspannhw/pulsar-airquality-function

32. Let’s Keep
in Touch!
Tim Spann
Developer Advocate
PaaSDev
https://www.linkedin.com/in/timothyspann
https://github.com/tspannhw

33. Thanks