Conf42 Python_ ML Enhanced Event Streaming Apps with Python Microservices
https://www.conf42.com/Python_2023_Tim_Spann_David_Kjerrumgaard_ml_enhanced_event_streaming_apps_microservices
Build ML Enhanced Event Streaming Apps with Python Microservices
TIM SPANN
PRINCIPAL DEVELOPER ADVOCATE @ STREAMNATIVE
Tim Spann's LinkedIn account Tim Spann's twitter account
DAVID KJERRUMGAARD
DEVELOPER ADVOCATE @ STREAMNATIVE
David Kjerrumgaard's LinkedIn account
Share
The easy way to build and scale machine learning apps.
2. 2
Tim Spann
Principal Developer
Advocate
FLiP(N)/FLaNK Stack = Flink, Kafka, Pulsar and NiFi Stack
Streaming Systems & Data Architecture Expert
Experience:
â—Ź 15+ years of experience with streaming technologies
including NiFi, Kafka, Pulsar, Flink, Spark, Big Data,
Cloud, ML, IoT, Spring, Java, SQL, Python and more.
â—Ź Today, he helps to grow the Pulsar community
sharing rich technical knowledge and experience at
both global conferences and through individual
conversations.
6. Install Python 3 Pulsar Client
pip3 install pulsar-client=='2.11.0[all]'
# Depending on Platform May Need C++ Client Built
For Python on Pulsar on Pi https://github.com/tspannhw/PulsarOnRaspberryPi
https://pulsar.apache.org/docs/en/client-libraries-python/
6
7. Building a Python3 Producer
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
producer
client.create_producer('persistent://conf/ete/first')
producer.send(('Simple Text Message').encode('utf-8'))
client.close()
7
26. • Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a sixty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control
DataFlows for Data Ingest, Movement and Routing
27. • Moving Binary, Unstructured, Image
and Tabular Data
• Enrichment
• Universal Visual Processor
• Simple Event Processor
• Routing
• Feeding data to Central Messaging
• Support for modern protocols
• Kafka Protocol Source/Sink
• Pulsar Protocol Source/Sink
The Power of Apache NiFi