The document presents a technical overview of using Apache Kafka and the Confluent platform for optimizing real-time analytics in financial services and IoT applications. Key topics include stream processing, event-driven architecture, and the integration of various data sources to enhance data management and analytics capabilities. It emphasizes the importance of evolving legacy systems to support global-scale, real-time data processing and storage.

1. Technical Deep Dive:
Using Kafka to Optimize Real-Time Analytics in
Financial Services & IoT Applications
February 6, 2019

2. Our Presenters
Peter Simpson
VP Panopticon Streaming Analytics
Responsible for driving the vision and strategy for the Panopticon
streaming analytics platform. Prior to Panopticon, he was Product
Manager at Instant Information, a provider of news analytics solutions
to trading desks. He also held analytical roles at HSBC Global Markets
within Technology, Equities, and Research for several years. Peter
holds a MSc in Info Systems Engineering and a BSc in Physics with
Space Science & Technology.
Tom Underhill
Confluent Partner Solutions Architect
Responsible for helping Confluent’s many partners become successful
implementing solutions using the Confluent Platform. Tom joined
Confluent after several years consulting in the Big Data and Analytics
space where he led technical teams delivering large scale integration
projects. His passion has always been around liberating data from
silos, turning batch into real time, and building systems that scale.

3. About Confluent
• Pub-sub messaging in real-time at scale
• Connectivity for all producers and consumers
• Data persistence with infinite retention
• Stream processing without coding
• Distributed architecture for global deployment
The streaming platform built by the creators of Apache Kafka

4. About Panopticon
Streaming Analytics Platform
Historic Time
Series Database Request / Response
Visual
Analytics
Stream
Processing
Predictive
Publish &
Subscribe
Publish & Subscribe
Fast
§ Data Access
§ Data Processing
§ Comprehension
§ Investigation
§ Insight
§ Deployment
Supporting
§ Trading
§ Compliance
§ Risk

5. 5
Build a Central
Nervous System for
your Modern Event-
driven Enterprise
As the creators of Apache Kafka,
Confluent delivers the only
enterprise-ready streaming platform
Relational DB
Apps Microservices
SaaS apps
Custom apps
Data warehouse

6. 6
A digital business represents events in dataEvents are at
the Heart of
Every
Business
A business is defined
through a series of
events and its ability to
respond to them
A Trade
A Sale
An Invoice
A Customer
Experience

7. 7
To become truly Event-driven, Organizations with
Legacy Architectures will need to Evolve

8. 8
● Global-scale
● Real-time
● Persistent Storage
● Stream Processing
Apache Kafka®:
the De-facto
Standard for
Real-Time Event
Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache Kafka

9. 9
Kafka is a Good Starting Point, Confluent
Completes the Journey
Set up secure Kafka
& build your first app
Understand streaming
Monitor & manage a
mission-critical solution
Set up secure Kafka &
build your first app
Understand streaming
Infrastructure & apps
across LOBs
Monitor & manage a
mission-critical solution
Set up secure Kafka &
build your first app
Understand streaming
Self-service on shared
Kafka
Infrastructure &
applications across
LOBs
Monitor & manage a
mission-critical solution
Set up secure Kafka &
build your first app
Understand streamingUnderstand streaming
Pre-streamingValue
Stream Everything
05
Break Silos
04
03
Go To Production
02
Learn Kafka
01
Investment & Time
Solve A Critical
Need

10. 10
Clients: Communicate with Kafka in a Broad
Variety of Languages
Apache Kafka
Confluent Platform Community Supported
Proxy http/REST
stdin/stdout
Confluent Platform Clients developed and fully supported by Confluent

11. 11
Apache Kafka Connect API: Import and Export
Data In & Out of Kafka
JDBC
Mongo
MySQL
Elastic
Cassandra
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of
data sources and sinks
Preserves data schema
Integrated within
Confluent Control Center

12. 12
Schema Registry: Make Data Backwards
Compatible and Future-Proof
● Define the expected fields for each Kafka topic
● Automatically handle schema changes (e.g. new
fields)
● Prevent backwards incompatible
changes
● Support multi-data center environments
Elastic
Cassandra
HDFS
Example Consumers
Serializer
App 1
Serializer
App 2
!
Kafka Topic!
Schema
Registry

13. 13
REST Proxy
Non-Java Applications
Native Kafka Java
Applications
Schema Registry
REST /
HTTP
Simplifies administrative
actions
Simplifies message
creation and consumption
Provides a RESTful
interface to a Kafka
cluster
REST Proxy: Talk to Non-native Kafka Apps and
Outside the Firewall
Community Feature

14. 14
Event Transformation with Stream Processing
streams
The streaming SQL engine for Apache
Kafka® to write real-time applications in SQL
You write only SQL. No Java, Python, or other
boilerplate to wrap around it!
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;
But you can create KSQL User Defined
Functions in Java
Apache Kafka® library to write
real-time applications and microservices
in Java and Scala
confluent.io/product/ksql
Confluent KSQL

15. 15
Replicator: Stretch Kafka Across Data Centers
and Public Cloud
Protect business-critical data and
metadata by replicating down to topic-level
configurations
Minimize recovery time objectives (RTO)
through automated failover and switchback
Meet recovery point objectives (RPO)
running more workers to increase
replication throughput
Bridge your data center to the
cloud with Confluent Cloud
Commercial Feature

16. 16
Management and Monitoring for the Enterprise
Monitor system health of your Kafka
cluster with curated dashboards
Monitor data streams with end to end
views of message delivery
Manage Kafka topics and Kafka
Connect operations
Confluent Control Center

17. 17
Complete Set of Development, Operations and
Management Capabilities to run Kafka at Scale
Apache Kafka®
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Enterprise Operations
Replicator | Auto Data Balancer | Connectors | MQTT Proxy | k8s Operator
Database
Changes
Log Events IoT Data Web Events
Other
Events
DATA
INTEGRATION
REAL-TIME
APPLICATIONS
COMMUNITY FEATURES
COMMERCIAL FEATURES
Datacenter Public Cloud Confluent Cloud
Confluent Platform
Management & Monitoring
Control Center | Security
Development & Connectivity
Connectors | REST Proxy | KSQL
Confluent fully-managedCustomer self-managed
Hadoop
Database
Data
Warehouse
CRM
Other
Transformation
Custom Apps
Analytics
Monitoring
Other

18. 18
Lowering the Bar to Enter the World of Streaming
Kafka User Population
CodingSophistication
Core Java developers
Core developers who don’t use Java/Scala
Data engineers, architects, DevOps/SRE
BI analysts
streams

19. Contact Us
Peter Simpson
VP Panopticon Streaming Analytics
info@panopticon.com
www.panopticon.com
Tom Underhill
Partner Solutions Architect
info@confluent.io
www.confluent.io