Streaming Visualization

@gschmutz
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Streaming Visualization
Guido Schmutz
Big Data Technology Summit 2019 – 27.2.19
@gschmutz guidoschmutz.wordpress.com

@gschmutz
Agenda
1. Introduction
2. Stream Data Integration & Stream Analytics Ecosystem
3. 3 Blueprints for Streaming Visualization
End-to-End Demo available here:
https://github.com/gschmutz/various-demos/tree/master/streaming-visualization

@gschmutz
Guido Schmutz
Working at Trivadis for more than 22 years
Oracle Groundbreaker Ambassador & Oracle ACE Director
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
146th edition

@gschmutz
Introduction

@gschmutz
Data Value Chain – Why Data Streaming?
Milliseconds
• Place Trace
• Serve ad
• Enrich Stream
• Approve Trans
Hundredths of Seconds
• Calculate Risk
• Leaderboard
• Aggregate
• Count
Second(s)
• Retrieve Click
Stream
• Show orders
Minutes
• Backtest algo
• BI
• Daily Reports
Hours
• Algo discovery
• Log analysis
• Fraud pattern match

@gschmutz
Keep the data in motion …
Data at Rest Data in Motion
Store
(Re)Act
Visualize/
Analyze
StoreAct
Analyze
11101
01010
10110
11101
01010
10110
vs.
Visualize

@gschmutz
Hadoop Clusterd
Hadoop Cluster
Big Data
Reference Architecture Modern Data Analytics Solutions
SQL
Search
Service
BI Tools
Enterprise Data
Warehouse
Search / Explore
File Import / SQL Import
Event
Hub
D
ata
Flow
D
ata
Flow
Change DataCapture Parallel
Processing
Storage
Storage
RawRefined
Results
SQL
Export
Microservice State
{ }
API
Stream
Processor
State
{ }
API
Event
Stream
Event
Stream
Search
Service
Stream Analytics
Microservices
Enterprise Apps
Logic
{ }
API
Edge Node
Rules
Event Hub
Storage
Bulk Source
Event Source
Location
DB
Extract
File
DB
IoT
Data
Mobile
Apps
Social
Event Stream
Telemetry

@gschmutz
Two Types of Stream Processing
(by Gartner)
Stream Data Integration
• focuses on the ingestion and processing of
data sources targeting real-time extract-
transform-load (ETL) and data integration
use cases
• filter and enrich the data
Stream Analytics
• targets analytics use cases
• calculating aggregates and detecting
patterns to generate higher-level, more
relevant summary information (complex
events)
• Complex events may signify threats or
opportunities that require a response from
the business
Gartner: Market Guide for Event Stream Processing, Nick Heudecker, W. Roy Schulte

@gschmutz
Stream Data Integration & Stream
Analytics Ecosystem

@gschmutz
”Data-in-Motion” Ecosystem
Stream Analytics
Event Hub
Open Source Closed Source
Stream Data Integration
Source: adapted from Tibco
Edge

@gschmutz
Apache Kafka – A Streaming Platform
Source
Connector
Sink
Connector
trucking_
driver
KSQL Engine
Kafka Streams
Kafka Broker

@gschmutz
Streaming Analytics: KSQL
STREAM and TABLE as first-class
citizens
• STREAM = data in motion
• TABLE = collected state of a stream
Stream Processing with zero coding
using SQL-like language
Built on top of Kafka Streams
Interactive (CLI) and headless (command
file)
ksql> CREATE STREAM order_s
WITH (kafka_topic=‘order',
value_format=‘AVRO');
Message
----------------
Stream created
ksql> SELECT * FROM order_s
WHERE address->country = ‘Switzerland’;
...
trucking_
driver
Kafka Broker
KSQL Engine
Kafka Streams
KSQL CLI Commands

@gschmutz
3 Blueprints for Streaming
Visualization

@gschmutz
Visualization: many many options! But do they support
Streaming Data?

@gschmutz
BP-1: Fast datastore with regular polling from consumer
Storage
Stream
Analytics
Event
Hub
Stream Data Integration & Stream Analytics
API
Data Store
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data In Motion Data at Rest
Data Flow

@gschmutz
BP-1: Elasticsearch / Kibana
Storage
Stream
Analytics
Event
Hub
API
Data Store
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data Flow
Alternatives:
Influx DB & Grafana
Prometheus & Grafana
Druid & Superset
…

@gschmutz
BP-1: Redis NoSQL & Custom Web
Storage
Stream
Analytics
Event
Hub
API
Data Store
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data Flow

@gschmutz
BP-1: Demo Redis NoSQL & Custom Web
https://opensky-network.org/

@gschmutz
BP-1: KafkaStreams Interactive Query & Custom Web App
Storage
Stream
Analytics
Event
Hub
API
Data Store
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data Flow

@gschmutz
BP-2: Direct Streaming to the Consumer
Stream
Analytics
Event
Hub
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data In Motion
Data Flow
Channel/
Protocol
API

@gschmutz
BP-2: Kafka Connect to Slack / WhatsApp / Twitter / …
Stream
Analytics
Event
Hub
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data In Motion
Data Flow
Channel/
Protocol
API

@gschmutz
BP-2: Demo Kafka Connect to Slack

@gschmutz
BP-2: Kafka to Tipboard (Dashboard Solution)
Stream
Analytics
Event
Hub
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data In Motion
Data Flow
Channel/
Protocol
API
Alternatives:
Dashing
Geckoboard
…

@gschmutz
BP-2: Demo Kafka to Tipboard (Dashboard Solution)
http://allegro.tech/tipboard/

@gschmutz
BP-2: Web Sockets / SSE & Custom Modern Web App
Stream
Analytics
Event
Hub
Streaming
Visualization
Data Flow
ConsumerData
Sources
Data In Motion
Data Flow
Channel/
Protocol
API
Sever Sent Event (SSE)

@gschmutz
BP-3: Streaming SQL Result to Consumer
Stream
Analytics
Event
Hub
Stream Data Integration & Stream Analytics ConsumerData
Sources
Data In Motion
Data Flow
API Streaming
Visualization

@gschmutz
BP-3: KSQL and Arcadia Data
Stream
Analytics
Event
Hub
Sources
Data In Motion
Data Flow
API Streaming
Visualization

@gschmutz
BP-3: Demo KSQL and Arcadia Data
https://www.arcadiadata.com/

@gschmutz
BP-3: KSQL with REST API to Custom Web App
Stream
Analytics
Event
Hub
Sources
Data In Motion
Data Flow
API Streaming
Visualization

@gschmutz
BP-3: Spark Streaming & Oracle Stream Analytics
Stream
Analytics
Event
Hub
Sources
Data In Motion
Data Flow
API Streaming
Visualization

@gschmutz
BP-3: Demo Spark Streaming & Oracle Stream Analytics
https://www.oracle.com/middleware/technologies/complex-event-processing.html

@gschmutz
Summary
BP-1: Fast Store &
Polling
• “classic” pattern
• Not end-to-end “data-in-
motion” -> “Data-at-rest”
before visualization
• Slight delay might not be
exceptable for
monitoring dashboard
• Can use full power of
data store
• In-memory option
• NoSQL databases
BP-2: Stream to
Consumer
• minimal latency
• More difficult on “client
side”
• good if stream holds
directly what should be
displayed
• More difficult if data in
stream needs to be
analyzed before
visualization
• No historical info
available
BP-3: Streaming SQL
• Minimal latency
• Power of SQL query
engine available for
visualization
• possibility for “self-
service” style
visualization
• Some analytics are more
difficult on streaming
data
• No historical info
available

@gschmutz
Technology on its own won't help you.
You need to know how to use it properly.

Streaming Visualization

More Related Content

What's hot

Similar to Streaming Visualization

More from Guido Schmutz

Recently uploaded

Streaming Visualization