Date: 13th November 2018 Location: Fast Data Theatre Time: 15:10 - 15:40 Speaker: Steve Wilkes Organisation: Striim About: Organisations realise that modernisation and adoption of new technologies are key to business success. Whether it is adopting hybrid cloud services for elastically scalable solutions, utilising machine learning to optimise business operations, or leveraging fast analytics for instant insight, they all rely on one thing: data. Modernisation requires moving away from batch-oriented architectures to streaming technologies, taking advantage of fast data, as well as integrating new and legacy resources.

1. © 2018 Striim, Inc. All rights reserved.
Steve Wilkes – Striim CTO/Co-Founder
Streaming Integration:
Powering Hybrid Cloud,
Machine Learning and Fast Analytics

2. © 2018 Striim, Inc. All rights reserved.
What Is Streaming Integration?
It’s all about
continuously
collecting and
moving any
enterprise data
in real-time,
while:
Handling extreme volumes of data
at scale with high throughput
Processing, analyzing, and
correlating data in flight
Making data valuable, verifiable,
and visible in real time

3. © 2018 Striim, Inc. All rights reserved.
Move and Process Data in Streams for Maximum Value
Collect, process, analyze,
visualize, and deliver
business events as they
happen, instantaneously
Run continuous SQL
queries on data-in-motion,
before it lands on disk

4. © 2018 Striim, Inc. All rights reserved.
Common Use Cases for Streaming Integration
For Cloud
For Real-Time
Applications
For Big Data Azure
HDInsight
AWS EMR
Cassandra
Streaming
Integration

5. © 2018 Striim, Inc. All rights reserved.
Customers Are Modernizing
Moving to a “streaming first” data architecture,
supporting cloud, streaming analytics, and IoT.
Bridge the old and new worlds of data.
Streaming Integration is the foundation for data
modernization initiatives.

6. © 2018 Striim, Inc. All rights reserved.
Data Arrives in Streams - Not Batches
database
humans
events
devices
logs
machines
… Streaming Integration
has emerged as a major
infrastructure requirement
streaming

7. © 2018 Striim, Inc. All rights reserved.
Incomplete +
Delayed
Customer
Insight
Missed
opportunities
for operational
efficiency
Increased Risk
of Security
Breach
Increased Risk
of Out of
Compliance
ETL tools
with basic CDC
Custom Code
Extract in
Batch
Land
Data
Transform &
Prepare
The Legacy Integration Problem
Data is Created Data Turns to
Intelligence
Verify &
Test
Reporting,Analytics,MachineLearning
Hours to Days Later

8. © 2018 Striim, Inc. All rights reserved.
Streaming Integration: Modern Data Movement
Data is Created Data Turns to
Operational
IntelligenceSub-seconds later
Continuous, Heterogenous Movement + Processing
Operational
ML
Real-Time
Analytics
Operational
Reporting
Timely
Customer
Interactions
Higher
Operational
Efficiency
Increased
Security
Improved
Compliance
and Risk
Management
Streaming
Integration

9. © 2018 Striim, Inc. All rights reserved.
Streaming Integration Capabilities
CONTINUOUS STREAMING
In-Memory SQL-Based Processing
ü Continuous Real-Time Collection
ü Filter, Transform, Aggregate, Enrich
ü Scalable, Guaranteed Delivery
ü Real-Time Analytics and Visualization
Monitor Pipeline Health
Databases
DWs
Kafka & Messaging
Microsoft Azure
Amazon AWS
Google Cloud Platform
Cloudera
Hortonworks
MapR
Cassandra
Real-Time
Ingestion
Real-Time Delivery
Files
Dashboard

10. © 2018 Striim, Inc. All rights reserved.
Data Movement Evolution
• Legacy technology
• Batch based
• Disk based transformations
• Used for advanced transformation
capabilities
• Limited to databases (CDC)
• 20 year old technology
• Lacks transformation capabilities
• Single point of failure (single node)
• Difficult to maintain
• Separate product for Cloud & Big
Data integration
• Evolution from Replication
• Includes real-time (CDC) support
• In-line denormalization and
transformations
• Data validation and visualization
• Advanced monitoring and alerting
• Distributed scalable architecture
• SQL-based processing
• Quick dev to production cycle
• Modern sources & targets
ETL Replication & CDC Streaming Integration
Evolution

11. © 2018 Striim, Inc. All rights reserved.
ON-PREMISE SOURCES
CLOUD SOURCES
PROCESSING
CONTINUOUS COLLECTION
Streaming Integration Supports Hybrid Cloud Initiatives

12. © 2018 Striim, Inc. All rights reserved.
Types of Hybrid Cloud Integrations
Migration
Cloud Bursting
Analytics

13. © 2018 Striim, Inc. All rights reserved.
Migration to the Cloud
Moving Apps Only Part of the Problem
Bigger Consideration is the Data
Target DB unlikely to be same
Cannot stop source app
Takes time to do initial load
Target DB needs current data
Verification can also take time ON-PREMISE CLOUD
? ? ?

14. © 2018 Striim, Inc. All rights reserved.
Key to this is Change Data Capture (CDC)
Update Delete
DML
Insert
Update
Delete
110100
001101
010011
Insert
Transaction
Logs
Streaming
Integration
CDC
Events

15. © 2018 Striim, Inc. All rights reserved.
Cloud Migration Steps With Streaming Integration
ON-PREMISE CLOUD
CDC
INITIAL LOAD
1.START CDC
2.DO INITIAL LOAD
3.APPLY CHANGE
4.CONTINUE TO
APPLY CHANGE
COLLECT APPLY

16. © 2018 Striim, Inc. All rights reserved.
Cloud Bursting With Streaming Integration
ON-PREMISE CLOUD
CDC
INITIAL LOAD
§SCALE ON DEMAND
§READ-ONLY DBS
§NEEDS LOAD + CDC
§EXTENDS TO
MULTI-CLOUD

17. © 2018 Striim, Inc. All rights reserved.
Cloud Bursting With Streaming Integration
ON-PREMISE CLOUD
CDC
INITIAL LOAD
§SCALE ON DEMAND
§READ-ONLY DBS
§NEEDS LOAD + CDC
§EXTENDS TO
MULTI-CLOUD

18. © 2018 Striim, Inc. All rights reserved.
Cloud Bursting With Streaming Integration
ON-PREMISE CLOUD
CDC
INITIAL LOAD
§SCALE ON DEMAND
§READ-ONLY DBS
§NEEDS LOAD + CDC
§EXTENDS TO
MULTI-CLOUD

19. © 2018 Striim, Inc. All rights reserved.
Cloud Analytics With Streaming Integration
ON-PREMISE CLOUD
ANALYTICS
CDC
INITIAL LOAD

20. © 2018 Striim, Inc. All rights reserved.
Cloud Analytics With Streaming Integration
ON-PREMISE CLOUD
ANALYTICS
CDC
INITIAL LOAD

21. © 2018 Striim, Inc. All rights reserved.
Cloud Analytics With Streaming Integration
ON-PREMISE CLOUD
ANALYTICS
CDC
INITIAL LOAD

22. © 2018 Striim, Inc. All rights reserved.
Real-Time Applications
Analytics
Alerts
Predictions

23. © 2018 Striim, Inc. All rights reserved.
Real-Time Applications Need Streaming Data Integration
Streaming
Integration • Increase customer
value with real-time
data
• Support for high-
volume, high-
velocity data
• Built-in validation,
HA, scalability
Real-Time
Analytics/Reports
Real-Time
Notifications
ContinuousIngestion
Continuous Streaming
In-Memory Pre-Processing:
Transform, Filter & Enrich
Pipeline Health
Monitoring
Scalable, Guaranteed Delivery
ContinuousDelivery
Databases
Log Files
Kafka/
Messaging
Sensors
Cloud
CDC

24. © 2018 Striim, Inc. All rights reserved.
Streaming Data to Apache Kudu
LOW LATENCY ANALYTICS
RAPIDLY CHANGING DATA
TRANSACTIONS & QUERIES
SINGULAR SCALABLE STORAGE

25. © 2018 Striim, Inc. All rights reserved.
Streaming Data to Apache Kudu
LOW LATENCY ANALYTICS
RAPIDLY CHANGING DATA
TRANSACTIONS & QUERIES
SINGULAR SCALABLE STORAGE
FOR FAST ANALYTICS
ON FAST DATA
YOU NEED FAST DATA

26. © 2018 Striim, Inc. All rights reserved.
Ingestion &
Pre-Processing
Post-Processing &
Delivery
Analytics & Visualization
Kafka Streaming Integration - Expanding Use Cases and Value
DEVELOPERS FOCUS ON
BUSINESS VALUE
Databases
Log Files
Kafka/
Messaging
Sensors
Cloud
Databases
Log Files
Cloud
Hadoop
& NoSQL

27. © 2018 Striim, Inc. All rights reserved.
Operational Machine Learning Streaming Architecture
ContinuousDataCollection ≈
Real-Time Dashboards & Alerts
Data Preparation
ContinuousDataStorage
Store long term raw real-time data
Prepare real-time data
Continuously Collect Data from All Sources
Databases, Logs, Sensors, Wire Data, etc.
Continuously in Real-Time
Store long term processed real-time data
Train ML from
long term data
Use Trained ML Model
to make real-time
decisions
ML/AI
Streaming
Store the results of
correlation, predictions, etc.
Enable real-time analytics
Trigger
re-training
ML model
Correlation & Anomaly Detection & Prediction

28. © 2018 Striim, Inc. All rights reserved.
Real-Time Big Data
HDFS / HBASE
HIVE / KUDU
CLOUDERA / MAPR
HORTONWORKS
MOVE TO CLOUD
AWS / AZURE / GCP

29. © 2018 Striim, Inc. All rights reserved.
Streaming Integration For Big Data
ContinuousIngestionCDC
Streaming
Integration
Continuous Streaming
In-Memory Pre-Processing:
Transform, Filter & Enrich
Pipeline Health
Monitoring
Scalable, Guaranteed Delivery ContinuousDelivery
Databases
Log Files
Kafka/
Messaging
Sensors
Cloud
HDInsight
EMR
DataProc

30. © 2018 Striim, Inc. All rights reserved.
Big Data Considerations
Streaming and non-streaming sources
Write frequency – real-time vs micro-batch
File formats (JSON / AVRO / ORC / PARQUET)
Pre-processing (especially enrichment)
Scalability, reliability and E1P

31. © 2018 Striim, Inc. All rights reserved.
Customer Cake Studies
REAL-WORLD STREAMING
INTEGRATION USE-CASES
OFTEN INCORPORATE
MANY OF THESE CONCEPTS

32. © 2018 Striim, Inc. All rights reserved.
Customer Case Studies
REAL-WORLD STREAMING
INTEGRATION USE-CASES
OFTEN INCORPORATE
MANY OF THESE CONCEPTS

33. © 2018 Striim, Inc. All rights reserved.
Customer Example: Banking
• North America financial institution
• Cloud-first strategy
• Striim runs on Azure
• Moves real-time transactions from
HPE NonStop to Azure Event Hubs
• Migrated Oracle reporting
database
• Feeds real-time data for financial
reporting, analytics, and
enterprise applications
Continuous, Real-Time Data Integration to Cloud
Azure Event
Hubs
Financial
Reporting and
Enterprise Apps
on Azure
HPE
NonStop
Oracle

34. © 2018 Striim, Inc. All rights reserved.
Customer Example: Communications
Real-Time Data Integration and Processing
Oracle and SQL Server
Customer Data
Including Orders
Message Bus and Cloud
Apps serving LOB
• Uses change data capture from
Oracle and SQL Server databases
• Performs stream processing for
Kafka including encryption for data
in motion
• Delivers continuous access to up-
to-date customer data across LOBs
• Enables real-time customer
authentication for CSRs.
• Keeps hybrid cloud systems in
sync.

35. © 2018 Striim, Inc. All rights reserved.
Customer Example: Finance
Many Log Sources
Multi-Log Correlation and Distribution
Security
Event Hub
Distributed Data Processing
Delivery & Analytics
Security
Event Hub
• Real-time security event hub
servicing all critical security
analytics efforts
• Ingests Syslogs, Website Logs,
CorreLog, Crowdstrike, Netflow
all in real time
• Enrichments with blacklisted IPs
• The right data, with the right
format for fast analytics
• Very large Kafka implementation

36. © 2018 Striim, Inc. All rights reserved.
Customer Example: Media
Real-Time Data Integration for Customer Analytics
Oracle
Database for
Siebel CRM
Customer
Behavior
Analytics
Salesforce.
• Needs real-time customer
behavior analytics
• Striim integrates real-time CRM
data for churn analytics
• Performs in-flight, rule-based
transformations
• Delivers to Kafka-based
messaging, Hive, and Kudu
• Different processing and
analytics for different targets

37. © 2018 Striim, Inc. All rights reserved.
About Striim

38. © 2018 Striim, Inc. All rights reserved.
Striim Executive Summary
Real-Time, Continuous Streaming Integration. On-Premises and Cloud.
Leadership Founded in 2012 by the former Leadership team from GoldenGate and BEA/WebLogic
Business Goal Modern Data Movement, at the Speed of Business
Solution Patented Streaming Data Integration Software Leverages In-Memory Computing
Customers Over 200 installs across Financial Services, Telco, Healthcare, Retail, Manufacturing

39. © 2018 Striim, Inc. All rights reserved.
Streaming Integration with Striim
IN-MEMORY DATA PIPELINE
VISUALIZATION & METRICS
DRAG & DROP UI
Delimited
JSON
XML
AVRO
Template
Oracle
MS SQL
MySQL
Teradata
PostgreSQL
MemSQL
JDBC/SQL
Files
MQTT
OPC-UA
Kafka
JMS
AMQP
HDFS
HBase
Hive
Cassandra
Kudu (preferred)
Impala
Microsoft (preferred)
Azure Blob/DL
Azure SQL DB
Azure SQL DW
Azure HDInsight
Amazon S3
Amazon Redshift
Amazon Kinesis
Google Big Query
Google Pub-Sub
Formats
CONTINUOUS DATA DELIVERY
DBs/DWs
Network
Messaging
Big Data
Cloud
Delimited
JSON
XML
Free Text
Binary
Name/Value
Zipped
AVRO
GoldenGate
Apache Log
Sys Log
MS Event Log
Mail Log
SNMP
CollectD
CEF
DHCP Log
WCF
+Others
Oracle CDC
MS SQL CDC
MySQL CDC
MongoDB CDC
HP NonStop CDC
Maria DB CDC
JDBC/SQL
Amazon S3
AWS RDS
Salesforce
Log Files
System Files
Batch Files
TCP
UDP
HTTP
MQTT
Netflow
PCAP
OPC-UA
Kafka
Flume
JMS
AMQP
HDFS
HBase
Hive
Parsers
CONTINUOUS DATA COLLECTION
DBs
Files
Network
Messaging
Big Data
Cloud

40. © 2018 Striim, Inc. All rights reserved.
Wizards Based-UI

41. © 2018 Striim, Inc. All rights reserved.
Integration and Analytics Through Data Flows

42. © 2018 Striim, Inc. All rights reserved.
Visualization Through Streaming Dashboards

43. © 2018 Striim, Inc. All rights reserved.
Striim Key Value Prepositions
Non-intrusively ingest
data from any relevant
source in real time for
operational intelligence
Transform data in motion
to a consumable format
without extensive coding
Monitor your data
pipelines continuously
without needing
another product
Deliver data to
where needed in
real time with
validation

44. © 2018 Striim, Inc. All rights reserved.
Thank You
@StriimTeam facebook.com/Striim
Resources
www.striim.com/resources/
Product Page
www.striim.com/product/
Download the Striim Platform
www.striim.com/download-striim/
Share