Speakers: Mathew Hawkins, Principal Solution Engineer, Kinetica + Chong Yan, Solutions Architect, Confluent The volume, complexity and unpredictability of streaming data is greater than ever before. Innovative organizations require instant insight from streaming data in order to make real-time business decisions. A new technology stack is emerging as traditional databases and data lakes are challenged to analyze streaming data and historical data together in real time. Confluent Platform, a more complete distribution of Apache Kafka®, works with Kinetica’s GPU-accelerated engine to transform data on the wire, instantly ingest data and analyze it at the same time. With the Kinetica Connector, end users can ingest streaming data from sensors, mobile apps, IoT devices and social media via Kafka into Kinetica’s database to combine it with data at rest. Together, the technologies deliver event-driven and real-time data to power the speed of thought analytics, improve customer experience, deliver targeted marketing offers and increase operational efficiencies. Register for this webinar to see: -How Kinetica enables businesses to leverage the streaming data delivered with Confluent Platform to gain actionable insights -How to leverage the Kafka Connect API to integrate data sources and destinations without writing cumbersome code -A KSQL demo showcasing an end-to-end flow of the complete data pipeline from a live source, to KSQL and finally into Kinetica Watch the recording: https://videos.confluent.io/watch/HdTKREFJzt6VcefneC4C7H?.

1. Mathew Hawkins, Principal Solutions Architect, Kinetica
Chong Yan, Solutions Architect, Confluent

2. Mathew Hawkins
Principal Solutions Architect
Kinetica
Chong Yan
Solutions Architect
Confluent

3. Explosion of
Data
• Structured
• Unstructured
• Smart devices
• Connected cars
• Sensors
Real-time
Demands
• Need for
action-driven
applications
• Demand for
analytics on latest
data not stale
data
Existing tech
doesn’t work
• Workloads are
I/O & compute
bound
• Too complex –
involves multiple
technologies
taped together
• Batch oriented

4. 100+ employees
worldwide
Exponential
growth
$50M Series A
Top Tier Investors

6. Location-based
Analysis, Rendering,
Discovery & Insights
Data-driven,
Streamlined Machine
Learning
BREAKTHROUGH
SPEED WITH +
Advanced Analytics on
Extreme Data:
Static & Streaming

7. OLAP
optimized
Native Geospatial
Datatypes & Functions
Distributed, Linear
Scale Out
User Defined
Functions (UDFs) &
Models
Native REST API Full Text
Search
SQL92
Visual
Dashboards
Ecosystem
Connectors

8. 1 NODE (1TB/2GPU)
PARALLEL
INGEST
1 NODE (1TB/2GPU)
1 NODE (1TB/2GPU)
Each node of the system can
share the task of data ingest,
provides more and faster
throughput. It can always be
made faster simply by adding
more nodes.
Leading In-memory DB NoSQL DB
150s
753s
1029s

9. ETL / STREAM
PROCESSING
SQL
Native
APIs
PARALLEL
INGEST
Geospatial
WMS
Custom
Connectors
BI DASHBOARDS
BI / GIS / APPS
CUSTOM APPS
& GEOSPATIAL
KINETICA ‘REVEAL’
STREAMINGDATA
UDF
s
ON DEMAND SCALE OUT +
Built-in Machine Learning
CUSTOM
LOGIC
BIDMach
ERP / CRM /
TRANSACTIONAL
CERTIFIED
CONNECTOR
CERTIFIED
CONNECTOR

10. MACHINE
LEARNING
MASSIVE
PARALLEL
COMPUTING
CUSTOM
APPLICATIONS
GEOSPATIAL
VISUALIZATION
STREAMING DATA
ANALYSIS
ADVANCED
ANALYTICS
BUSINESS USERS DATA SCIENTISTS /DEVELOPERS

11. LARGE US CABLE PROVIDER
BUSINESS OBJECTIVE
Real-time analysis of live viewership
across all broadcasted channels
particularly for live events ( Ex.
Super bowl, Olympics)
NEW CAPABILITIES DELIVERED
Ability to collect data streaming
from set-top boxes and analyze it in
real-time to track viewership by
senior executive team

12. BUSINESS OBJECTIVE
Be first to market with game
changing technologies that put
publishers’ needs first
NEW CAPABILITIES DELIVERED
High-speed ingest, store, and
persist data processing
capabilities
Ad-hoc analytics on ad impression
and bid data

13. 13Confidential
Confluent Platform
Technical Overview
Chong Yan, Solutions Architect, Confluent

14. “What is APACHE Kafka®
?

15. True Storage
Real-Time
Processing
Scalability
More than messaging

16. 16
Pub/Sub
Messaging
ETL
Connectors
Stream
Processing

17. “Why do we need Kafka?

20. to
streams of data
Similar to a message
queue or enterprise
messaging system
110101
010111
001101
100010
streams
of data
In a fault tolerant way110101
010111
001101
100010
streams
of data
In real time,
as they occur
110101
010111
001101
100010

21. •
•
•

23. Kafka
● distributed commit log
●
●
●
●
Reads are a single seek and scan
Writes are
append only

24. Kafka Streams APIKafka Connect API
Orders
Table
Customers
Kafka Streams API

25. Connectors and ClientsKSQL
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

26. REST ProxySchema Registry
{
"type": "record",
"name": "LOGON",
"namespace": "ORCL.SOE2",
"fields": [
{
"name": "table",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "op_type",
"type": [
"null",
"string"
],
"default": null
},

27. Docker images, deb/yum installersConfluent CLI

28. Multi-Datacenter ReplicationControl Center

29. Auto Data BalancerJMS Client
Enhanced Security

31. ✓
✓
✓
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
CREATE STREAM vip_actions AS
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u
ON c.userid = u.userid
WHERE u.level = 'Platinum';
KSQL is the simplest way to process streams of data in real-time
✓
✓
https://github.com/confluentinc/ksql

32. CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

33. CREATE TABLE error_counts AS
SELECT error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1 MINUTE)
WHERE type = 'ERROR'
GROUP BY error_code;
CREATE STREAM vip_actions AS
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
1 2 3

34. •
•
•
> bin/ksql-cli local
•
> bin/ksql-cli local –-properties-file foo/bar/ksql.properties

35. •
• > bin/ksql-server-start
•
• >bin/ksql-cli remote http://myserver:8090
•
•
•

36. •
•
> bin/ksql-node query-file=foo/bar.sql
•
•
•
•
•

37. Live data from
our San
Francisco
Firewall
Machine Learning
Reveal Dashboard for Analyst
SQL Developer to produce
INSIGHTS
Real time joins on
streaming data
INSIGHTS
Kafka
Netflow
Blacklist
KSQL Threats

38. SQL DEVELOPER
FRONT-END
USER
DATA SCIENTIST

39. Fast
moving data
Bringing together all types of
analysts on real time data
Manipulate real
time data to trigger
events
INSIGHTS
Kafka
Netflow
Blacklist
KSQL Threats

40. Kinetica Partner Page
https://www.confluent.io/connector/
kinetica-db-connector/
Website
www.confluent.io
Twitter
@ConfluentInc
Confluent Partner Page
https://www.kinetica.com/partner/
confluent/
Website
www.kinetica.com
Twitter
@KineticaHQ