The document discusses the integration of Kafka with MemSQL, highlighting their combined capabilities for operational analytics and machine learning at scale. It details MemSQL as a cloud-native database designed for speed and performance, while Kafka serves as a real-time data hub for processing large volumes of data. The introduction of the MemSQL Kafka connector further enhances data streaming and management between the two platforms.

1. —
How Kafka and MemSQL Became
the Dynamic Duo
Sarung Tripathi

2. 2 CONFIDENTIAL

3. 3 CONFIDENTIAL
—
Traditional Analytics

4. 4 CONFIDENTIAL
—
Operational Analytics & ML at Scale

5. 5
Scalable relational database for
transactions and analytics

6. 6
MemSQL: The Database of Now
The cloud-native operational database built for speed and scale
SPEED SCALE
SQL
Accelerate time to insight
with ultra fast ingest and
high performance query
Get the familiarity & ease of integration
of a traditional RDBMS and SQL, but with
a groundbreaking, modern architecture
Build on a cloud-native data
platform designed for today’s
most demanding applications
and analytical systems

7. 7 CONFIDENTIAL
—
Kafka: The Hub for Real-Time, Operational Data
Real-Time
Millions of records per second
Consistent
Exactly-once writes
Distributed, Fault Tolerant
Parallel ingest
Developer-Friendly
Publish-and-subscribe model
The Demands of Operational Analytics...

8. 8 CONFIDENTIAL
—
Operational
Analytics &
ML at Scale
Highly Performant
Ingest & Queries
Converged
Data Platform
SingleStore

9. 9 CONFIDENTIAL
—
MemSQL Pipelines Sequence
ARCHITECTURE
MemSQL pulls for
changes from a
data source system.
KAFKA PIPELINES MEMSQL
MemSQL pulls the data into its
memory space (no commit)
where a transform can be
applied.
The data is committed
in a transaction (and in
parallel)

10. 10 CONFIDENTIAL
MemSQL Pipelines
10MUPSERTS PER SECOND
WITH KAFKA + MEMSQL
CREATE OR REPLACE PIPELINE trade_data
AS LOAD DATA KAFKA 'hostname:9092/trades'
WITH TRANSFORM (‘score_data.py','','') #optional
INTO TABLE live_predictions
FIELDS TERMINATED BY ',';
—

11. 11 CONFIDENTIAL
—
MemSQL Pipelines
ARCHITECTURE

12. 12 CONFIDENTIAL
—
MemSQL Pipelines
TRANSFORMS
● Build transforms using any language!
● Transforms are an optional user-defined
program that receives data from a pipeline’s
extractor and outputs modified data (JSON,
Avro, CSV)
○ Examples: Data modification,
aggregation, feature engineering,
model execution, and more!
● Linux distribution must have the required
dependencies to execute the transform
● Data streamed to the transform is byte-
length encoded
Stream Transform Load

13. 13 CONFIDENTIAL
—
Introducing: The MemSQL Kafka Connector
● MemSQL Connector on the Confluent Hub
● Integration with Confluent Kafka Connect to stream
data into MemSQL
● Management and deployment capabilities of Confluent
make this incredibly easy to get started
● Cloud-first: Kafka Connector sits Kafka-side,
eliminating many potential security constraints
● Initial release: Kafka Connect Sink Connector
5X
THAN JDBC CONNECTOR
FASTER

14. 14 CONFIDENTIAL

15. 15 CONFIDENTIAL
—
SELECT ... INTO KAFKA ...
● Allows users to leverage MemSQL as a true operational data hub with
downstream decisioning
● “SELECT … INTO KAFKA …” runs a SELECT query, constructs Kafka message for
each row in the result set, and publishes the messages to a Kafka topic
● Includes every column value in the result set’s row and separates the column
values by a delimiter
● Configure security credentials within the statement easily

16. —
16
Thank You
Get Started with
Kafka Connector
Install it today from the Confluent Hub!