SlideShare a Scribd company logo
1 of 14
Eventador SQLStreamBuilder
A rich streaming SQL interface for creating and scaling stateful
stream processing jobs
June 2019
Eventador.io
Quick Background
● Streaming data systems like
Apache Kafka have become
massively popular for a variety of
workloads.
● Stream processing frameworks
like Apache Flink have been
widely adopted for performing
computations on these streams.
● But, there is no simple way to
author, iterate, execute, and
manage jobs against boundless
streams of data without writing
lots of Java or Scala code.
Using SQL makes
building stream
processing jobs
EASY and FAST.
Why SQL?
Structured Query Language (SQL) is a tried-and-true language for managing structured data, so it is a
mature and ubiquitous language with a massive user base.
With streaming SQL engines for Apache Kafka, Apache Flink and other stream processing technologies,
enable organizations without Java and Scala to benefit from rapid and simple streaming application
development.
SQLStreamBuilder Overview
Eventador SQLStreamBuilder (SSB) is
an interface to create and manage
stream processing jobs using SQL.
SSB provides:
● An Interactive Streaming SQL Console
● A SQL Parser
● Compatible with a wide variety of
sources and sinks
● Schema definition and management
● SQL job management
● A production grade engine
SQLStreamBuilder Feature: Console Interface
SSB includes a rich editing interface for authoring, validating and launching streaming SQL jobs and
includes a number of unique features.
Theming and
color coding
for SQL
syntax
SQL history
VIM Mode
SQL editor
with context
sensitive
menu
Apache Calcite-
standard SQL
syntax
SQL comment
support
SQLStreamBuilder Feature: Built-in SQL Parser
SSB delivers a built-in Apache Calcite
SQL parser to give immediate feedback
on syntax and validity.
SSB’s SQL parser:
● doesn’t launch the job if you’re still
iterating on your syntax,
● guides you with helpful feedback
● enables you to interactively author SQL
by iterating on the statement until it
produces the result you want
● enables quick trial and error of SQL and
speeds time to market for streaming
jobs.
SQLStreamBuilder Feature: Comprehensive Sources and
Sinks
SSB enables you to create a virtual
table for a source or sink, then specify
them in SQL statements (source) and in
the console itself (sink)—all in a
compact interface.
SSB supports:
● Kafka-based sources and sinks via
○ the Eventador platform,
○ Amazon MSK, or
○ your own endpoint
● The comprehensive Flink ecosystem of
sources and sinks
○ including ones from projects like
Apache Bahir
SQLStreamBuilder Feature: Schema Management
SSB simplifies schema management by:
● defining a schema for each source at
creation time
● exposing that schema using metadata
commands and SQL queries in the editor
interface.
SSB is currently compatible with JSON
data, and the schema is defined using
JSON-schema compatible syntax. Avro
support is coming soon.
SQLStreamBuilder Feature: Simple Apache Flink Job
Management
SSB can seamlessly scale from one to
hundreds of concurrent processes all
servicing your one SQL job.
With SSB:
● each statement becomes a job
● a collection of one or more jobs makes
up some business
need/process/feature-a streaming
topology
● managing jobs is done via the SQL
Jobs tab where jobs can be observed,
canceled, and edited.
● clusters can be expanded to serve as
many concurrent SQL jobs as required.
SQLStreamBuilder: Production-grade Stream Processing
SSB being production grade was a core design tenant. One unique capability of the SSB engine is that
users can iterate and perfect SQL, then when ready, simply run against production data sources.
Additionally, SSB is a robust platform that:
● runs in a highly scalable and fault-tolerant manner by default
● utilizes checkpoints to ensure job restart-ability,
● is a rich streaming SQL UI fully integrated with Apache Kafka such that restarting picks up from the proper consumer
offset, and the state store itself scales and is fault tolerant
● runs redundant Flink management components and a best-in-breed stack for Flink
SSB also comes with Eventador’s 24x7x365 hands-on support.
SQLStreamBuilder: Eventador Architecture
Getting Started with SQLStreamBuilder
To request access to SQLStreamBuilder, please visit: https://eventador.io/sql-streambuilder/
1. Create a SSB deployment:
● Click on SQLStreamBuilder on the side menu
● Set a unique name for the SQL Job Name. Each query is a job and requires a name. If you are iterating quickly on your
SQL statement, just use the default name.
● If you already have a deployment, you can select it and skip down to the next section.
● If you will be creating a new deployment, select Add Deployment from the Target Deployment dropdown box.
● In the Add Deployment box, choose a logical Name for this deployment.
● Give it a good Description.
● Select a Region/Zone.
● Click Create. Over the next few moments, The deployment will provision a cluster for running SSB jobs.
2. Create virtual tables for sources and sinks
3. Run SQL
It’s as simple as that. To see more detail on getting started, creating sources and sinks, etc. with SQLStreamBuilder, please
visit: https://docs.eventador.io/streambuilder/
SQLStreamBuilder Streaming SQL Examples
Metadata Commands Example SQL Statement for
Fraud Detection:
Aggregation
Timestamps, intervals
and time
Union ALL
Math
Misc SQL Tricks
Visit
https://eventador.io
For More Information
hello@eventador.io

More Related Content

What's hot

A Pluggable Autoscaling System @ UCC
A Pluggable Autoscaling System @ UCCA Pluggable Autoscaling System @ UCC
A Pluggable Autoscaling System @ UCCChris Bunch
 
Un'introduzione a Kafka Streams e KSQL... and why they matter!
Un'introduzione a Kafka Streams e KSQL... and why they matter!Un'introduzione a Kafka Streams e KSQL... and why they matter!
Un'introduzione a Kafka Streams e KSQL... and why they matter!Paolo Castagna
 
KSQL and Kafka Streams – When to Use Which, and When to Use Both
KSQL and Kafka Streams – When to Use Which, and When to Use BothKSQL and Kafka Streams – When to Use Which, and When to Use Both
KSQL and Kafka Streams – When to Use Which, and When to Use Bothconfluent
 
Tech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosTech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosVishanth Bala
 
AppScale + Neptune @ HPCDB
AppScale + Neptune @ HPCDBAppScale + Neptune @ HPCDB
AppScale + Neptune @ HPCDBChris Bunch
 
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent
 
Confluent building a real-time streaming platform using kafka streams and k...
Confluent   building a real-time streaming platform using kafka streams and k...Confluent   building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...Thomas Alex
 
AppScale @ LA.rb
AppScale @ LA.rbAppScale @ LA.rb
AppScale @ LA.rbChris Bunch
 
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...LINE Corporation
 
Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Chris Bunch
 
Kafka Connect by Datio
Kafka Connect by DatioKafka Connect by Datio
Kafka Connect by DatioDatio Big Data
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
 
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...HostedbyConfluent
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)Simba Khadder
 
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflowAutomatic Forecasting using Prophet, Databricks, Delta Lake and MLflow
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflowDatabricks
 
Streaming ETL with Apache Kafka and KSQL
Streaming ETL with Apache Kafka and KSQLStreaming ETL with Apache Kafka and KSQL
Streaming ETL with Apache Kafka and KSQLNick Dearden
 

What's hot (20)

A Pluggable Autoscaling System @ UCC
A Pluggable Autoscaling System @ UCCA Pluggable Autoscaling System @ UCC
A Pluggable Autoscaling System @ UCC
 
Un'introduzione a Kafka Streams e KSQL... and why they matter!
Un'introduzione a Kafka Streams e KSQL... and why they matter!Un'introduzione a Kafka Streams e KSQL... and why they matter!
Un'introduzione a Kafka Streams e KSQL... and why they matter!
 
Serverless Summit - Quiz
Serverless Summit - QuizServerless Summit - Quiz
Serverless Summit - Quiz
 
KSQL and Kafka Streams – When to Use Which, and When to Use Both
KSQL and Kafka Streams – When to Use Which, and When to Use BothKSQL and Kafka Streams – When to Use Which, and When to Use Both
KSQL and Kafka Streams – When to Use Which, and When to Use Both
 
Tech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosTech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache Stratos
 
AppScale + Neptune @ HPCDB
AppScale + Neptune @ HPCDBAppScale + Neptune @ HPCDB
AppScale + Neptune @ HPCDB
 
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
 
A Short Presentation on Kafka
A Short Presentation on KafkaA Short Presentation on Kafka
A Short Presentation on Kafka
 
Confluent building a real-time streaming platform using kafka streams and k...
Confluent   building a real-time streaming platform using kafka streams and k...Confluent   building a real-time streaming platform using kafka streams and k...
Confluent building a real-time streaming platform using kafka streams and k...
 
Neptune @ SoCal
Neptune @ SoCalNeptune @ SoCal
Neptune @ SoCal
 
AppScale @ LA.rb
AppScale @ LA.rbAppScale @ LA.rb
AppScale @ LA.rb
 
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...
Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...
 
KSQL Intro
KSQL IntroKSQL Intro
KSQL Intro
 
Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10Active Cloud DB at CloudComp '10
Active Cloud DB at CloudComp '10
 
Kafka Connect by Datio
Kafka Connect by DatioKafka Connect by Datio
Kafka Connect by Datio
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
 
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)
 
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflowAutomatic Forecasting using Prophet, Databricks, Delta Lake and MLflow
Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow
 
Streaming ETL with Apache Kafka and KSQL
Streaming ETL with Apache Kafka and KSQLStreaming ETL with Apache Kafka and KSQL
Streaming ETL with Apache Kafka and KSQL
 

Similar to Introduction to SQLStreamBuilder: Rich Streaming SQL Interface for Creating and Scaling Stateful Stream Processing Jobs

Integrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your EnvironmentIntegrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your Environmentconfluent
 
KSQL---Streaming SQL for Apache Kafka
KSQL---Streaming SQL for Apache KafkaKSQL---Streaming SQL for Apache Kafka
KSQL---Streaming SQL for Apache KafkaMatthias J. Sax
 
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...Kai Wähner
 
Writing an Interactive Interface for SQL on Flink
Writing an Interactive Interface for SQL on FlinkWriting an Interactive Interface for SQL on Flink
Writing an Interactive Interface for SQL on FlinkEventador
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Codemotion
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Codemotion
 
KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
KSQL Deep Dive - The Open Source Streaming Engine for Apache KafkaKSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
KSQL Deep Dive - The Open Source Streaming Engine for Apache KafkaKai Wähner
 
KSQL- Streaming Sql for Kafka
KSQL- Streaming Sql for KafkaKSQL- Streaming Sql for Kafka
KSQL- Streaming Sql for KafkaKnoldus Inc.
 
Interactive workflow management using Azkaban
Interactive workflow management using AzkabanInteractive workflow management using Azkaban
Interactive workflow management using Azkabandatamantra
 
Event Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBEvent Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBScyllaDB
 
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018Paolo Castagna
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Flink Forward
 
Introduction to Kafka Streams - Knolx.pdf
Introduction to Kafka Streams - Knolx.pdfIntroduction to Kafka Streams - Knolx.pdf
Introduction to Kafka Streams - Knolx.pdfKnoldus Inc.
 
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLRethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLKai Wähner
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using KafkaKnoldus Inc.
 
Data Pipeline for The Big Data/Data Science OKC
Data Pipeline for The Big Data/Data Science OKCData Pipeline for The Big Data/Data Science OKC
Data Pipeline for The Big Data/Data Science OKCMark Smith
 
Confluent Tech Talk Korea
Confluent Tech Talk KoreaConfluent Tech Talk Korea
Confluent Tech Talk Koreaconfluent
 

Similar to Introduction to SQLStreamBuilder: Rich Streaming SQL Interface for Creating and Scaling Stateful Stream Processing Jobs (20)

Integrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your EnvironmentIntegrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your Environment
 
KSQL---Streaming SQL for Apache Kafka
KSQL---Streaming SQL for Apache KafkaKSQL---Streaming SQL for Apache Kafka
KSQL---Streaming SQL for Apache Kafka
 
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...
Confluent Platform 5.5 + Apache Kafka 2.5 => New Features (JSON Schema, Proto...
 
Writing an Interactive Interface for SQL on Flink
Writing an Interactive Interface for SQL on FlinkWriting an Interactive Interface for SQL on Flink
Writing an Interactive Interface for SQL on Flink
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
 
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...
 
KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
KSQL Deep Dive - The Open Source Streaming Engine for Apache KafkaKSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka
 
KSQL- Streaming Sql for Kafka
KSQL- Streaming Sql for KafkaKSQL- Streaming Sql for Kafka
KSQL- Streaming Sql for Kafka
 
Interactive workflow management using Azkaban
Interactive workflow management using AzkabanInteractive workflow management using Azkaban
Interactive workflow management using Azkaban
 
Event Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBEvent Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDB
 
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018
Confluent and Elastic: a Lovely Couple - Elastic Stack in a Day 2018
 
Jug - ecosystem
Jug -  ecosystemJug -  ecosystem
Jug - ecosystem
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
 
Introduction to Kafka Streams - Knolx.pdf
Introduction to Kafka Streams - Knolx.pdfIntroduction to Kafka Streams - Knolx.pdf
Introduction to Kafka Streams - Knolx.pdf
 
Chti jug - 2018-06-26
Chti jug - 2018-06-26Chti jug - 2018-06-26
Chti jug - 2018-06-26
 
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLRethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Data Pipeline for The Big Data/Data Science OKC
Data Pipeline for The Big Data/Data Science OKCData Pipeline for The Big Data/Data Science OKC
Data Pipeline for The Big Data/Data Science OKC
 
Confluent Tech Talk Korea
Confluent Tech Talk KoreaConfluent Tech Talk Korea
Confluent Tech Talk Korea
 
SFScon18 - Stefano Pampaloni - The SQL revenge
SFScon18 - Stefano Pampaloni - The SQL revengeSFScon18 - Stefano Pampaloni - The SQL revenge
SFScon18 - Stefano Pampaloni - The SQL revenge
 

Recently uploaded

Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfjimielynbastida
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfngoud9212
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April  Automation LPDGAPIForce Zurich 5 April  Automation LPDG
APIForce Zurich 5 April Automation LPDGMarianaLemus7
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 

Recently uploaded (20)

Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdf
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdf
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
APIForce Zurich 5 April Automation LPDG
APIForce Zurich 5 April  Automation LPDGAPIForce Zurich 5 April  Automation LPDG
APIForce Zurich 5 April Automation LPDG
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 

Introduction to SQLStreamBuilder: Rich Streaming SQL Interface for Creating and Scaling Stateful Stream Processing Jobs

  • 1. Eventador SQLStreamBuilder A rich streaming SQL interface for creating and scaling stateful stream processing jobs June 2019 Eventador.io
  • 2. Quick Background ● Streaming data systems like Apache Kafka have become massively popular for a variety of workloads. ● Stream processing frameworks like Apache Flink have been widely adopted for performing computations on these streams. ● But, there is no simple way to author, iterate, execute, and manage jobs against boundless streams of data without writing lots of Java or Scala code. Using SQL makes building stream processing jobs EASY and FAST.
  • 3. Why SQL? Structured Query Language (SQL) is a tried-and-true language for managing structured data, so it is a mature and ubiquitous language with a massive user base. With streaming SQL engines for Apache Kafka, Apache Flink and other stream processing technologies, enable organizations without Java and Scala to benefit from rapid and simple streaming application development.
  • 4. SQLStreamBuilder Overview Eventador SQLStreamBuilder (SSB) is an interface to create and manage stream processing jobs using SQL. SSB provides: ● An Interactive Streaming SQL Console ● A SQL Parser ● Compatible with a wide variety of sources and sinks ● Schema definition and management ● SQL job management ● A production grade engine
  • 5. SQLStreamBuilder Feature: Console Interface SSB includes a rich editing interface for authoring, validating and launching streaming SQL jobs and includes a number of unique features. Theming and color coding for SQL syntax SQL history VIM Mode SQL editor with context sensitive menu Apache Calcite- standard SQL syntax SQL comment support
  • 6. SQLStreamBuilder Feature: Built-in SQL Parser SSB delivers a built-in Apache Calcite SQL parser to give immediate feedback on syntax and validity. SSB’s SQL parser: ● doesn’t launch the job if you’re still iterating on your syntax, ● guides you with helpful feedback ● enables you to interactively author SQL by iterating on the statement until it produces the result you want ● enables quick trial and error of SQL and speeds time to market for streaming jobs.
  • 7. SQLStreamBuilder Feature: Comprehensive Sources and Sinks SSB enables you to create a virtual table for a source or sink, then specify them in SQL statements (source) and in the console itself (sink)—all in a compact interface. SSB supports: ● Kafka-based sources and sinks via ○ the Eventador platform, ○ Amazon MSK, or ○ your own endpoint ● The comprehensive Flink ecosystem of sources and sinks ○ including ones from projects like Apache Bahir
  • 8. SQLStreamBuilder Feature: Schema Management SSB simplifies schema management by: ● defining a schema for each source at creation time ● exposing that schema using metadata commands and SQL queries in the editor interface. SSB is currently compatible with JSON data, and the schema is defined using JSON-schema compatible syntax. Avro support is coming soon.
  • 9. SQLStreamBuilder Feature: Simple Apache Flink Job Management SSB can seamlessly scale from one to hundreds of concurrent processes all servicing your one SQL job. With SSB: ● each statement becomes a job ● a collection of one or more jobs makes up some business need/process/feature-a streaming topology ● managing jobs is done via the SQL Jobs tab where jobs can be observed, canceled, and edited. ● clusters can be expanded to serve as many concurrent SQL jobs as required.
  • 10. SQLStreamBuilder: Production-grade Stream Processing SSB being production grade was a core design tenant. One unique capability of the SSB engine is that users can iterate and perfect SQL, then when ready, simply run against production data sources. Additionally, SSB is a robust platform that: ● runs in a highly scalable and fault-tolerant manner by default ● utilizes checkpoints to ensure job restart-ability, ● is a rich streaming SQL UI fully integrated with Apache Kafka such that restarting picks up from the proper consumer offset, and the state store itself scales and is fault tolerant ● runs redundant Flink management components and a best-in-breed stack for Flink SSB also comes with Eventador’s 24x7x365 hands-on support.
  • 12. Getting Started with SQLStreamBuilder To request access to SQLStreamBuilder, please visit: https://eventador.io/sql-streambuilder/ 1. Create a SSB deployment: ● Click on SQLStreamBuilder on the side menu ● Set a unique name for the SQL Job Name. Each query is a job and requires a name. If you are iterating quickly on your SQL statement, just use the default name. ● If you already have a deployment, you can select it and skip down to the next section. ● If you will be creating a new deployment, select Add Deployment from the Target Deployment dropdown box. ● In the Add Deployment box, choose a logical Name for this deployment. ● Give it a good Description. ● Select a Region/Zone. ● Click Create. Over the next few moments, The deployment will provision a cluster for running SSB jobs. 2. Create virtual tables for sources and sinks 3. Run SQL It’s as simple as that. To see more detail on getting started, creating sources and sinks, etc. with SQLStreamBuilder, please visit: https://docs.eventador.io/streambuilder/
  • 13. SQLStreamBuilder Streaming SQL Examples Metadata Commands Example SQL Statement for Fraud Detection: Aggregation Timestamps, intervals and time Union ALL Math Misc SQL Tricks