SlideShare a Scribd company logo
Streaming SQL
Julian Hyde
Hadoop Summit
San Jose, 2016/06/29
@julianhyde
SQL
Query planning
Query federation
OLAP
Streaming
Hadoop
Apache member
VP Apache Calcite
PMC Apache Arrow, Drill, Kylin
Thanks:
● Milinda Pathirage & Yi Pan (Apache Samza)
● Haohui Mai (Apache Storm)
● Fabian Hueske & Stephan Ewen (Apache Flink)
Data center
Streaming data sources
Sources:
● Devices / sensors
● Web servers
● (Micro-)services
● Databases (CDC)
● Synthetic streams
● Logging / tracing
Transports:
● Kafka
● Nifi
IoT
Devices
Services DatabaseWeb
server
How much is your data worth?
Recent data is more valuable
➢ ...if you act on it in time
Data moves from expensive
memory to cheaper disk as it cools
Old + new data is more valuable
still
➢ ...if we have a means to
combine them Time
Value of
data
($/GB)
Now1 hour
ago
1 day
ago
1 week
ago
1 year
ago
Hot data
Read often
Likely to be modified
High value
In memory
Cold data
Read rarely
Unlikely to be modified
Low value
On disk
Why query streams?
Stream - Database Duality:
● “Your database is just a cache of my stream”
● “Your stream is just change-capture of my database”
“Data is the new oil”
● Treating events/messages as data allows you to extract and refine them
Declarative approach to streaming applications
Why SQL? ● API to your database
● Ask for what you want,
system decides how to get it
● Query planner (optimizer)
converts logical queries to
physical plans
● Mathematically sound
language (relational algebra)
● For all data, not just data in a
database
● Opportunity for novel data
organizations & algorithms
● Standard
https://www.flickr.com/photos/pere/523019984/ (CC BY-NC-SA 2.0)
➢ API to your database
➢ Ask for what you want,
system decides how to get it
➢ Query planner (optimizer)
converts logical queries to
physical plans
➢ Mathematically sound
language (relational algebra)
➢ For all data, not just “flat”
data in a database
➢ Opportunity for novel data
organizations & algorithms
➢ Standard
Why SQL?
Data workloads
● Batch
● Transaction processing
● Single-record lookup
● Search
● Interactive / OLAP
● Exploration / profiling
● Continuous execution generating alerts (CEP)
● Continuous load
A variety of workloads, requiring specialized engines, but to the user it’s all “just
data”.
Building a streaming SQL standard via
consensus
Please! No more “SQL-like” languages!
Key technologies are open source (many are Apache projects)
Calcite is providing leadership: developing example queries, TCK
(Optional) Use Calcite’s framework to build a streaming SQL parser/planner for
your engine
Several projects are working with us: Samza, Storm, Flink. (Also non-streaming
SQL in Cassandra, Drill, Druid, Elasticsearch, Flink, Hive, Kylin, Phoenix.)
Simple queries
select *
from Products
where unitPrice < 20
select stream *
from Orders
where units > 1000
➢ Traditional (non-streaming)
➢ Products is a table
➢ Retrieves records from -∞ to now
➢ Streaming
➢ Orders is a stream
➢ Retrieves records from now to +∞
➢ Query never terminates
Stream-table duality
select *
from Orders
where units > 1000
➢ Yes, you can use a stream as
a table
➢ And you can use a table as a
stream
➢ Actually, Orders is both
➢ Use the stream keyword
➢ Where to actually find the
data? That’s up to the system
select stream *
from Orders
where units > 1000
Combining past and future
select stream *
from Orders as o
where units > (
select avg(units)
from Orders as h
where h.productId = o.productId
and h.rowtime > o.rowtime - interval ‘1’ year)
➢ Orders is used as both stream and table
➢ System determines where to find the records
➢ Query is invalid if records are not available
Semantics of streaming queries
The replay principle:
A streaming query produces the same result as the corresponding non-
streaming query would if given the same data in a table.
Output must not rely on implicit information (arrival order, arrival time,
processing time, or watermarks/punctuations)
(Some triggering schemes allow records to be emitted early and re-stated if
incorrect.)
Making progress
It’s not enough to get the right result. We
need to give the right result at the right
time.
Ways to make progress without
compromising safety:
➢ Monotonic columns (e.g. rowtime)
and expressions (e.g. floor
(rowtime to hour))
➢ Punctuations (aka watermarks)
➢ Or a combination of both
select stream productId,
count(*) as c
from Orders
group by productId;
ERROR: Streaming aggregation
requires at least one
monotonic expression in
GROUP BY clause
8
75
4
10:00 10:15 10:30 11:00 11:15
Arrival
time
1
2
3 5
6
Event
time 8
10:00 10:15 10:30 11:00 11:15
Arrival
time
1
2
3
6
Event
time
4 Drop out-of-sequence
records
Emit 10:00-11:00 window
when first record after 11:
00 arrives
W 11:00
Emit 10:00-11:00
window when 11:
00 watermark
arrives
W 11:00’
7
New
watermark.
Re-state 10:
00-11:00
window
Policies for emitting results
Monotonic column Watermark
Aggregation and windows
on streams
GROUP BY aggregates multiple rows into sub-
totals
➢ In regular GROUP BY each row contributes
to exactly one sub-total
➢ In multi-GROUP BY (e.g. HOP, GROUPING
SETS) a row can contribute to more than
one sub-total
Window functions (OVER) leave the number of
rows unchanged, but compute extra
expressions for each row (based on
Multi
GROUP BY
Window
functions
GROUP BY
GROUP BY select stream productId,
floor(rowtime to hour) as rowtime,
sum(units) as u,
count(*) as c
from Orders
group by productId,
floor(rowtime to hour)
rowtime productId units
09:12 100 5
09:25 130 10
09:59 100 3
10:00 100 19
11:05 130 20
rowtime productId u c
09:00 100 8 2
09:00 130 10 1
10:00 100 19 1
not emitted yet; waiting
for a row >= 12:00
Window types
Tumbling
window
“Every T seconds, emit the total for T seconds”
Hopping
window
“Every T seconds, emit the total for T2 seconds”
Session
window
“Emit groups of records that are separated by gaps of no
more than T seconds”
Sliding
window
“Every record, emit the total for the surrounding T
seconds”
“Every record, emit the total for the surrounding R records”
Tumbling, hopping & session windows in SQL
Tumbling window
Hopping window
Session window
select stream … from Orders
group by floor(rowtime to hour)
select stream … from Orders
group by tumble(rowtime, interval ‘1’ hour)
select stream … from Orders
group by hop(rowtime, interval ‘1’ hour,
interval ‘2’ hour)
select stream … from Orders
group by session(rowtime, interval ‘1’ hour)
Sliding windows in SQL
select stream
sum(units) over w (partition by productId) as units1hp,
sum(units) over w as units1h,
rowtime, productId, units
from Orders
window w as (order by rowtime range interval ‘1’ hour preceding)
rowtime productId units
09:12 100 5
09:25 130 10
09:59 100 3
10:17 100 10
units1hp units1h rowtime productId units
5 5 09:12 100 5
10 15 09:25 130 10
8 18 09:59 100 3
23 13 10:17 100 10
The “pie chart” problem
➢ Task: Write a web page summarizing
orders over the last hour
➢ Problem: The Orders stream only
contains the current few records
➢ Solution: Materialize short-term history
Orders over the last hour
Beer
48%
Cheese
30%
Wine
22%
select productId, count(*)
from Orders
where rowtime > current_timestamp - interval ‘1’ hour
group by productId
Join stream to a table
Inputs are the Orders stream and the
Products table, output is a stream.
Acts as a “lookup”.
Execute by caching the table in a hash-
map (if table is not too large) and
stream order will be preserved.
select stream *
from Orders as o
join Products as p
on o.productId = p.productId
Join stream to a changing table
Execution is more difficult if the
Products table is being changed
while the query executes.
To do things properly (e.g. to get the
same results when we re-play the
data), we’d need temporal database
semantics.
(Sometimes doing things properly is
too expensive.)
select stream *
from Orders as o
join Products as p
on o.productId = p.productId
and o.rowtime
between p.startEffectiveDate
and p.endEffectiveDate
Join stream to a stream
We can join streams if the join
condition forces them into “lock
step”, within a window (in this case,
1 hour).
Which stream to put input a hash
table? It depends on relative rates,
outer joins, and how we’d like the
output sorted.
select stream *
from Orders as o
join Shipments as s
on o.productId = p.productId
and s.rowtime
between o.rowtime
and o.rowtime + interval ‘1’ hour
Planning queries
MySQL
Splunk
join
Key: productId
group
Key: productName
Agg: count
filter
Condition:
action = 'purchase'
sort
Key: c desc
scan
scan
Table: products
select p.productName, count(*) as c
from splunk.splunk as s
join mysql.products as p
on s.productId = p.productId
where s.action = 'purchase'
group by p.productName
order by c desc
Table: splunk
Optimized query
MySQL
Splunk
join
Key: productId
group
Key: productName
Agg: count
filter
Condition:
action = 'purchase'
sort
Key: c desc
scan
scan
Table: splunk
Table: products
select p.productName, count(*) as c
from splunk.splunk as s
join mysql.products as p
on s.productId = p.productId
where s.action = 'purchase'
group by p.productName
order by c desc
Apache Calcite
Apache top-level project since October, 2015
Query planning framework
➢ Relational algebra, rewrite rules
➢ Cost model & statistics
➢ Federation via adapters
➢ Extensible
Packaging
➢ Library
➢ Optional SQL parser, JDBC server
➢ Community-authored rules, adapters
Embedded Adapters Streaming
Apache Drill
Apache Hive
Apache Kylin
Apache Phoenix*
Cascading
Lingual
Apache
Cassandra
Apache Spark
CSV
Druid*
Elasticsearch*
In-memory
JDBC
JSON
MongoDB
Splunk
Web tables
Apache Flink*
Apache Samza
Apache Storm
* Under development
Architecture
Conventional database Calcite
Relational algebra (plus streaming)
Core operators:
➢ Scan
➢ Filter
➢ Project
➢ Join
➢ Sort
➢ Aggregate
➢ Union
➢ Values
Streaming operators:
➢ Delta (converts relation to
stream)
➢ Chi (converts stream to
relation)
In SQL, the STREAM keyword
signifies Delta
Streaming algebra
➢ Filter
➢ Route
➢ Partition
➢ Round-robin
➢ Queue
➢ Aggregate
➢ Merge
➢ Store
➢ Replay
➢ Sort
➢ Lookup
Optimizing streaming queries
The usual relational transformations still apply: push filters and projects towards
sources, eliminate empty inputs, etc.
The transformations for delta are mostly simple:
➢ Delta(Filter(r, predicate)) → Filter(Delta(r), predicate)
➢ Delta(Project(r, e0, ...)) → Project(Delta(r), e0, …)
➢ Delta(Union(r0, r1), ALL) → Union(Delta(r0), Delta(r1))
But not always:
➢ Delta(Join(r0, r1, predicate)) → Union(Join(r0, Delta(r1)), Join(Delta(r0), r1)
➢ Delta(Scan(aTable)) → Empty
Sort
Sorting a streaming query is
valid as long as the system can
make progress.
Need a monotonic or
watermark-enabled expression
in the ORDER BY clause.
select stream productId,
floor(rowtime to hour) as rowtime,
sum(units) as u,
count(*) as c
from Orders
group by productId,
floor(rowtime to hour)
order by rowtime, c desc
Union
As in a typical database, we rewrite x union y
to select distinct * from (x union all y)
We can implement x union all y by simply combining the inputs in arrival
order but output is no longer monotonic. Monotonicity is too useful to squander!
To preserve monotonicity, we merge on the sort key (e.g. rowtime).
DML
➢ View & standing INSERT give same
results
➢ Useful for chained transforms
➢ But internals are different
insert into LargeOrders
select stream * from Orders
where units > 1000
create view LargeOrders as
select stream * from Orders
where units > 1000
upsert into OrdersSummary
select stream productId,
count(*) over lastHour as c
from Orders
window lastHour as (
partition by productId
order by rowtime
range interval ‘1’ hour preceding)
Use DML to maintain a “window”
(materialized stream history).
Summary: Streaming SQL features
Standard SQL over streams and relations
Streaming queries on relations, and relational queries on streams
Joins between stream-stream and stream-relation
Queries are valid if the system can get the data, with a reasonable latency
➢ Monotonic columns and punctuation are ways to achieve this
Views, materialized views and standing queries
Summary: The benefits of streaming SQL
Relational algebra covers needs of data-in-flight and data-at-rest applications
High-level language lets the system optimize quality of service (QoS) and data
location
Give DB tools and traditional users to access streaming data;
give message-oriented tools access to historic data
Combine real-time and historic data, and produce actionable results
Discussion continues at Apache Calcite, with contributions from Samza, Flink,
Storm and others. Please join in!
Thank you!
@julianhyde
@ApacheCalcite
calcite.apache.org
calcite.apache.org/docs/stream.html
Next talk (with @maryannxue) tomorrow at 12:20pm: “How We Re-
Engineered Phoenix with a Cost-Based Optimizer Based on Calcite”

More Related Content

What's hot

Streaming SQL with Apache Calcite
Streaming SQL with Apache CalciteStreaming SQL with Apache Calcite
Streaming SQL with Apache Calcite
Julian Hyde
 
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Julian Hyde
 
Why you care about
 relational algebra (even though you didn’t know it)
Why you care about
 relational algebra (even though you didn’t know it)Why you care about
 relational algebra (even though you didn’t know it)
Why you care about
 relational algebra (even though you didn’t know it)
Julian Hyde
 
Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!
Julian Hyde
 
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache CalciteCost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde
 
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
Julian Hyde
 
Apache Calcite Tutorial - BOSS 21
Apache Calcite Tutorial - BOSS 21Apache Calcite Tutorial - BOSS 21
Apache Calcite Tutorial - BOSS 21
Stamatis Zampetakis
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
DataWorks Summit/Hadoop Summit
 
Spatial query on vanilla databases
Spatial query on vanilla databasesSpatial query on vanilla databases
Spatial query on vanilla databases
Julian Hyde
 
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
Julian Hyde
 
SQL on Big Data using Optiq
SQL on Big Data using OptiqSQL on Big Data using Optiq
SQL on Big Data using Optiq
Julian Hyde
 
Don't optimize my queries, organize my data!
Don't optimize my queries, organize my data!Don't optimize my queries, organize my data!
Don't optimize my queries, organize my data!
Julian Hyde
 
Data Profiling in Apache Calcite
Data Profiling in Apache CalciteData Profiling in Apache Calcite
Data Profiling in Apache Calcite
Julian Hyde
 
Cost-based Query Optimization
Cost-based Query Optimization Cost-based Query Optimization
Cost-based Query Optimization
DataWorks Summit/Hadoop Summit
 
Drill / SQL / Optiq
Drill / SQL / OptiqDrill / SQL / Optiq
Drill / SQL / Optiq
Julian Hyde
 
Bucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
Bucketing 2.0: Improve Spark SQL Performance by Removing ShuffleBucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
Bucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
Databricks
 
Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14
Julian Hyde
 
Redshift performance tuning
Redshift performance tuningRedshift performance tuning
Redshift performance tuning
Carlos del Cacho
 
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
Amazon Web Services
 

What's hot (20)

Streaming SQL with Apache Calcite
Streaming SQL with Apache CalciteStreaming SQL with Apache Calcite
Streaming SQL with Apache Calcite
 
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
 
Why you care about
 relational algebra (even though you didn’t know it)
Why you care about
 relational algebra (even though you didn’t know it)Why you care about
 relational algebra (even though you didn’t know it)
Why you care about
 relational algebra (even though you didn’t know it)
 
Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!Don’t optimize my queries, optimize my data!
Don’t optimize my queries, optimize my data!
 
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache CalciteCost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
 
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
 
Tactical data engineering
Tactical data engineeringTactical data engineering
Tactical data engineering
 
Apache Calcite Tutorial - BOSS 21
Apache Calcite Tutorial - BOSS 21Apache Calcite Tutorial - BOSS 21
Apache Calcite Tutorial - BOSS 21
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
 
Spatial query on vanilla databases
Spatial query on vanilla databasesSpatial query on vanilla databases
Spatial query on vanilla databases
 
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...
 
SQL on Big Data using Optiq
SQL on Big Data using OptiqSQL on Big Data using Optiq
SQL on Big Data using Optiq
 
Don't optimize my queries, organize my data!
Don't optimize my queries, organize my data!Don't optimize my queries, organize my data!
Don't optimize my queries, organize my data!
 
Data Profiling in Apache Calcite
Data Profiling in Apache CalciteData Profiling in Apache Calcite
Data Profiling in Apache Calcite
 
Cost-based Query Optimization
Cost-based Query Optimization Cost-based Query Optimization
Cost-based Query Optimization
 
Drill / SQL / Optiq
Drill / SQL / OptiqDrill / SQL / Optiq
Drill / SQL / Optiq
 
Bucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
Bucketing 2.0: Improve Spark SQL Performance by Removing ShuffleBucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
Bucketing 2.0: Improve Spark SQL Performance by Removing Shuffle
 
Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14Cost-based query optimization in Apache Hive 0.14
Cost-based query optimization in Apache Hive 0.14
 
Redshift performance tuning
Redshift performance tuningRedshift performance tuning
Redshift performance tuning
 
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
Redshift at Lightspeed: How to continuously optimize and modify Redshift sche...
 

Viewers also liked

Apache Calcite: One planner fits all
Apache Calcite: One planner fits allApache Calcite: One planner fits all
Apache Calcite: One planner fits all
Julian Hyde
 
Apache Calcite overview
Apache Calcite overviewApache Calcite overview
Apache Calcite overview
Julian Hyde
 
Streaming SQL
Streaming SQLStreaming SQL
Streaming SQL
Julian Hyde
 
Discardable In-Memory Materialized Queries With Hadoop
Discardable In-Memory Materialized Queries With HadoopDiscardable In-Memory Materialized Queries With Hadoop
Discardable In-Memory Materialized Queries With Hadoop
Julian Hyde
 
The twins that everyone loved too much
The twins that everyone loved too muchThe twins that everyone loved too much
The twins that everyone loved too much
Julian Hyde
 
What's new in Mondrian 4?
What's new in Mondrian 4?What's new in Mondrian 4?
What's new in Mondrian 4?
Julian Hyde
 
Optiq: A dynamic data management framework
Optiq: A dynamic data management frameworkOptiq: A dynamic data management framework
Optiq: A dynamic data management framework
Julian Hyde
 
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Christian Tzolov
 
How BigQuery broke my heart
How BigQuery broke my heartHow BigQuery broke my heart
How BigQuery broke my heart
Gabriel Hamilton
 
Rust (Ginzarb 20161114)
Rust (Ginzarb 20161114)Rust (Ginzarb 20161114)
Rust (Ginzarb 20161114)
Kevin Toyoda
 
データベース12 - トランザクションと同時実行制御
データベース12 - トランザクションと同時実行制御データベース12 - トランザクションと同時実行制御
データベース12 - トランザクションと同時実行制御
Kenta Oku
 
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと  ~teratailのDevRel担当、ゼロからの奮闘記~ @a...【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと  ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
Yusaku Kinoshita
 
Cache obliviousの話
Cache obliviousの話Cache obliviousの話
Cache obliviousの話
Kumazaki Hiroki
 
Solrを使ったレシピ検索のプロトタイピング
Solrを使ったレシピ検索のプロトタイピングSolrを使ったレシピ検索のプロトタイピング
Solrを使ったレシピ検索のプロトタイピング
genta kaneyama
 
高速な倍精度指数関数expの実装
高速な倍精度指数関数expの実装高速な倍精度指数関数expの実装
高速な倍精度指数関数expの実装
MITSUNARI Shigeo
 
Prometheus meets Consul -- Consul Casual Talks
Prometheus meets Consul -- Consul Casual TalksPrometheus meets Consul -- Consul Casual Talks
Prometheus meets Consul -- Consul Casual Talks
Satoshi Suzuki
 
Windows Server+Photon Server環境でも Fluentd+Elasticsearch+Kibanaを活用して 各種情報を可視化する...
Windows Server+Photon Server環境でもFluentd+Elasticsearch+Kibanaを活用して各種情報を可視化する...Windows Server+Photon Server環境でもFluentd+Elasticsearch+Kibanaを活用して各種情報を可視化する...
Windows Server+Photon Server環境でも Fluentd+Elasticsearch+Kibanaを活用して 各種情報を可視化する...
GMO GlobalSign Holdings K.K.
 
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポートJAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
真吾 吉田
 

Viewers also liked (18)

Apache Calcite: One planner fits all
Apache Calcite: One planner fits allApache Calcite: One planner fits all
Apache Calcite: One planner fits all
 
Apache Calcite overview
Apache Calcite overviewApache Calcite overview
Apache Calcite overview
 
Streaming SQL
Streaming SQLStreaming SQL
Streaming SQL
 
Discardable In-Memory Materialized Queries With Hadoop
Discardable In-Memory Materialized Queries With HadoopDiscardable In-Memory Materialized Queries With Hadoop
Discardable In-Memory Materialized Queries With Hadoop
 
The twins that everyone loved too much
The twins that everyone loved too muchThe twins that everyone loved too much
The twins that everyone loved too much
 
What's new in Mondrian 4?
What's new in Mondrian 4?What's new in Mondrian 4?
What's new in Mondrian 4?
 
Optiq: A dynamic data management framework
Optiq: A dynamic data management frameworkOptiq: A dynamic data management framework
Optiq: A dynamic data management framework
 
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
 
How BigQuery broke my heart
How BigQuery broke my heartHow BigQuery broke my heart
How BigQuery broke my heart
 
Rust (Ginzarb 20161114)
Rust (Ginzarb 20161114)Rust (Ginzarb 20161114)
Rust (Ginzarb 20161114)
 
データベース12 - トランザクションと同時実行制御
データベース12 - トランザクションと同時実行制御データベース12 - トランザクションと同時実行制御
データベース12 - トランザクションと同時実行制御
 
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと  ~teratailのDevRel担当、ゼロからの奮闘記~ @a...【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと  ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
【非エンジニアが 「明日からDevRelやってよ」って 突然言われて1年半やってきたこと ~teratailのDevRel担当、ゼロからの奮闘記~ @a...
 
Cache obliviousの話
Cache obliviousの話Cache obliviousの話
Cache obliviousの話
 
Solrを使ったレシピ検索のプロトタイピング
Solrを使ったレシピ検索のプロトタイピングSolrを使ったレシピ検索のプロトタイピング
Solrを使ったレシピ検索のプロトタイピング
 
高速な倍精度指数関数expの実装
高速な倍精度指数関数expの実装高速な倍精度指数関数expの実装
高速な倍精度指数関数expの実装
 
Prometheus meets Consul -- Consul Casual Talks
Prometheus meets Consul -- Consul Casual TalksPrometheus meets Consul -- Consul Casual Talks
Prometheus meets Consul -- Consul Casual Talks
 
Windows Server+Photon Server環境でも Fluentd+Elasticsearch+Kibanaを活用して 各種情報を可視化する...
Windows Server+Photon Server環境でもFluentd+Elasticsearch+Kibanaを活用して各種情報を可視化する...Windows Server+Photon Server環境でもFluentd+Elasticsearch+Kibanaを活用して各種情報を可視化する...
Windows Server+Photon Server環境でも Fluentd+Elasticsearch+Kibanaを活用して 各種情報を可視化する...
 
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポートJAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
JAWS-UGアーキテクチャ専門支部 ServerlessConfレポート
 

Similar to Streaming SQL

Julian Hyde - Streaming SQL
Julian Hyde - Streaming SQLJulian Hyde - Streaming SQL
Julian Hyde - Streaming SQL
Flink Forward
 
Streaming SQL w/ Apache Calcite
Streaming SQL w/ Apache Calcite Streaming SQL w/ Apache Calcite
Streaming SQL w/ Apache Calcite
Hortonworks
 
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/TridentQuerying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
DataWorks Summit/Hadoop Summit
 
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 KeynoteAdvanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
StreamNative
 
Towards sql for streams
Towards sql for streamsTowards sql for streams
Towards sql for streams
Radu Tudoran
 
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2
 
Building Streaming Applications with Streaming SQL
Building Streaming Applications with Streaming SQLBuilding Streaming Applications with Streaming SQL
Building Streaming Applications with Streaming SQL
Mohanadarshan Vivekanandalingam
 
WSO2 Analytics Platform: The one stop shop for all your data needs
WSO2 Analytics Platform: The one stop shop for all your data needsWSO2 Analytics Platform: The one stop shop for all your data needs
WSO2 Analytics Platform: The one stop shop for all your data needs
Sriskandarajah Suhothayan
 
Onyx data processing the clojure way
Onyx   data processing  the clojure wayOnyx   data processing  the clojure way
Onyx data processing the clojure way
Bahadir Cambel
 
Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0
Anyscale
 
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
WSO2
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Guido Schmutz
 
Introduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK StackIntroduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK Stack
Ahmed AbouZaid
 
Google cloud Dataflow & Apache Flink
Google cloud Dataflow & Apache FlinkGoogle cloud Dataflow & Apache Flink
Google cloud Dataflow & Apache Flink
Iván Fernández Perea
 
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache FlinkUnifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
DataWorks Summit/Hadoop Summit
 
Strtio Spark Streaming + Siddhi CEP Engine
Strtio Spark Streaming + Siddhi CEP EngineStrtio Spark Streaming + Siddhi CEP Engine
Strtio Spark Streaming + Siddhi CEP Engine
Myung Ho Yun
 
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
A Deep Dive into Structured Streaming:  Apache Spark Meetup at Bloomberg 2016 A Deep Dive into Structured Streaming:  Apache Spark Meetup at Bloomberg 2016
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
Databricks
 
Couchbase@live person meetup july 22nd
Couchbase@live person meetup   july 22ndCouchbase@live person meetup   july 22nd
Couchbase@live person meetup july 22nd
Ido Shilon
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward KeynoteK. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward Keynote
Flink Forward
 
Apache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and FriendsApache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and Friends
Stephan Ewen
 

Similar to Streaming SQL (20)

Julian Hyde - Streaming SQL
Julian Hyde - Streaming SQLJulian Hyde - Streaming SQL
Julian Hyde - Streaming SQL
 
Streaming SQL w/ Apache Calcite
Streaming SQL w/ Apache Calcite Streaming SQL w/ Apache Calcite
Streaming SQL w/ Apache Calcite
 
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/TridentQuerying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
 
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 KeynoteAdvanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
 
Towards sql for streams
Towards sql for streamsTowards sql for streams
Towards sql for streams
 
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
 
Building Streaming Applications with Streaming SQL
Building Streaming Applications with Streaming SQLBuilding Streaming Applications with Streaming SQL
Building Streaming Applications with Streaming SQL
 
WSO2 Analytics Platform: The one stop shop for all your data needs
WSO2 Analytics Platform: The one stop shop for all your data needsWSO2 Analytics Platform: The one stop shop for all your data needs
WSO2 Analytics Platform: The one stop shop for all your data needs
 
Onyx data processing the clojure way
Onyx   data processing  the clojure wayOnyx   data processing  the clojure way
Onyx data processing the clojure way
 
Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0Continuous Application with Structured Streaming 2.0
Continuous Application with Structured Streaming 2.0
 
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
WSO2Con ASIA 2016: WSO2 Analytics Platform: The One Stop Shop for All Your Da...
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
 
Introduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK StackIntroduction to InfluxDB and TICK Stack
Introduction to InfluxDB and TICK Stack
 
Google cloud Dataflow & Apache Flink
Google cloud Dataflow & Apache FlinkGoogle cloud Dataflow & Apache Flink
Google cloud Dataflow & Apache Flink
 
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache FlinkUnifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
Unifying Stream, SWL and CEP for Declarative Stream Processing with Apache Flink
 
Strtio Spark Streaming + Siddhi CEP Engine
Strtio Spark Streaming + Siddhi CEP EngineStrtio Spark Streaming + Siddhi CEP Engine
Strtio Spark Streaming + Siddhi CEP Engine
 
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
A Deep Dive into Structured Streaming:  Apache Spark Meetup at Bloomberg 2016 A Deep Dive into Structured Streaming:  Apache Spark Meetup at Bloomberg 2016
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
 
Couchbase@live person meetup july 22nd
Couchbase@live person meetup   july 22ndCouchbase@live person meetup   july 22nd
Couchbase@live person meetup july 22nd
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward KeynoteK. Tzoumas & S. Ewen – Flink Forward Keynote
K. Tzoumas & S. Ewen – Flink Forward Keynote
 
Apache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and FriendsApache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and Friends
 

More from Julian Hyde

Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Julian Hyde
 
Building a semantic/metrics layer using Calcite
Building a semantic/metrics layer using CalciteBuilding a semantic/metrics layer using Calcite
Building a semantic/metrics layer using Calcite
Julian Hyde
 
Cubing and Metrics in SQL, oh my!
Cubing and Metrics in SQL, oh my!Cubing and Metrics in SQL, oh my!
Cubing and Metrics in SQL, oh my!
Julian Hyde
 
Adding measures to Calcite SQL
Adding measures to Calcite SQLAdding measures to Calcite SQL
Adding measures to Calcite SQL
Julian Hyde
 
Morel, a data-parallel programming language
Morel, a data-parallel programming languageMorel, a data-parallel programming language
Morel, a data-parallel programming language
Julian Hyde
 
Is there a perfect data-parallel programming language? (Experiments with More...
Is there a perfect data-parallel programming language? (Experiments with More...Is there a perfect data-parallel programming language? (Experiments with More...
Is there a perfect data-parallel programming language? (Experiments with More...
Julian Hyde
 
Morel, a Functional Query Language
Morel, a Functional Query LanguageMorel, a Functional Query Language
Morel, a Functional Query Language
Julian Hyde
 
Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)
Julian Hyde
 
The evolution of Apache Calcite and its Community
The evolution of Apache Calcite and its CommunityThe evolution of Apache Calcite and its Community
The evolution of Apache Calcite and its Community
Julian Hyde
 
What to expect when you're Incubating
What to expect when you're IncubatingWhat to expect when you're Incubating
What to expect when you're Incubating
Julian Hyde
 
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Open Source SQL - beyond parsers: ZetaSQL and Apache CalciteOpen Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Julian Hyde
 
Efficient spatial queries on vanilla databases
Efficient spatial queries on vanilla databasesEfficient spatial queries on vanilla databases
Efficient spatial queries on vanilla databases
Julian Hyde
 
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Julian Hyde
 
Lazy beats Smart and Fast
Lazy beats Smart and FastLazy beats Smart and Fast
Lazy beats Smart and Fast
Julian Hyde
 
Data profiling with Apache Calcite
Data profiling with Apache CalciteData profiling with Apache Calcite
Data profiling with Apache Calcite
Julian Hyde
 
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
A smarter Pig: Building a SQL interface to Apache Pig using Apache CalciteA smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
Julian Hyde
 

More from Julian Hyde (17)

Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Building a semantic/metrics layer using Calcite
Building a semantic/metrics layer using CalciteBuilding a semantic/metrics layer using Calcite
Building a semantic/metrics layer using Calcite
 
Cubing and Metrics in SQL, oh my!
Cubing and Metrics in SQL, oh my!Cubing and Metrics in SQL, oh my!
Cubing and Metrics in SQL, oh my!
 
Adding measures to Calcite SQL
Adding measures to Calcite SQLAdding measures to Calcite SQL
Adding measures to Calcite SQL
 
Morel, a data-parallel programming language
Morel, a data-parallel programming languageMorel, a data-parallel programming language
Morel, a data-parallel programming language
 
Is there a perfect data-parallel programming language? (Experiments with More...
Is there a perfect data-parallel programming language? (Experiments with More...Is there a perfect data-parallel programming language? (Experiments with More...
Is there a perfect data-parallel programming language? (Experiments with More...
 
Morel, a Functional Query Language
Morel, a Functional Query LanguageMorel, a Functional Query Language
Morel, a Functional Query Language
 
Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)
 
The evolution of Apache Calcite and its Community
The evolution of Apache Calcite and its CommunityThe evolution of Apache Calcite and its Community
The evolution of Apache Calcite and its Community
 
What to expect when you're Incubating
What to expect when you're IncubatingWhat to expect when you're Incubating
What to expect when you're Incubating
 
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Open Source SQL - beyond parsers: ZetaSQL and Apache CalciteOpen Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite
 
Efficient spatial queries on vanilla databases
Efficient spatial queries on vanilla databasesEfficient spatial queries on vanilla databases
Efficient spatial queries on vanilla databases
 
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
 
Lazy beats Smart and Fast
Lazy beats Smart and FastLazy beats Smart and Fast
Lazy beats Smart and Fast
 
Data profiling with Apache Calcite
Data profiling with Apache CalciteData profiling with Apache Calcite
Data profiling with Apache Calcite
 
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
A smarter Pig: Building a SQL interface to Apache Pig using Apache CalciteA smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
 

Recently uploaded

Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Edge AI and Vision Alliance
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness  Posters-Social Engineering part 3FREE A4 Cyber Security Awareness  Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
Data Hops
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 

Recently uploaded (20)

Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness  Posters-Social Engineering part 3FREE A4 Cyber Security Awareness  Posters-Social Engineering part 3
FREE A4 Cyber Security Awareness Posters-Social Engineering part 3
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 

Streaming SQL

  • 1. Streaming SQL Julian Hyde Hadoop Summit San Jose, 2016/06/29
  • 2. @julianhyde SQL Query planning Query federation OLAP Streaming Hadoop Apache member VP Apache Calcite PMC Apache Arrow, Drill, Kylin Thanks: ● Milinda Pathirage & Yi Pan (Apache Samza) ● Haohui Mai (Apache Storm) ● Fabian Hueske & Stephan Ewen (Apache Flink)
  • 3. Data center Streaming data sources Sources: ● Devices / sensors ● Web servers ● (Micro-)services ● Databases (CDC) ● Synthetic streams ● Logging / tracing Transports: ● Kafka ● Nifi IoT Devices Services DatabaseWeb server
  • 4. How much is your data worth? Recent data is more valuable ➢ ...if you act on it in time Data moves from expensive memory to cheaper disk as it cools Old + new data is more valuable still ➢ ...if we have a means to combine them Time Value of data ($/GB) Now1 hour ago 1 day ago 1 week ago 1 year ago Hot data Read often Likely to be modified High value In memory Cold data Read rarely Unlikely to be modified Low value On disk
  • 5. Why query streams? Stream - Database Duality: ● “Your database is just a cache of my stream” ● “Your stream is just change-capture of my database” “Data is the new oil” ● Treating events/messages as data allows you to extract and refine them Declarative approach to streaming applications
  • 6. Why SQL? ● API to your database ● Ask for what you want, system decides how to get it ● Query planner (optimizer) converts logical queries to physical plans ● Mathematically sound language (relational algebra) ● For all data, not just data in a database ● Opportunity for novel data organizations & algorithms ● Standard https://www.flickr.com/photos/pere/523019984/ (CC BY-NC-SA 2.0) ➢ API to your database ➢ Ask for what you want, system decides how to get it ➢ Query planner (optimizer) converts logical queries to physical plans ➢ Mathematically sound language (relational algebra) ➢ For all data, not just “flat” data in a database ➢ Opportunity for novel data organizations & algorithms ➢ Standard Why SQL?
  • 7. Data workloads ● Batch ● Transaction processing ● Single-record lookup ● Search ● Interactive / OLAP ● Exploration / profiling ● Continuous execution generating alerts (CEP) ● Continuous load A variety of workloads, requiring specialized engines, but to the user it’s all “just data”.
  • 8. Building a streaming SQL standard via consensus Please! No more “SQL-like” languages! Key technologies are open source (many are Apache projects) Calcite is providing leadership: developing example queries, TCK (Optional) Use Calcite’s framework to build a streaming SQL parser/planner for your engine Several projects are working with us: Samza, Storm, Flink. (Also non-streaming SQL in Cassandra, Drill, Druid, Elasticsearch, Flink, Hive, Kylin, Phoenix.)
  • 9. Simple queries select * from Products where unitPrice < 20 select stream * from Orders where units > 1000 ➢ Traditional (non-streaming) ➢ Products is a table ➢ Retrieves records from -∞ to now ➢ Streaming ➢ Orders is a stream ➢ Retrieves records from now to +∞ ➢ Query never terminates
  • 10. Stream-table duality select * from Orders where units > 1000 ➢ Yes, you can use a stream as a table ➢ And you can use a table as a stream ➢ Actually, Orders is both ➢ Use the stream keyword ➢ Where to actually find the data? That’s up to the system select stream * from Orders where units > 1000
  • 11. Combining past and future select stream * from Orders as o where units > ( select avg(units) from Orders as h where h.productId = o.productId and h.rowtime > o.rowtime - interval ‘1’ year) ➢ Orders is used as both stream and table ➢ System determines where to find the records ➢ Query is invalid if records are not available
  • 12. Semantics of streaming queries The replay principle: A streaming query produces the same result as the corresponding non- streaming query would if given the same data in a table. Output must not rely on implicit information (arrival order, arrival time, processing time, or watermarks/punctuations) (Some triggering schemes allow records to be emitted early and re-stated if incorrect.)
  • 13. Making progress It’s not enough to get the right result. We need to give the right result at the right time. Ways to make progress without compromising safety: ➢ Monotonic columns (e.g. rowtime) and expressions (e.g. floor (rowtime to hour)) ➢ Punctuations (aka watermarks) ➢ Or a combination of both select stream productId, count(*) as c from Orders group by productId; ERROR: Streaming aggregation requires at least one monotonic expression in GROUP BY clause
  • 14. 8 75 4 10:00 10:15 10:30 11:00 11:15 Arrival time 1 2 3 5 6 Event time 8 10:00 10:15 10:30 11:00 11:15 Arrival time 1 2 3 6 Event time 4 Drop out-of-sequence records Emit 10:00-11:00 window when first record after 11: 00 arrives W 11:00 Emit 10:00-11:00 window when 11: 00 watermark arrives W 11:00’ 7 New watermark. Re-state 10: 00-11:00 window Policies for emitting results Monotonic column Watermark
  • 15. Aggregation and windows on streams GROUP BY aggregates multiple rows into sub- totals ➢ In regular GROUP BY each row contributes to exactly one sub-total ➢ In multi-GROUP BY (e.g. HOP, GROUPING SETS) a row can contribute to more than one sub-total Window functions (OVER) leave the number of rows unchanged, but compute extra expressions for each row (based on Multi GROUP BY Window functions GROUP BY
  • 16. GROUP BY select stream productId, floor(rowtime to hour) as rowtime, sum(units) as u, count(*) as c from Orders group by productId, floor(rowtime to hour) rowtime productId units 09:12 100 5 09:25 130 10 09:59 100 3 10:00 100 19 11:05 130 20 rowtime productId u c 09:00 100 8 2 09:00 130 10 1 10:00 100 19 1 not emitted yet; waiting for a row >= 12:00
  • 17. Window types Tumbling window “Every T seconds, emit the total for T seconds” Hopping window “Every T seconds, emit the total for T2 seconds” Session window “Emit groups of records that are separated by gaps of no more than T seconds” Sliding window “Every record, emit the total for the surrounding T seconds” “Every record, emit the total for the surrounding R records”
  • 18. Tumbling, hopping & session windows in SQL Tumbling window Hopping window Session window select stream … from Orders group by floor(rowtime to hour) select stream … from Orders group by tumble(rowtime, interval ‘1’ hour) select stream … from Orders group by hop(rowtime, interval ‘1’ hour, interval ‘2’ hour) select stream … from Orders group by session(rowtime, interval ‘1’ hour)
  • 19. Sliding windows in SQL select stream sum(units) over w (partition by productId) as units1hp, sum(units) over w as units1h, rowtime, productId, units from Orders window w as (order by rowtime range interval ‘1’ hour preceding) rowtime productId units 09:12 100 5 09:25 130 10 09:59 100 3 10:17 100 10 units1hp units1h rowtime productId units 5 5 09:12 100 5 10 15 09:25 130 10 8 18 09:59 100 3 23 13 10:17 100 10
  • 20. The “pie chart” problem ➢ Task: Write a web page summarizing orders over the last hour ➢ Problem: The Orders stream only contains the current few records ➢ Solution: Materialize short-term history Orders over the last hour Beer 48% Cheese 30% Wine 22% select productId, count(*) from Orders where rowtime > current_timestamp - interval ‘1’ hour group by productId
  • 21. Join stream to a table Inputs are the Orders stream and the Products table, output is a stream. Acts as a “lookup”. Execute by caching the table in a hash- map (if table is not too large) and stream order will be preserved. select stream * from Orders as o join Products as p on o.productId = p.productId
  • 22. Join stream to a changing table Execution is more difficult if the Products table is being changed while the query executes. To do things properly (e.g. to get the same results when we re-play the data), we’d need temporal database semantics. (Sometimes doing things properly is too expensive.) select stream * from Orders as o join Products as p on o.productId = p.productId and o.rowtime between p.startEffectiveDate and p.endEffectiveDate
  • 23. Join stream to a stream We can join streams if the join condition forces them into “lock step”, within a window (in this case, 1 hour). Which stream to put input a hash table? It depends on relative rates, outer joins, and how we’d like the output sorted. select stream * from Orders as o join Shipments as s on o.productId = p.productId and s.rowtime between o.rowtime and o.rowtime + interval ‘1’ hour
  • 24. Planning queries MySQL Splunk join Key: productId group Key: productName Agg: count filter Condition: action = 'purchase' sort Key: c desc scan scan Table: products select p.productName, count(*) as c from splunk.splunk as s join mysql.products as p on s.productId = p.productId where s.action = 'purchase' group by p.productName order by c desc Table: splunk
  • 25. Optimized query MySQL Splunk join Key: productId group Key: productName Agg: count filter Condition: action = 'purchase' sort Key: c desc scan scan Table: splunk Table: products select p.productName, count(*) as c from splunk.splunk as s join mysql.products as p on s.productId = p.productId where s.action = 'purchase' group by p.productName order by c desc
  • 26. Apache Calcite Apache top-level project since October, 2015 Query planning framework ➢ Relational algebra, rewrite rules ➢ Cost model & statistics ➢ Federation via adapters ➢ Extensible Packaging ➢ Library ➢ Optional SQL parser, JDBC server ➢ Community-authored rules, adapters Embedded Adapters Streaming Apache Drill Apache Hive Apache Kylin Apache Phoenix* Cascading Lingual Apache Cassandra Apache Spark CSV Druid* Elasticsearch* In-memory JDBC JSON MongoDB Splunk Web tables Apache Flink* Apache Samza Apache Storm * Under development
  • 28. Relational algebra (plus streaming) Core operators: ➢ Scan ➢ Filter ➢ Project ➢ Join ➢ Sort ➢ Aggregate ➢ Union ➢ Values Streaming operators: ➢ Delta (converts relation to stream) ➢ Chi (converts stream to relation) In SQL, the STREAM keyword signifies Delta
  • 29. Streaming algebra ➢ Filter ➢ Route ➢ Partition ➢ Round-robin ➢ Queue ➢ Aggregate ➢ Merge ➢ Store ➢ Replay ➢ Sort ➢ Lookup
  • 30. Optimizing streaming queries The usual relational transformations still apply: push filters and projects towards sources, eliminate empty inputs, etc. The transformations for delta are mostly simple: ➢ Delta(Filter(r, predicate)) → Filter(Delta(r), predicate) ➢ Delta(Project(r, e0, ...)) → Project(Delta(r), e0, …) ➢ Delta(Union(r0, r1), ALL) → Union(Delta(r0), Delta(r1)) But not always: ➢ Delta(Join(r0, r1, predicate)) → Union(Join(r0, Delta(r1)), Join(Delta(r0), r1) ➢ Delta(Scan(aTable)) → Empty
  • 31. Sort Sorting a streaming query is valid as long as the system can make progress. Need a monotonic or watermark-enabled expression in the ORDER BY clause. select stream productId, floor(rowtime to hour) as rowtime, sum(units) as u, count(*) as c from Orders group by productId, floor(rowtime to hour) order by rowtime, c desc
  • 32. Union As in a typical database, we rewrite x union y to select distinct * from (x union all y) We can implement x union all y by simply combining the inputs in arrival order but output is no longer monotonic. Monotonicity is too useful to squander! To preserve monotonicity, we merge on the sort key (e.g. rowtime).
  • 33. DML ➢ View & standing INSERT give same results ➢ Useful for chained transforms ➢ But internals are different insert into LargeOrders select stream * from Orders where units > 1000 create view LargeOrders as select stream * from Orders where units > 1000 upsert into OrdersSummary select stream productId, count(*) over lastHour as c from Orders window lastHour as ( partition by productId order by rowtime range interval ‘1’ hour preceding) Use DML to maintain a “window” (materialized stream history).
  • 34. Summary: Streaming SQL features Standard SQL over streams and relations Streaming queries on relations, and relational queries on streams Joins between stream-stream and stream-relation Queries are valid if the system can get the data, with a reasonable latency ➢ Monotonic columns and punctuation are ways to achieve this Views, materialized views and standing queries
  • 35. Summary: The benefits of streaming SQL Relational algebra covers needs of data-in-flight and data-at-rest applications High-level language lets the system optimize quality of service (QoS) and data location Give DB tools and traditional users to access streaming data; give message-oriented tools access to historic data Combine real-time and historic data, and produce actionable results Discussion continues at Apache Calcite, with contributions from Samza, Flink, Storm and others. Please join in!
  • 36. Thank you! @julianhyde @ApacheCalcite calcite.apache.org calcite.apache.org/docs/stream.html Next talk (with @maryannxue) tomorrow at 12:20pm: “How We Re- Engineered Phoenix with a Cost-Based Optimizer Based on Calcite”