Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stream Analytics with SQL on Apache Flink - Fabian Hueske


Published on

SQL is undoubtedly the most widely used language for data analytics for many good reasons. It is declarative,

many database systems and query processors feature advanced query optimizers and highly efficient execution engines, and last but not least it is the standard that everybody knows and uses. With stream processing technology becoming mainstream a question arises: “Why isn’t SQL widely supported by open source stream processors?”. One answer is that SQL’s semantics and syntax have not been designed with the characteristics of streaming data in mind. Consequently, systems that want to provide support for SQL on data streams have to overcome a conceptual gap. One approach is to support standard SQL which is known by users and tools but comes at the cost of cumbersome workarounds for many common streaming computations. Other approaches are to design custom SQL-inspired stream analytics languages or to extend SQL with streaming-specific keywords. While such solutions tend to result in more intuitive syntax, they suffer from not being established standards and thereby exclude many users and tools.
Apache Flink is a distributed stream processing system with very good support for streaming analytics. Flink features two relational APIs, the Table API and SQL. The Table API is a language-integrated relational API with stream-specific features. Flink’s SQL interface implements the plain SQL standard. Both APIs are semantically compatible and share the same optimization and execution path based on Apache Calcite.

In this talk we present the future of Apache Flink’s relational APIs for stream analytics, discuss their conceptual model, and showcase their usage. The central concept of these APIs are dynamic tables. We explain how streams are converted into dynamic tables and vice versa without losing information due to the stream-table duality. Relational queries on dynamic tables behave similar to materialized view definitions and produce new dynamic tables. We show how dynamic tables are converted back into changelog streams or are written as materialized views to external systems, such as Apache Kafka or Apache Cassandra, and are updated in place with low latency. We conclude our talk demonstrating the power and expressiveness of Flink’s relational APIs by presenting how common stream analytics use cases can be realized.

Published in: Data & Analytics
  • Be the first to comment

Stream Analytics with SQL on Apache Flink - Fabian Hueske

  1. 1. FEBRUARY 9, 2017, WARSAW Stream Analytics with SQL on Apache Flink® Fabian Hueske | Apache Flink PMC member | Co-founder dataArtisans
  2. 2. FEBRUARY 9, 2017, WARSAW Streams are Everywhere
  3. 3. FEBRUARY 9, 2017, WARSAW Data Analytics on Streaming Data • Periodic batch processing • Lots of duct tape and baling wire • It’s up to you to make everything work… reliably! • High latency • Continuous stream processing • Framework takes care of failures • Low latency
  4. 4. FEBRUARY 9, 2017, WARSAW Stream Processing in Apache Flink • Platform for scalable stream processing • Fast • Low latency and high throughput • Accurate • Stateful streaming processing in event time • Reliable • Exactly-once state guarantees • Highly available cluster setup
  5. 5. FEBRUARY 9, 2017, WARSAW Streaming Applications Powered by Flink 30 Flink applications in production for more than one year. 10 billion events (2TB) processed daily Complex jobs of > 30 operators running 24/7, processing 30 billion events daily, maintaining state of 100s of GB with exactly-once guarantees Largest job has > 20 operators, runs on > 5000 vCores in 1000-node cluster, processes millions of events per second
  6. 6. FEBRUARY 9, 2017, WARSAW Stream Processing is not for Everybody, … yet • APIs of open source stream processors target developers • Implementing streaming applications requires knowledge & skill • Stream processing concepts (time, state, windows, triggers, ...) • Programming experience (Java / Scala APIs) • Stream processing technology spreads rapidly • There is a talent gap
  7. 7. FEBRUARY 9, 2017, WARSAW What about SQL? • SQL is the most widely used language for data analytics • Many good reasons to use SQL • Declarative specification • Optimization • Efficient execution • “Everybody” knows SQL • SQL would make stream processing much more accessible, but…
  8. 8. FEBRUARY 9, 2017, WARSAW No OS Stream Processor Offers Decent SQL Support • SQL was not designed with streaming data in mind • Relations are sets. Streams are infinite sequences. • Records arrive over time. • Syntax • Time-based operations are cumbersome to specify (aggregates, joins) • Semantics • A SQL query should compute the same result on a batch table and a stream
  9. 9. FEBRUARY 9, 2017, WARSAW • Standard SQL and LINQ-style Table API • Unified APIs for batch & streaming data • Common translation layers • Optimization based on Apache Calcite • Type system & code-generation • Table sources & sinks • Streaming SQL & Table API is work in progress Flink’s SQL Support & Table API
  10. 10. FEBRUARY 9, 2017, WARSAW What are the Use Cases for Stream SQL? • Continuous ETL & Data Import • Live Dashboards & Reports • Ad-hoc Analytics & Exploration
  11. 11. FEBRUARY 9, 2017, WARSAW Dynamic Tables • Core concept is a “Dynamic Table” • Dynamic tables change over time • Dynamic tables are treated like static batch tables • Dynamic tables are queried with standard SQL • A query returns another dynamic table • Stream ←→ Dynamic Table conversions without information loss • “Stream / Table Duality”
  12. 12. FEBRUARY 9, 2017, WARSAW Stream → Dynamic Table • Append • Replace by Key time k 1 A 2 B 4 A 5 C 7 B 8 A 9 B … … time k 2, B4, A5, C7, B8, A9, B 1, A 2, B4, A5, C7, B8, A9, B 1, A 8 A 9 B 5 C … …
  13. 13. FEBRUARY 9, 2017, WARSAW Querying a Dynamic Table • Dynamic tables change over time • A[t]: Table A at time t • Dynamic tables are queried with regular SQL • Result of a query changes as input table changes • q(A[t]): Evaluate query q on table A at time t • As time t progresses, the query result is continuously updated • similar to maintaining a materialized view • t is current event time
  14. 14. FEBRUARY 9, 2017, WARSAW Querying a Dynamic Table time k k cnt A 3 B 2 C 1 9 B k cnt A 3 B 3 C 1 12 C k cnt A 3 B 3 C 2 A[8] A[9] A[12] q(A[8]) q(A[9]) q(A[12]) Table A q: SELECT k, COUNT(k) as cnt FROM A GROUP BY k 1 A 2 B 4 A 5 C 7 B 8 A
  15. 15. FEBRUARY 9, 2017, WARSAW time k A[5] A[10] A[15] q(A[5]) q(A[10]) q(A[15]) Table A Querying a Dynamic Table 7 B 8 A 9 B 11 A 12 C 14 C 15 A k cnt endT A 2 5 B 1 5 C 1 5 q(A) A 1 10 B 2 10 A 2 15 C 2 15 q: SELECT k, COUNT(k) AS cnt, TUMBLE_END( time, INTERVAL '5' SECONDS) AS endT FROM A GROUP BY k, TUMBLE( time, INTERVAL '5' SECONDS) 1 A 2 B 4 A 5 C
  16. 16. FEBRUARY 9, 2017, WARSAW Can We Run Any Query on Dynamic Tables? • No  • There are state and computation constraints • State may not grow infinitely as more data arrives • Clean-up timeout must be defined • Input updates may only trigger partial re-computation of the result • Queries with possibly unbounded state or computation are rejected • Optimizer performs validation
  17. 17. FEBRUARY 9, 2017, WARSAW Bounding the State of a Query • State grows infinitely with domain of grouping attribute • Bound query input by time • Query aggregates data of last 24 hours. Older data is discarded. SELECT k, COUNT(k) AS cnt FROM A GROUP BY k SELECT k, COUNT(k) AS cnt FROM A WHERE last(time, INTERVAL ‘1’ DAY) GROUP BY k STOP! UNBOUNED STATE!
  18. 18. FEBRUARY 9, 2017, WARSAW Updating Results and Late Arriving Data • Sometimes emitted results need to be updated • Results which are continuously updated • Results for which relevant records arrived late • Results that might be updated must be kept as state • Clean-up timeout • When a table is converted into a stream, updates must be propagated • Update mode • Add/Retract mode
  19. 19. FEBRUARY 9, 2017, WARSAW Dynamic Table → Stream: Update Mode time k Table A B, 1A, 2C, 1B, 2A, 3 A, 1 SELECT k, COUNT(k) AS cnt FROM A GROUP BY k 1 A 2 B 4 A 5 C 7 B 8 A … … Update by Key
  20. 20. FEBRUARY 9, 2017, WARSAW Dynamic Table → Stream: Add/Retract Mode time k Table A + B, 1+ A, 2+ C, 1+ B, 2+ A, 3 + A, 1- A, 1- B, 1- A, 2 1 A 2 B 4 A 5 C 7 B 8 A … … SELECT k, COUNT(k) AS cnt FROM A GROUP BY k Add (+) / Retract (-)
  21. 21. FEBRUARY 9, 2017, WARSAW Current State of SQL and Table API • Huge interest and many contributors • Current development efforts • Adding more window operators • Introducing dynamic tables • And there is a lot more to do • New operators and features for streaming and batch • Performance improvements • Tooling and integration • Try it out, give feedback, and start contributing!
  22. 22. FEBRUARY 9, 2017, WARSAW Ready for More Stream Processing with Flink? Preview will be available via O’Reilly Early Release in the next weeks
  23. 23. FEBRUARY 9, 2017, WARSAW Stream Analytics with SQL on Apache Flink Fabian Hueske | @fhueske