Download as PDF, PPTX
Paris jug ksql - 2018-06-28
- 1. KSQLOpen-source streaming forApache Kafka
- 2. let's look at KafkaStreams First! But ...is a
- 3. Trade-Offs • subscribe() • poll() •send() • flush() Consumer, Producer Flexibility Simplicity
- 4. Trade-Offs • subscribe() • poll() •send() • flush() Consumer, Producer • filter() • join() • aggregate() Kafka Streams Flexibility Simplicity
- 5.
- 6.
- 7. Things Kafka StreamsDoes Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
- 8.
- 9.
- 10. No Panic, it'sa walktrough! https://github.com/framiere/a-kafka-story/tree/master/step6
- 11. Expressive ... butverbose
- 12. Trade-Offs • subscribe() • poll() •send() • flush() Consumer, Producer • filter() • join() • aggregate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity
- 13.
- 14.
- 15. Stream Processing byAnalogy Kafka Cluster Connect API Stream Processing Connect API $ cat < in.txt | grep “ksql” | tr a-z A-Z > out.txt
- 16.
- 17. KSQL for DataExploration SELECT status, bytes FROM clickstream WHERE user_agent = ‘Mozilla/5.0 (compatible; MSIE 6.0)’; An easy way to inspect data in a running cluster
- 18. KSQL for StreamingETL • Kafka is popular for data pipelines. • KSQL enables easy transformations of data within the pipe. • Transforming data while moving from Kafka to another system. CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
- 19. KSQL for AnomalyDetection CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3; Identifying patterns or anomalies in real-time data, surfaced in milliseconds
- 20. KSQL implements foryou the Kafka Stream application you would have implemented if you had • ... the time • ... the experience • ... the KSQL as a spec • ... the willingness to do boring code
- 21. KSQL for DataTransformation CREATE STREAM views_by_userid WITH (PARTITIONS=6, VALUE_FORMAT=‘JSON’, TIMESTAMP=‘view_time’) AS SELECT * FROM clickstream PARTITION BY user_id; Make simple derivations of existing topics from the command line
- 22.
- 23. Where is KSQLnot such a great fit? BI reports (Tableau etc.) • No indexes • No JDBC (most BI tools are not good with continuous results!) Ad-hoc queries • Limited span of time usually retained in Kafka • No indexes
- 24. Do you thinkthat’s a table you are querying ?
- 25.
- 26. alice 1 alice 1 charlie1 alice 2 charlie 1 alice 2 charlie 1 bob 1 TABLE STREAM TABLE (“alice”, 1) (“charlie”, 1) (“alice”, 2) (“bob”, 1) alice 1 alice 1 charlie 1 alice 2 charlie 1 alice 2 charlie 1 bob 1
- 27. CREATE STREAM clickstream( time BIGINT, url VARCHAR, status INTEGER, bytes INTEGER, userid VARCHAR, agent VARCHAR) WITH ( value_format = ‘JSON’, kafka_topic=‘my_clickstream_topic’ ); Creating a Stream
- 28. CREATE TABLE users( user_id INTEGER, registered_at LONG, username VARCHAR, name VARCHAR, city VARCHAR, level VARCHAR) WITH ( key = ‘user_id', kafka_topic=‘clickstream_users’, value_format=‘JSON'); Creating a Table
- 29. CREATE STREAM vip_actionsAS SELECT userid, fullname, url, status FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum'; Joins for Enrichment
- 30. Trade-Offs • subscribe() • poll() •send() • flush() Consumer, Producer • filter() • join() • aggregate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity
- 31. Kafka Cluster JVM KSQL ServerKSQLCLI #1 STAND-ALONE AKA ‘LOCAL MODE’ How to run KSQL
- 32. How to runKSQL JVM KSQL Server KSQL CLI JVM KSQL Server JVM KSQL Server Kafka Cluster #2 CLIENT-SERVER
- 33. How to runKSQL Kafka Cluster JVM KSQL Server JVM KSQL Server JVM KSQL Server #3 AS A STANDALONE APPLICATION
- 34. How to runKSQL Kafka Cluster #4 EMBEDDED IN AN APPLICATION JVM App Instance KSQL Engine Application Code JVM App Instance KSQL Engine Application Code JVM App Instance KSQL Engine Application Code
- 35. Confluent 5.0 -Nested Types SELECT userid, address.city FROM users WHERE address.state = 'CA' https://github.com/confluentinc/ksql/pull/1114
- 36. Confluent 5.0 -Remaining joins SELECT orderid, shipmentid FROM orders INNER JOIN shipments ON order.id = shipmentid;
- 37. Confluent 5.0 -User defined functions SELECT address, stringLength(address->street) from orders;
- 38. KSQL Capacity Planning– Sizing https://docs.confluent.io/current/ksql/docs/capacity-planning.html
- 39.
- 40.
- 41.
- 42.
- 43. Resources – CommunitySlack and Mailing List https://slackpass.io/confluentcommunity https://groups.google.com/forum/#!forum/confluent-platform
- 44. Resources and NextSteps https://github.com/confluentinc/ksql http://confluent.io/ksql https://slackpass.io/confluentcommunity #ksql
- 45. Links • https://www.confluent.io/blog/taking-ksql-spin-using-real-time-device-data/ • https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16 •https://asciinema.org/a/nNfYKsVFwzOKMIWfdj5L16uzW • https://github.com/confluentinc/ksql/blob/master/ksql-examples/src/main/java/io/confluent/ksql/datagen/ DataGen.java • https://github.com/confluentinc/ksql/blob/master/ksql-parser/src/main/antlr4/io/confluent/ksql/parser/SqlBase.g4 • https://github.com/confluentinc/ksql/blob/v5.1.0-beta201806200051/ksql-engine/src/main/java/io/confluent/ksql/ KsqlContext.java • https://github.com/mmolimar/ksql-jdbc-driver • https://twitter.com/alexott_en/status/1008384806693097472 • https://github.com/framiere/a-kafka-story • https://docs.confluent.io/current/ksql/docs/syntax-reference.html • https://github.com/confluentinc/cp-demo • https://github.com/confluentinc/cp-ansible • https://www.confluent.io/blog/introducing-the-confluent-operator-apache-kafka-on-kubernetes/ • https://www.confluent.io/blog/author/robin/
- 46.
- 47.