Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lightbend Fast Data Platform

0 views

Published on

Dean Wampler, O’Reilly author and Big Data Strategist in the office of the CTO at Lightbend discusses practical tips for architecting stream-processing applications and explains how you can tame some of the complexity in moving from data at rest to data in motion.

Published in: Technology

Lightbend Fast Data Platform

  1. 1. Lightbend Fast Data Platform A Technical Overview
  2. 2. A Bit of History: Monoliths to Microservices
  3. 3. lightbend.com/reactive-microservices-architecture
  4. 4. • Tangled responsibilities, lead to… • Infrequent, “big-bang” deployments • Runtimes: months to years • Latencies: μsecs to seconds Monoliths
  5. 5. • Each does one thing, so… • Updates easier to deploy frequently • Message driven & asynchronous • Runtimes: minutes! to forever • Latencies: higher than function calls Microservices
  6. 6. Reactive Systems reactivemanifesto.org
  7. 7. Open Source Core For Development in Scala / Java Produc8on Suite For Opera8ons / DevOps Lightbend Reactive Platform
  8. 8. A Bit of History: Big Data
  9. 9. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Hadoop
  10. 10. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Hadoop 2013: Embrace Spark Spark job #1 Spark job #2
  11. 11. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Go beyond batch? Spark job #1 Spark job #2
  12. 12. The Emergence of Fast Data: (Time Is Money)
  13. 13. Respond to change
  14. 14. IoT
  15. 15. 16 Logs Sockets Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
  16. 16. Fast Data and Microservices; Are they Converging?
  17. 17. •Each (stream|μservice): •does one responsibility •has unending (data|messages) Synergies 18 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
  18. 18. •Each (stream|μservice): •encourages asynchrony •offers never-ending service Synergies 19 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
  19. 19. •Reasons these architectures will converge: 1.Similar design problems 2.Data becomes dominate problem Thesis 20 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
  20. 20. Lightbend Fast Data Platform
  21. 21. 1. An accelerated on ramp for building streaming data systems and applications. Value Three Ways
  22. 22. 2. Best practices guidance, through sample apps, documentation, and enablement services for what tools and techniques to use for solving specific design problems. Value Three Ways
  23. 23. 3. Intelligent monitoring and management, driven by machine learning, to keep your systems resilient, scalable, and responsive with minimal user intervention. Value Three Ways
  24. 24. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  25. 25. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  26. 26. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  27. 27. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N * M links ConsumersProducers
  28. 28. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N + M links ConsumersProducers
  29. 29. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  30. 30. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  31. 31. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  32. 32. •Low latency? How low? •High volume? How high? Streaming Tradeoffs (1/3) 33
  33. 33. •Which kinds of data processing & analytics are required? •How will this processing be done? •Individual processing of events? •Bulk processing of records? Streaming Tradeoffs (2/3) 34
  34. 34. •Which tools and data sources/sinks must interoperate with your streaming tool? Streaming Tradeoffs (3/3) 35
  35. 35. 36
  36. 36. 37 •Low latency •Low volume •Complex flows •Complex Event Processing
  37. 37. 38 •Med. latency •High volume •Data flows, SQL •En masse processing
  38. 38. 39 •Low latency •High volume •Data flows, correctness •En masse processing
  39. 39. 40 •Low latency •Med. volume •ETL, “tables” •Data flow or Per Event
  40. 40. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  41. 41. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  42. 42. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  43. 43. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  44. 44. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
  45. 45. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS, Marathon On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted Graviton Cluster Analysis Machine LearningMicroservices ProducCon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles For More Information: lightbend.com/fast-data-platform
  46. 46. bit.ly/lightbend-fast-data For More Information
  47. 47. • FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data

×