Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassandra And Kafka


Published on

It’s become clear to many business that the ability to extract real-time actionable insights from data is not only a source of competitive advantage, but also a way to defend their existing business models from disruption. So while legacy models such as nightly batch jobs aren’t disappearing, an era of fast, streaming data (aka “Fast Data”) is upon us, and represents the state of the art for gaining real-time perishable insights that can then be used to serve existing customers better, acquiring new markets and keep the competition at bay.

That said, distributed, Fast Data architectures are much harder to build, and carry their own set of challenges. Enterprises looking to move quickly are presented with a growing ecosystem of technologies, which often delays fast decisions and provides its own set of risks:

* With so many choices, what tools should you use?
* How do you avoid making rookie mistakes?
* What are the best patterns and practices for streaming applications?

In this webinar with Sean Glover, Senior Consultant at Lightbend and industry veteran, we examine the rise of streaming systems built around Spark, Mesos, Akka, Cassandra and Kafka, their role in handling endless streams of data to gain real-time insights. Sean then reviews how the Lightbend Fast Data Platform (FDP) brings them together in a comprehensive, easy to use, integrated platform, which includes installation, integration, and monitoring tools tuned for various deployment scenarios, plus sample applications.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassandra And Kafka

  1. 1. WEBINAR Building Streaming and Fast Data Applications with Spark, Mesos, Akka, Cassandra and Kafka Sean Glover (@seg1o), Senior Consultant at Lightbend
  2. 2. A Bit of History: IT Services vs. Big Data
  3. 3. Big DataServices Some Overlap: Concerns, Architecture The Recent Past…
  4. 4. Microservices & Fast Data Much More Overlap The Future?
  5. 5. Drilling Down: Monoliths to Microservices
  6. 6.
  7. 7. •Tangled responsibilities, lead to infrequent, “big-bang” deployments •App lifetimes months to forever! Monoliths
  8. 8. •Each does one thing, so must be message driven & asynchronous •Updates easier, deployments frequent •App lifetimes: minutes! to forever Microservices
  9. 9. •But be careful: •Message overhead is much better than function calls! Microservices
  10. 10. Reactive Systems
  11. 11. The Reactive Platform
  12. 12. Drilling Down: Big Data
  13. 13. Hadoop
  14. 14. Hadoop 2013: Embrace Spark
  15. 15. Go beyond batch?
  16. 16. The Emergence of Fast Data: (Time Is Money)
  17. 17. Respond to change
  18. 18. IoT
  19. 19. 20
  20. 20. Fast Data and Microservices; Are they Converging?
  21. 21. •Each [stream app or μservice]: •does one responsibility •ingests unending [data or messages] Synergies 22
  22. 22. •Each [stream app or μservice] must: •operate asynchronously •offer never-ending service 23 Synergies
  23. 23. •These architectures are converging: 1.Similar design problems 2.Data becomes dominant problem 24 Thesis
  24. 24. Lightbend Fast Data Platform
  25. 25. 1. An accelerated on ramp for building streaming data systems, data applications, and other microservices. Value Three Ways
  26. 26. 2. Best practices guidance for solving specific design problems: - Sample apps - Documentation - Enablement services Value Three Ways
  27. 27. 3. Machine learning-based monitoring and management: -Keep your systems resilient, scalable, and responsive with minimal user intervention. Value Three Ways
  28. 28. •Low latency? How low? •High volume? How high? Streaming Tradeoffs (1/3) 38
  29. 29. •Which kinds of data processing & analytics are required? •How will this processing be done? •Individual processing of events? •Bulk processing of records? Streaming Tradeoffs (2/3) 39
  30. 30. •Which tools and data sources/ sinks must interoperate with your streaming tool? Streaming Tradeoffs (3/3) 40
  31. 31. 41
  32. 32. 42 •Low latency •Low volume •Complex flows •Complex Event Processing
  33. 33. 43 •Med. latency •High volume •Data flows, SQL •En masse processing
  34. 34. 44 •Low latency •High volume •Data flows, correctness •En masse processing
  35. 35. 45 •Low latency •Med. volume •ETL, “tables” •Data flow or per event
  36. 36. Kafka & Spark Metrics
  37. 37. Correlated Troubleshooting
  38. 38. •Only metrics are uploaded, not sensitive domain data. •One less service for you to manage. •… Why Only Hosted Services?
  39. 39. •… •We can rapidly evolve this service. without impacting your environment. •You benefit from aggregated knowledge from all FDP clusters. Why Only Hosted Services?
  40. 40. Upgrade your grey matter!
 Get the free O’Reilly book by Dr. Dean Wampler, 
 VP of Fast Data Engineering at Lightbend
  41. 41. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS, Marathon On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted Graviton Cluster Analysis Machine LearningMicroservices ProducCon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles For More Information: