Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apache Mesos and DC/OS

280 views

Published on

Apache Mesos allows operators to run distributed applications across an entire datacenter and is attracting ever increasing interest. As much as distributed applications see increased use enabled by Mesos, Mesos also sees increasing use due to a growing ecosystem of well integrated applications. One of the latest additions to the Mesos family is Apache Flink. Flink is one of the most popular open source systems for real-time high scale data processing and allows users to deal with low-latency streaming analytical workloads on Mesos. In this talk we explain the challenges solved while integrating Flink with Mesos, including how Flink’s distributed architecture can be modeled as a Mesos framework, and how Flink was integrated with Fenzo. Next, we describe how Flink was packaged to easily run on DC/OS.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apache Mesos and DC/OS

  1. 1. Till Rohrmann till@data-artisans.com @stsffap Apache Flink® Meets Apache Mesos® and DC/OS Jörg Schad joerg@mesosphere.io @joerg_schad
  2. 2. MapReduce is crunching Data
  3. 3. We need to turn faster!
  4. 4. Evolution of Data Analytics Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analytics Reports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time Advertising Billing,
 Chargeback Product recommendations
  5. 5. FMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Flink ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data & build data driven apps Mesos/ DC/OS Sensors Devices Clients
  6. 6. Datacenter
  7. 7. Naive Approach Typical Datacenter
 siloed, over-provisioned servers,
 low utilization Industry Average
 12-15% utilization mySQL microservice Cassandra Flink Kafka
  8. 8. © 2017 Mesosphere, Inc. All Rights Reserved. 10
  9. 9. Apache Mesos Typical Datacenter
 siloed, over-provisioned servers,
 low utilization Industry Average
 12-15% utilization mySQL microservice Cassandra Flink Kafka Mesos
 automated schedulers, workload multiplexing onto the same machines
  10. 10. Apache Mesos 
 Why Mesos? ! 2-level scheduling ! Fault-tolerant, battle-tested ! Scalable to 10,000+ nodes ! Created by Mesosphere founder @ UC Berkeley; used in production by 100+ web-scale companies [1] [1] http://mesos.apache.org/documentation/latest/powered-by-mesos/
  11. 11. Apache Flink & Apache Mesos
  12. 12. Why Apache Mesos? ▪ Mesos offers full functionality to implement fault tolerant and elastic distributed applications ▪ 30% of survey respondents were running Flink on Mesos (prior to proper Mesos support, September 2016)
  13. 13. Flink’s Mesos Integration Apache Flink Framework Mesos Master Mesos App Master Flink Mesos
 ResourceManager JobManager Mesos Task TaskManager Mesos Task TaskManager Allocate Resources Launch Mesos tasks Register Execute Job
  14. 14. Resource Manager Components ▪ Monitors connection to Mesos Connection Monitor Launch Coordinator ▪ Resource offer processing and task scheduling ▪ Gathers offers and matches them to tasks using Fenzo Task Monitor Reconciliation Coordinator ▪ Monitors Mesos tasks ▪ Triggers reconciliation ▪ Makes sure tasks are properly killed ▪ Reconciles tasks view between ResourceManager and Mesos Master
  15. 15. Component Interplay ResourceManager Connection Monitor Launch Coordinator Task MonitorReconciliation Coordinator Mesos Master Resource offers Launch tasks Monitor tasks Status messages Trigger reconciliation Status messages Mesos Task Reconcile tasks Start TaskManagers Recover tasks Kill task
  16. 16. Fenzo ▪ Generic task scheduler for Mesos frameworks ▪ Developed by Netflix ▪ Matching between tasks and resource offers ▪ Pluggable fitness evaluator Fenzo Mesos Launch Coordinator Periodic resource offers Tell Fenzo offered resources & tasks Fenzo returns resource task matchings Tasks to launch
  17. 17. New Distributed Architecture Mesos Master Mesos Cluster Client (2) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos
 ResourceManager JobManager (4) Start Process (and supervise) (8) Deploy
 Tasks (7) Register (5) Request slots Flink Mesos Dispatcher (3) Allocate container
 for Flink master (6) Allocate containers
 for TaskManagers Marathon (1) Start and monitor dispatcher Mesos Task TaskManager Mesos Task TaskManager
  18. 18. DC/OS Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Any Infrastructure (Physical, Virtual, Cloud)
  19. 19. Demo Time Generator ▪ Financial data generated by generator ▪ Written to Kafka topics ▪ Kafka topics consumed by Flink ▪ Flink pipeline operates on Kafka data ▪ Results written back into Kafka
  20. 20. Conclusion
  21. 21. TL;DL ▪ Apache Flink runs on Mesos using Fenzo ▪ New distributed architecture supports dynamic resource allocation ▪ DC/OS offers easy to use Flink package
  22. 22. 25 Thank you! @joerg_schad @stsffap @ApacheFlink @dataArtisans @dcos
  23. 23. We are hiring! data-artisans.com/careers

×