Downloaded 40 times
Uploaded by{code} by Dell EMC
Data Analytics Using Container Persistence Through SMACK - Manny Rodriguez-Perez and Kendrick Coleman - Dell EMC World 2017
The document discusses data analytics through the use of the SMACK stack, which consists of Spark, Mesos, Akka, Cassandra, and Kafka, emphasizing the importance of streaming analytics for real-time decision-making in business. It details the architecture and functionality of the SMACK stack, including container persistence management and the role of various components like Kafka for event transport and Cassandra for data storage. The document also highlights the integration of open-source tools like Rex-Ray for storage orchestration within the Mesos framework.
More Related Content
Similar to Data Analytics Using Container Persistence Through SMACK - Manny Rodriguez-Perez and Kendrick Coleman - Dell EMC World 2017
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Data Analytics Using Container Persistence Through SMACK - Manny Rodriguez-Perez and Kendrick Coleman - Dell EMC World 2017
- 1. Data Analytics Using ContainerPersistence Through SMACK Manny Rodriguez-Perez FL Enterprise Advisory SE @MannyRodP {code} by Dell EMC Kendrick Coleman Developer Advocate @KendrickColeman {code} by Dell EMC
- 2. © Copyright 2017Dell Inc.2 Open source at Dell EMC – Contribute to meaningful OSS projects – Create new thought leading OSS applications – Drive awareness of OSS opportunities with Dell EMC product teams – Participate in relevant community engagement projects – Act in the interest of building a community {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. Platinum Sponsor
- 3. © Copyright 2017Dell Inc.3
- 4. © Copyright 2017Dell Inc.4 Digital Business Cycle … AD&D pros aren’t fully exploiting real-time streaming data that flows from IoT devices and mobile, web, and enterprise apps. Streaming analytics is essential for bringing real-time context to apps. - The Forrester Wave™: Big Data Streamin Analytics, Q1 2016 Applications/IoT Transforms Business Build Data Generated By New Applications is Captured / Stored Measure Analytics Insights Drive New Functionality, Which Drives New Data Learn Validated Learning
- 5. © Copyright 2017Dell Inc.5 What has happened? What could happen? What should we do? Batch Micro-Batch Event driven A brief current history of business analytics Days/weeks Hours Minutes Seconds Microseconds Need to make decision in …. Descriptive Prescriptive Predictive
- 6. © Copyright 2017Dell Inc.6 the world is streaming live… Right now!
- 7. © Copyright 2017Dell Inc.7 Modern Digital Application Src: Mesosphere , Ben Hindham Message Driven & Non-Blocking
- 8. © Copyright 2017Dell Inc.8 The SMACK Stack Spark micro-batch processing Mesos cluster management Akka event processing Cassandra persistence Kafka event transport
- 9. © Copyright 2017Dell Inc.9 Digital Business Cycle with SMACK Stack Applications/IoT Transforms Business Build Data Generated By New Applications is Captured / Stored Measure Analytics Insights Drive New Functionality, Which Drives New Data Learn Validated Learning
- 10. © Copyright 2017Dell Inc.10 The SMACK Stack – Stateless and Stateful Services Spark The engine Mesos The container Akka The model Cassandra The storage Kafka The broker Stateless Microservices Stateful Data Services s s s s a a a a k k k k c c c c
- 11. © Copyright 2017Dell Inc.11 Mesos (Mesosphere DC/OS) Architecture SRC: http://datastrophic.io/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka/ Agent Agent Agent 1. Leader election & service discovery via ZooKeeper 2. Agents publish available resources to master 3. Master sends resource offers to frameworks 4. Scheduler replies with tasks and resources needed per task 5. Master sends task to slaves 1 2 34 5
- 12. © Copyright 2017Dell Inc.12 Kafka Data Protection (queues) • Kafka brokers = Kafka nodes in containers • Kafka topic = publish/subscribe queue • Topics can be partitioned , partitions are spread over brokers • Data fault tolerance is set using a replication factor for partitions at the topic level bin/kafka-topics.sh --create --replication-factor 2 --partitions 3 --topic test topic-part0 topic-part0’ topic-part1 topic-part2 topic-part1’ topic-part2’ broker 1 broker 2 broker 3 leader leader leader follower follower follower
- 13. © Copyright 2017Dell Inc.13 Cassandra Data Protection (database) • Keyspace defines how data replication happens for each row using two settings • Replication Strategy - Where to place next replica • Replication Factor - Total number of replicas placed on different nodes • Tables are defined primary keys (columns) that are used to distribute the data among nodes using hash of the key CREATE KEYSPACE mykeyspace WITH replication = {'class':’SimpleStrategy', 'replication_factor' : 3}; USE mykeyspace CREATE TABLE emp( emp_id int PRIMARY KEY, emp_name text, emp_cel ); 56A79BAA…. John emp_id emp_name emp_cel 555-555-5555 C387A57A…. Peter 999-999-9999 F93B2CAE…. Paul 777-777-7777 1 3 2 56A79BAA…. John emp_id emp_name emp_cel 555-555-5555 56A79BAA…. John emp_id emp_name emp_cel 555-555-5555 56A79BAA…. John emp_id emp_name emp_cel 555-555-5555 node 1 node 2 node 3
- 14. © Copyright 2017Dell Inc.14 Kafka & Cassandra on Mesos • Kafka Brokers and Cassandra Nodes are run by Marathon container scheduler on Agents • Configuration must be set to make sure containers are scheduled to run on separate nodes to manage fault agent domains – Kafka on Mesos , placement_strategy =NODE – Cassandra on Mesos , placement_constraint in JSON config flie • Mesos does not provide replication. Data protection must be provided by the app, framework or other. node 1 node 2 node 3broker 2broker 1 broker 3 k1c1 k2 k3c2 c3 Agent Agent Agent What are my storage options?
- 15. © Copyright 2017Dell Inc.15 Native Data Protection vs. Highly Available Database Shards 🔥 • Cassandra Native Data Protection – Can take hours or days to re-protect data – Increases load on remaining nodes › Nodes must both handle workload & re-balance data • Not Limited to Node Failures – Database Software Updates – Security Patches – Node Hardware Upgrades
- 16. © Copyright 2017Dell Inc.16 What are my storage options? node 1 broker 1 k1c1 Local Persistent Storage1 What’s Missing? • Scale beyond node capacity • Performance during rebuilds • Offsite replication • Snapshots • Node migration • Off-site federation
- 17. © Copyright 2017Dell Inc.17 What are my storage options? Shared Persistent Storage2 node 1 broker 1 c1 SAN LAN WAN Cloud k1 … the amount of storage capacity that can be directly attached to a single agent is limited— certain applications (e.g., databases) would like to access more data than can easily be attached to a single node. Using external storage can also simplify data migration between agents/containers, and can make backups and disaster recovery easier. http://mesos.apache.org/documentation/latest/docker-volume/ Enterprise Storage Features & Flexibility
- 18. © Copyright 2017Dell Inc.18 End-to-End Container Persistence Leadership for Mesos What does it do Enables external storage to be created / mounted / unmounted with each agent task Abstracts Docker Volume Drivers for Mesos Storage orchestration engine Cool factor • First Mesos Agent external storage module • Merged into Mesos 1.0 (2016) • Allows the use of any Docker volume driver with Mesos! • Vendor agnostic • DC/OS 1.7+ framework (2016) mesos-module-dvdi Docker Volume Driver Client CLI DVDCLI REX-Ray
- 19. © Copyright 2017Dell Inc.19 Mesos 0.23 and earlier mesos- module-dvdi 0.1.0-rc1 for mesos 0.23.0 September 2015 October 2015 mesos- module- dvdi 0.1.0 GA v0.3 0.23.1 0.24 0.25 December 2015 August 2016 mesos- module-dvdi 0.4.5 Mesos 1.0 Merge! mesos-module- dvdi 0.4.7 Mesos: 1.1.0, 1.0.1, 0.28.3, 0.27.2, 0.26.1, 0.25.1, 0.24.2, 0.23.1 January 2017 ships with REX-Ray March 2017 Defining CSI with Mesos
- 20. © Copyright 2017Dell Inc.20 Introducing REX-Ray REX-Ray The leading container storage orchestration engine enabling persistence for cloud native workloads rexray.codedellemc.com • Cloud Native Interoperability • Open Source • Enterprise Ready – High Availability – CLI Intuitiveness – Effortless Deployment – Architectural Choices • Multi-Platform Storage Management – Storage agnostic (block/file/object)
- 21. © Copyright 2017Dell Inc.21 Native Data Protection vs. Highly Available Database Shards • ScaleIO External Storage & Container Benefits – Containers are rescheduled immediately on other nodes – No data rebalance performed – Node operations (reboots, patches, upgrades) have much less operational cost › Complete in seconds / minutes – Containers can be replaced without overwriting data of configuration – REX-Ray provides interoperability and storage orchestration Scheduler (Mesos) Container Persistent Volumes Software Defined Volume Service
- 22. © Copyright 2017Dell Inc.22 Persistence supporting SMACK… Spark The engine Akka The model Cassandra The storage Kafka The broker Supporting Frameworks / Apps Needing Shared Storage Persistence • Jenkins • Solr • Prvt Docker Registry • ELK stack • Marathon • etc…
- 23. © Copyright 2017Dell Inc.23 ScaleIO Framework for Mesos • A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere • Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster. • Deploy and Configure ScaleIO without knowing operational details • Centralized Monitoring of the storage platform • Persistent storage native to the container scheduling platform using REX-Ray • https://github.com/codedellemc/scaleio-framework
- 24. © Copyright 2017Dell Inc.24 {code} + Mesos Mesos Module DVDI is officially merged into Mesos 1.0 and allows any Docker Volume Driver to be used with Mesos Frameworks REX-Ray is installed on every host and performs storage orchestration MARATHON ScaleIO-Framework installs ScaleIO software on all Mesos Agents SUPPORTED STORAGE PLATFORMS WITH REX-RAY
- 25.
- 26. Take our projectsfor a spin at the {code} Labs http://github.com/codedellemc/labs
- 27. © Copyright 2017Dell Inc.27 codedellemc.com community.codedellemc.com @codeDellEMC blog.codedellemc.com {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. rexray.codedellemc.com github.com/codedellemc/labs HOL01 Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes Kendrick Coleman @KendrickColeman github.com/kacole2 Manny Rodriguez-Perez FL Ent Advisriy SE @MannyRodP
Editor's Notes
- #3 Add talking points to each point OSS projects: Docker, Mesos, Kubernetes, Cloud Foundry
- #4 My first job out of college was at a company that analyzed Sprint Call Center statistics. One of my initial responsibilities was handed over to me from other people and it required going to 15 or so different internal sites and pulling data, put it into an excel spreadsheet, and have it ready for the call center management staff by 8:00am everyday before the phones opened. This required me getting to office at 6am since it was a manual process. Over the course of 1 month, I was able to automate everything through VBA macros and I got an extra hour and a half of sleep since it only took 5 minutes to collect data instead of two hours. However, everyone was analyzing day old data. Things have progressed and people need less than just a few second lag time.
- #5 To be a modern software-enabled company you have to embrace the validated learning innovation loop , made popular by Eric Ries in his book The Lean Startup. In addition we believe the companies or teams that operate the loop at the fastest cycles will excel beyond competitors and find new markets. These faster innovation cycles are based on faster software development and deployments. The modern technology and practices that enable faster software development and deployments form a Cloud Native Strategy. Being cloud native meanings that the applications and their supporting ecosystem allows business to pivot as insights are gained. … [teams] that accelerate this feedback loop quickly accumulate validated learning while minimizing wasted time, money and effort. - Eric Ries , the Lean Startup
- #6 Innovations in analytics processing in the last 10 years have served to reduce the cycle time of the validated learning loop and fundamentally change the questions business can ask. Lets look at a brief history of analytics processing in terms of the technology, capabilities and business alignment… When I started in IT in the 1990s , focus was on modeling, organizing, ETL-ing data for the sake of reporting – collect then analyzing process As data volume, velocity, and variability increased , and to take advantage of increased host and storage per node performance , parallel processing architectures such as hadoop have gained popularity Hadoop and others provided a faster collection and analysis process , reducing the cycle time considerably, However it was still a collect then analyze model and at best a micro-batch process, and while some companies have build near-realtime engagement models with technology like Hadoop , its still very difficult to do at sacle and with cost efficiencies To get to a more prescriptive model and respond to devices and customers in real time we can no longer wait for our haddop batch jobs to load and process, we need to look at the data coming in real time and process it, make a decision. This is a new world of event drive analytics which helps us answer What should we do? This is a paradigm shift from collect-analyze to analyze-collect data analysis needs to occur almost in tandem with data collection I say analyze – then collect because its not like we are going to throw away the legacy analytics capabilities
- #7 Because the world is not a batch oriented, the world is streaming live … right now … from billions of people and devices – imagine 50B streams with their own message, uniqueness, waiting to be heard and engaged Just imagine one application , like twitter which has a potential for 320MM streams of data at any one time [Need actual data on how many streams does certain companies , web scale and not web scale see/process, example] and not web scale see/process? Perishable insights
- #8 So we need a way of building applications and analytics capabilities that can run our cloud native applications , handle complex event processing and store historical data , all while
- #9 Key Apache project
- #11 Key Apache project
- #18 Benefits of shared storage Flexibility Class of service Consistent operations across multiple storage endpoints Enterprise storage features – Snaps etc Simplify recovery from downed node Easily Add/Remove additional nodes
- #19 Docker Volume Driver Isolator Module No remote storage capability using Mesos before September 2015 {code} was first to bring an abstracted persistent storage model Freedom to use the Mesos Containerizer with ANY docker volume driver (that means you don’t need Docker) Merged upstream into Mesos 1.0 to provide storage platform support for any vendor who has written a docker volume driver, including REX-Ray Every company that boasts storage integration with Mesos uses the {code} contributed module. DVDCLI Sss Sss DC/OS 1.7 Release ships with REX-Ray embedded DC/OS 1.8 Release ships with DVDI embedded for Mesos 1.0 Featured in the documentation - https://dcos.io/docs/1.8/usage/storage/external-storage/
- #21 Kenny
- #23 Key Apache project
- #28 First, a few things about the team that has made this possible. The Dell EMC {code} team is a team made up of open source software engineers and developer advocates, focused on making EMC a well-known name within the open source community. We will focus on one of their projects, REX-Ray, in this presentation.