Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Get started with Developing Frameworks in Go on Apache MesosJoe Stein
Apache Mesos provides a platform for building distributed systems. Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. How to use that platform and what to make of it becomes a complex task requiring not only understanding of where the system has been but also where it is going. Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework in Go, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but should also be used for custom internal R&D built software applications too.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but can also be used for custom internal R&D built software applications too.
Get started with Developing Frameworks in Go on Apache MesosJoe Stein
Apache Mesos provides a platform for building distributed systems. Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. How to use that platform and what to make of it becomes a complex task requiring not only understanding of where the system has been but also where it is going. Using schedulers like Marathon and Aurora help to get your applications scheduled and executing on Mesos. In many cases it makes sense to build a framework and integrate directly. This talk will breakdown what is involved in building a framework in Go, how to-do this with examples and why you would want to-do this. Frameworks are not only for generally available software applications (like Kafka, HDFS, Spark ,etc) but should also be used for custom internal R&D built software applications too.
Deploying Docker Containers at Scale with Mesos and MarathonDiscover Pinterest
Connor Doyle from Mesosphere.
Deploying Docker Containers at Scale with Mesos and Marathon
The norm these days is to operate apps at web scale. But that’s out of reach for most companies. Deploying Docker containers with Mesos and Marathon makes it easier. See how they help deploy and manage Docker containers at scale and how the Mesos cluster scheduler builds highly-available, fault-tolerant web scale apps.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Adobe has packaged HBase in Docker containers and uses Marathon and Mesos to schedule them—allowing them to decouple the HBase RegionServer from the host, express resource requirements declaratively, and open the door for unassisted real-time deployments, elastic (up and down) real-time scalability, and more.
Apache Mesos is the first open source cluster manager that handles the workload efficiently in a distributed environment through dynamic resource sharing and isolation.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
O'Reilly Media - Strata SC 2014
Apache Mesos is an open source cluster manager that provides efficient resource isolation for distributed frameworks—similar to Google’s “Borg” and “Omega” projects for warehouse scale computing. It is based on isolation features in the modern kernel: “cgroups” in Linux, “zones” in Solaris.
Google’s “Omega” research paper shows that while 80% of the jobs on a given cluster may be batch (e.g., MapReduce), 55-60% of cluster resources go toward services. The batch jobs on a cluster are the easy part—services are much more complex to schedule efficiently. However by mixing workloads, the overall problem of scheduling resources can be greatly improved.
Given the use of Mesos as the kernel for a “data center OS”, two additional open source components Chronos (like Unix “cron”) and Marathon (like Unix “init.d”) serve as the building blocks for creating distributed, fault-tolerant, highly-available apps at scale.
This talk will examine case studies of Mesos uses in production at scale: ranging from Twitter (100% on prem) to Airbnb (100% cloud), plus MediaCrossing, Categorize, HubSpot, etc. How have these organizations leveraged Mesos to build better, more scalable and efficient distributed apps? Lessons from the Mesos developer community show that one can port an existing framework with a wrapper in approximately 100 line of code. Moreover, an important lesson from Spark is that based on “data center OS” building blocks one can rewrite a distributed system much like Hadoop to be 100x faster within a relatively small amount of source code.
These case studies illustrate the obvious benefits over prior approaches based on virtualization: scalability, elasticity, fault-tolerance, high availability, improved utilization rates, etc. Less obvious outcomes also include: reduced time for engineers to ramp-up new services at scale; reduced latency between batch and services, enabling new high-ROI use cases; and enabling dev/test apps to run on a production cluster without disrupting operations.
Introduction of mesos persistent storageZhou Weitao
1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
As a company starts dealing with large amounts of data, operation engineers are challenged with managing the influx of information while ensuring the resilience of data. Hadoop HDFS, Mesos and Spark help reduce issues with a scheduler that allows data cluster resources to be shared. It provides a common ground where data scientists and engineers can meet, develop high performance data processing applications and deploy their own tools.
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDropsolid
In this presentation I explain using video examples how kubernetes works and how this can be used to host your Drupal 7 or 8 site. There are obviously also gotcha's and I'd like to warn you to not use this in production until you've verified it
Anatomy of the libvirt virtualization library
http://www.ibm.com/developerworks/library/l-libvirt/
libvirt
http://libvirt.org/index.html
Scheduling
http://docs.openstack.org/icehouse/config-reference/content/section_compute-scheduler.html
Openstack Zoning – Region/Availability Zone/Host Aggregate
https://kimizhang.wordpress.com/2013/08/26/openstack-zoning-regionavailability-zonehost-aggregate/
Availability Zones and Host Aggregates in OpenStack Compute (Nova)
http://blog.russellbryant.net/2013/05/21/availability-zones-and-host-aggregates-in-openstack-compute-nova/
An Introduction to Droplet Metadata
https://www.digitalocean.com/community/tutorials/an-introduction-to-droplet-metadata
HOW WE USE CLOUDINIT IN OPENSTACK HEAT
http://sdake.io/2013/03/03/how-we-use-cloudinit-in-openstack-heat/
How to inject file/meta/ssh key/root password/userdata/config drive to a VM during nova boot
https://kimizhang.wordpress.com/2014/03/18/how-to-inject-filemetassh-keyroot-passworduserdataconfig-drive-to-a-vm-during-nova-boot/
Cloud-init
https://cloudinit.readthedocs.org/en/latest/
This presentation describes how hortonworks is delivering Hadoop on Docker for a cloud-agnostic deployment approach which presented in Cisco Live 2015.
Deploying Docker Containers at Scale with Mesos and MarathonDiscover Pinterest
Connor Doyle from Mesosphere.
Deploying Docker Containers at Scale with Mesos and Marathon
The norm these days is to operate apps at web scale. But that’s out of reach for most companies. Deploying Docker containers with Mesos and Marathon makes it easier. See how they help deploy and manage Docker containers at scale and how the Mesos cluster scheduler builds highly-available, fault-tolerant web scale apps.
Real-Time Distributed and Reactive Systems with Apache Kafka and Apache AccumuloJoe Stein
In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...DataStax
Traditionally, machines were statically partitioned across the different services at Uber. In an effort to increase the machine utilization, Uber has recently started transitioning most of its services, including the storage services, to run on top of Mesos. This presentation will describe the initial experience building and operating a framework for running Cassandra on top of Mesos running across multiple datacenters at Uber. This framework automates several Cassandra operations such as node repairs, addition of new nodes and backup/restore. It improves efficiency by co-locating CPU-intensive services as well as multiple Cassandra nodes on the same Mesos agent. It handles failure and restart of Mesos agents by using persistent volumes and dynamic reservations. This talk includes statistics about the number of Cassandra clusters in production, time taken to start a new cluster, add a new node, detect a node failure; and the observed Cassandra query throughput and latency.
About the Speaker
Abhishek Verma Software Engineer, Uber
Dr. Abhishek Verma is currently working on running Cassandra on top of Mesos at Uber. Prior to this, he worked on BorgMaster at Google and was the first author of the Borg paper published in Eurosys 2015. He received an MS in 2010 and a PhD in 2012 in Computer Science from the University of Illinois at Urbana-Champaign, during which he authored more than 20 publications in conferences, journals and books and presented tens of talks.
Adobe has packaged HBase in Docker containers and uses Marathon and Mesos to schedule them—allowing them to decouple the HBase RegionServer from the host, express resource requirements declaratively, and open the door for unassisted real-time deployments, elastic (up and down) real-time scalability, and more.
Apache Mesos is the first open source cluster manager that handles the workload efficiently in a distributed environment through dynamic resource sharing and isolation.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
O'Reilly Media - Strata SC 2014
Apache Mesos is an open source cluster manager that provides efficient resource isolation for distributed frameworks—similar to Google’s “Borg” and “Omega” projects for warehouse scale computing. It is based on isolation features in the modern kernel: “cgroups” in Linux, “zones” in Solaris.
Google’s “Omega” research paper shows that while 80% of the jobs on a given cluster may be batch (e.g., MapReduce), 55-60% of cluster resources go toward services. The batch jobs on a cluster are the easy part—services are much more complex to schedule efficiently. However by mixing workloads, the overall problem of scheduling resources can be greatly improved.
Given the use of Mesos as the kernel for a “data center OS”, two additional open source components Chronos (like Unix “cron”) and Marathon (like Unix “init.d”) serve as the building blocks for creating distributed, fault-tolerant, highly-available apps at scale.
This talk will examine case studies of Mesos uses in production at scale: ranging from Twitter (100% on prem) to Airbnb (100% cloud), plus MediaCrossing, Categorize, HubSpot, etc. How have these organizations leveraged Mesos to build better, more scalable and efficient distributed apps? Lessons from the Mesos developer community show that one can port an existing framework with a wrapper in approximately 100 line of code. Moreover, an important lesson from Spark is that based on “data center OS” building blocks one can rewrite a distributed system much like Hadoop to be 100x faster within a relatively small amount of source code.
These case studies illustrate the obvious benefits over prior approaches based on virtualization: scalability, elasticity, fault-tolerance, high availability, improved utilization rates, etc. Less obvious outcomes also include: reduced time for engineers to ramp-up new services at scale; reduced latency between batch and services, enabling new high-ROI use cases; and enabling dev/test apps to run on a production cluster without disrupting operations.
Introduction of mesos persistent storageZhou Weitao
1. How to run stateful service against current Mesos-0.22
2. Disk isolation and monitoring
3. Persistent Volumes
4. Dynamic Reservations
5. What we can contribute for Mesos persistent storage
As a company starts dealing with large amounts of data, operation engineers are challenged with managing the influx of information while ensuring the resilience of data. Hadoop HDFS, Mesos and Spark help reduce issues with a scheduler that allows data cluster resources to be shared. It provides a common ground where data scientists and engineers can meet, develop high performance data processing applications and deploy their own tools.
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDropsolid
In this presentation I explain using video examples how kubernetes works and how this can be used to host your Drupal 7 or 8 site. There are obviously also gotcha's and I'd like to warn you to not use this in production until you've verified it
Anatomy of the libvirt virtualization library
http://www.ibm.com/developerworks/library/l-libvirt/
libvirt
http://libvirt.org/index.html
Scheduling
http://docs.openstack.org/icehouse/config-reference/content/section_compute-scheduler.html
Openstack Zoning – Region/Availability Zone/Host Aggregate
https://kimizhang.wordpress.com/2013/08/26/openstack-zoning-regionavailability-zonehost-aggregate/
Availability Zones and Host Aggregates in OpenStack Compute (Nova)
http://blog.russellbryant.net/2013/05/21/availability-zones-and-host-aggregates-in-openstack-compute-nova/
An Introduction to Droplet Metadata
https://www.digitalocean.com/community/tutorials/an-introduction-to-droplet-metadata
HOW WE USE CLOUDINIT IN OPENSTACK HEAT
http://sdake.io/2013/03/03/how-we-use-cloudinit-in-openstack-heat/
How to inject file/meta/ssh key/root password/userdata/config drive to a VM during nova boot
https://kimizhang.wordpress.com/2014/03/18/how-to-inject-filemetassh-keyroot-passworduserdataconfig-drive-to-a-vm-during-nova-boot/
Cloud-init
https://cloudinit.readthedocs.org/en/latest/
This presentation describes how hortonworks is delivering Hadoop on Docker for a cloud-agnostic deployment approach which presented in Cisco Live 2015.
Big Data in Container; Hadoop Spark in Docker and MesosHeiko Loewe
3 examples for Big Data analytics containerized:
1. The installation with Docker and Weave for small and medium,
2. Hadoop on Mesos w/ Appache Myriad
3. Spark on Mesos
Lessons Learned Running Hadoop and Spark in Docker ContainersBlueData, Inc.
Many initiatives for running applications inside containers have been scoped to run on a single host. Using Docker containers for large-scale production environments poses interesting challenges, especially when deploying distributed big data applications like Apache Hadoop and Apache Spark. This session at Strata + Hadoop World in New York City (September 2016) explores various solutions and tips to address the challenges encountered while deploying multi-node Hadoop and Spark production workloads using Docker containers.
Some of these challenges include container life-cycle management, smart scheduling for optimal resource utilization, network configuration and security, and performance. BlueData is "all in” on Docker containers—with a specific focus on big data applications. BlueData has learned firsthand how to address these challenges for Fortune 500 enterprises and government organizations that want to deploy big data workloads using Docker.
This session by Thomas Phelan, co-founder and chief architect at BlueData, discusses how to securely network Docker containers across multiple hosts and discusses ways to achieve high availability across distributed big data applications and hosts in your data center. Since we’re talking about very large volumes of data, performance is a key factor, so Thomas shares some of the storage options implemented at BlueData to achieve near bare-metal I/O performance for Hadoop and Spark using Docker as well as lessons learned and some tips and tricks on how to Dockerize your big data applications in a reliable, scalable, and high-performance environment.
http://conferences.oreilly.com/strata/hadoop-big-data-ny/public/schedule/detail/52042
Using Clojure for Sentiment Analysis of the TwittersphereThoughtworks
Sentiment analysis of online social networks, such as Twitter, is a rapidly growing research area with an increasing number of applications such as: informing disaster response; tracking spread of contagious disease; early forecast of civil unrest. This talk will present an approach for building scalable social media analysis platforms in Clojure
Accumulo Summit 2014: Four Orders of Magnitude: Running Large Scale Accumulo ...Accumulo Summit
Speaker: Aaron Cordova
Most users of Accumulo start developing applications on a single machine and will to scale to up to four orders of magnitude more machines without having to rewrite. In this talk we describe techniques for designing applications for scale, planning a large scale cluster, tuning the cluster for high speed ingest, dealing with a large amount of data over time, and unique features of Accumulo for taking advantage of up to ten thousand nodes in a single instance. We also include the largest public metrics gathered on Accumulo clusters to date and include a discussion of overcoming practical limits to scaling in the future.
Accumulo Summit 2015: Tracing in Accumulo and HDFS [Internals]Accumulo Summit
Talk Abstract
Having the ability to diagnose and understand what is happening in distributed systems is essential. Tracing is one mechanism that enables analysis of operations in distributed systems by dividing each operation into a tree of measurable sub-tasks. HDFS, Accumulo, and HBase are now converging on a single tracing system utilizing HTrace, an open source tracing instrumentation library that recently became a new Apache Incubator project. This talk will cover tracing fundamentals, the instrumentation that has been added to HDFS to support tracing, and changes that have been made in Accumulo's tracing. It will also cover options for collecting and visualizing traces, as well as the current status of the HTrace podling.
Speaker
Billie Rinaldi
Sr. Member of Technical Staff, Hortonworks
Billie Rinaldi is a Senior Member of Technical Staff at Hortonworks, Inc., currently prototyping new features related to application monitoring and deployment in the Apache Hadoop ecosystem. Prior to August 2012, Billie engaged in big data science and research at the National Security Agency. Since 2008, she has been providing technical leadership regarding the software that is now Apache Accumulo. Billie is the VP of Apache Accumulo, the Accumulo Project Management Committee Chair, and a member of the Apache Software Foundation. She holds a Ph.D. in applied mathematics from Rensselaer Polytechnic Institute.
Accumulo Summit 2016: Accumulo in the EnterpriseAccumulo Summit
Many organizations are looking to Hadoop clusters in order to store and manage an ever-increasing amount of data. As the volume and variety of data in these systems grows, administrators are being confronted with more information, from more sources, than they have ever seen concentrated in a single place. The responsibility for securing all this data can be daunting to an administrator, even intimidating. Could the answer lie in Accumulo?
Conventional approaches to data security usually do not suffice for this scenario. They are often coarse-grained, applying only at the file or table level. In a world where arbitrary compute tasks can be pushed into the cluster, defining a security perimeter is difficult or impossible. On the other hand, relegating access policy enforcement to the application level instead of the database level ultimately invites a security disaster.
This is the world that Chief Security Officers, Chief Information Officers, and Chief Data Officers live in, and the problem of security for big data is the single biggest impediment to delivering a Hadoop-based solution in the enterprise’s production network. Numerous organizations have implemented Hadoop as a pilot, but find themselves blocked by similar considerations when the time to move into production:
• How do you implement fine-grained access controls in a Hadoop system?
• What about encryption at rest? Encryption in motion?
• How will this tie into our identity infrastructure?
• How will this fit into our operational workflow?
This keynote will explore the ways in which Apache Accumulo is uniquely positioned to mitigate or resolve problems around access control and security for big data, thus enabling Hadoop clusters to move from pilot to production.
– Speaker –
Russ Weeks
Software Architect, PHEMI Systems
Russ Weeks is a Software Architect at PHEMI. Prior to joining PHEMI Systems, Russ worked in the network management groups at Ericsson and Cray Supercomputers, where he discovered a passion for distributed data structures and algorithms. PHEMI Systems is a Vancouver, BC-based startup focused on the storage, retention and governance of structured and unstructured data.
— More Information —
For more information see http://www.accumulosummit.com/
Aaron Cordova outlines how Accumulo helps provide the essential features of a "Data Lake": a system in which all types of data from all sources can be imported, secured, analyzed, and delivered to decision makers.
Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?Accumulo Summit
Speaker: Mike Drob
Apache Accumulo has long held a reputation for enabling high-throughput operations in write-heavy workloads. In this talk, we use the Yahoo! Cloud Serving Benchmark (YCSB) to put real numbers on Accumulo performance. We then compare these numbers to previous versions, to other databases, and wrap up with a discussion of parameters that can be tweaked to improve them.
Accumulo Summit 2016: Embedding Authenticated Data Structures in AccumuloAccumulo Summit
Accumulo requires its users to trust each Accumulo installation with their data — a malicious server or user could easily compromise critical data or learn secrets they are not authorized to access. One particular threat is a malicious Accumulo server compromising data’s integrity, by tampering with query results and returning forged, modified, or incomplete results to a user. In prior work, we implemented a lightweight client-side tool to protect against this kind of threat. We now present improvements to this tool that handle a wider range of attacks by a malicious server and reduce overhead for the client.
In our solution, Accumulo clients use Authenticated Data Structures (ADSs) to verify their range queries’ integrity. ADS metadata is stored in Accumulo, so that after each query, the server must construct a proof that the query has not been tampered with. We use Accumulo iterators to compute these proofs on the server without requiring an unnecessary computational burden from the client. We will present our approach to adding ADSs to Accumulo, our schema for storing the ADS metadata, and opportunities for future work in efficiency and expressiveness.
– Speaker –
Leo St. Amour
Military Fellow, MIT Lincoln Laboratory
Leo St. Amour is a master’s student at Northeastern University and a military fellow at MIT Lincoln Laboratory. He graduated from the United States Military Academy in May 2015, where he worked on a TLS library with enhanced usability and security. In addition to his work on TLS and Accumulo, he is currently working on binary analysis, with a focus on discovering and hardening security properties.
— More Information —
For more information see http://www.accumulosummit.com/
Accumulo Summit 2015: Accumulo In-Depth: Building Bulk Ingest [Sponsored]Accumulo Summit
Talk Abstract
Bulk ingest enables Accumulo to import externally-prepared data into existing tables. Unlike ingest via batch writers, much of the work of organizing data can be left to external processing frameworks such as MapReduce and scaled independently of the Accumulo cluster itself. This reduces the work required of the tablet servers to support ingest, freeing resources to support other operations.
Under the hood, bulk ingest involves a number a moving parts and accounting for a variety of failure scenarios. This talk covers the components of the bulk ingest process in-depth and describes past, current and future implementations of this capability. Attendees will leave this session with an understanding of bulk ingest that will enable troubleshooting, capacity estimation and performance management.
Speaker
Eric Newton
Senior Software Developer, SWComplete
Eric Newton has been a programmer for over 30 years, and has worked on Accumulo since 2009. He has been an open-source contributor and consumer since 1988. Through the years, his distributed communications systems work has included Air Traffic Control, Systems Monitoring and Databases. Eric has started 3 of his own companies and helped several other businesses start.
Running and managing large scale applications with microservices architectures is difficult and often requires operating complex container management infrastructure. Amazon EC2 Container Service (ECS) is a highly scalable, high performance service for running and managing Docker applications. In this webinar, we will walk through a number of patterns and tools used by our customers to run their applications on Amazon ECS. We will show you how to set up, manage and scale your Amazon ECS resources, keep them secure and deploy your applications to an Amazon ECS cluster. We will also provide best practices for monitoring, logging and service discovery.
Learning Objectives:
• Learn how to set up and manage Amazon ECS for production applications
• Learn how to schedule containers on production clusters using Amazon ECS
Who Should Attend:
•Developers, DevOps, Sys Admin
Speaker: Jacob Aae Mikkelsen
Once you have successfully developped your application in Grails, Ratpack or your other favorite framework, you would like to see it deployed as fast and painless as possible, right?
This talk will cover some of the supporting cast members of a succesful modern infrastructure, that developers can understand and use efficiently, and with good DevOps practices.
Key elements are
Docker
Infrastructure as Code
Container Orchestration
The demo-goods will hopefully be on our side, as this talk includes quite some live demos!
Architecting world class azure resource manager templatesMarc Mercuri
This session will provide details on consumption scenarios, architecture, and implementation patterns identified during our design sessions and real-world template implementations with customers. Far from academic, these are proven practices informed by the development of ARM templates for 12 of the top Linux-based OSS technologies, including: Apache Kafka, Apache Spark, Cloudera, Couchbase, Hortonworks HDP, DataStax Enterprise powered by Apache Cassandra, Elasticsearch, Jenkins, MongoDB, Nagios, PostgreSQL, Redis, and Nagios. The majority of these templates were developed with a well-known vendor of a given distribution and influenced by the requirements of Microsoft’s enterprise and SI customers during recent projects.
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and moreDropsolid
Given at DrupalJam 2015 - Netherlands.
This presentation explains some of the fundamental issues you have to overcome when designing software for distributed systems that can fail. Also called "Cloud" in other terminologies. The presentation uses AWS components to explain these fundamentals and uses Drupal as the example application. The example is by no means perfect, but gives you a good idea how to design your system from scratch.
Technologies used:
Cloudformation
EC2 Instances
RDS MySQL Database
Elastic Load Balancer
ElastiCache (Memcache)
Example can be found here:
https://gist.github.com/nickveenhof/601c5dc1b76ff26896bf
Take note that the example does not include components such as VPC for simplicity, but it is highly recommended to add this.
(APP313) NEW LAUNCH: Amazon EC2 Container Service in Action | AWS re:Invent 2014Amazon Web Services
Container technology, particularly Docker, is all the rage these days. At AWS, our customers have been running Linux containers at scale for several years, and we are increasingly seeing customers adopt Docker, especially as they build loosely coupled distributed applications. However, to do so they have to run their own cluster management solutions, deal with configuration management, and manage their containers and associated metadata. We believe that those capabilities should be a core building block technology, just like EC2. Today, we are announcing the preview of Amazon EC2 Container Service, a new AWS service that makes is easy to run and manage Docker-enabled distributed applications using powerful APIs that allow you to launch and stop containers, get complete cluster state information, and manage linked containers. In this session we will discuss why we built the EC2 Container Service, some of the core concepts, and walk you through how you can use the service for your applications.
(BDT401) Big Data Orchestra - Harmony within Data Analysis Tools | AWS re:Inv...Amazon Web Services
Yes, you can build a data analytics solution with a relational database, but should you? What about scalability? What about flexibility? What about cost? In this session, we demonstrate how to build a real world solution for location-based data analytics, with the combination of Amazon Kinesis, Amazon DynamoDB, Amazon Redshift, Amazon CloudSearch, and Amazon EMR. We discuss how to integrate these services to create a robust solution in terms of security, simplicity, speed, and low cost.
Apache Ambari at the Apache Big Data Conference in Miami on May 18, 2017
presented by Alejandro Fernandez
Using Apache Ambari for enterprises with Blueprints, Custom Services, Stack Advisor, Kerberos, Large Scale, Rolling/Express Upgrades, Alerts, Metrics, and Log Search.
Burn down the silos! Helping dev and ops gel on high availability websitesLindsay Holmwood
HA websites are where the rubber meets the road - at 200km/h. Traditional separation of dev and ops just doesn't cut it.
Everything is related to everything. Code relies on performant and resilient infrastructure, but highly performant infrastructure will only get a poorly written application so far. Worse still, root cause analysis in HA sites will more often than not identify problems that don't clearly belong to either devs or ops.
The two options are collaborate or die.
This talk will introduce 3 core principles for improving collaboration between operations and development teams: consistency, repeatability, and visibility. These principles will be investigated with real world case studies and associated technologies audience members can start using now. In particular, there will be a focus on:
- fast provisioning of test environments with configuration management
- reliable and repeatable automated deployments
- application and infrastructure visibility with statistics collection, logging, and visualisation
SMACK Stack 1.0 has been Spark, Mesos, Akka, Cassandra and Kafka working into different cohesive systems delivering different solutions for different use cases. Haven't heard about it before? Oh man! Where have you been? https://www.google.com/search?q=smack+stack+1.0
SMACK Stack 1.1 we go a step further Streaming, Mesos, Analytics, Cassandra and Kafka and Joe Stein will walk through in detail some of the different viable options for Streaming and Analytics with Mesos, Kafka and Cassandra.
Real-Time Log Analysis with Apache Mesos, Kafka and CassandraJoe Stein
Slides for our solution we developed for using Mesos, Docker, Kafka, Spark, Cassandra and Solr (DataStax Enterprise Edition) all developed in Go for doing realtime log analysis at scale. Many organizations either need or want log analysis in real time where you can see within a second what is happening within your entire infrastructure. Today, with the hardware available and software systems we have in place, you can develop, build and use as a service these solutions.
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.
Developing Realtime Data Pipelines With Apache KafkaJoe Stein
Developing Realtime Data Pipelines With Apache Kafka. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Real-time streaming and data pipelines with Apache KafkaJoe Stein
Get up and running quickly with Apache Kafka http://kafka.apache.org/
* Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
* Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
* Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
* Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Detail behind the Apache Cassandra 2.0 release and what is new in it including Lightweight Transactions (compare and swap) Eager retries, Improved compaction, Triggers (experimental) and more!
• CQL cursors
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
2. Joe Stein
• Developer, Architect & Technologist
• Founder & Principal Consultant => Big Data Open Source Security LLC - http://stealth.ly
Big Data Open Source Security LLC provides professional services and product solutions for the collection,
storage, transfer, real-time analytics, batch processing and reporting for complex data streams, data sets and
distributed systems. BDOSS is all about the "glue" and helping companies to not only figure out what Big Data
Infrastructure Components to use but also how to change their existing (or build new) systems to work with
them.
• Apache Kafka Committer & PMC member
• Blog & Podcast - http://allthingshadoop.com
• Twitter @allthingshadoop
3. Overview
● Quick Intro To Mesos
● Architecture
● Frameworks
● Roles, Resources, Attributes
● Marathon
● Constraints
● Creating your own “executor” for Marathon
● Persisting Data
● Kafka, HDFS, Accumulo, And More!!
4.
5. Origins
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
http://static.usenix.org/event/nsdi11/tech/full_papers/Hindman_new.pdf
Datacenter Management with Apache Mesos
https://www.youtube.com/watch?v=YB1VW0LKzJ4
Omega: flexible, scalable schedulers for large compute clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf
2011 GAFS Omega John Wilkes
https://www.youtube.com/watch?v=0ZFMlO98Jkc&feature=youtu.be
14. Apache Mesos
● Scalability to 10,000s of nodes
● Fault-tolerant replicated master and slaves using ZooKeeper
● Support for Docker containers
● Native isolation between tasks with Linux Containers
● Multi-resource scheduling (memory, CPU, disk, and ports)
● Java, Python and C++ APIs for developing new parallel applications
● Web UI for viewing cluster state
15.
16.
17. Sample Frameworks
C++ - https://github.com/apache/mesos/tree/master/src/examples
Java - https://github.com/apache/mesos/tree/master/src/examples/java
Python - https://github.com/apache/mesos/tree/master/src/examples/python
Scala - https://github.com/mesosphere/scala-sbt-mesos-framework.g8
Go - https://github.com/mesosphere/mesos-go
18. Resources & Attributes
The Mesos system has two basic methods to describe the
slaves that comprise a cluster. One of these is managed by the
Mesos master, the other is simply passed onwards to the
frameworks using the cluster.
Attributes
The attributes are simply key value string pairs that Mesos passes along when it sends offers to frameworks.
attributes : attribute ( ";" attribute )*
attribute : labelString ":" ( labelString | "," )+
19. Resources
The Mesos system can manage 3 different types of resources: scalars, ranges, and sets. These are used to represent
the different resources that a Mesos slave has to offer. For example, a scalar resource type could be used to represent
the amount of memory on a slave. Each resource is identified by a key string.
resources : resource ( ";" resource )*
resource : key ":" ( scalar | range | set )
key : labelString ( "(" resourceRole ")" )?
scalar : floatValue
range : "[" rangeValue ( "," rangeValue )* "]"
rangeValue : scalar "-" scalar
set : "{" labelString ( "," labelString )* "}"
resourceRole : labelString | "*"
labelString : [a-zA-Z0-9_/.-]
floatValue : ( intValue ( "." intValue )? ) | ...
intValue : [0-9]+
20. Predefined Uses & Conventions
The Mesos master has a few resources that it pre-defines in how it handles them. At the current time, this list consist of:
● cpu
● mem
● disk
● ports
In particular, a slave without cpu and mem resources will never have its resources advertised to any frameworks. Also, the Master’s
user interface interprets the scalars inmem and disk in terms of MB. IE: the value 15000 is displayed as 14.65GB.
21. Examples
Here are some examples for configuring the Mesos slaves.
--resources='cpu:24;mem:24576;disk:409600;ports:[21000-24000];bugs:{a,b,c}'
--attributes='rack:abc;zone:west;os:centos5,full'
In this case, we have three different types of resources, scalars, a range, and a set. They are called cpu, mem, disk, and the range
type is ports.
● scalar called cpu, with the value 24
● scalar called mem, with the value 24576
● scalar called disk, with the value 409600
● range called ports, with values 21000 through 24000 (inclusive)
● set called bugs, with the values a, b and c
In the case of attributes, we end up with three attributes:
● rack with value abc
22. Roles
Total consumable resources per slave, in the form 'name(role):value;name(role):value...'. This value can be set to
limit resources per role, or to overstate the number of resources that are available to the slave.
--resources="cpus(*):8; mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
--resources="cpus(prod):8; cpus(stage):2 mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
All * roles will be detected, so you can specify only the resources that are not all roles (*). --
resources="cpus(prod):8; cpus(stage)"
Frameworks bind a specific roles or any. A default roll (instead of *) can also be configured.
Roles can be used to isolate and segregate frameworks.
24. Constraints
Constraints control where apps run to allow optimizing for fault tolerance or locality. Constraints are made up of three parts: a field
name, an operator, and an optional parameter. The field can be the slave hostname or any Mesos slave attribute.
Fields
Hostname field
hostname field matches the slave hostnames, see UNIQUE operator for usage example.
hostname field supports all operators of Marathon.
Attribute field
If the field name is none of the above, it will be treated as a Mesos slave attribute. Mesos slave attribute is a way to tag a slave
node, see mesos-slave --help to learn how to set the attributes.
25. Unique
UNIQUE tells Marathon to enforce uniqueness of the attribute across all of an app's tasks. For example the following constraint
ensures that there is only one app task running on each host:
via the Marathon gem:
$ marathon start -i sleep -C 'sleep 60' -n 3 --constraint hostname:UNIQUE
via curl:
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-unique",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["hostname", "UNIQUE"]]
}'
26. CLUSTER allows you to run all of your app's tasks on slaves that share a certain attribute.
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-cluster",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["rack_id", "CLUSTER", "rack-1"]]
}'
You can also use this attribute to tie an application to a specific node by using the hostname property:
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-cluster",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["hostname", "CLUSTER", "a.specific.node.com"]]
}'
Cluster
27. Group By
GROUP_BY can be used to distribute tasks evenly across racks or datacenters for high availability.
via the Marathon gem:
$ marathon start -i sleep -C 'sleep 60' -n 3 --constraint rack_id:GROUP_BY
via curl:
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-group-by",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["rack_id", "GROUP_BY"]]
}'
Optionally, you can specify a minimum number of groups to try and achieve.
28. Like
LIKE accepts a regular expression as parameter, and allows you to run your tasks only on the slaves whose field values match the
regular expression.
via the Marathon gem:
$ marathon start -i sleep -C 'sleep 60' -n 3 --constraint rack_id:LIKE:rack-[1-3]
via curl:
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-group-by",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["rack_id", "LIKE", "rack-[1-3]"]]
}'
29. Unlike
Just like LIKE operator, but only run tasks on slaves whose field values don't match the regular expression.
via the Marathon gem:
$ marathon start -i sleep -C 'sleep 60' -n 3 --constraint rack_id:UNLIKE:rack-[7-9]
via curl:
$ curl -X POST -H "Content-type: application/json" localhost:8080/v1/apps/start -d '{
"id": "sleep-group-by",
"cmd": "sleep 60",
"instances": 3,
"constraints": [["rack_id", "UNLIKE", "rack-[7-9]"]]
}'
30. Persisting Data
● Write data outside the sandbox, the other kids will not mind.
○ /var/lib/data/<tag>/<app>
● Careful about starvation, use the constraints and roles to manage this.
● Future improvements https://issues.apache.org/jira/browse/MESOS-2018 This is a
feature to provide better support for running stateful services on Mesos such as
HDFS (Distributed Filesystem), Cassandra (Distributed Database), or MySQL (Local
Database). Current resource reservations (henceforth called "static" reservations)
are statically determined by the slave operator at slave start time, and individual
frameworks have no authority to reserve resources themselves. Dynamic
reservations allow a framework to dynamically/lazily reserve offered resources, such
that those resources will only be re-offered to the same framework (or other
frameworks with the same role). This is especially useful if the framework's task
stored some state on the slave, and needs a guaranteed set of resources reserved
so that it can re-launch a task on the same slave to recover that state.
32. yourScript
#think of it as a distributed application launching in the cloud
HOST=$1
PORT0=$2
PORT1=$3
#talk to zookeeper
#call marathon rest api
#spawn another process, they are all in your cgroup =8^) woot woot
33. Scripts for running Kafka on Marathon
https://github.com/brndnmtthws/kafka-on-marathon
● initialize, zk_connect
● get_missing_brokers
● become_broker
● run
loop do
puts "About to run:"
puts env, cmd
GC.start # clean up memory
system env, cmd
finished = Time.now.to_f
if finished - last_finished < 120
# If the process was running for less than 2 minutes, abort. We're
# probably 'bouncing'. Occasional restarts are okay, but not continuous restarting.
$stderr.puts "Kafka exited too soon!"
exit 1
end
last_finished = finished
34. The future of Kafka on Mesos
● A Kafka framework https://github.com/stealthly/kafka-mesos
35. ● Go the script option, use Marathon, it works!!!
● Pick the language you like (bash, python, ruby, go) and work a way to pass vars through.
○ One script for your journal nodes
○ One script for your namenodes
○ One script for your datanodes
● Use constraints
● Store local, seperate based on your marathon /var/lib/<tag>/hdfs/...
● HDFS Framework in progress https://github.com/mesosphere/hdfs
● Accumulo on Mesos, Go the script option, it works!!!
● Use service discovery from Marathon https://mesosphere.github.io/marathon/docs/service-
discovery-load-balancing.html
● Focus on variable substitutions
○ “env”:{ “HDFS”,”http://localhost:$HDFS_APP_PORT”}'
HDFS & Accumulo on Mesos