Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software

•

2 likes•506 views

Presentation made at ApacheCon: Core Europe 2015 Container technology are being evaluated by software developers and administrators with a great deal of interest. Developers want to focus on what they do best: Creating and coding new applications. That shouldn't have to change just because they need to deploy an application to a different environment. Administrators want the environment to stay reliable and stable, keeping changes at a minimum. By following a strategy that embraces good Architecture, use of Containers, DevOps philosophy, Apache Mesos and a Cloud based environment, developers and operators can create, consume and collaborate on the infrastructure configuration over the time, deploy Java EE applications and test your application infrastructure consistently regardless of the stage of the development life cycle.

Containers, DevOps, Apache Mesos and Cloud
Reshaping how we develop and delivery software
https://twitter.com/marceloancelmo
Marcelo Sousa Ancelmo
marcelo.souzaancelmo@ig.com
marceloancelmo@gmail.com
marceloancelmo
http://www.slideshare.net/marceloancelmo
https://www.linkedin.com/in/marceloancelmo

Decompose the applications into self-contained services
Be stateless (As much as possible)
NFR's are king, be aware on latency, throughput, etc...
Expect an impact on your application architecture
Fault tolerant
Scalable
Elastic
Multi-tenant

“There are 17 million shipping containers in existence, packed with every
physical good imaginable. Every single one of them can be loaded onto the
same boats, by the same cranes, in the same facilities, and sent anywhere
in the World with incredible efficiency. It is embarrassing to think that a 30
ton shipment of coffee can safely travel half-way across the World in less
time than it takes a software team to deliver its code from one datacenter to
another sitting 10 miles away.”
5th
principle of Open Container Initiative -
https://github.com/opencontainers/specs/blob/master/principles.md
Why Use Containers?

And potentially can use all your computing power

Introducing Apache Mesos
Program against your datacenter like it’s a single pool of resources
“Apache Mesos abstracts CPU, memory, storage, and other compute resources
away from machines (physical or virtual), enabling fault-tolerant and elastic
distributed systems to easily be built and run effectively.”
http://mesos.apache.org/
Used by Twitter, AirBNB, eBay, Netflix, Apple and many others
http://mesos.apache.org/documentation/latest/powered-by-mesos/

Apache Mesos Overview
Batch Processing Long Running Services

Starting a new node for the application
300 users300 users

Continuous Delivery
Blue-Green Deployments

Test and Evaluate the new version
Production
80%
Production
20%

Putting into perspective
“If a Docker application is a Lego brick, Kubernetes would be like a kit for
building the Millennium Falcon and the Mesos cluster would be like a whole
Star Wars universe made of Legos.” ~ Solomon Hykes

Cloud + Microservices + Containers + CD
Is this the Perfect Storm?

Thank You
Merci
Grazie
Gracias
Obrigado
Danke
Japanese
English
French
Russian
German
Italian
Spanish
Brazilian Portuguese
Arabic
Traditional Chinese
Simplified Chinese
Hindi
Tamil
Thai
Korean
Dziękuję
Polish
Marcelo Sousa Ancelmo
marcelo.souzaancelmo@ig.com
marceloancelmo@gmail.com

Real-life Machine Learning (ML) workloads typically require more than training and predicting: data often needs to be pre-processed and post-processed, sometimes in multiple steps. Thus, developers and data scientists have to train and deploy not just a single algorithm, but a sequence of algorithms that will collaborate in delivering predictions from raw data. In this session, we’ll first show you how to use Apache Spark MLlib to build ML pipelines, and we’ll discuss scaling options when datasets grow huge. We’ll then show how to how implement inference pipelines on Amazon SageMaker, using Apache Spark, Scikit-learn, as well as ML algorithms implemented by Amazon.

WebAPIs & Apps - Mozilla London

Robert Nyman

Firefox OS allows developers to build apps and customize the user interface using HTML5, CSS, JavaScript, and Web APIs. Apps are developed as regular web apps, with the addition of an app manifest file. Apps can be published to the Firefox Marketplace or installed directly. The Firefox OS platform provides both regular web APIs as well as more privileged APIs that require permissions. Future plans include additional APIs for features like spell checking, peer-to-peer connectivity, and WebRTC. Developers can get help through IRC channels or mailing lists and try things out using emulators, boilerplate apps, and the Firefox OS developer preview.

Hadoop Introduction

sheetal sharma

Introduction to Apache Kafka and why it matters - Madrid

Paolo Castagna

This document provides an introduction to Apache Kafka and discusses why it is an important distributed streaming platform. It outlines how Kafka can be used to handle streaming data flows in a reliable and scalable way. It also describes the various Apache Kafka APIs including Kafka Connect, Streams API, and KSQL that allow organizations to integrate Kafka with other systems and build stream processing applications.

Hello, kafka! (an introduction to apache kafka)

Timothy Spann

How Apache Pulsar Helps Tencent Process Tens of Billions of Transactions Effi...

StreamNative

As the largest provider of Internet products and services in China, Tencent serves billions of users and over a million merchants—and these numbers are growing fast! Tencent’s enterprises generate a huge volume of financial transactions, placing a tremendous load on their billing service, which processes hundreds of millions of dollars in revenue each day. Because Tencent had been unable to scale its current billing service to handle its rapidly growing business, the possibility of data loss had become an escalating concern. To ensure data consistency, the company decided to redesign its system’s transaction processing pipeline. After evaluating the pros and cons of several messaging systems, Tencent chose to implement its billing service using Apache Pulsar. As a result, Tencent can now run their billing service on a very large scale with virtually no data loss. In this talk, Ningguo Chen, the Chief Architect from Tencent Billing will share their journal of adopting Pulsar in their core transaction processing engine to process tens of billions of events every day. He will also discuss the problems they have encountered in using Pulsar and the improvements they have made for meeting their scale.

Kafka clients and emitters

Edgar Domingues

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

StreamNative

OVHcloud is the biggest European cloud provider. From dedicated servers to Managed Kubernetes, from VMware® based Hosted Private Cloud to OpenStack-based Public Cloud, we have over 1.4 million customers worldwide. Internally, we have been running Apache Kafka for years, and despite all the skills obtained operating multiples clusters with millions of messages per second, we decided to shift and build the foundation of our 'topic-as-a-service' product called ioStream on Apache Pulsar. In this talk, you will have the insights of why we decided to use Apache Pulsar instead of Apache Kafka as the core of ioStream. We will tell you our journey to use Apache Pulsar, from our deployments to the management, what did work and what did not.

(Stephane Maarek, DataCumulus) Kafka Summit SF 2018 Security in Kafka is a cornerstone of true enterprise production-ready deployment: It enables companies to control access to the cluster and limit risks in data corruption and unwanted operations. Understanding how to use security in Kafka and exploiting its capabilities can be complex, especially as the documentation that is available is aimed at people with substantial existing knowledge on the matter. This talk will be delivered in a “hero journey” fashion, tracing the experience of an engineer with basic understanding of Kafka who is tasked with securing a Kafka cluster. Along the way, I will illustrate the benefits and implications of various mechanisms and provide some real-world tips on how users can simplify security management. Attendees of this talk will learn about aspects of security in Kafka, including: -Encryption: What is SSL, what problems it solves and how Kafka leverages it. We’ll discuss encryption in flight vs. encryption at rest. -Authentication: Without authentication, anyone would be able to write to any topic in a Kafka cluster, do anything and remain anonymous. We’ll explore the available authentication mechanisms and their suitability for different types of deployment, including mutual SSL authentication, SASL/GSSAPI, SASL/SCRAM and SASL/PLAIN. -Authorization: How ACLs work in Kafka, ZooKeeper security (risks and mitigations) and how to manage ACLs at scale

Hands-on Workshop: Apache Pulsar

Sijie Guo

Introducing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pu...

StreamNative

Kafka-on-Pulsar has been one of the most anticipated features in the Pulsar ecosystem. The Kafka-on-Pulsar project was initiated by StreamNative and the OVHCloud team quickly joined the project to collaborate on its development. Kafka-on-Pulsar enables Kafka applications to leverage Pulsar’s powerful features, such as streamlined operations with enterprise-grade multi-tenancy, without modifying code. In this webinar, Sijie Guo, from StreamNative, and Pierre Zemb, from OVHCloud, will introduce KoP and discuss the following: 1. What are the key benefits? 2. What is the protocol handler and how does it work? 3. How KoP is implemented? 4. What are the new use cases it unlocks? 5. Watch a Live Demo!

Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo

Joe Stein

In this talk we will walk through how Apache Kafka and Apache Accumulo can be used together to orchestrate a de-coupled, real-time distributed and reactive request/response system at massive scale. Multiple data pipelines can perform complex operations for each message in parallel at high volumes with low latencies. The final result will be inline with the initiating call. The architecture gains are immense. They allow for the requesting system to receive a response without the need for direct integration with the data pipeline(s) that messages must go through. By utilizing Apache Kafka and Apache Accumulo, these gains sustain at scale and allow for complex operations of different messages to be applied to each response in real-time.

I Heart Log: Real-time Data and Apache Kafka

Jay Kreps

kafka for db as postgres

PivotalOpenSourceHub

Devoxx Morocco 2016 - Microservices with Kafka

László-Róbert Albert

This document discusses using microservices with Kafka. It describes how Kafka can be used to connect microservices for asynchronous communication. It outlines various features of Kafka like high throughput, replication, partitioning, and how it can provide reliability. Examples are given of how microservices could use Kafka for logging, filtering messages, and dispatching to different topics. Performance benefits of Kafka are highlighted like scalability and ability to handle high volumes of messages.

Building Kafka-powered Activity Stream

Oleksiy Holubyev

This document discusses using Apache Kafka to build a message bus for aggregating activity information from various services and enabling communication between services. It outlines the challenges of needing a way to aggregate activity data and needing a messaging backbone. It then explains how Apache Kafka provides a scalable, durable, distributed solution as a publish-subscribe messaging system to address these needs. Key features of Kafka like speed, scalability, durability and distributed design are highlighted. An example setup and usage with Ruby is also briefly described.

Introduction to Apache Kafka and Confluent... and why they matter

confluent

Milano Apache Kafka Meetup by Confluent (First Italian Kafka Meetup) on Wednesday, November 29th 2017. Il talk introduce Apache Kafka (incluse le APIs Kafka Connect e Kafka Streams), Confluent (la società creata dai creatori di Kafka) e spiega perché Kafka è un'ottima e semplice soluzione per la gestione di stream di dati nel contesto di due delle principali forze trainanti e trend industriali: Internet of Things (IoT) e Microservices.

How Apache Kafka is transforming Hadoop, Spark and Storm

Edureka!

This document provides an overview of Apache Kafka and how it is transforming Hadoop, Spark, and Storm. It begins with explaining why Kafka is needed, then defines what Kafka is and describes its architecture. Key components of Kafka like topics, producers, consumers and brokers are explained. The document also shows how Kafka can be used with Hadoop, Spark, and Storm for stream processing. It lists some companies that use Kafka and concludes by advertising an Edureka course on Apache Kafka.

Apache Pulsar Seattle - Meetup

Karthik Ramasamy

Data processing use cases, from transformation to analytics, perform tasks that require various combinations of queuing, streaming & lightweight processing steps. Until now, supporting all of those needs has required different systems for each task--stream processing engines, messaging queuing middleware, & streaming messaging systems. That has led to increased complexity for development & operations. In this session, well discuss the need to unify these capabilities in a single system & how Apache Pulsar was designed to address that. Apache Pulsar is a next generation distributed pub-sub system that was developed & deployed at Yahoo. Streamlios Karthik Ramasamy, will explain how the architecture & design of Pulsar provides the flexibility to support developers & applications needing any combination of queuing, messaging, streaming & lightweight compute.

Design Patterns for working with Fast Data

MapR Technologies

Interactive querying of streams using Apache Pulsar_Jerry peng

StreamNative

As applications become more reliant on real-time data, streaming/messaging platforms have become more and more popular and crucial to any data pipeline. Currently, many streaming/messaging platforms are only used to access the most recent events from streams of data, however, there is tremendous value that can be unlocked if the full history of streams can be queried in an interactive fashion. Pulsar SQL is a query layer built on top of Apache Pulsar (a next-gen messaging platform), that enables users to dynamically query all streams, old and new, stored inside of Pulsar. Thus, users can unlock insights from querying both new and historical streams of data in a single system. Pulsar SQL leverages Presto and Apache Pulsar’s unique architecture to execute queries in a highly scalable fashion irregardless of the number of partitions of topics that make up the streams. In this talk, we will examine the use cases and advantages of being able to interactively query events within an streaming messaging platform and how Pulsar enables users to do that in the most user-friendly and efficient manner.

Apache Mesos and the new Open Source Architecture of the Modern Datacenter

Data Con LA

Abstract Apache Mesos has the ability to run on every private and cloud instance, anywhere. In this talk, Aaron will explain the momentum behind the “single computer” abstraction that has put Mesos at the center of one of the most exciting architecture shifts in recent information technology history. He will explain how Mesos is enabling application developers and devops to redefine their responsibilities and shorten the amount of time it takes to write and ship production code. He will outline how Mesos is empowering the new class of “datacenter developers” to program directly against datacenter resources, and draw correlations to how the Linux revolutionized the server industry. Bio Aaron Williams is Head of Developer Advocacy at Mesosphere, where he enjoys helping to build the vibrant Mesos community and drive adoption of the Mesosphere Datacenter Operating System (DCOS). Prior to Mesosphere, he was a two-time startup CEO at SocialSamba and Picotent. Earlier in Aaron’s career he led the developer ecosystem at SAP and ran the Java Community Process at Sun Microsystems. Aaron has an MS in computer science from Case Western. When not immersing himself in technology, he plays basketball, runs a few half marathons a year, and is a television junkie (BTW: he is a 2-time Emmy Nominee). Going way back, he grew up in Columbus, Ohio and is a die-hard Ohio State Buckeye.

Real time Messages at Scale with Apache Kafka and Couchbase

Will Gardella

Kafka is a scalable, distributed publish subscribe messaging system that's used as a data transmission backbone in many data intensive digital businesses. Couchbase Server is a scalable, flexible document database that's fast, agile, and elastic. Because they both appeal to the same type of customers, Couchbase and Kafka are often used together. This presentation from a meetup in Mountain View describes Kafka's design and why people use it, Couchbase Server and its uses, and the use cases for both together. Also covered is a description and demo of Couchbase Server writing documents to a Kafka topic and consuming messages from a Kafka topic. using the Couchbase Kafka Connector.

Developing Real-Time Data Pipelines with Apache Kafka

Joe Stein

Apache Kafka is a distributed streaming platform that allows for building real-time data pipelines and streaming apps. It provides a publish-subscribe messaging system with persistence that allows for building real-time streaming applications. Producers publish data to topics which are divided into partitions. Consumers subscribe to topics and process the streaming data. The system handles scaling and data distribution to allow for high throughput and fault tolerance.

Scaling customer engagement with apache pulsar

StreamNative

Iterable's platform is used by marketers to reach hundreds of millions of users every day, and those numbers are quickly growing. Iterable's infrastructure is built with pub-sub messaging at it's core, so the reliability, scalability and flexibility provided by that system are business critical. In this talk we'll discuss why Iterable chose Pulsar as a pub-sub messaging system, as well as how Iterable is taking advantage of some of more recently added features in Pulsar. We'll also talk about some of the challenges we encountered, where we think Pulsar can improve, and some contributions we've made to the open source community around Pulsar.

Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar

Karthik Ramasamy

Presented at SF Big Analytics Meetup Online event processing applications often require the ability to ingest, store, dispatch and process events. Until now, supporting all of these needs has required different systems for each task -- stream processing engines, messaging queuing middleware, and pub/sub messaging systems. This has led to the unnecessary complexity for the development of such applications and operations leading to increased barrier to adoption in the enterprises. In this talk, Karthik will outline the need to unify these capabilities in a single system and make it easy to develop and operate at scale. Karthik will delve into how Apache Pulsar was designed to address this need with an elegant architecture. Apache Pulsar is a next generation distributed pub-sub system that was originally developed and deployed at Yahoo and running in production in more than 100+ companies. Karthik will explain how the architecture and design of Pulsar provides the flexibility to support developers and applications needing any combination of queuing, messaging, streaming and lightweight compute for events. Furthermore, he will provide real life use cases how Apache Pulsar is used for event processing ranging from data processing tasks to web processing applications.

Intro to Apache Kafka

Jason Hubbard

Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...

StreamNative

This document summarizes a presentation about Apache Pulsar multi-tenancy and security features at Verizon Media. It discusses how Pulsar implements tenant and namespace isolation through storage quotas, throttling policies, broker and bookie isolation. It also covers authentication, authorization, encryption in transit and at rest, and how Pulsar proxy supports SNI routing for hybrid cloud deployments and cross-organization replication. Future plans include tenant-based broker virtualization and hybrid cloud deployments with geo-replication.

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and del...

Marcelo Sousa Ancelmo

Presentation made at DevOpsDays Berlin 2015 Container technology are being evaluated by software developers and administrators with a great deal of interest. Developers want to focus on what they do best: Creating and coding new applications. That shouldn't have to change just because they need to deploy an application to a different environment. Administrators want the environment to stay reliable and stable, keeping changes at a minimum. By following a strategy that embraces good Architecture, use of Containers, DevOps philosophy, Apache Mesos and a Cloud based environment, developers and operators can create, consume and collaborate on the infrastructure configuration over the time, deploy Java EE applications and test your application infrastructure consistently regardless of the stage of the development life cycle.

Amazon ECS (December 2015)

Julien SIMON

What's hot

Kafka Security 101 and Real-World Tips

confluent

Hands-on Workshop: Apache Pulsar

Sijie Guo

Introducing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pu...

StreamNative

Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo

Joe Stein

I Heart Log: Real-time Data and Apache Kafka

Jay Kreps

kafka for db as postgres

PivotalOpenSourceHub

Devoxx Morocco 2016 - Microservices with Kafka

László-Róbert Albert

Building Kafka-powered Activity Stream

Oleksiy Holubyev

Introduction to Apache Kafka and Confluent... and why they matter

confluent

How Apache Kafka is transforming Hadoop, Spark and Storm

Edureka!

Apache Pulsar Seattle - Meetup

Karthik Ramasamy

Design Patterns for working with Fast Data

MapR Technologies

Interactive querying of streams using Apache Pulsar_Jerry peng

StreamNative

Apache Mesos and the new Open Source Architecture of the Modern Datacenter

Data Con LA

Real time Messages at Scale with Apache Kafka and Couchbase

Will Gardella

Developing Real-Time Data Pipelines with Apache Kafka

Joe Stein

Scaling customer engagement with apache pulsar

StreamNative

Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar

Karthik Ramasamy

Intro to Apache Kafka

Jason Hubbard

Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...

StreamNative

What's hot (20)

Kafka Security 101 and Real-World Tips

Hands-on Workshop: Apache Pulsar

Introducing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pu...

Real-Time Distributed and Reactive Systems with Apache Kafka and Apache Accumulo

I Heart Log: Real-time Data and Apache Kafka

kafka for db as postgres

Devoxx Morocco 2016 - Microservices with Kafka

Building Kafka-powered Activity Stream

Introduction to Apache Kafka and Confluent... and why they matter

How Apache Kafka is transforming Hadoop, Spark and Storm

Apache Pulsar Seattle - Meetup

Design Patterns for working with Fast Data

Interactive querying of streams using Apache Pulsar_Jerry peng

Apache Mesos and the new Open Source Architecture of the Modern Datacenter

Real time Messages at Scale with Apache Kafka and Couchbase

Developing Real-Time Data Pipelines with Apache Kafka

Scaling customer engagement with apache pulsar

Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar

Intro to Apache Kafka

Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...

Similar to Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and del...

Marcelo Sousa Ancelmo

Amazon ECS (December 2015)

Julien SIMON

DevOps vs. Site Reliability Engineering (SRE) in Age of Kubernetes

DevOps.com

There is a transformation brewing for DevOps in age of Kubernetes. The tools of the trade, configuration management solutions, have been superseded in agility and preference by development teams who want the declarative choreography of containerized applications. The new preference for mixing developer and operations is the site reliability engineering (SRE) model championed by Google. In this new structure, the need to automate doesn’t stop at the containerized application and DevOps professionals should seek to automate the Kubernetes service itself. In this webinar, Chris Gaun, Product Marketing Manager at Mesosphere, will cover: The transformation of DevOps to SRE How Kubernetes and DC/OS were catalyst for this change How DevOps professionals can get started with Kubernetes WHO SHOULD ATTEND Tech Professionals Developer Managers IT Managers Note the material is technical and is not intended as sales and marketing training

[DO16] Mesosphere : Microservices meet Fast Data on Azure

de:code 2017

The document discusses how microservices and fast data can be combined on Azure using the SMACK stack. It describes each component of the stack: Apache Kafka for ingesting data streams, Apache Spark for stream processing, Apache Cassandra for storage, and Akka for acting on the data. It outlines how Mesos provides a datacenter operating system to run these diverse workloads together efficiently using resource multiplexing across servers.

DOD 2016 - Jörg Schad - How Fast Data and Microservices Change the Datacenter.

PROIDEA

The application landscape inside our datacenter is changing: there are a number new distributed data processing frameworks such as Kafka or Flink being released on a weekly basis and also the trend towards microservices and container. This has implications for the ways we are running our datacenter. With this growing need of computing power, distributed applications, and larger data centers also the need for a reliable and simple use cluster manager and programming abstraction grows. This talk explains how Apache Mesos and DC/OS allows combining Microservice management and Fast Data systems on a single platform.

DevOps in Age of Kubernetes

Mesosphere Inc.

The cloud infrastructure with eucalyptus

Giuseppe Agrillo

Eucalyptus is open source software that implements cloud computing within a datacenter. It provides an Infrastructure as a Service (IaaS) solution for enterprises and service providers to build their own on-premise private clouds. Eucalyptus architecture consists of 5 components - the Cloud Controller (CLC) for high-level scheduling, Cluster Controllers (CC) to manage virtual machines, Node Controllers (NC) that execute VMs on host machines, Storage Controller (SC) for block storage, and Walrus for persistent object storage. Together these components allow users to deploy and manage virtual computing environments in a cloud-like manner.

Journey to the Modern App with Containers, Microservices and Big Data

Lightbend

Drupal In The Cloud

Bret Piatt

The secret is out – Drupal has become the ‘go-to’ open source software for the publication and management of website content. By pairing Drupal with cloud technologies there is a whole new world of user benefits well beyond scale and performance. In this session, Bret Piatt, director, technical alliances at Rackspace Hosting will discuss how to best take advantage of cloud technologies with Drupal sites. The panel presentation will address: • Leveraging the cloud ecosystem for managing configuration, code, and backups • How to scale Drupal clusters by integrating with cloud APIs • Enhancing site scale and performance by taking advantage of cloud file storage/CDN • Cloud/Drupal success stories such as Chapter Three’s ( http://www.chapterthree.com ) on Mercury, a Drupal PaaS built on The Rackspace Cloud’s Cloud Servers

EMC World 2016 - code.08 Introduction to Mesos and Mesosphere

{code}

Thinking in Terms of a Data Center Operating System

Weston Bassler

Weston Bassler will discuss the concept of a data center operating system and introduce Apache Mesos. A data center operating system treats all the machines in a data center as a single machine by abstracting resources into a single pool and allowing for better utilization of resources and ease of deploying and scaling applications. Apache Mesos is an open source project that implements this concept through a master-slave architecture and frameworks that manage tasks and services. Bassler will provide demonstrations of launching an application with Marathon, scaling an application on Marathon, and configuring Jenkins to run jobs on Mesos. He will also discuss ways to get started using Mesos through documentation, books, his GitHub repository, or by contacting him directly.

EMC World 2016 - Introduction to Mesos and Mesosphere

David vonThenen

This document provides an introduction to Mesos and Mesosphere. It begins with presenters and an agenda. It then defines Mesos as a cluster manager that pools server resources and a scheduler that dispatches workloads. Mesos components include a master, Zookeeper, and agents. Mesos supports generic and framework applications. Mesosphere was founded to deliver Mesos for enterprises and provides a datacenter operating system with rich ecosystem. The document demonstrates how EMC Code integrates persistent external storage with Mesos and Mesosphere through projects like REX-Ray and mesos-module-dvdi. It concludes with an invitation to ask questions.

Building high scalable distributed framework on apache mesos

Sigmoid

Building High Scalable Distributed Framework on Apache Mesos

Rahul Kumar

This document discusses building a high scalable framework on Apache Mesos. It provides an overview of Mesos and describes how to create your own framework by developing a scheduler and executor. The scheduler interacts with the Mesos master to launch tasks in response to resource offers, while the executor runs and manages tasks on slave nodes. Protocol buffers are used for messaging and serializing tasks. Common frameworks built on Mesos include Aurora, Marathon, Spark, Chronos and Elasticsearch.

Kubernetes on DC/OS

Cloud Technology Experts

Join us on Wednesday, January 9 as Mesosphere will demo how to install and run Kubernetes in under 10 minutes on DC/OS. We will walk you step-by-step through installing and running Kubernetes on Mesosphere DC/OS 1.10, discuss the benefits of container orchestrators, and answer frequently asked questions. Topics include: Live demo showing how to deploy and manage 100% pure Kubernetes distribution on DC/OS How to run multiple Kubernetes clusters (of different versions) alongside each other How to run both stateless and stateful workloads on the same infrastructure Live Q&A

Datacenter Computing with Apache Mesos - シリコンバレー日本人駐在員Meetup

Paco Nathan

Meetup Mesos : Mesos, Chronos and Marathon in CI/CD factory

Laurent Grangeau

The document discusses Mesos, Chronos, and Marathon and how they can be integrated into a CI/CD factory. Mesos is an open-source cluster manager that provides efficient resource sharing across distributed applications. Chronos is a replacement for cron that runs on Mesos and can be used for job orchestration. Marathon is an Apache Mesos framework for long-running applications. The document provides examples of how Jenkins can be integrated with Mesos to provide high availability and optimal utilization of resources by dynamically provisioning build slaves as needed.

Enterprise-Ready Private and Hybrid Cloud Computing Today

RightScale

RightScale User Conference NYC 2011: Enterprise-Ready Private and Hybrid Cloud Computing Today Rich Wolski - Founder and CTO, Eucalyptus In this session, we'll discuss the use of Eucalyptus and RightScale to build enterprise-grade cloud computing environments. By combining on-premise clouds with Amazon Web Services (AWS) through a common cloud management interface, Eucalyptus and AWS form a coherent platform for reliable and cost-effective enterprise cloud computing. The RightScale Cloud Management Platform delivers the high-level framework for cost-effectively automating and managing this ensemble of technologies.

A Journey to Modern Apps with Containers, Microservices and Big Data

Edward Hsu

This document discusses the transition to modern enterprise applications using containers, microservices, and big data technologies. It provides examples of how Mayo Clinic and Uber have revolutionized their industries using these technologies. The key benefits of DC/OS are that it provides a datacenter operating system that simplifies deploying and managing modern apps at scale. It allows turning infrastructure into a unified pool of resources and installing distributed services like Spark, Kafka with one command. The DC/OS community contributes many open services and it can be tried locally in under 20 minutes. Mesosphere Enterprise DC/OS provides additional support for production use.

Containers and the Evolution of Computing

Amazon Web Services

In recent years, Docker containers have become a key component of modern application design. Increasingly, developers are breaking their applications apart into smaller components and distributing them across a pool of compute resources. Using Docker on your local development machine is simple, but running Docker applications at scale in production can be difficult. In this session, we will discuss the difficulties of running Docker in production and how Amazon EC2 Container Service (ECS) can be used to reduce the operational burdens. We will give an overview of the core architectural principles underlying Amazon ECS., and we will walk through a number of patterns used by our customers to run their microservices platforms, to run batch jobs, and for deployments and continuous integration. We will also demonstrate how to define multi-container applications, deploy and scale them seamlessly on a cluster with Amazon ECS.

Similar to Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software (20)

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and del...

Amazon ECS (December 2015)

DevOps vs. Site Reliability Engineering (SRE) in Age of Kubernetes

[DO16] Mesosphere : Microservices meet Fast Data on Azure

DOD 2016 - Jörg Schad - How Fast Data and Microservices Change the Datacenter.

DevOps in Age of Kubernetes

The cloud infrastructure with eucalyptus

Journey to the Modern App with Containers, Microservices and Big Data

Drupal In The Cloud

EMC World 2016 - code.08 Introduction to Mesos and Mesosphere

Thinking in Terms of a Data Center Operating System

EMC World 2016 - Introduction to Mesos and Mesosphere

Building high scalable distributed framework on apache mesos

Building High Scalable Distributed Framework on Apache Mesos

Kubernetes on DC/OS

Datacenter Computing with Apache Mesos - シリコンバレー日本人駐在員Meetup

Meetup Mesos : Mesos, Chronos and Marathon in CI/CD factory

Enterprise-Ready Private and Hybrid Cloud Computing Today

A Journey to Modern Apps with Containers, Microservices and Big Data

Containers and the Evolution of Computing

More from Marcelo Sousa Ancelmo

From Developer and Beyond - The IT Architect Career

Marcelo Sousa Ancelmo

DevOps @ Enterprise - DevOps Meetup Zurich

Marcelo Sousa Ancelmo

This document discusses strategies for implementing DevOps practices within large enterprises. It notes that DevOps requires cultural change to break down silos between development and operations teams. Executive leadership is needed to get teams on the "same page" and defend the initiative. A pilot project was used to prove the value of DevOps and gain support. Subsequent releases introduced new practices over time to address business challenges. The document emphasizes focusing on feedback loops, collaboration, and continual learning and experimentation when adopting DevOps within a large organization.

DevOps @ Enterprise - Lessons from the trenches

Marcelo Sousa Ancelmo

This document discusses DevOps adoption within large enterprises. It notes that DevOps requires cultural change to break down silos between development and operations teams. Executive leadership is needed to engage all teams and help them understand how DevOps can improve collaboration, transparency and feedback. The document advocates for mixing team members and using common tools to increase collaboration across the software development lifecycle. It also describes one company's experience piloting DevOps and gradually implementing practices and tools across their organization over multiple releases.

Quem quer ser um arquiteto?

Marcelo Sousa Ancelmo

O documento discute o que é necessário para ser um arquiteto de software. Ele destaca que é preciso ter conhecimento técnico em vários domínios como arquitetura de negócios, infraestrutura, software e segurança, além de tecnologias e linguagens de programação. Também é importante ter habilidades de liderança, comunicação e mentoria. Um arquiteto precisa adotar uma abordagem ágil e enxuta, evitando soluções prematuras para problemas ainda não conhecidos.

Práticas, Técnicas e Ferramentas para Continuous Delivery com ALM

Marcelo Sousa Ancelmo

Agile and DevOps for Services

Marcelo Sousa Ancelmo

The document discusses implementing continuous integration, delivery, and deployment practices for service-oriented architecture (SOA) environments. It recommends having a centralized code repository, automated builds, and making latest deliverables easily accessible. Key tools include source control management for workflow and collaboration, build automation to compile and test code, an asset repository to store and track assets, and deployment automation to provision resources and deploy services. The document provides an example workflow where a team uses these practices to develop a service through multiple sprints with continuous integration, delivery, and deployment.

More from Marcelo Sousa Ancelmo (6)

From Developer and Beyond - The IT Architect Career

DevOps @ Enterprise - DevOps Meetup Zurich

DevOps @ Enterprise - Lessons from the trenches

Quem quer ser um arquiteto?

Práticas, Técnicas e Ferramentas para Continuous Delivery com ALM

Agile and DevOps for Services

Recently uploaded

What next after learning python programming basics

Rakesh Kumar R

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

Peter Muessig

The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.

Oracle 23c New Features For DBAs and Developers.pptx

Remote DBA Services

SQL Accounting Software Brochure Malaysia

GohKiangHock

14 th Edition of International conference on computer vision

ShulagnaSarkar2

About the event 14th Edition of International conference on computer vision Computer conferences organized by ScienceFather group. ScienceFather takes the privilege to invite speakers participants students delegates and exhibitors from across the globe to its International Conference on computer conferences to be held in the Various Beautiful cites of the world. computer conferences are a discussion of common Inventions-related issues and additionally trade information share proof thoughts and insight into advanced developments in the science inventions service system. New technology may create many materials and devices with a vast range of applications such as in Science medicine electronics biomaterials energy production and consumer products. Nomination are Open!! Don't Miss it Visit: computer.scifat.com Award Nomination: https://x-i.me/ishnom Conference Submission: https://x-i.me/anicon For Enquiry: Computer@scifat.com

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

fiscal year variant fiscal year variant.

AnkitaPandya11

zOS Mainframe JES2-JES3 JCL-JECL Differences

YousufSait3

Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...

XfilesPro

All you need to know about Spring Boot and GraalVM

Alina Yurenko

GreenCode-A-VSCode-Plugin--Dario-Jurisic

Green Software Development

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

mz5nrf0n

原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

Enums On Steroids - let's look at sealed classes !

Marcin Chrost

Fundamentals of Programming and Language Processors

Rakesh Kumar R

How to write a program in any programming language

Rakesh Kumar R

Modelling Up - DDDEurope 2024 - Amsterdam

Alberto Brandolini

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Green Software Development

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

Quickdice ERP

YAML crash COURSE how to write yaml file for adding configuring details

NishanthaBulumulla1

Recently uploaded (20)

What next after learning python programming basics

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

Oracle 23c New Features For DBAs and Developers.pptx

SQL Accounting Software Brochure Malaysia

14 th Edition of International conference on computer vision

Energy consumption of Database Management - Florina Jonuzi

fiscal year variant fiscal year variant.

zOS Mainframe JES2-JES3 JCL-JECL Differences

Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...

All you need to know about Spring Boot and GraalVM

GreenCode-A-VSCode-Plugin--Dario-Jurisic

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

Mobile App Development Company In Noida | Drona Infotech

Enums On Steroids - let's look at sealed classes !

Fundamentals of Programming and Language Processors

How to write a program in any programming language

Modelling Up - DDDEurope 2024 - Amsterdam

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

YAML crash COURSE how to write yaml file for adding configuring details

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software

1. Containers, DevOps, Apache Mesos and Cloud Reshaping how we develop and delivery software https://twitter.com/marceloancelmo Marcelo Sousa Ancelmo marcelo.souzaancelmo@ig.com marceloancelmo@gmail.com marceloancelmo http://www.slideshare.net/marceloancelmo https://www.linkedin.com/in/marceloancelmo

2. Are your application ready?

3. Decompose the applications into self-contained services Be stateless (As much as possible) NFR's are king, be aware on latency, throughput, etc... Expect an impact on your application architecture Fault tolerant Scalable Elastic Multi-tenant

4. !!!MICROSERVICES!!!

5. Microservices... … or more Spaghetti

6. “There are 17 million shipping containers in existence, packed with every physical good imaginable. Every single one of them can be loaded onto the same boats, by the same cranes, in the same facilities, and sent anywhere in the World with incredible efficiency. It is embarrassing to think that a 30 ton shipment of coffee can safely travel half-way across the World in less time than it takes a software team to deliver its code from one datacenter to another sitting 10 miles away.” 5th principle of Open Container Initiative - https://github.com/opencontainers/specs/blob/master/principles.md Why Use Containers?

7. Collaboration @ Container

8. Your usual Delivery Pipeline...

9. … Is now Baking your image

10. And potentially can use all your computing power

11. Introducing Apache Mesos Program against your datacenter like it’s a single pool of resources “Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.” http://mesos.apache.org/ Used by Twitter, AirBNB, eBay, Netflix, Apple and many others http://mesos.apache.org/documentation/latest/powered-by-mesos/

12. Apache Mesos Overview Batch Processing Long Running Services

13. Way of work

14. Apache Mesos Architecture

15. Resource Offer example

16. Auto-Scale

17. Application 300 users

18. Autoscaling UP 600 users

19. Autoscaling DOWN 400 users

20. High Availability

21. Application Running 300 users

22. Node Failure 300 users300 users

23. Starting a new node for the application 300 users300 users

24. Application Running 300 users

25. Continuous Delivery Blue-Green Deployments

26. Continuous Delivery – Blue / Green

27. Deploy the application Production

28. Test the application Production

29. Flip the application Production

30. Empowering everyone Production

31. Continuous Delivery Canary Release

32. Continuous Delivery – Canary Release

33. Deploy the new version Production 100%

34. Test and Evaluate the new version Production 80% Production 20%

35. Rollout the new Release

36. Putting into perspective “If a Docker application is a Lego brick, Kubernetes would be like a kit for building the Millennium Falcon and the Mesos cluster would be like a whole Star Wars universe made of Legos.” ~ Solomon Hykes

37. Cloud + Microservices + Containers + CD Is this the Perfect Storm?

38. Questions?

39. Thank You Merci Grazie Gracias Obrigado Danke Japanese English French Russian German Italian Spanish Brazilian Portuguese Arabic Traditional Chinese Simplified Chinese Hindi Tamil Thai Korean Dziękuję Polish Marcelo Sousa Ancelmo marcelo.souzaancelmo@ig.com marceloancelmo@gmail.com

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software

Similar to Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software (20)

More from Marcelo Sousa Ancelmo

More from Marcelo Sousa Ancelmo (6)

Recently uploaded

Recently uploaded (20)

Containers, DevOps, Apache Mesos and Cloud - Reshaping how we develop and delivery software