Mesos and Container Schedulers
•
0 likes•185 views
Mesos and Container Schedulers Landscape and Practical Experiences This document discusses Piksel's evolution from static VM partitioning to using Mesos as a container scheduler. It summarizes their experiences using different frameworks like YARN and Mesos directly. Apache Aurora is highlighted as a framework that provides capabilities for multi-tenancy, quotas, and preemption needed for Piksel's heterogeneous workloads and multiple customers. Customizing Aurora's job definition language is discussed.
Report
Share
Report
Share
Download to read offline
Recommended
AWS ECS workshop
This document outlines an ECS workshop agenda that covers: continuous delivery using infrastructure as code; containerizing services; deploying to ECS and maintaining uptime; platform thinking for infrastructure as code; ECS service and cluster scaling; blue/green deployments; and demos of cluster updates, multiple environments, and on-demand environments. The workshop teaches development of microservices like an Asgard portal with Odin and Thor gods, containerizing the services, deploying to ECS, and scaling the cluster and services. It addresses shortcomings of the initial solution and introduces platform thinking to provision clusters and services together in an adaptive, disposable way.
Immutable infrastructure with Terraform
This document summarizes a talk on immutable infrastructure with Terraform. It begins by covering the devops journey from manually managed on-premise hardware to configuration management tools. It then discusses the rise of cloud computing and challenges it introduced. Next, it defines immutable infrastructure and contrasts it with mutable infrastructure. The rest of the document focuses on Terraform, explaining what it is, why it is useful, and demonstrating how to use it through examples like deploying a web app on EC2, handling failures and incremental changes, using remote state and locking, and best practices for teams. It also shows examples of rolling deployments, output variables, supporting multiple environments, and using modules.
Boris Stoyanov - some new features in Apache cloudStack
A look at some new CloudStack features around VM deployment and configuration. These new features include: more sophisticated options for specifying pod and cluster while deploying a VM; running and retrieving diagnostics on the VR; sending additional configuration to VMs; and adding options to cleanup additional data disks when destroying a VM.
Cloud computing - an insight into "how does it really work ?"
Using "Grails" and utilizing SpringSource, we shall offer a hands-on demo introducing a typical "Grails" development environment and a classical cloud computing application deployed and managed on top of Amazon CC services.
Terraform day 1
This document provides an overview and tutorial on using Terraform for DevOps. It introduces Terraform as a tool for defining and managing infrastructure as code. It then covers installing Terraform, deploying AWS infrastructure like EC2 instances using Terraform configurations, managing variables and outputs, using provisioners, organizing code with modules and workspaces, and managing Terraform state. The document aims to help users get started with Terraform for infrastructure as code.
Building a PaaS with Docker and AWS
Eric Holmes from Remind discussed building an internal Platform as a Service (PaaS) called Empire using Docker and Amazon EC2 Container Service (ECS). Remind started on Heroku but encountered issues with scaling and visibility. Empire provides a management layer on top of ECS for deploying and scaling microservices. It implements a subset of the Heroku API and provides a single binary and CLI. Empire is running 15 of Remind's production services on ECS with improved performance over Heroku. A demo was shown of deploying a sample app with Empire.
Deploying and running Grails in the cloud
Taking a look at different cloud providers and how easy it is to deploy a basic Grails application to them. Created for the http://sfgrails.com meetup Feb 2011.
Making cloud portability a practical reality (i pad)
The term "cloud portability" is often considered a synonym for "Cloud API portability," which implies a series of misconceptions. If we break away from dogma, we can find that what we really looking for in cloud portability is Application portability between clouds which can be a vastly simpler requirement, as we can achieve application portability without settling on a common Cloud API. ..
The following presentation shows how I could use the ideas from this post and provide a practical cloud portability solution today using Cloudify and JClouds.
See details in this post:
http://natishalom.typepad.com/nati_shaloms_blog/2011/12/making-cloud-portability-a-practical-reality.html
Recommended
AWS ECS workshop
This document outlines an ECS workshop agenda that covers: continuous delivery using infrastructure as code; containerizing services; deploying to ECS and maintaining uptime; platform thinking for infrastructure as code; ECS service and cluster scaling; blue/green deployments; and demos of cluster updates, multiple environments, and on-demand environments. The workshop teaches development of microservices like an Asgard portal with Odin and Thor gods, containerizing the services, deploying to ECS, and scaling the cluster and services. It addresses shortcomings of the initial solution and introduces platform thinking to provision clusters and services together in an adaptive, disposable way.
Immutable infrastructure with Terraform
This document summarizes a talk on immutable infrastructure with Terraform. It begins by covering the devops journey from manually managed on-premise hardware to configuration management tools. It then discusses the rise of cloud computing and challenges it introduced. Next, it defines immutable infrastructure and contrasts it with mutable infrastructure. The rest of the document focuses on Terraform, explaining what it is, why it is useful, and demonstrating how to use it through examples like deploying a web app on EC2, handling failures and incremental changes, using remote state and locking, and best practices for teams. It also shows examples of rolling deployments, output variables, supporting multiple environments, and using modules.
Boris Stoyanov - some new features in Apache cloudStack
A look at some new CloudStack features around VM deployment and configuration. These new features include: more sophisticated options for specifying pod and cluster while deploying a VM; running and retrieving diagnostics on the VR; sending additional configuration to VMs; and adding options to cleanup additional data disks when destroying a VM.
Cloud computing - an insight into "how does it really work ?"
Using "Grails" and utilizing SpringSource, we shall offer a hands-on demo introducing a typical "Grails" development environment and a classical cloud computing application deployed and managed on top of Amazon CC services.
Terraform day 1
This document provides an overview and tutorial on using Terraform for DevOps. It introduces Terraform as a tool for defining and managing infrastructure as code. It then covers installing Terraform, deploying AWS infrastructure like EC2 instances using Terraform configurations, managing variables and outputs, using provisioners, organizing code with modules and workspaces, and managing Terraform state. The document aims to help users get started with Terraform for infrastructure as code.
Building a PaaS with Docker and AWS
Eric Holmes from Remind discussed building an internal Platform as a Service (PaaS) called Empire using Docker and Amazon EC2 Container Service (ECS). Remind started on Heroku but encountered issues with scaling and visibility. Empire provides a management layer on top of ECS for deploying and scaling microservices. It implements a subset of the Heroku API and provides a single binary and CLI. Empire is running 15 of Remind's production services on ECS with improved performance over Heroku. A demo was shown of deploying a sample app with Empire.
Deploying and running Grails in the cloud
Taking a look at different cloud providers and how easy it is to deploy a basic Grails application to them. Created for the http://sfgrails.com meetup Feb 2011.
Making cloud portability a practical reality (i pad)
The term "cloud portability" is often considered a synonym for "Cloud API portability," which implies a series of misconceptions. If we break away from dogma, we can find that what we really looking for in cloud portability is Application portability between clouds which can be a vastly simpler requirement, as we can achieve application portability without settling on a common Cloud API. ..
The following presentation shows how I could use the ideas from this post and provide a practical cloud portability solution today using Cloudify and JClouds.
See details in this post:
http://natishalom.typepad.com/nati_shaloms_blog/2011/12/making-cloud-portability-a-practical-reality.html
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform. Infographics on IaaS, PaaS, SaaS. Commandments of developing a cloud based distributed application.
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
Giles Sirett, Chairman CSEUG, PMC member, Apache CloudStack shared in-depth insight about the new features and functionalities in CloudStack 4.15. He also provided info on when 4.16 is expected, presented the new VP of Apache CloudStack, latest integrations of CloudStack, improvements in the UI, new OS supported, advanced capabilities of vSphere, OVF support, dynamic roles enhancements and more.
Find out more for Giles Sirett: https://www.shapeblue.com/shapeblue-leadership-team/
------------------------------------------
CloudStack European User Group Virtual happened on May 27th. The first CSEUG Virtual proved to be a huge success. It collected people from 23 countries – Germany, the United Kingdom, Switzerland, India, Bulgaria, Greece, Poland, Serbia, Brazil, Chile, Russia, USA, Canada, Japan, France, Uruguay, Korea …
We also had a record number of registrations and attendees for a CloudStack User Group Event. The physical distance was not a stopper for our speakers, who joined the event from 6 different countries.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Giles Sirett: Introduction and CloudStack news
Giles will talk about all that's new and happening within the Apache CloudStack community, and about new and future releases, exciting features, upcoming events and more!
Managing Your Cloud Assets
Cost is often the conversation starter when customers think about moving to the cloud. AWS helps lower costs for customers through its “pay only for what you use” pricing model, frequent price drops, and pricing model choice to support variable & stable workloads. In this session, you will learn about the financial considerations of owning and operating a traditional data center or managed hosting provider versus utilizing AWS. We will detail our TCO methodology and showcase cost comparisons for some common customer use-cases. We’ll also cover a few AWS cost optimization areas, including Spot and Reserved Instances, EC2 Auto Scaling, and consolidated billing.
Presenter:
Amit Sharma, Solution Architect, Amazon Internet Services
Krishnenjit Roy, Director IT Operations, Freshdesk
Open Source Docs: The Good, The Bad, and the WTF?!
Open source documentation is generally awful, often frightening, and sometimes it’s just outright weird. In this discussion, I will show you examples from the entire spectrum of open source documentation, in an effort to try and help you avoid some of the worst mistakes made by others before you. We will discuss why documentation is important for open source projects, and how to go about getting it written, and written well, without making too much of a fool out of yourself.
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
If you have tried Docker but are unsure about how to run it at scale, you will benefit from this session. Like virtualization before, containerization (à; la Docker) is increasing the elastic nature of cloud infrastructure by an order of magnitude. But maybe you still have questions: How many containers can you run on a given Amazon EC2 instance type? Which metric should you look at to measure contention? How do you manage fleets of containers at scale?
Datadog is a monitoring service for IT, operations, and development teams who write and run applications at scale. In this session, the cofounder of Datadog presents the challenges and benefits of running containers at scale and how to use quantitative performance patterns to monitor your infrastructure at this magnitude and complexity. Sponsored by Datadog.
Virtualization
This document discusses virtualization of test environments. It defines virtualization as creating virtual versions of computer hardware, operating systems, or network resources. Virtualization allows software to be tested under different platforms and configurations without needing to set up each environment separately. Popular virtualization tools include VmWare, Oracle VirtualBox, KVM, and XEN. Vagrant is also introduced as a tool that allows portable development environments to be created and deployed easily across different infrastructures using provisioning scripts. The document provides commands for initializing, running, connecting to, and deleting Vagrant environments.
Aeolus - Clouds Flying in Assembly
As clouds become commonplace, there is a need to manage virtual instances across a great variety of clouds and cloud-like environments. Aeolus, an open source project recently launched by Red Hat, lives in the space just below where traditional config management tools like Puppet live.
It provides tools for managing virtual instances and images across public, private and hybrid clouds, from image creation to the launch of complex deployments. It enhances existing clouds by providing user management, monitoring and quality-of-service metrics to make intelligent decisions about resource placement, or automate them based on policy, across multiple clouds. All fronted by a friendly self-service web application, with an API in the works.
Google App Engine overview (GAE/J)
This document provides an overview of Google App Engine (GAE), a platform as a service that allows users to run web applications on Google's infrastructure. It discusses key aspects of GAE like language support, automatic scaling, quotas and limits. The document then demonstrates how to set up a basic "Hello World" application on GAE for Java (GAE/J) using Eclipse as the IDE. It shows the project structure, deploying the application to GAE, and potential next steps like using Guice or the Python runtime.
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Nashorn is a JavaScript engine for Java that is integrated with JSR-223 and released with Java 8. It aims to provide fast and compliant ECMAScript 5.1 JavaScript execution using invokedynamic and MethodHandles to dynamically link and optimize code at runtime. This allows Nashorn to handle JavaScript's dynamic nature better than previous engines like Rhino by hotswapping optimized codepaths. Nashorn is small, fast, and fully compliant while providing tight integration between JavaScript and Java through JSR-223.
Suning OpenStack Cloud and Heat
An experience sharing of the OpenStack deployment at Suning.com, a large online retailer in China. The talk presents the challenges and opportunities on orchestrating the enterprise workloads using Heat.
Impact of cloud services on the work of oracle technology experts
The document discusses how cloud services are impacting the work of Oracle technology experts. It notes that many database administrator and fusion middleware administrator roles will transition to cloud providers as more systems move to the cloud. It outlines a roadmap for technology experts that includes trialing cloud services, ongoing learning, and adopting a hybrid approach using both on-premises and cloud systems. It concludes that while some tasks will shift to cloud providers, technology experts still have opportunities consulting on cloud services, developing cloud software, and supporting hybrid environments.
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28CloudStack - Open Source Cloud Computing Project
Cloudstack is an open source Infrastructure-as-a-Service (IaaS) software platform available under the GPLv3 license, which enables users to build, manage and deploy compute cloud environments. The community edition is based on the latest, leading edge features and bits that the Cloud.com team of engineers are working on and is supported by our open source community.
Using CloudStack a free and open source cloud computing software to build a private cloud. During the training attendees will be instructed on how to install Cloudstack to manage virtual infrastructure in a private cloud computing configuration. At the conclusion of the Build a Private Cloud section users will have the knowledge needed to create a simple private cloud computing environment.Continuous Delivery with AWS Services
Continuous delivery is a software development practice where code changes are automatically built, tested, and prepared for a release. The document discusses the prerequisites, artifacts, and process for continuous delivery from development to testing environments. Key aspects include infrastructure setup using tools like AWS OpsWorks, committing code to branches for different environments, updating configuration scripts, and deployments that are automated, tracked, and can rollback on failure. The goal is to continuously deploy changes through environments with integration and testing at each stage.
AWS Re:Invent - Optimizing Costs with AWS
AWS Re:Invent 2012 presentation from Netflix which covers how to optimize cost and usage of your AWS resources. Areas of focus are Autoscaling EC2 instances, batch access of SQS, and improved S3 usage.
Scalable On-Demand Hadoop Clusters with Docker and Mesos
This document discusses using Docker and Mesos to provide scalable on-demand Hadoop clusters. It outlines how Mesos acts as a multi-tenant resource pool to allow multiple frameworks like Hadoop, Spark, and Storm to dynamically share resources. Docker is proposed as the new "unit of work" to provide a flexible and developer-friendly form factor. The document recommends starting small and scaling fast, using the most appropriate framework for each job, and planning for rolling restarts to take advantage of this approach for on-demand Hadoop clusters.
Dev ops for big data cluster management tools
What are the tools that we can find to day to manage Hadoop cluster and its ecosystem?
There are two tools ready today:
Cloudera Manager and Ambari from Hortonworks.
In this presentation I explain what they do and why to use them, as well as Pros. and Cons.
Containerised ASP.NET Core apps with Kubernetes
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containerized applications into logical units for easy management and discovery called pods. It can be used to deploy a containerized ASP.NET Core application. It provides basic mechanisms for self-healing, auto-scaling and updates through concepts like deployments, services, replica sets and labels.
EMEA OpenStack Day, July 13th 2011 in London - Jim Curry intro
Jim Curry kicks off the first OpenStack day in London on July 13th, 2011 with an overview of the OpenStack technology.
The Aeolus Project
Aeolus is a project that aims to provide cloud portability and proliferation by hiding individual cloud infrastructure and allowing resources to be remixed. It consists of several component projects including a HA Manager, Conductor, Composer, and Orchestrator. The Conductor manages resource pools and placement across clouds, while the Composer generates cloud-independent images. Aeolus integrates with projects like Apache Deltacloud to provide cross-cloud APIs and abstraction.
2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure
The document discusses the need for hybrid infrastructure and hybrid DevOps to manage different cloud platforms and physical infrastructure in a consistent way. It notes that while no single API or platform can meet all needs, AWS dominance means its operational patterns have become the benchmark. The key is developing composable infrastructure modules that can be orchestrated together to provide portability across environments using a common operational process.
OpenStack Preso: DevOps on Hybrid Infrastructure
Discusses the approach for making hybrid DevOps workable including what obstacles must be overcome. Includes demo of multiple OpenStack clouds & Kubernetes deploy on AWS, Google and OpenStack
More Related Content
What's hot
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform. Infographics on IaaS, PaaS, SaaS. Commandments of developing a cloud based distributed application.
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
Giles Sirett, Chairman CSEUG, PMC member, Apache CloudStack shared in-depth insight about the new features and functionalities in CloudStack 4.15. He also provided info on when 4.16 is expected, presented the new VP of Apache CloudStack, latest integrations of CloudStack, improvements in the UI, new OS supported, advanced capabilities of vSphere, OVF support, dynamic roles enhancements and more.
Find out more for Giles Sirett: https://www.shapeblue.com/shapeblue-leadership-team/
------------------------------------------
CloudStack European User Group Virtual happened on May 27th. The first CSEUG Virtual proved to be a huge success. It collected people from 23 countries – Germany, the United Kingdom, Switzerland, India, Bulgaria, Greece, Poland, Serbia, Brazil, Chile, Russia, USA, Canada, Japan, France, Uruguay, Korea …
We also had a record number of registrations and attendees for a CloudStack User Group Event. The physical distance was not a stopper for our speakers, who joined the event from 6 different countries.
------------------------------------------
About CloudStack: https://cloudstack.apache.org/
Giles Sirett: Introduction and CloudStack news
Giles will talk about all that's new and happening within the Apache CloudStack community, and about new and future releases, exciting features, upcoming events and more!
Managing Your Cloud Assets
Cost is often the conversation starter when customers think about moving to the cloud. AWS helps lower costs for customers through its “pay only for what you use” pricing model, frequent price drops, and pricing model choice to support variable & stable workloads. In this session, you will learn about the financial considerations of owning and operating a traditional data center or managed hosting provider versus utilizing AWS. We will detail our TCO methodology and showcase cost comparisons for some common customer use-cases. We’ll also cover a few AWS cost optimization areas, including Spot and Reserved Instances, EC2 Auto Scaling, and consolidated billing.
Presenter:
Amit Sharma, Solution Architect, Amazon Internet Services
Krishnenjit Roy, Director IT Operations, Freshdesk
Open Source Docs: The Good, The Bad, and the WTF?!
Open source documentation is generally awful, often frightening, and sometimes it’s just outright weird. In this discussion, I will show you examples from the entire spectrum of open source documentation, in an effort to try and help you avoid some of the worst mistakes made by others before you. We will discuss why documentation is important for open source projects, and how to go about getting it written, and written well, without making too much of a fool out of yourself.
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
If you have tried Docker but are unsure about how to run it at scale, you will benefit from this session. Like virtualization before, containerization (à; la Docker) is increasing the elastic nature of cloud infrastructure by an order of magnitude. But maybe you still have questions: How many containers can you run on a given Amazon EC2 instance type? Which metric should you look at to measure contention? How do you manage fleets of containers at scale?
Datadog is a monitoring service for IT, operations, and development teams who write and run applications at scale. In this session, the cofounder of Datadog presents the challenges and benefits of running containers at scale and how to use quantitative performance patterns to monitor your infrastructure at this magnitude and complexity. Sponsored by Datadog.
Virtualization
This document discusses virtualization of test environments. It defines virtualization as creating virtual versions of computer hardware, operating systems, or network resources. Virtualization allows software to be tested under different platforms and configurations without needing to set up each environment separately. Popular virtualization tools include VmWare, Oracle VirtualBox, KVM, and XEN. Vagrant is also introduced as a tool that allows portable development environments to be created and deployed easily across different infrastructures using provisioning scripts. The document provides commands for initializing, running, connecting to, and deleting Vagrant environments.
Aeolus - Clouds Flying in Assembly
As clouds become commonplace, there is a need to manage virtual instances across a great variety of clouds and cloud-like environments. Aeolus, an open source project recently launched by Red Hat, lives in the space just below where traditional config management tools like Puppet live.
It provides tools for managing virtual instances and images across public, private and hybrid clouds, from image creation to the launch of complex deployments. It enhances existing clouds by providing user management, monitoring and quality-of-service metrics to make intelligent decisions about resource placement, or automate them based on policy, across multiple clouds. All fronted by a friendly self-service web application, with an API in the works.
Google App Engine overview (GAE/J)
This document provides an overview of Google App Engine (GAE), a platform as a service that allows users to run web applications on Google's infrastructure. It discusses key aspects of GAE like language support, automatic scaling, quotas and limits. The document then demonstrates how to set up a basic "Hello World" application on GAE for Java (GAE/J) using Eclipse as the IDE. It shows the project structure, deploying the application to GAE, and potential next steps like using Guice or the Python runtime.
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Nashorn is a JavaScript engine for Java that is integrated with JSR-223 and released with Java 8. It aims to provide fast and compliant ECMAScript 5.1 JavaScript execution using invokedynamic and MethodHandles to dynamically link and optimize code at runtime. This allows Nashorn to handle JavaScript's dynamic nature better than previous engines like Rhino by hotswapping optimized codepaths. Nashorn is small, fast, and fully compliant while providing tight integration between JavaScript and Java through JSR-223.
Suning OpenStack Cloud and Heat
An experience sharing of the OpenStack deployment at Suning.com, a large online retailer in China. The talk presents the challenges and opportunities on orchestrating the enterprise workloads using Heat.
Impact of cloud services on the work of oracle technology experts
The document discusses how cloud services are impacting the work of Oracle technology experts. It notes that many database administrator and fusion middleware administrator roles will transition to cloud providers as more systems move to the cloud. It outlines a roadmap for technology experts that includes trialing cloud services, ongoing learning, and adopting a hybrid approach using both on-premises and cloud systems. It concludes that while some tasks will shift to cloud providers, technology experts still have opportunities consulting on cloud services, developing cloud software, and supporting hybrid environments.
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28CloudStack - Open Source Cloud Computing Project
Cloudstack is an open source Infrastructure-as-a-Service (IaaS) software platform available under the GPLv3 license, which enables users to build, manage and deploy compute cloud environments. The community edition is based on the latest, leading edge features and bits that the Cloud.com team of engineers are working on and is supported by our open source community.
Using CloudStack a free and open source cloud computing software to build a private cloud. During the training attendees will be instructed on how to install Cloudstack to manage virtual infrastructure in a private cloud computing configuration. At the conclusion of the Build a Private Cloud section users will have the knowledge needed to create a simple private cloud computing environment.Continuous Delivery with AWS Services
Continuous delivery is a software development practice where code changes are automatically built, tested, and prepared for a release. The document discusses the prerequisites, artifacts, and process for continuous delivery from development to testing environments. Key aspects include infrastructure setup using tools like AWS OpsWorks, committing code to branches for different environments, updating configuration scripts, and deployments that are automated, tracked, and can rollback on failure. The goal is to continuously deploy changes through environments with integration and testing at each stage.
AWS Re:Invent - Optimizing Costs with AWS
AWS Re:Invent 2012 presentation from Netflix which covers how to optimize cost and usage of your AWS resources. Areas of focus are Autoscaling EC2 instances, batch access of SQS, and improved S3 usage.
Scalable On-Demand Hadoop Clusters with Docker and Mesos
This document discusses using Docker and Mesos to provide scalable on-demand Hadoop clusters. It outlines how Mesos acts as a multi-tenant resource pool to allow multiple frameworks like Hadoop, Spark, and Storm to dynamically share resources. Docker is proposed as the new "unit of work" to provide a flexible and developer-friendly form factor. The document recommends starting small and scaling fast, using the most appropriate framework for each job, and planning for rolling restarts to take advantage of this approach for on-demand Hadoop clusters.
Dev ops for big data cluster management tools
What are the tools that we can find to day to manage Hadoop cluster and its ecosystem?
There are two tools ready today:
Cloudera Manager and Ambari from Hortonworks.
In this presentation I explain what they do and why to use them, as well as Pros. and Cons.
Containerised ASP.NET Core apps with Kubernetes
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containerized applications into logical units for easy management and discovery called pods. It can be used to deploy a containerized ASP.NET Core application. It provides basic mechanisms for self-healing, auto-scaling and updates through concepts like deployments, services, replica sets and labels.
EMEA OpenStack Day, July 13th 2011 in London - Jim Curry intro
Jim Curry kicks off the first OpenStack day in London on July 13th, 2011 with an overview of the OpenStack technology.
The Aeolus Project
Aeolus is a project that aims to provide cloud portability and proliferation by hiding individual cloud infrastructure and allowing resources to be remixed. It consists of several component projects including a HA Manager, Conductor, Composer, and Orchestrator. The Conductor manages resource pools and placement across clouds, while the Composer generates cloud-independent images. Aeolus integrates with projects like Apache Deltacloud to provide cross-cloud APIs and abstraction.
What's hot (20)
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
What’s New in CloudStack 4.15 - CloudStack European User Group Virtual, May 2021
Open Source Docs: The Good, The Bad, and the WTF?!
Open Source Docs: The Good, The Bad, and the WTF?!
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Impact of cloud services on the work of oracle technology experts
Impact of cloud services on the work of oracle technology experts
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
vBACD - Deploying Infrastructure-as-a-Service with CloudStack - 2/28
Scalable On-Demand Hadoop Clusters with Docker and Mesos
Scalable On-Demand Hadoop Clusters with Docker and Mesos
EMEA OpenStack Day, July 13th 2011 in London - Jim Curry intro
EMEA OpenStack Day, July 13th 2011 in London - Jim Curry intro
Similar to Mesos and Container Schedulers
2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure
The document discusses the need for hybrid infrastructure and hybrid DevOps to manage different cloud platforms and physical infrastructure in a consistent way. It notes that while no single API or platform can meet all needs, AWS dominance means its operational patterns have become the benchmark. The key is developing composable infrastructure modules that can be orchestrated together to provide portability across environments using a common operational process.
OpenStack Preso: DevOps on Hybrid Infrastructure
Discusses the approach for making hybrid DevOps workable including what obstacles must be overcome. Includes demo of multiple OpenStack clouds & Kubernetes deploy on AWS, Google and OpenStack
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Kubernetes is the most popular container orchestration system that is natively designed for Cloud. At Lyft and Cloudera, we have both emerged the next-generation, cloud-native infrastructure based on Kubernetes, which supports various distributed workloads.
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
This document provides an overview of OCCIware, a project that aims to create a cloud consumer platform using the Open Cloud Computing Interface (OCCI) standard. It discusses the need for such a platform given the fragmented state of existing cloud solutions. OCCIware takes a model-driven engineering approach, using Eclipse modeling tools to generate an OCCI extension, designer, and runtime configuration from a domain model. The document demonstrates using these tools to model a Linked Data application and deploy its configuration to Docker. Upcoming work on OCCIware includes improving existing generators, integrating additional capabilities like simulation, and contributing back to the OCCI standard.
OCCIware Project at EclipseCon France 2016, by Marc Dutoo, Open Wide
Hear hear dev & ops alike - ever got bitten by the fragmentation of the Cloud space at deployment time, By AWS vs Azure, Open Shift vs Heroku ? in a word, ever dreamt of configuring at once your Cloud application along with both its VMs and database ? Well, the extensible Open Cloud Computing Interface (OCCI) REST API (see http://occi-wg.org/) allows just that, by addressing the whole XaaS spectrum.
And now, OCCI is getting powerboosted by Eclipse Modeling and formal foundations. Enter Cloud Designer and other outputs of the OCCIware project (See http://www.occiware.org) : multiple visual representations, one per Cloud layer and technology. XaaS Cloud extension model validation, documentation & ops scripting generation. Simulation, decision-making comparison. Connectors that bring those models to life by getting their status from common Cloud services. Runtime middleware, deployed, monitored, adminstrated. And tackling the very interesting challenge of modeling a meta API in EMF's metamodel, while staying true to EMF, Eclipse tools and the OCCI standard.
Featuring Eclipse Sirius, Acceleo generators, EMF at runtime. Coming soon to a new Eclipse Foundation project near you, if so you'd like.
This talk includes a demonstration of the Docker connector and of how to use Cloud Designer to configure a simple Cloud application's deployment on the Roboconf PaaS system and OpenStack infrastructure.
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
"Bootstrapping Puppet and Application Deployment" by Robert de Macedo Soares, Application Security Engineer, BusinessWire.
Presentation Overview: A dive into the problems faced when first launching Puppet across existing, heterogeneous servers, outlining possible solutions using our experience as an example. In addition, this session will touch on application management and deployment using subversion and rake tasks, what works and what is a little rough around the edges.
Speaker Bio: Robert is an engineer who has spent the past several years attempting to automate away the need for the work that he does. Focusing on server automation and security work for BusinessWire, Robert also develops web services such as tee.ms, a chat service, and designs and develops games. Trism, which he co-designed, was nominated for Cellular Game of the Year by the Academy of Interactive Arts & Sciences in the 2009 Interactive Achievement Awards.
2016 - Easing Your Way Into Docker: Lessons From a Journey to Production
Presentation by Steve Woodruff
The story of how SpareFoot broke up its monolithic application into micro services, deployed Docker into production, and established a "contract" between Dev and Ops.
Puppet overview
Overview of Puppet, Puppet Labs, and the Puppet "ecosystem". Recent changes in Puppet, community resources, and learning resources.
Openstack Summit Tokyo 2015 - Building a private cloud to efficiently handle ...
What do you do when your usual setup or turnkey solution isn’t suited for your workload?
Most of the documentation and user feedback that you can find about OpenStack is written for the use-case of running a public facing cloud serving several external customers. When you want to host a single tenant with a single application the problem is completely different, you don't want publicly exposed APIs. You want to ensure optimal resource allocation to maximize your application performance. You want to leverage the fact that you own the infrastructure layer to optimize your instance placement strategy, and to get the best latency and to avoid creating SPOFs using affinity (or anti affinity rules).
This talk will focus on what we learned during a two years journey; from getting OpenStack up and running reliably, to investigating performance bottlenecks, to maximizing the performance of our private cloud.
堵俊平：Hadoop virtualization extensions
This document discusses big data in the cloud and provides an overview of YARN. It begins with introducing the speaker and their experience with VMware and Apache Hadoop. The rest of the document covers: 1) trends in big data like the rise of YARN, faster query engines, and focus on enterprise capabilities, 2) how YARN addresses limitations of MapReduce by splitting responsibilities, 3) how YARN serves as a hub for various big data applications, and 4) how YARN can integrate with cloud infrastructure for elastic resource management between the two frameworks. The document advocates for open source contribution to help advance big data technologies.
create auto scale jboss cluster with openshift
The document discusses running a JBoss EAP cluster on OpenShift. It introduces containers and Docker as a way to package and run applications. OpenShift is presented as a way to automate deployment and scaling of containerized applications on Kubernetes. The document demonstrates deploying a simple Java application using JBoss EAP on OpenShift, configuring the cluster, and enabling auto-scaling based on CPU usage.
MongoDB Ops Manager and Kubernetes - James Broadhead
This document provides an overview of how to deploy and manage MongoDB in Kubernetes using the MongoDB Enterprise Kubernetes Operator. It begins with introductions to Kubernetes, Operators, and MongoDB Ops Manager. It then demonstrates deploying a sharded MongoDB cluster with the Operator, including examples of operational tasks like version upgrades, horizontal and vertical scaling. The document also covers topics like fault tolerance, performance, and automation benefits of the Operator. It concludes by stating the current beta status of the Operator and encouraging questions.
Building Modular Cloud Applications in Java - Lessons Learned
Over the past couple of years we have been building large scale Java applications and deployed them in the cloud. While dealing with typical cloud issues such as auto scaling, failover, and incremental deployments we meanwhile had to refactor them for significant functional changes.
Being able to deal with change and dynamics are important reasons for companies to resort to the cloud for application development. Short time to market and rapidly changing requirements can lead to experimental software design however. Dealing with constant change in a codebase is non-trivial. In order to facilitate much change a modular codebase is of great importance.
In this talk we will show you what we have learned on how to create modular Java applications and how to deploy them in the cloud using an Open Source approach without locking into one of the PaaS providers. We will discuss OSGi, scalable web application architecture, and Apache ACE for deployment and provisioning.
The Unbearable Lightness of Ephemeral Processing
Ephemeral clusters can be launched quickly (minutes), are pre-configured for a specific processing purpose, and can be brought down quickly as soon as their usefulness has expired. The ability to launch Ephemeral clusters for on-demand processing, quickly and efficiently, is transforming how organizations design, deploy and Manage applications. The velocity and elasticity of fast cluster deployment enables seamless peak-demand provisioning, enables cost optimization by leveraging significantly lower cloud spot pricing, and maximizes utilization of existing compute capacity. Additionally, being able to launch bespoke clusters for specific compute needs in a repeatable fashion and within a shared infrastructure provides flexibility for special purpose processing needs. Organizations can leverage Ephemeral Clusters for parallel compute intensive applications which require short bursts of power but are short lived. In this session we will explore how to design Ephemeral clusters, how to launch, modify and bring them down, as well as application design considerations to maximize Ephemeral clusters usability.
Configuration Management and Transforming Legacy Applications in the Enterpri...
Share the continuity of Société Générale's journey with Docker Enterprise from different points of view, from executives to devops, with CD platform as an enabler. Creating a Dockerfile that runs a container on a developer's laptop is pretty straightforward. But extending that to stacks of containers running on a dozen environments (development, integration, testing, staging, production, etc.) with different configuration and topologies can be a challenge. This talk will cover aspects of our journey to Docker Enterprise:
What configuration should go in an image?
Where to put different types of configuration? Images, environment variables, entrypoint, ...?
How to store assets for building images and configuration for deployment in version control.
We will discuss how Société Générale has implemented these, and what we plan next for Docker Enterprise deployment.
Running Spark on Cloud
This document discusses running Spark on the cloud, including the advantages, challenges, and how Qubole addresses them. Some key advantages include using S3 for storage which allows independent scaling of storage and compute, ability to create ephemeral clusters on demand, and autoscaling capabilities. Challenges involve cluster lifecycle management, different interfaces needed, Spark autoscaling, debuggability across clusters, and handling spot instances. Qubole provides tools that automate cluster management, enable autoscaling of Spark, and make experiences seamless across clusters and interfaces.
Using Docker and Elastic Beanstalk in Production
This document discusses using Docker and Elastic Beanstalk for deploying microservices architectures in production. It covers moving from monolithic to service-oriented architectures, and advantages of treating infrastructure as "cattle" rather than "pets". Docker is introduced as a way to package applications into lightweight, portable containers instead of virtual machines. Configuration steps are outlined for building Docker images, pushing them to Docker Hub, and deploying containers to Elastic Beanstalk. Health checks, Node.js configuration, and application monitoring strategies are also briefly discussed.
A docker love story
1) Docker Swarm provides a simple way to orchestrate containers across multiple nodes using built-in tools for scheduling, load balancing, and configuration through environment variables.
2) Kubernetes and Mesosphere are more fully featured orchestration platforms that can schedule both containers and tasks, but have additional dependencies and more complex configurations.
3) When moving to microservices, the document recommends containerizing the application components rather than the infrastructure, starting small, having contingency plans for failure, and managing orchestration without hosted solutions.
Percona presentation v2
This document discusses challenges and lessons learned with OpenStack deployments and MySQL. It provides background on the author and their experience with OpenStack. Key points include that OpenStack is not simple to deploy and lacks capabilities required for production use out of the box. The document also discusses eNovance's OpenStack product, which aims to deliver a fully highly available OpenStack deployment with support for features like high availability, upgrades, and multi-data center capabilities. MySQL is commonly used as the database for OpenStack services, and the document shares experiences using Galera clustering for MySQL.
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
The Science and Technology Facilities Council is a UK Research Council which funds research and provides large facilities to the UK Scientific Community. This includes running a Tier 1 site for the LHC computing project, the JASMIN Super Data Cluster and a number of other HPC and HTC facilities. The Scientific Computing Department at the Rutherford Appleton Laboratory has been developing a cloud for use across both sites of the Department and in the wider scientific community. This is an OpenNebula backed by Ceph block storage. I will give a brief background of the project, describe our set up, some use cases and the work we have done around OpenNebula (including a simplified web front-end and a number of hooks to provide us with traceability). I will also discuss how we are creating an elastic boundary between our HTC batch farm and cloud.
Author Biography
I am a Systems Administrator in the Scientific Computing Department of the UK’s Science and Technology Facilities Council. I work as part of the cloud team and I also work on a number of Grid services including our HTC batch farm for the LHC computing project.
Prior to my position here I worked in IT at a SMB focusing on Storage and Virtualisation, in particular Hyper-V and VMWare.
Similar to Mesos and Container Schedulers (20)
2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure
2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Cloud-Native Apache Spark Scheduling with YuniKorn Scheduler
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
OCCIware Project at EclipseCon France 2016, by Marc Dutoo, Open Wide
OCCIware Project at EclipseCon France 2016, by Marc Dutoo, Open Wide
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
Bootstrapping Puppet and Application Deployment - PuppetConf 2013
2016 - Easing Your Way Into Docker: Lessons From a Journey to Production
2016 - Easing Your Way Into Docker: Lessons From a Journey to Production
Openstack Summit Tokyo 2015 - Building a private cloud to efficiently handle ...
Openstack Summit Tokyo 2015 - Building a private cloud to efficiently handle ...
MongoDB Ops Manager and Kubernetes - James Broadhead
MongoDB Ops Manager and Kubernetes - James Broadhead
Building Modular Cloud Applications in Java - Lessons Learned
Building Modular Cloud Applications in Java - Lessons Learned
Configuration Management and Transforming Legacy Applications in the Enterpri...
Configuration Management and Transforming Legacy Applications in the Enterpri...
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
OpenNebulaConf2015 1.07 Cloud for Scientific Computing @ STFC - Alexander Dibbo
More from J On The Beach
Massively scalable ETL in real world applications: the hard way
Big Data examples always give the correct answers. However, in the real world, Big Data might be corrupt, contradictory or consist of so many small files it becomes extremely hard to keep track - let alone scale. A solid architecture will help to overcome many of the difficulties.
Floris will talk about a real-world implementation of a massively scalable ETL architecture. Two years ago, at the time of the implementation, Airflow just became part of Apache and still left many features to be desired for. However, requirements from the start were thousands of ETL tasks per day on average, but on occasion, this could become hundreds of thousands. The script-based method that was in place was already not capable to meet the requirements on a day to day basis and needed to be replaced as soon as possible. So this custom framework was rolled out in just 8 weeks of development time.
Big Data On Data You Don’t Have
Traditional Big Data is done on Data you have. You load the data into a repository and perform map reduce or other style calculations on the data. However, certain industries need to perform complex operations on data you might not have. Data you can acquire, Data that can be shared with you, and Data that you can model are all types of data you may not have but may need to integrate instantly into a complex data analysis. Problem is: you may not even know you need this data until deep into the execution stack at runtime. This talk discusses a new functional language paradigm for dealing naturally with data you don’t have and about how to make all data first-class citizens, regardless of whether you have it or you don’t, and we will give a demo of a project written in Scala to deal exactly with this issue.
Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...
Industry 4.0, aka the "Fourth Industrial Revolution," refers to the computerization of manufacturing. One important aspect of Industry 4.0 is the ability to monitor the health and reliability of a physical manufacturing plant using low-cost IoT sensors. For example, machine learning models can be trained to predict the physical degradation of a manufacturing system as a function of acoustic measurements obtained from strategically placed microphones; however, the same acoustic measurements can be used to reverse engineer proprietary information about the manufacturing process and/or precisely what is being manufactured at the time of recording. Thus, improved reliability and fault tolerance is achieved at the cost of what appears to be an unprecedented new class of security vulnerabilities related to the acoustic side channel.
As a case study, we report a novel acoustic side channel attack against a commercial DNA synthesizer, a commonly used instrument in fields such as synthetic biology. Using a smart phone-quality microphone placed on or in the near vicinity of a DNA synthesizer, we were able to determine with 88.07% accuracy the sequence of DNA being produced; using a database of biologically relevant known-sequences, we increased the accuracy of our model to 100%. An academic or industrial research project may use the synthetic DNA to engineer an organism with desired traits or functions; however, while the organism is still under development, prior to publication, patent, and/or copyright, the research remains vulnerable to academic intellectual property theft and/or industrial espionage. On the other hand, this attack could also be used for benevolent purposes, for example, to determine whether a suspected criminal or terrorist is engineering a harmful pathogen. Thus, it is essential to recognize both the benefits and risks inherent to the cyber-physical systems that will inevitably control Industry 4.0 manufacturing processes and to take steps to mitigate them whenever possible.
Pushing it to the edge in IoT
Where is the edge in IoT and how much can you do there? Data collection? Analytics? I’ll show you how to build and deploy an embedded IoT edge platform that can do data collection, analytics, dashboarding and much more. All using Open Source.
As IoT deployments move forward, the need to collect, analyze, and respond to data further out on the edge becomes a critical factor in the success – or failure – of any IoT project. Network bandwidth costs may be dropping, and storage is cheaper than ever, but at IoT scale, these costs can still quickly overrun a project’s budget and ultimately doom it to failure.
The more you centralize your data collection and storage, the higher these costs become. Edge data collection and analysis can dramatically lower these costs, plus decrease the time to react to critical sensor data. With most data platforms, it simply isn’t practical, or even possible, to push collection AND analytics to the edge. In this talk I’ll show how I’ve done exactly this with a combination of open source hardware – Pine64 – and open source software – InfluxDB – to build a practical, efficient and scalable data collection and analysis gateway device for IoT deployments. The edge is where the data is, so the edge is where the data collection and analytics needs to be.
Drinking from the firehose, with virtual streams and virtual actors
Event Stream Processing is a popular paradigm for building robust and performant systems in many different domains, from IoT to fraud detection to high-frequency trading. Because of the wide range of scenarios and requirements, it is difficult to conceptualize a unified programming model that would be equally applicable to all of them. Another tough challenge is how to build streaming systems with cardinalities of topics ranging from hundreds to billions while delivering good performance and scalability.
In this session, Sergey Bykov will talk about the journey of building Orleans Streams that originated in gaming and monitoring scenarios, and quickly expanded beyond them. He will cover the programming model of virtual streams that emerged as a natural extension of the virtual actor model of Orleans, the architecture of the underlying runtime system, the compromises and hard choices made in the process. Sergey will share the lessons learned from the experience of running the system in production, and future ideas and opportunities that remain to be explored.
How do we deploy? From Punched cards to Immutable server pattern
A short retrospective of deployment approaches evolution and the key features of the most up-to-date concepts.
Java, Turbocharged
Over the last twenty years, there has been a paradigm shift in software development: from meticulously planned release cycles to an experimental way of working in which lead times are becoming shorter and shorter.
How can Java ever keep up with this trend when we have Docker containers that are several hundred megabytes in size, with warm-up times of ten minutes or longer? In this talk, I'll demonstrate how we can use Quarkus so that we can create super small, super fast Java containers! This will give us better possibilities for scaling up and down - which can be a game-changer, especially in a serverless environment. It will also provide the shortest possible lead times, as well as a much better use of cloud performance with the added bonus of lower costs.
When Cloud Native meets the Financial Sector
We live in our own bubble of microservices and endlessly horizontal scaling infrastructure, but there is still critical infrastructure that runs the world of financial systems depending on Windows boxes, FTP servers, and single-threaded protocols. This talk is about how to glue these two worlds together, what works for us and what doesn't.
The big data Universe. Literally.
The document discusses the challenges of big data in astronomy. It describes several upcoming and current astronomical surveys that will produce enormous amounts of data, such as Euclid which will collect 15 terabytes per day starting in 2022. Dealing with such large datasets requires new techniques for storage, processing, analyzing, classifying, compressing and visualizing the data. Machine learning and citizen science are helping to process the data more efficiently. Future surveys like the Square Kilometer Array telescope will pose even greater challenges by producing data at terabytes per second.
Streaming to a New Jakarta EE
The world is moving from a model where data sits at rest, waiting for people to make requests of it, to where data is constantly moving and streams of data flow to and from devices with or without human interaction. Decisions need to be made based on these streams of data in real-time, models need to be updated, and intelligence needs to be gathered. In this context, our old-fashioned approach of CRUD REST APIs serving CRUD database calls just doesn't cut it. It's time we moved to a stream-centric view of the world.
The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...
Our increasingly connected world leveraging the Internet of Things (IoT) creates great value, in connected healthcare, smart cities, and more. The increasing use of IoT also creates great risk. We will discuss the challenges and risks we need to address as developers in TIPPSS - Trust, Identity, Privacy, Protection, Safety, and Security - for devices, systems and solutions we deliver and use. Florence leads IEEE workstreams on clinical IoT and data interoperability with blockchain addressing TIPPSS issues. She is an author of IEEE articles on "Enabling Trust and Security - TIPPSS for IoT" and "Wearables and Medical Interoperability - the Evolving Frontier", "TIPPSS for Smart Cities" in the 2017 book "Creating, Analysing and Sustaining Smarter Cities: A Systems Perspective" , and Editor in Chief for an upcoming book on "Women Securing the Future with TIPPSS for IoT."
Pushing AI to the Client with WebAssembly and Blazor
Want to run your AI algorithms directly in the browser on the client-side? Now you can with WebAssembly and Blazor. Join us as we write code directly in WebAssembly. Then, we’ll look at Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser.
Want to run your AI algorithms directly in the browser on the client-side without the need for transpilers or browser plug-ins? Well, now you can with WebAssembly and Blazor. WebAssembly (WASM) is the W3C specification that will be used to provide the next generation of development tools for the web and beyond. Blazor is Microsoft’s experiment that allows ASP.Net developers to create web pages that do much of the scripting work in C# using WASM. Come join us as we learn to write code directly in WebAssembly’s human-readable format. Then, we’ll look at the current state of Blazor and how you can use it, along with WebAssembly to run your tooling client side in the browser.
Axon Server went RAFTing
RAFT protocol is a well-known protocol for consensus in Distributed Systems. Want to learn how consensus is achieved in a system with a large amount of data such as Axon Server’s Event Store? Join this talk to hear about all specifics regarding data replication in highly available Event Store!
Axon is a free and open source Java framework for writing Java applications following DDD, event sourcing, and CQRS principles. While especially useful in a microservices context, Axon provides great value in building structured monoliths that can be broken down into microservices when needed.
Axon Server is a messaging platform specifically built to support distributed Axon applications. One of its key benefits is storing events published by Axon applications. In not so rare cases, the number of these events is over millions, even billions. Availability of Axon Server plays a significant role in the product portfolio. To keep event replication reliable we chose RAFT protocol for consensus implementation of our clustering features.
In short, consensus involves multiple servers agreeing on values. Once they reach a decision on a value, that decision is final. Typical consensus algorithms make progress when any majority of their servers is available; for example, a cluster of 5 servers can continue to operate even if 2 servers fail. If more servers fail, they stop making progress (but will never return an incorrect result).
Join this talk to learn why we chose RAFT; what were our findings during the design, the implementation, and testing phase; and what does it mean to replicate an event store holding billions of events!
The Six Pitfalls of building a Microservices Architecture (and how to avoid t...
Thinking of moving to Microservices? Watch out! That quest is full of traps, social traps. If you are not able to handle it, you may be blocked by meetings, frustration, endless challenges that will make you miss the monolith. In this talk, I share my experience and mistakes, so you can avoid them.
Creating or migrating to a Microservices architecture might easily become a big mess, not only due to technical challenges but mostly because of human factors: it’s a major change in the software culture of a company. In this talk, I’ll share my past experience as the technical lead of an ambitious Microservices-based product, I’ll go through the parts we struggled with, and give you some advice on how to deal with what I call the Six Pitfalls:
The Common Patterns Phobia
The Book Club Cult
The Never-Decoupled Story
The Buzz Words Syndrome
The Agile Trap
The Conway’s Law Hackers
Madaari : Ordering For The Monkeys
Instead of randomly injecting faults ( i.e. Chaos Monkey), what if we could order our experiments to perform min number of experiments for maximum yield? We present a solution(& results) to the problem of experiment selection using Lineage Driven Fault Injection to reduce the search space of faults.
Lineage Driven Fault Injection (LDFI) is a state of the art technique in chaos engineering experiment selection. LDFI since its inception has used an SAT solver under the hood which presents solutions to the decision problem (which faults to inject) in no particular order. As SRE’s we would like to perform experiments that reveal the bugs that the customers are most likely to hit first. In this talk, we present new improvements to LDFI that orders the experiment suggestions.
In the first the half of the talk we will show LDFI is a technique that can be widely used within an enterprise. We present the motivation for ordering the chaos experiments along with some prioritization we utilized while conducting the experiments. We also highlight how ordering is a general purpose technique that we can use to encode the peculiarities of a heterogeneous microservices architecture. LDFI can work in an enterprise by harnessing the observability infrastructure to model the redundancy of the system.
Next, we present experiments conducted within our organization using ordered LDFI and some preliminary results. We show examples of services where we discovered bugs, and how carefully controlling the order of experiments allowed LDFI to avoid running unnecessary experiments. We also present an example of an application where we declared the service shippable under crash stop model. We also present a comparison with Chaos Monkey and show how LDFI found the known bugs in a given application using orders of magnitude fewer experiments than a random fault injection tool like Chaos Monkey.
Finally, we discuss how we plan to take LDFI forward. We discuss open problems and possible solutions for scalarizing probabilities of failure, latency injection, integration with service mesh technologies like envoy for fine-grained fault injection, fault injection for stateful systems.
Key takeaways: 1) Understand how LDFI can be integrated in the enterprise by harnessing the observability infrastructure. 2) Limitations of LDFI w.r.t unordered solutions and why ordering matters for chaos engineering experiments. 3) Preliminary results of prioritized LDFI and a future direction for the community.
Servers are doomed to fail
Complexity in systems should be defeated if it is possible to do. But the default nature of our computer systems are complex and servers are doomed to fail. In this talk, we will go through new approaches in modern architectures to design and evaluate new computer systems.
Interaction Protocols: It's all about good manners
Distributed systems collaborate to achieve collective goals via a system of rules. Rules that affords good hygiene, fault tolerance, effective communication and trusted feedback. These rules form protocols which enable the system to achieve its goals.
Distributed and concurrent systems can be considered a social group that collaborates to achieve collective goals. In order to collaborate a system of rules must be applied, that affords good hygiene, fault tolerance, and effective communication to coordinate, share knowledge, and provide feedback in a polite trusted manner. These rules form a number of protocols which enable the group to act as a system which is greater than the sum of the individual components.
In this talk, we will explore the history of protocols and their application when building distributed systems.
A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...
Do you want to check the efficiency of the new, state of the art, GraalVM JIT Compiler in comparison to the old but mostly used JIT C2? Let’s have a side by side comparison from a performance standpoint on the same source code.
The talk reveals how traditional Just In Time Compiler (e.g. JIT C2) from HotSpot/OpenJDK internally manages runtime optimizations for hot methods in comparison to the new, state of the art, GraalVM JIT Compiler on the same source code, emphasizing all of the internals and strategies used by each Compiler to achieve better performance in most common situations (or code patterns). For each optimization, there is Java source code and corresponding generated assembly code in order to prove what really happens under the hood.
Each test is covered by a dedicated benchmark (JMH), timings and conclusions. Main topics of the agenda: - Scalar replacement - Null Checks - Virtual calls - Lock coarsening - Lock elision - Virtual calls - Scalar replacement - Lambdas - Vectorization (few cases)
The tools used during my research study are JITWatch, Java Measurement Harness, and perf. All test scenarios will be launched against the latest official Java release (e.g. version 11).
Leadership at every level
Leadership is easy when you're a manager, or an expert in a field, or a conference speaker! In a Kanban organisation, though, we "encourage acts of leadership at every level". In this talk, we look at what it means to be a leader in the uncertain, changing and high-learning environment of software development. We learn about the importance of safety in encouraging others to lead and follow, and how to get that safety using both technical and human practices; the necessity of a clear, compelling vision and provision of information on how we're achieving it; and the need to be able to ask awkward and difficult questions... especially the ones without easy answers.
Machine Learning: The Bare Math Behind Libraries
During this presentation, we will answer how much you’ll need to invest in a superhero costume to be as popular as Superman. We will generate a unique logo which will stand against the ever popular Batman and create new superhero teams. We shall achieve it using linear regression and neural networks.
Machine learning is one of the hottest buzzwords in technology today as well as one of the most innovative fields in computer science – yet people use libraries as black boxes without basic knowledge of the field. In this session, we will strip them to bare math, so next time you use a machine learning library, you’ll have a deeper understanding of what lies underneath.
During this session, we will first provide a short history of machine learning and an overview of two basic teaching techniques: supervised and unsupervised learning.
We will start by defining what machine learning is and equip you with an intuition of how it works. We will then explain the gradient descent algorithm with the use of simple linear regression to give you an even deeper understanding of this learning method. Then we will project it to supervised neural networks training.
Within unsupervised learning, you will become familiar with Hebb’s learning and learning with concurrency (winner takes all and winner takes most algorithms). We will use Octave for examples in this session; however, you can use your favourite technology to implement presented ideas.
Our aim is to show the mathematical basics of neural networks for those who want to start using machine learning in their day-to-day work or use it already but find it difficult to understand the underlying processes. After viewing our presentation, you should find it easier to select parameters for your networks and feel more confident in your selection of network type, as well as be encouraged to dive into more complex and powerful deep learning methods.
More from J On The Beach (20)
Massively scalable ETL in real world applications: the hard way
Massively scalable ETL in real world applications: the hard way
Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...
Acoustic Time Series in Industry 4.0: Improved Reliability and Cyber-Security...
Drinking from the firehose, with virtual streams and virtual actors
Drinking from the firehose, with virtual streams and virtual actors
How do we deploy? From Punched cards to Immutable server pattern
How do we deploy? From Punched cards to Immutable server pattern
The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...
The TIPPSS Imperative for IoT - Ensuring Trust, Identity, Privacy, Protection...
Pushing AI to the Client with WebAssembly and Blazor
Pushing AI to the Client with WebAssembly and Blazor
The Six Pitfalls of building a Microservices Architecture (and how to avoid t...
The Six Pitfalls of building a Microservices Architecture (and how to avoid t...
Interaction Protocols: It's all about good manners
Interaction Protocols: It's all about good manners
A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...
A race of two compilers: GraalVM JIT versus HotSpot JIT C2. Which one offers ...
Recently uploaded
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Recently uploaded (20)
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Mesos and Container Schedulers
- 1. Mesos and Container Schedulers Landscape and Practical Experiences
- 2. Piksel: We do video stuff
- 3. Piksel’s big data toolset
- 4. Multi-tenant PaaS for Heterogeneous Workloads - E.g. Metadata Enrichment
- 5. Piksel Video Platform - Cluster Evolution
- 6. Previous - Static Partitioning at VM Level
- 7. Impossible to efficiently utilise resources across machines
- 8. Too much EC2 We estimated that this was about 20-30% utilized
- 10. Puppet - VM Focused Deployment With provisioning tools VMs are provisioned with a specific role in mind.
- 11. Teams are thinking about machines rather than applications
- 12. Cluster Manager: VMs are provisioned to contribute resources With cluster manager the VMs are provisioned to contribute resources
- 14. Java big data frameworks
- 15. We dipped our toe in the cluster pool
- 16. We ran with YARN and with Mesos
- 17. API services: Docker, marathon, mesos
- 18. Heterogeneous API services, homogeneous workload
- 20. Multiple clusters - Myriad
- 21. Where we wanted to get to ● Single platform for deploying applications ● Development teams don’t care about machines ● Development teams do care about applications ● Development teams describe their application and operational characteristics declaratively
- 22. Heterogeneous Workload, single PaaS
- 23. Let’s Talk about Workload Types
- 24. Video Platform at a Glance
- 26. But Mesos Needs a Framework to Schedule work
- 27. The Role of a Framework Mesos provides resource offers to frameworks which can choose to accept or decline them subject to various constraints
- 28. Mesos is Low-Level - Requires a Framework Credit: https://twitter.com/wendigo/status/646394656700375040
- 29. Framework capabilities - the BIG THREE
- 30. Framework capabilities ● Choice between Aurora and the others boils down to the need for batch jobs and pre-emption ● Do you run DAG jobs? ● Do you have a large engineering team? Or multiple isolated customers that need quotas?
- 32. Noisy neighbours are a problem!
- 33. Customers…. ● Some customers pay more than others, and need more capacity ● Some jobs are higher priority than others ● We don’t need all capacity, all the time
- 34. Aurora - Roles Aurora has a concept of a Role, e.g. a Customer or Internal Team… Work is submitted into a Role. Quotas can be assigned to Roles.
- 35. Aurora - Goal ● Ensure high priority work gets done ● Provide quota isolation across Roles ● Allow low priority work to use all resources unless high priority work arrives
- 36. Aurora - Pre-emption Aurora supports pre-emption; killing and rescheduling work when higher priority work arrives. It does this under 2 conditions...
- 37. Aurora - Priority killing 1. Role submits a job with higher priority that existing job in same Role
- 38. Aurora - Intra-Role Prioritization 2. A production job requires quota from a non-production job of any Role
- 39. Multi-Tenancy Scheduling - Production Quotas ● Aurora allows us to give a quota to Roles for their critical work
- 40. Multi-Tenancy Scheduling - Environments and Pre-emption
- 41. Multi-Tenancy Scheduling - Slot Non-Critical Work Around Critical Work Aurora allows us to slot low-priority (typically batch) work around critical work, yet discard it if more critical work arrives
- 42. Frameworks that provision their own resources need Aurora integration
- 43. Problem: Workloads that provision their own resources
- 44. Problem: Workloads that provision their own resources
- 45. PVP - Current state of affairs
- 47. How do we submit work to Aurora
- 48. Job DAGs Aurora allows Jobs to contain parallel and sequential stages, resulting a DAG. However all processes will be ran in the same container. It is not a high level job orchestrator.
- 49. Domain Specific Job Definition Language With Templating transcode_proc = Process( name=’transcode’, cmdline=’ffmpeg -i {{input}} {{output}} ’ ) transcode_task = Task( name = ‘transcode_task ’, processes = [transcode_proc] resources = Resources(cpu = 1, ram = 128*MB, disk=1*GB) ) jobs = [ Job(cluster = 'devcluster', environment = 'sandbox', role = 'customer_a', name = 'transcode_job', task = transcode_task, container = Docker(image = 'ffmpeg_image:16.04 ') ) ]
- 50. Customer Extensions - Extending the DSL transcode_task = transcode() .input(“s3://input.mp4”) .output(“s3://output.avi”) jobs = Flow.create(transcode_task) No nice hook exists to plug in DSL extensions. We need to patch and build the Aurora client and then register this within the configuration loader.