Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2jk5yDt.
Jie Yu overviews Mesos and its API which allows users to deploy stateless and stateful services. She discusses how containers are managed in Mesos, the future of container support in Mesos, and shows some of the new container networking and storage features that are built recently. Filmed at qconsf.com.
Jie Yu is a Software Engineer at Mesosphere, leading the containerization development in Mesos. He is a PMC member of the Apache Mesos project, and one of the top committers. Before joining Mesosphere, he was a Software Engineer at Twitter.
Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
The document compares the cluster management systems Mesos and Kubernetes. Mesos is an Apache project that provides efficient resource sharing across distributed applications. Kubernetes, developed by Google, provides cluster management capabilities for virtual machines and containers. Both support Docker containers and scheduling of workloads but Mesos gives more control while Kubernetes has easier setup and built-in scheduling. The document also discusses their architectures, major customers, and the future of these systems.
Deploying Containers in Production and at ScaleMesosphere Inc.
This presentation was part of "Deploying Containers in Production and at Scale" by Sunil Shah (Engineer at Mesosphere) at ContainerCon 2015
Try Mesosphere for Free: https://mesosphere.com/try
Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It divides resources into CPU, memory, storage, and other compute resources and shares those resources dynamically and efficiently across applications. Mesos abstracts the underlying infrastructure to provide a unified API to applications while employing operating system-level virtualization through interfaces like Docker to maximize resource utilization. It works by having a Mesos master that negotiates resources among Mesos slaves to run applications or frameworks, which are made up of a scheduler to negotiate for resources and executors to run tasks. Common frameworks that run on Mesos include Spark, Hadoop and Docker containers.
This document discusses DC/OS, an open source tool that provides container orchestration and management on top of Apache Mesos. It addresses common cluster computing problems like resource management, task scheduling, and self-healing infrastructure. It also introduces key DC/OS components like Mesos, Marathon, the DC/OS CLI, Universe package repository, and how they work together to deploy and manage containerized applications on a cluster.
Mesos: The Operating System for your DatacenterDavid Greenberg
Maybe you’ve heard of Mesos—that thing that you can run Hadoop on. I think it powers Twitter? Isn’t it an Apache project, or something?
In this talk, we’ll learn all about Mesos—what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
The document discusses using DC/OS for big data operations. It outlines presenting on DC/OS for big data ops and deploying a data pipeline on a DC/OS cluster. The presentation includes deploying HDFS on a DC/OS cluster, demonstrating running a Twitter data analytics pipeline on a 6-node DC/OS cluster in AWS, and recap that DC/OS simplifies production and operations for big data.
Presented at MesosCon EU 2015.
Linux containers, popularized by Docker, have been a game-changer in data center computing in recent years. Mesos has supported container isolation since its early days and has been supporting Docker since 0.20. This talk gives an overview of the evolution of Mesos containerization and an introduction to an upcoming Mesos feature that provisions container (filesystem) images such as Appc and Docker and does filesystem isolation all natively through one unified containerizer without requiring any additional container image runtime. Lastly, it includes a case study for introducing container images to large running clusters both in terms of the number of hosts and the size of their host images and what we have learned along the way.
The document compares the cluster management systems Mesos and Kubernetes. Mesos is an Apache project that provides efficient resource sharing across distributed applications. Kubernetes, developed by Google, provides cluster management capabilities for virtual machines and containers. Both support Docker containers and scheduling of workloads but Mesos gives more control while Kubernetes has easier setup and built-in scheduling. The document also discusses their architectures, major customers, and the future of these systems.
Deploying Containers in Production and at ScaleMesosphere Inc.
This presentation was part of "Deploying Containers in Production and at Scale" by Sunil Shah (Engineer at Mesosphere) at ContainerCon 2015
Try Mesosphere for Free: https://mesosphere.com/try
Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It divides resources into CPU, memory, storage, and other compute resources and shares those resources dynamically and efficiently across applications. Mesos abstracts the underlying infrastructure to provide a unified API to applications while employing operating system-level virtualization through interfaces like Docker to maximize resource utilization. It works by having a Mesos master that negotiates resources among Mesos slaves to run applications or frameworks, which are made up of a scheduler to negotiate for resources and executors to run tasks. Common frameworks that run on Mesos include Spark, Hadoop and Docker containers.
This document discusses DC/OS, an open source tool that provides container orchestration and management on top of Apache Mesos. It addresses common cluster computing problems like resource management, task scheduling, and self-healing infrastructure. It also introduces key DC/OS components like Mesos, Marathon, the DC/OS CLI, Universe package repository, and how they work together to deploy and manage containerized applications on a cluster.
Mesos: The Operating System for your DatacenterDavid Greenberg
Maybe you’ve heard of Mesos—that thing that you can run Hadoop on. I think it powers Twitter? Isn’t it an Apache project, or something?
In this talk, we’ll learn all about Mesos—what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
The document discusses using DC/OS for big data operations. It outlines presenting on DC/OS for big data ops and deploying a data pipeline on a DC/OS cluster. The presentation includes deploying HDFS on a DC/OS cluster, demonstrating running a Twitter data analytics pipeline on a 6-node DC/OS cluster in AWS, and recap that DC/OS simplifies production and operations for big data.
Containers, cluster management, microservices, Kubernetes and many other buzzwords are flying around us all the time. Our team is building solutions that make it easy to cope with all the complexity around cluster infrastructure. In this talk we present the project we are working on, namely running Kubernetes on top of the Mesos cluster scheduler. Furthermore we show DCOS which makes it easy to deploy and run Kubernetes with a single command.
Scalable On-Demand Hadoop Clusters with Docker and Mesosnelsonadpresent
This document discusses using Docker and Mesos to provide scalable on-demand Hadoop clusters. It outlines how Mesos acts as a multi-tenant resource pool to allow multiple frameworks like Hadoop, Spark, and Storm to dynamically share resources. Docker is proposed as the new "unit of work" to provide a flexible and developer-friendly form factor. The document recommends starting small and scaling fast, using the most appropriate framework for each job, and planning for rolling restarts to take advantage of Mesos for on-demand Hadoop clusters.
How to Survive an OpenStack Cloud Meltdown with CephSean Cohen
What if you lost your datacenter completely in a catastrophe, but your users hardly noticed? Sounds like a mirage, but it’s absolutely possible.
This talk will showcase OpenStack features enabling multisite and disaster recovery functionalities. We’ll present the latest capabilities of OpenStack and Ceph for Volume and Image Replication using Ceph Block and Object as the backend storage solution, as well as look at the future developments they are driving to improve and simplify the relevant architecture use cases, such as Distributed NFV, an emerging use case that rationalizes your IT by using less control planes and allows you to spread your VNF on multiple datacenters and edge deployments.
In this session you will learn about wew OpenStack features enabling Multisite and distributed deployments, as well as review key use cases, architecture design and best practices to help operations avoid the OpenStack cloud Meltdown nightmare.
https://youtu.be/n2S7uNC_KMw
https://goo.gl/cRNGBK
This document discusses the ceph-mesos framework, which implements a Mesos scheduler and executor for Ceph. The goal is to provide RADOS services like the RADOS gateway in a Mesos cluster. The scheduler has callback modules that interact with the Mesos master and provide a REST API and static file server. The executor launches Ceph Docker containers as tasks. The framework is still in early development and future work includes improving support for host hardware selection and networking configurations to optimize Ceph performance. A video demo of ceph-mesos is available online.
Elastic HBase on Mesos aims to improve resource utilization of HBase clusters by running HBase in Docker containers managed by Mesos and Marathon. This allows HBase clusters to dynamically scale based on varying workload demands, increases utilization by running mixed workloads on shared resources, and simplifies operations through standard containerization. Key benefits include easier management, higher efficiency through elastic scaling and resource sharing, and improved cluster tunability.
The document discusses various virtualization and storage features in VMware's vSphere solution that can help optimize datacenter costs and efficiency. Key points include:
1) Thin provisioning allows virtual disks to only use allocated storage as needed, improving utilization over traditional thick provisioning.
2) Enhancements to software iSCSI and new storage management capabilities in vCenter help improve performance and flexibility.
3) Features like hot extend and storage vMotion allow live expanding of virtual disks and migrating VMs between storage systems with minimal disruption.
GlusterFS is an open-source networked file system that provides scalable, distributed storage for OpenStack. It aggregates disk storage from multiple servers into a single global namespace that is accessible from anywhere. GlusterFS uses a distributed hashing technique to spread files across servers and can be configured for replication or erasure coding for redundancy. It integrates with OpenStack components like Swift, Cinder, and Glance to provide block, object, and file-level storage services for virtual machines. Future integration plans include a Cinder-like file service (FaaS) and support for Windows and NFS shares.
The Future of Cloud Software Defined Storage with Ceph: Andrew Hatfield, Red HatOpenStack
Audience: Intermediate
About: Learn how cloud storage differs to traditional storage systems and how that delivers revolutionary benefits.
Starting with an overview of how Ceph integrates tightly into OpenStack, you’ll see why 62% of OpenStack users choose Ceph, we’ll then take a peek into the very near future to see how rapidly Ceph is advancing and how you’ll be able to achieve all your childhood hopes and dreams in ways you never thought possible.
Speaker Bio: Andrew Hatfield – Practice Lead–Cloud Storage and Big Data, Red Hat
Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined storage future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Red Hat Storage - Introduction to GlusterFSGlusterFS
Red Hat Storage introduces GlusterFS, an open source scale-out file system. GlusterFS provides scalable, affordable storage using commodity hardware. It allows linearly scaling performance and capacity by adding servers. GlusterFS has a global namespace and supports various protocols, enabling flexible deployment across private and public clouds. Many enterprises rely on GlusterFS for applications, virtual machines, Hadoop, and hybrid cloud solutions.
This session will examine the many options the data scientist has for running Spark clusters in public and private clouds. We will discuss various environments employing AWS, Mesos, containers, docker, and BlueData EPIC technologies and the benefits and challenges of each.
Speakers:
Tom Phelan, Co-founder and Chief Architect - BlueData Inc. Tom has spent the last 25 years as a senior architect, developer, and team lead in the computer software industry in Silicon Valley. Prior to co-founding BlueData, Tom spent 10 years at VMware as a senior architect and team lead in the core R&D Storage and Availability group. Most recently, Tom led one of the key projects – vFlash, focusing on integration of server-based Flash into the vSphere core hypervisor. Prior to VMware, Tom was part of the early team at Silicon Graphics that developed XFS, one of the most successful open source file systems. Earlier in his career, he was a key member of the Stratus team that ported the Unix operating system to their highly available computing platform. Tom received his Computer Science degree from the University of California, Berkeley.
BlueStore: a new, faster storage backend for CephSage Weil
Traditionally Ceph has made use of local file systems like XFS or btrfs to store its data. However, the mismatch between the OSD's requirements and the POSIX interface provided by kernel file systems has a huge performance cost and requires a lot of complexity. BlueStore, an entirely new OSD storage backend, utilizes block devices directly, doubling performance for most workloads. This talk will cover the motivation a new backend, the design and implementation, the improved performance on HDDs, SSDs, and NVMe, and discuss some of the thornier issues we had to overcome when replacing tried and true kernel file systems with entirely new code running in userspace.
This document outlines a course on Ceph storage. It covers Ceph history and components. Key components discussed include Object Storage Daemons (OSDs) that store data, Monitors that maintain the cluster map and provide consensus, and the Ceph journal. Other topics are the Ceph Gateway for object storage access, Ceph Block Device (RBD) for block storage, and CephFS for file storage. The goal is to understand Ceph concepts and deploy a Ceph cluster.
StatefulSet is used to run PostgreSQL pods across Kubernetes nodes for high availability. When a pod fails, StatefulSet will restart the pod on the same node. However, if the entire node fails, the PostgreSQL pod will not failover to another node by default. To manually failover the pod, it needs to be force deleted and it will restart on a different ready node. However, manual failovers are not recommended for production use.
Divide and conquer: resource segregation in the OpenStack cloudStephen Gordon
This document discusses resource segregation techniques in OpenStack clouds. It describes how infrastructure resources like compute hosts can be logically grouped using regions, cells, host aggregates, and availability zones. It also discusses workload segregation using server groups that define affinity and anti-affinity rules for instance placement. The goals of segregation include isolating workloads, ensuring high availability, and enabling horizontal scaling of the infrastructure.
Ceph began as a research project in 2005 to create a scalable object storage system. It was incubated at DreamHost from 2007-2012 and spun out as an independent company called Inktank in 2012. Key developments included the RADOS distributed storage cluster, erasure coding, and the Ceph filesystem. The project has grown a large community and is used in many production deployments, focusing on areas like tiering, erasure coding, replication, and integrating with the Linux kernel. Future plans include improving CephFS, expanding the ecosystem through different storage backends, strengthening governance, and targeting new use cases in big data and the enterprise.
The ability to create global namespace, high availability, and volume economics enables unique capabilities when Red Hat Storage Server is used on Amazon Web Services (AWS). In this session, you’ll learn about the capabilities and the potential of using AWS ephemeral storage, availability zones, and regions to deliver a truly distributed cloud storage solution that is geographically diverse, cost effective, and highly available.
New Ceph capabilities and Reference ArchitecturesKamesh Pemmaraju
Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph? Then this is the session for you!
In this two part session you learn details of:
• the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas.
• best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.
This document discusses deploying the DC/OS container management system on bare metal servers using Juju charms. It describes bootstrapping DC/OS masters and agents using Juju, relating them, and scaling agents. Additional charms can be deployed for monitoring with Nagios and logging with Logstash. Containers can then be deployed to DC/OS using the dcos-nginx charm. The current implementation supports deploying DC/OS masters and agents in a scalable way and deploying docker containers. Future work includes improving security, simplifying upgrades, and adding more actions.
Containers, cluster management, microservices, Kubernetes and many other buzzwords are flying around us all the time. Our team is building solutions that make it easy to cope with all the complexity around cluster infrastructure. In this talk we present the project we are working on, namely running Kubernetes on top of the Mesos cluster scheduler. Furthermore we show DCOS which makes it easy to deploy and run Kubernetes with a single command.
Scalable On-Demand Hadoop Clusters with Docker and Mesosnelsonadpresent
This document discusses using Docker and Mesos to provide scalable on-demand Hadoop clusters. It outlines how Mesos acts as a multi-tenant resource pool to allow multiple frameworks like Hadoop, Spark, and Storm to dynamically share resources. Docker is proposed as the new "unit of work" to provide a flexible and developer-friendly form factor. The document recommends starting small and scaling fast, using the most appropriate framework for each job, and planning for rolling restarts to take advantage of Mesos for on-demand Hadoop clusters.
How to Survive an OpenStack Cloud Meltdown with CephSean Cohen
What if you lost your datacenter completely in a catastrophe, but your users hardly noticed? Sounds like a mirage, but it’s absolutely possible.
This talk will showcase OpenStack features enabling multisite and disaster recovery functionalities. We’ll present the latest capabilities of OpenStack and Ceph for Volume and Image Replication using Ceph Block and Object as the backend storage solution, as well as look at the future developments they are driving to improve and simplify the relevant architecture use cases, such as Distributed NFV, an emerging use case that rationalizes your IT by using less control planes and allows you to spread your VNF on multiple datacenters and edge deployments.
In this session you will learn about wew OpenStack features enabling Multisite and distributed deployments, as well as review key use cases, architecture design and best practices to help operations avoid the OpenStack cloud Meltdown nightmare.
https://youtu.be/n2S7uNC_KMw
https://goo.gl/cRNGBK
This document discusses the ceph-mesos framework, which implements a Mesos scheduler and executor for Ceph. The goal is to provide RADOS services like the RADOS gateway in a Mesos cluster. The scheduler has callback modules that interact with the Mesos master and provide a REST API and static file server. The executor launches Ceph Docker containers as tasks. The framework is still in early development and future work includes improving support for host hardware selection and networking configurations to optimize Ceph performance. A video demo of ceph-mesos is available online.
Elastic HBase on Mesos aims to improve resource utilization of HBase clusters by running HBase in Docker containers managed by Mesos and Marathon. This allows HBase clusters to dynamically scale based on varying workload demands, increases utilization by running mixed workloads on shared resources, and simplifies operations through standard containerization. Key benefits include easier management, higher efficiency through elastic scaling and resource sharing, and improved cluster tunability.
The document discusses various virtualization and storage features in VMware's vSphere solution that can help optimize datacenter costs and efficiency. Key points include:
1) Thin provisioning allows virtual disks to only use allocated storage as needed, improving utilization over traditional thick provisioning.
2) Enhancements to software iSCSI and new storage management capabilities in vCenter help improve performance and flexibility.
3) Features like hot extend and storage vMotion allow live expanding of virtual disks and migrating VMs between storage systems with minimal disruption.
GlusterFS is an open-source networked file system that provides scalable, distributed storage for OpenStack. It aggregates disk storage from multiple servers into a single global namespace that is accessible from anywhere. GlusterFS uses a distributed hashing technique to spread files across servers and can be configured for replication or erasure coding for redundancy. It integrates with OpenStack components like Swift, Cinder, and Glance to provide block, object, and file-level storage services for virtual machines. Future integration plans include a Cinder-like file service (FaaS) and support for Windows and NFS shares.
The Future of Cloud Software Defined Storage with Ceph: Andrew Hatfield, Red HatOpenStack
Audience: Intermediate
About: Learn how cloud storage differs to traditional storage systems and how that delivers revolutionary benefits.
Starting with an overview of how Ceph integrates tightly into OpenStack, you’ll see why 62% of OpenStack users choose Ceph, we’ll then take a peek into the very near future to see how rapidly Ceph is advancing and how you’ll be able to achieve all your childhood hopes and dreams in ways you never thought possible.
Speaker Bio: Andrew Hatfield – Practice Lead–Cloud Storage and Big Data, Red Hat
Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined storage future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Red Hat Storage - Introduction to GlusterFSGlusterFS
Red Hat Storage introduces GlusterFS, an open source scale-out file system. GlusterFS provides scalable, affordable storage using commodity hardware. It allows linearly scaling performance and capacity by adding servers. GlusterFS has a global namespace and supports various protocols, enabling flexible deployment across private and public clouds. Many enterprises rely on GlusterFS for applications, virtual machines, Hadoop, and hybrid cloud solutions.
This session will examine the many options the data scientist has for running Spark clusters in public and private clouds. We will discuss various environments employing AWS, Mesos, containers, docker, and BlueData EPIC technologies and the benefits and challenges of each.
Speakers:
Tom Phelan, Co-founder and Chief Architect - BlueData Inc. Tom has spent the last 25 years as a senior architect, developer, and team lead in the computer software industry in Silicon Valley. Prior to co-founding BlueData, Tom spent 10 years at VMware as a senior architect and team lead in the core R&D Storage and Availability group. Most recently, Tom led one of the key projects – vFlash, focusing on integration of server-based Flash into the vSphere core hypervisor. Prior to VMware, Tom was part of the early team at Silicon Graphics that developed XFS, one of the most successful open source file systems. Earlier in his career, he was a key member of the Stratus team that ported the Unix operating system to their highly available computing platform. Tom received his Computer Science degree from the University of California, Berkeley.
BlueStore: a new, faster storage backend for CephSage Weil
Traditionally Ceph has made use of local file systems like XFS or btrfs to store its data. However, the mismatch between the OSD's requirements and the POSIX interface provided by kernel file systems has a huge performance cost and requires a lot of complexity. BlueStore, an entirely new OSD storage backend, utilizes block devices directly, doubling performance for most workloads. This talk will cover the motivation a new backend, the design and implementation, the improved performance on HDDs, SSDs, and NVMe, and discuss some of the thornier issues we had to overcome when replacing tried and true kernel file systems with entirely new code running in userspace.
This document outlines a course on Ceph storage. It covers Ceph history and components. Key components discussed include Object Storage Daemons (OSDs) that store data, Monitors that maintain the cluster map and provide consensus, and the Ceph journal. Other topics are the Ceph Gateway for object storage access, Ceph Block Device (RBD) for block storage, and CephFS for file storage. The goal is to understand Ceph concepts and deploy a Ceph cluster.
StatefulSet is used to run PostgreSQL pods across Kubernetes nodes for high availability. When a pod fails, StatefulSet will restart the pod on the same node. However, if the entire node fails, the PostgreSQL pod will not failover to another node by default. To manually failover the pod, it needs to be force deleted and it will restart on a different ready node. However, manual failovers are not recommended for production use.
Divide and conquer: resource segregation in the OpenStack cloudStephen Gordon
This document discusses resource segregation techniques in OpenStack clouds. It describes how infrastructure resources like compute hosts can be logically grouped using regions, cells, host aggregates, and availability zones. It also discusses workload segregation using server groups that define affinity and anti-affinity rules for instance placement. The goals of segregation include isolating workloads, ensuring high availability, and enabling horizontal scaling of the infrastructure.
Ceph began as a research project in 2005 to create a scalable object storage system. It was incubated at DreamHost from 2007-2012 and spun out as an independent company called Inktank in 2012. Key developments included the RADOS distributed storage cluster, erasure coding, and the Ceph filesystem. The project has grown a large community and is used in many production deployments, focusing on areas like tiering, erasure coding, replication, and integrating with the Linux kernel. Future plans include improving CephFS, expanding the ecosystem through different storage backends, strengthening governance, and targeting new use cases in big data and the enterprise.
The ability to create global namespace, high availability, and volume economics enables unique capabilities when Red Hat Storage Server is used on Amazon Web Services (AWS). In this session, you’ll learn about the capabilities and the potential of using AWS ephemeral storage, availability zones, and regions to deliver a truly distributed cloud storage solution that is geographically diverse, cost effective, and highly available.
New Ceph capabilities and Reference ArchitecturesKamesh Pemmaraju
Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph? Then this is the session for you!
In this two part session you learn details of:
• the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas.
• best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.
This document discusses deploying the DC/OS container management system on bare metal servers using Juju charms. It describes bootstrapping DC/OS masters and agents using Juju, relating them, and scaling agents. Additional charms can be deployed for monitoring with Nagios and logging with Logstash. Containers can then be deployed to DC/OS using the dcos-nginx charm. The current implementation supports deploying DC/OS masters and agents in a scalable way and deploying docker containers. Future work includes improving security, simplifying upgrades, and adding more actions.
The document discusses running Spark analytics on Mesos using Docker containers. It demonstrates running a Spark scheduler within a Docker container on Marathon and executing Spark tasks on Mesos. It shows launching the Spark TeraSort benchmark to generate, sort, and validate 100 million records of data, and resizing the cluster using AWS Auto Scaling Groups. The document also lists some use cases for running Spark on Mesos such as analytics, data warehousing, machine learning, and stream processing.
The document discusses Docker networking and Kubernetes networking concepts. It provides an overview of Docker networking and how containers on the same host can communicate. It then summarizes key Kubernetes concepts like pods, replication controllers, services and networking. It demonstrates how to set up a sample application topology in Kubernetes using replication controllers and services. It also discusses exposing services externally and additional resources for learning about Docker and Kubernetes.
Container Network Interface: Network Plugins for Kubernetes and beyondKubeAcademy
With the rise of modern containers comes new problems to solve – especially in networking. Numerous container SDN solutions have recently entered the market, each best suited for a particular environment. Combined with multiple container runtimes and orchestrators available today, there exists a need for a common layer to allow interoperability between them and the network solutions.
As different environments demand different networking solutions, multiple vendors and viewpoints look to a specification to help guide interoperability. Container Network Interface (CNI) is a specification started by CoreOS with the input from the wider open source community aimed to make network plugins interoperable between container execution engines. It aims to be as common and vendor-neutral as possible to support a wide variety of networking options — from MACVLAN to modern SDNs such as Weave and flannel.
CNI is growing in popularity. It got its start as a network plugin layer for rkt, a container runtime from CoreOS. Today rkt ships with multiple CNI plugins allowing users to take advantage of virtual switching, MACVLAN and IPVLAN as well as multiple IP management strategies, including DHCP. CNI is getting even wider adoption with Kubernetes adding support for it. Kubernetes accelerates development cycles while simplifying operations, and with support for CNI is taking the next step toward a common ground for networking. For continued success toward interoperability, Kubernetes users can come to this session to learn the CNI basics.
This talk will cover the CNI interface, including an example of how to build a simple plugin. It will also show Kubernetes users how CNI can be used to solve their networking challenges and how they can get involved.
KubeCon schedule link: http://sched.co/4VAo
Open access building block in developmentHELIGLIASA
This document discusses open access and LIASA's support of open access initiatives in South Africa. It provides an overview of LIASA, the challenges to accessing information, and the evolution and benefits of open access. LIASA is committed to promoting open access and contributing to the development of open access policies and frameworks in South Africa to help ensure all people have access to information.
This document provides an overview of a presentation on transformative curriculum development. It discusses defining curriculum and transformation, and developing a shared understanding of these concepts. It also covers curriculum responsiveness, constructing transformative curricula, and the challenges of curriculum alignment. The purpose of higher education and dominant ideas shaping curriculum at NMMU are debated. Transformation is linked to addressing societal needs through curriculum content and activities. Constructive alignment between learning outcomes, teaching, and assessment is emphasized.
The document criticizes and mocks common gifts given by companies to employees such as branded polo shirts, paperweights, mugs, gift cards with small amounts, calendars, cheap pens, laser pointers, frisbees, stress balls, personalized sticky notes, and mousepads. It argues that most recipients do not want or need these promotional items and sees them as cheap and useless gifts.
Ipsos Global Business Influencers USA 2016Ipsos UK
22nd September 2016 - Ipsos launched the new Global Business Influencers Survey (GBI) USA 2016 in New York. The GBI survey is the world’s leading study, tracking the media, business, financial, luxury and travel habits of the most senior global business executives. It runs in 16 markets – from the US, to Europe, to Asia including China.
The presentation is an introduction to the survey and its heritage, sharing some of the results from both the GBI survey and the GBI Barometer 2016.
For more information, please contact James Torr.
The document is a presentation on DC/OS and Mesos. It discusses how Mesos and DC/OS were created to address the challenges of running modern distributed applications across large clusters. It provides an overview of the key features of DC/OS, including service discovery, load balancing, storage integration, and centralized operations. The presentation encourages attendees to try out DC/OS and provides contact information for support.
The document is a presentation on DC/OS, an open source tool for managing containerized applications in a clustered environment. It discusses how DC/OS builds on Apache Mesos to provide service discovery, load balancing, security and management of applications. It also summarizes some key capabilities of DC/OS like running various frameworks including Marathon, Docker and Kubernetes, integrated storage options, and providing a centralized view of operations across all frameworks and clusters.
Downtime is not an option - day 2 operations - Jörg SchadCodemotion
The document discusses container orchestration and microservices on Apache Mesos and DC/OS. It introduces concepts like microservices, containers, container orchestration and scheduling. It then summarizes key Mesos and DC/OS capabilities like running various workloads, multiplexing resources for higher utilization and running distributed applications. It also touches on day 2 operations for containerized workloads like monitoring, maintenance and troubleshooting.
Elastic data services on Apache Mesos via Mesosphere’s DCOSharrythewiz
Adam Bordelon and Mohit Soni demonstrate how projects like Apache Myriad (incubating) can install Hadoop on Mesosphere DC/OS alongside other data center-scale applications, enabling efficient resource sharing and isolation across a variety of distributed applications while sharing the same cluster resources and hence breaking silos.
OSDC 2018 | From batch to pipelines – why Apache Mesos and DC/OS are a soluti...NETWAYS
Apache Mesos is a distributed system for running other distributed systems, often described as a distributed kernel. It’s in use at massive scale at some of the worlds largest companies like Netflix, Uber and Yelp, abstracting entire data centres of hardware to allow for workloads to be distributed efficiently. DC/OS is an open source distribution of Mesos, which adds all the functionality to run Mesos in production across any substrate, both on-premise and in the cloud. In this talk, I’ll introduce both Mesos and DC/OS and talk about how they work under the hood, and what the benefits are of running these new kinds of systems for emerging cloud native workloads.
Cloud Native Night, January 2018, Munich: Workshop led by Jörg Schad (@joerg_schad, Technical Lead Community Projects at Mesosphere)
Join our Meetup: https://www.meetup.com/de-DE/cloud-native-muc
PLEASE NOTE: During this workshop, Jörg showed many demos and the audience could participate on their laptops. Unfortunately, we can't provide these demos. Nevertheless, Jörg's slides give a deep dive into the topic.
ABSTRACT: Kubernetes has been one of the topics in 2017 and will probably remain so in 2018. In this hands-on technical workshop you will learn how best to deploy, operate, and scale Kubernetes clusters from one to hundreds of nodes using DC/OS. You will learn how to integrate and run Kubernetes alongside traditional applications and fast data services of your choice (e.g. Apache Cassandra, Apache Kafka, Apache Spark, TensorFlow, and more) on any infrastructure.
This workshop best suits operators focussed on keeping their apps and services up and running in production and developers focussed on quickly delivering internal and customer facing apps into production.
You will learn how to:
- Introduction to Kubernetes and DC/OS (including the differences between both)
- Deploy Kubernetes on DC/OS in a secure, highly available, and fault-tolerant manner
- Solve operational challenges of running a large/multiple Kubernetes cluster
- One-click deploy big data stateful and stateless services alongside a Kubernetes cluster
Webinar: End-to-End CI/CD with GitLab and DC/OSMesosphere Inc.
Seven years ago, Apache Mesos was born as a platform to bring the distributed computing capabilities that powered the largest digital companies to the masses. Today, Mesosphere DC/OS technologies power more containers in production than any other software stack in the world, and has emerged as the premier platform for building and elastically scaling data-rich, modern applications and the associated CI/CD infrastructure across any infrastructure, public or private.
GitLab is an end-to-end software development and delivery platform with built-in CI/CD, monitoring, and performance metrics. With a unified experience for every step of the development lifecycle and seamless integration with container schedulers, GitLab provides the most efficient approach to reduce cycle time, increase velocity, and improve software quality.
In this webinar, you will learn how to combine DC/OS and GitLab to easily build a CI/CD infrastructure and build a complete CI/CD pipeline in minutes.
Slides cover:
1. An introduction to Apache Mesos and Mesosphere DC/OS and overview of DC/OS features and capabilities for developing, deploying, and operating containerized applications, microservices and CI/CD
2. An introduction to GitLab
3. How to use DC/OS and GitLab to build a CI/CD solution and go from idea to production
The document discusses evolving patterns in data processing from early approaches like MapReduce to modern approaches using containers and microservices. It introduces the SMACK stack (Streaming, Messaging, Analytics, Catalogs, Knowledge) as a model for modern data applications and discusses how the Mesos/DC/OS platform provides the infrastructure to run these applications, including resource sharing, scheduling, and deployment of containerized services. It also briefly mentions how Kubernetes has been added to the DC/OS ecosystem.
The document discusses Mesosphere's Datacenter Operating System (DCOS). It provides an overview of containers, Apache Mesos, Marathon, and how DCOS integrates these components to function as an operating system for datacenters. Key points include that DCOS treats resources across a cluster as a single shared pool, provides scheduling and orchestration of workloads, and enables easy deployment and management of applications.
Mesosphere & Magnetic: Take the pain out of running complex and critical serv...Mesosphere Inc.
In software development and delivery, costly and potentially deadly risks appear in the form of performance issues, errors and even downtime. These risks mostly surface when upgrading software or migrating applications. The more power and speed is being introduced, the more explosive these risks can become. To reduce these risks new safety-systems need to be introduced to reduce the risks involved with speeding up while leveraging the full potential of new technologies like containers and microservices.
VAMP reduces the risks of performance issues and downtime when upgrading software by providing proven “canary testing & releasing” features to high-power container and microservices systems. One of the most potent systems to scale and speed up the datacenter is DC/OS. Combined with VAMP, this delivers a powerful package for DevOps engineers.
In this webinar Olaf Molenveld, CEO of Magnetic and Amr Hamed Abdelrazik, Product Marketing Manager of Mesosphere will explain:
The vision and architecture of both VAMP and DC/OS
Benefits and requirements of the canary release process
How VAMP and DC/OS work together to provide Canary release for docker based microservices
Provide a working demo of the canary release process with VAMP on top of DC/OS.
Containerizing couchbase with microservice architecture on mesosphere.pptxRavi Yadav
Ravi Yadav, Mesosphere
Anil Kumar, Couchbase
Organizations focused on delivering exceptional customer experiences are building applications using microservice architectures because of the flexibility, speed of delivery, and maintainability that they provide. In this session, you will learn how Couchbase can fit into a microservice architecture using containers and orchestration. We will explore how Couchbase and Mesosphere work together to simplify application development and delivery. Additionally, you will see a demonstration of exactly how to create a Couchbase cluster on Mesosphere DC/OS Enterprise.
Kubesprawl is becoming a real challenge - with over 70 ways to deploy and on average 15 clusters, how can we keep going fast without being slowed down by operations overhead? Kubernetes is being adopted by most teams and they each have their own way of doing things, which leads to inconsistency and takes us back to the bad days of snowflakes and separate environments. How can we show both the development productivity benefits of Kubernetes as well as the financial and operational ones if we still treat Kubernetes clusters as pets? Learn how to achieve high density multiple Kubernetes clusters bin-packed on the same OS instances without virtualization, while still retaining isolation. We’ll talk about the challenges of resource usage, network separation and service automation for both the developer and operator, and how we can avoid them.
[DO16] Mesosphere : Microservices meet Fast Data on Azure de:code 2017
The document discusses how microservices and fast data can be combined on Azure using the SMACK stack. It describes each component of the stack: Apache Kafka for ingesting data streams, Apache Spark for stream processing, Apache Cassandra for storage, and Akka for acting on the data. It outlines how Mesos provides a datacenter operating system to run these diverse workloads together efficiently using resource multiplexing across servers.
This document discusses deploying the Alluxio distributed file system on Mesosphere DC/OS. It begins with an overview of the SMACK and SMAACK data stacks that include Apache Spark, Kafka, Cassandra and Akka. It then summarizes the benefits of Alluxio in providing unified access to data across storage systems at memory speed. The document demonstrates deploying Alluxio on DC/OS, noting how this provides on-demand provisioning, simplified operations and an elastic data infrastructure. It concludes by recommending users get started with Alluxio on DC/OS to process data from multiple storage systems faster.
Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications running on a shared pool of nodes. It provides APIs for resource management and scheduling and can run applications like Hadoop, Spark, and Kafka across large compute clusters. Mesos was created at Berkeley and adopted by companies like Twitter, Airbnb, and Hubspot to improve resource utilization, allow dynamic scaling of services, and reduce the operational complexity of managing large fleets of servers. Mesos also integrates with Docker and Kubernetes to enable containerized workloads and services to be deployed and orchestrated on Mesos clusters.
EMC World 2016 - Introduction to Mesos and MesosphereDavid vonThenen
This document provides an introduction to Mesos and Mesosphere. It begins with presenters and an agenda. It then defines Mesos as a cluster manager that pools server resources and a scheduler that dispatches workloads. Mesos components include a master, Zookeeper, and agents. Mesos supports generic and framework applications. Mesosphere was founded to deliver Mesos for enterprises and provides a datacenter operating system with rich ecosystem. The document demonstrates how EMC Code integrates persistent external storage with Mesos and Mesosphere through projects like REX-Ray and mesos-module-dvdi. It concludes with an invitation to ask questions.
Using DC/OS for Continuous Delivery - DevPulseCon 2017pleia2
The document discusses new features in DC/OS 1.9 including improved operations through remote container shells, unified logging and metrics, deployment failure debugging, and upgrades/configuration updates. It also covers new workload features such as pods for co-locating containers and GPU-based scheduling. A variety of data services are now available in the DC/OS ecosystem.
Similar to Mesos: A State-of-the-art Container Orchestrator (20)
Streaming a Million Likes/Second: Real-Time Interactions on Live VideoC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39NIjLV.
Akhilesh Gupta does a technical deep-dive into how Linkedin uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers. Filmed at qconlondon.com.
Akhilesh Gupta is the technical lead for LinkedIn's Real-time delivery infrastructure and LinkedIn Messaging. He has been working on the revamp of LinkedIn’s offerings to instant, real-time experiences. Before this, he was the head of engineering for the Ride Experience program at Uber Technologies in San Francisco.
Next Generation Client APIs in Envoy MobileC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2x0Fav8.
Jose Nino guides the audience through the journey of Mobile APIs at Lyft. He focuses on how the team has reaped the benefits of API generation to experiment with the network transport layer. He also discusses recent developments the team has made with Envoy Mobile and the roadmap ahead. Filmed at qconlondon.com.
Jose Nino works as a Software Engineer at Lyft.
Software Teams and Teamwork Trends Report Q1 2020C4Media
The document discusses key trends in software teams and teamwork in 2020 according to a report by InfoQ. Some of the trends discussed include the sudden shift to remote work due to COVID-19, with many teams not fully prepared; the continued spread of agile practices to other areas of organizations beyond software development; and a growing focus on diversity, inclusion, and creating more humanistic and sustainable workplaces. The report aims to help technical leaders and individual contributors navigate these trends and challenges to improve team experiences and organizational success.
Understand the Trade-offs Using Compilers for Java ApplicationsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2QCmmJ0.
Mark Stoodley examines some of the strengths and weaknesses of the different Java compilation technologies, if one was to apply them in isolation. Stoodley discusses how production JVMs are assembling a combination of these tools that work together to provide excellent performance across the large spectrum of applications written in Java and JVM based languages. Filmed at qconsf.com.
Mark Stoodley joined IBM Canada to build Java JIT compilers for production use and led the team that delivered AOT compilation in the IBM SDK for Java 6. He spent the last five years leading the effort to open source nearly 4.3 million lines of source code from the IBM J9 Java Virtual Machine to create the two open source projects Eclipse OMR and Eclipse OpenJ9, and now co-leads both projects.
Kafka is evolving to remove its dependency on Zookeeper. The Kafka Improvement Proposal 500 (KIP-500) aims to manage Kafka's metadata log with a self-managed Raft consensus algorithm and controller quorum rather than relying on Zookeeper. This will improve scalability, robustness, and make deployment easier. It will take multiple releases to fully implement KIP-500, beginning with removing Zookeeper from clients and ending with a release where Zookeeper is no longer required.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2SXXXiD.
Katharina Probst talks about what it means to act like an owner and why teams need ownership to be high-performing. When team members, regardless of whether they have a formal leadership role or not, act like owners, magical things can happen. She shares ideas that we can apply to our own work, and talks about how to recognize when we don’t live up to our own expectations of acting like an owner. Filmed at qconsf.com.
Katharina Probst is a Senior Engineering Leader, Kubernetes & SaaS at Google. Before this, she was leading engineering teams at Netflix, being responsible for the Netflix API, which helps bring Netflix streaming to millions of people around the world. Prior to joining Netflix, she was in the cloud computing team at Google, where she saw cloud computing from the provider side.
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2T04Lw4.
Sergey Kuksenko talks about the performance benefits inline types bring to Java and how to exploit them. Inline/value types are the key part of experimental project Valhalla, which should bring new abilities to the Java language. Filmed at qconsf.com.
Sergey Kuksenko is a Java Performance Engineer at Oracle working on a variety of Java and JVM performance enhancements. He started working as Java Engineer in 1996 and as Java Performance Engineer in 2005. He has had a passion for exploring how Java works on modern hardware.
Do you need service meshes in your tech stack?
This on-line guide aims to answer pertinent questions for software architects and technical leaders, such as: what is a service mesh?, do I need a service mesh?, how do I evaluate the different service mesh offerings? In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy.
The document discusses Cloud Native CI/CD and the Tekton project. It begins with an overview of cloud native concepts like containers, Kubernetes, and microservices. It then defines characteristics of cloud native CI/CD like being serverless, using open standards, reusable components, and config as code. The document introduces Tekton as a cloud native CI/CD building block on Kubernetes that uses custom resources for tasks, pipelines, triggers and more. It highlights a demo of Tekton before concluding with the project's roadmap and how to get involved.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S7lDiS.
Sasha Rosenbaum shows how a CI/CD pipeline for Machine Learning can greatly improve both productivity and reliability. Filmed at qconsf.com.
Sasha Rosenbaum is a Program Manager on the Azure DevOps engineering team, focused on improving the alignment of the product with open source software. She is a co-organizer of the DevOps Days Chicago and the DeliveryConf conferences, and recently published a book on Serverless computing in Azure with .NET.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/36epVKg.
Todd Montgomery discusses the techniques and lessons learned from implementing Aeron Cluster. His focus is on how Raft can be implemented on Aeron, minimizing the network round trip overhead, and comparing single process to a fully distributed cluster. Filmed at qconsf.com.
Todd Montgomery is a networking hacker who has researched, designed, and built numerous protocols, messaging-oriented middleware systems, and real-time data systems, done research for NASA, contributed to the IETF and IEEE, and co-founded two startups. He currently works as an independent consultant and is active in several open source projects.
Architectures That Scale Deep - Regaining Control in Deep SystemsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2FWc5Sk.
Ben Sigelman talks about "Deep Systems", their common properties and re-introduces the fundamentals of control theory from the 1960s, including the original conceptualizations of Observability & Controllability. He uses examples from Google & other companies to illustrate how deep systems have damaged people's ability to observe software, and what needs to be done in order to regain control. Filmed at qconsf.com.
Ben Sigelman is a co-founder and the CEO at LightStep, a co-creator of Dapper (Google’s distributed tracing system), and co-creator of the OpenTracing and OpenTelemetry projects (both part of the CNCF). His work and interests gravitate towards observability, especially where microservices, high transaction volumes, and large engineering organizations are involved.
ML in the Browser: Interactive Experiences with Tensorflow.jsC4Media
This document discusses machine learning in the browser using Tensorflow.js. It begins with an introduction and overview of Tensorflow.js, including how it can be used for both authoring models and importing pre-trained models for inference. Examples are provided of using the Ops API to fit a polynomial function and the Layers API to build and train an autoencoder in the browser. Challenges of developing machine learning applications in the browser are also discussed.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2s9T3Vl.
Colin Eberhardt looks at some of the internals of WebAssembly, explores how it works “under the hood”, and looks at how to create a (simple) compiler that targets this runtime. Filmed at qconsf.com.
Colin Eberhardt is the Technology Director at Scott Logic, a UK-based software consultancy where they create complex application for their financial services clients. He is an avid technology enthusiast, spending his evenings contributing to open source projects, writing blog posts and learning as much as he can.
User & Device Identity for Microservices @ Netflix ScaleC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2S9tOgy.
Satyajit Thadeshwar provides useful insights on how Netflix implemented a secure, token-agnostic, identity solution that works with services operating at a massive scale. He shares some of the lessons learned from this process, both from architectural diagrams and code. Filmed at qconsf.com.
Satyajit Thadeshwar is an engineer on the Product Edge Access Services team at Netflix, where he works on some of the most critical services focusing on user and device authentication. He has more than a decade of experience building fault-tolerant and highly scalable, distributed systems.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ezs08q.
Justin Ryan talks about Netflix’ scalability issues and some of the ways they addressed it. He shares successes they’ve had from unintuitively partitioning computation into multiple services to get better runtime characteristics. He introduces us to useful probabilistic data structures, innovative bi-directional data passing, open-source projects available from Netflix that make this all possible. Filmed at qconsf.com.
Justin Ryan is Playback Edge Engineering at Netflix. He works on some of the most critical services at Netflix, specifically focusing on user and device authentication. Years of building developer tools has also given him a healthy set of opinions on developer productivity.
Make Your Electron App Feel at Home EverywhereC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Z4ZJjn.
Kilian Valkhof discusses the process of making an Electron app feel at home on all three platforms: Windows, MacOS and Linux, making devs aware of the pitfalls and how to avoid them. Filmed at qconsf.com.
Kilian Valkhof is a Front-end Developer & User-experience Designer at Firstversionist. He writes about various topics, from design to machine learning, on his personal website, kilianvalkhof.com and is a frequent contributer to open source software. He is part of the Electron governance team that oversees the development of the Electron framework.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/344PnB1.
Steve Klabnik goes over the deep details of how async/await works in Rust, covering concepts like coroutines, generators, stack-less vs stack-ful, "pinning", and more. Filmed at qconsf.com.
Steve Klabnik is on the core team of Rust, leads the documentation team, and is an author of "The Rust Programming Language." He is a frequent speaker at conferences and is a prolific open source contributor, previously working on projects such as Ruby and Ruby on Rails.
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2OUz6dt.
Chris Riccomini talks about the current state-of-the-art in data pipelines and data warehousing, and shares some of the solutions to current problems dealing with data streaming and warehousing. Filmed at qconsf.com.
Chris Riccomini works as a Software Engineer at WePay.
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreC4Media
The document discusses testing infrastructure as code using automated tests. It recommends writing unit tests to test individual components in isolation by deploying real infrastructure, validating it works through methods like HTTP requests or API calls, and then undeploying it. The document provides an example of using Terratest to write a unit test for a Terraform module that deploys a "Hello World" web app. It shows how to build and deploy the infrastructure, validate it works by making an HTTP request, and clean it up after the test.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Dive into the realm of operating systems (OS) with Pravash Chandra Das, a seasoned Digital Forensic Analyst, as your guide. 🚀 This comprehensive presentation illuminates the core concepts, types, and evolution of OS, essential for understanding modern computing landscapes.
Beginning with the foundational definition, Das clarifies the pivotal role of OS as system software orchestrating hardware resources, software applications, and user interactions. Through succinct descriptions, he delineates the diverse types of OS, from single-user, single-task environments like early MS-DOS iterations, to multi-user, multi-tasking systems exemplified by modern Linux distributions.
Crucial components like the kernel and shell are dissected, highlighting their indispensable functions in resource management and user interface interaction. Das elucidates how the kernel acts as the central nervous system, orchestrating process scheduling, memory allocation, and device management. Meanwhile, the shell serves as the gateway for user commands, bridging the gap between human input and machine execution. 💻
The narrative then shifts to a captivating exploration of prominent desktop OSs, Windows, macOS, and Linux. Windows, with its globally ubiquitous presence and user-friendly interface, emerges as a cornerstone in personal computing history. macOS, lauded for its sleek design and seamless integration with Apple's ecosystem, stands as a beacon of stability and creativity. Linux, an open-source marvel, offers unparalleled flexibility and security, revolutionizing the computing landscape. 🖥️
Moving to the realm of mobile devices, Das unravels the dominance of Android and iOS. Android's open-source ethos fosters a vibrant ecosystem of customization and innovation, while iOS boasts a seamless user experience and robust security infrastructure. Meanwhile, discontinued platforms like Symbian and Palm OS evoke nostalgia for their pioneering roles in the smartphone revolution.
The journey concludes with a reflection on the ever-evolving landscape of OS, underscored by the emergence of real-time operating systems (RTOS) and the persistent quest for innovation and efficiency. As technology continues to shape our world, understanding the foundations and evolution of operating systems remains paramount. Join Pravash Chandra Das on this illuminating journey through the heart of computing. 🌟
Trusted Execution Environment for Decentralized Process MiningLucaBarbaro3
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
Digital Marketing Trends in 2024 | Guide for Staying AheadWask
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
mesos-api
3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com
37. 35
Responsible for process management
● Spawn containers
● Kill and wait containers
Supported launchers:
● Posix launcher
● Linux launcher
● Windows launcher
Launcher
Unified Containerizer
38. 36
Interface for extensions during the life cycle of a container
● Pre-launch - prepare()
● Post-launch (both in parent and child context) - isolate()
● Termination - cleanup()
● Resources update - update()
● Resources limitation reached - watch()
● Agent restart and recovery - recover()
● Stats and status pulling - usage()
Isolator
Unified Containerizer
Sufficient for most of
the extensions!
39. 37
Isolator example: cgroups memory isolator
Unified Containerizer
Agent Process
Launcher
creates
Subprocess Container
Process
execve()
LaunchInfo = Isolator::prepare()
* Create a cgroup for the container
in memory cgroup hierarchy:
/sys/fs/cgroup/memory/mesos/…
* Start listening for OOM event
Isolator::isolate(pid)
Block on pipe
Move ‘pid’ to the
memory cgroup just
created
Invoke ‘LaunchInfo.script’
Exec the executor
Signal the Child to continue
40. 38
Isolator example: cgroups memory isolator
Unified Containerizer
Agent Process
Container
Process
Isolator::update()
Change cgroup control:
memory.limit_in_bytes
Sending a new Task to
Executor, ‘resources’ of
the Executor changes
Send Task to Executor
41. 39
Isolator example: cgroups memory isolator
Unified Containerizer
Agent Process
Container
Process
Isolator::cleanup()
Remove the memory
cgroup associated
with the container
Shutdown Executor
or kill Task Destroy container
Container terminated
42. 40
Cgroups isolators: cgroups/cpu, cgroups/mem, ...
Disk isolators: disk/du, disk/xfs
Filesystem isolators: filesystem/posix, filesystem/linux
Volume isolators: docker/volume
Network isolators: network/cni, network/port_mapping
GPU isolators: gpu/nvidia
…... and more! Need your contribution!
Built-in isolators
Unified Containerizer
43. 41
Start from 0.28, you can run your Docker container on
Mesos without a Docker daemon installed!
● One less dependency in your stack
● Agent restart handled gracefully, task not affected
● Compose well with all existing isolators
● Easier to add extensions
Container image support
Unified Containerizer
44. 42
● Mesos supports multiple container image format
○ Docker (without docker daemon)
○ Appc (without rkt)
○ OCI (ready soon)
○ CVMFS (experimental)
○ Host filesystem with tars/jars
○ Your own image format!
Pluggable container image format
Unified Containerizer
Used in large scale
production clusters
45. 43
● Manage container images
○ Store: fetch and cache image layers
○ Backend: assemble rootfs from image layers
■ E.g., copy, overlayfs, bind, aufs
● Store can be extended
○ Currently supported: Docker, Appc
○ Plan to support: OCI (ongoing), CVMFS
○ Custom fetching (e.g., p2p)
Provisioner
Unified Containerizer
47. 45
● Support Container Network Interface (CNI) from 1.0
○ A spec for container networking
○ Supported by most network vendors
● Implemented as an isolator
○ --isolation=network/cni,...
Container network support
Unified Containerizer
48. 46
● Proposed by CoreOS :
https://github.com/containernetworking/cni
● Simple contract between container
runtime and CNI plugin defined in the
form of a JSON schema
○ CLI interface
○ ADD: attach to network
○ DEL: detach from network
Container Network Interface (CNI)
Unified Containerizer
Mesos Agent
Containerizer
Container
Executor
T1 T2
CNI Plugin
IPAM
veth
Network
49. ● Simpler and less dependencies than Docker CNM
● Backed by Kubernetes community as well
● Rich plugins from network vendors
● Clear separation between container and network management
● IPAM has its own pluggable interface
47
Why CNI?
Unified Containerizer
50. 48
Existing CNI plugins
● ipvlan
● macvlan
● bridge
● flannel
● calico
● contiv
● contrail
● weave
● …
CNI plugins
Unified Containerizer
You can write your own plugin,
and Mesos supports it!
51. 49
● Support Docker volume plugins from 1.0
○ Define the interface between container runtime and storage provider
○ https://docs.docker.com/engine/extend/plugins_volume/
● A variety of Docker volume plugins
○ Ceph
○ Convoy
○ Flocker
○ Glusterfs
○ Rexray
Container storage support
Unified Containerizer
52. 50
Launcher
● Custom container processes management
Isolator
● Extension to the life cycle of a container
Provisioner
● New type of images
● Custom fetching and caching
Extensions
Unified Containerizer
62. 60
CNI support using an isolator
Unified Containerizer
Agent Process
Launcher
creates
Subprocess Container
Process
execve()
LaunchInfo = Isolator::prepare()
Tell the launcher to create the
child process in a new NET, UTS
and MNT namespace.
Isolator::isolate(pid)
Block on pipe
Bind mount the NET
namespace to keep
it open
Invoke ‘ADD’ of the
CNI plugin with the
NET namespace
associated with pid
Setup network
related /etc/xx files
for the container
Invoke ‘LaunchInfo.script’
Exec the executor
Signal the Child to continue
63. 61
CNI support using an isolator
Unified Containerizer
Container
Process
Isolator::cleanup()
Invoke ‘DEL’ of the
CNI plugin with the
NET namespace
handle
Umount and remove
the NET namespace
handle
Shutdown Executor
or kill Task Destroy container
Container terminated
Agent Process