Apache Mesos is a cluster manager that provides efficient resource sharing for distributed applications across a shared pool of nodes. It allows organizations to run applications like Hadoop, Spark, and Storm on large clusters with high utilization. Mesos addresses issues with prior solutions that constrained everything as "jobs" or required static partitioning. It has been adopted by companies like Twitter, Airbnb, and Hubspot to improve efficiency and allow applications to dynamically scale resources.
Mesos: The Operating System for your DatacenterDavid Greenberg
Maybe you’ve heard of Mesos—that thing that you can run Hadoop on. I think it powers Twitter? Isn’t it an Apache project, or something?
In this talk, we’ll learn all about Mesos—what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
O'Reilly Media - Strata SC 2014
Apache Mesos is an open source cluster manager that provides efficient resource isolation for distributed frameworks—similar to Google’s “Borg” and “Omega” projects for warehouse scale computing. It is based on isolation features in the modern kernel: “cgroups” in Linux, “zones” in Solaris.
Google’s “Omega” research paper shows that while 80% of the jobs on a given cluster may be batch (e.g., MapReduce), 55-60% of cluster resources go toward services. The batch jobs on a cluster are the easy part—services are much more complex to schedule efficiently. However by mixing workloads, the overall problem of scheduling resources can be greatly improved.
Given the use of Mesos as the kernel for a “data center OS”, two additional open source components Chronos (like Unix “cron”) and Marathon (like Unix “init.d”) serve as the building blocks for creating distributed, fault-tolerant, highly-available apps at scale.
This talk will examine case studies of Mesos uses in production at scale: ranging from Twitter (100% on prem) to Airbnb (100% cloud), plus MediaCrossing, Categorize, HubSpot, etc. How have these organizations leveraged Mesos to build better, more scalable and efficient distributed apps? Lessons from the Mesos developer community show that one can port an existing framework with a wrapper in approximately 100 line of code. Moreover, an important lesson from Spark is that based on “data center OS” building blocks one can rewrite a distributed system much like Hadoop to be 100x faster within a relatively small amount of source code.
These case studies illustrate the obvious benefits over prior approaches based on virtualization: scalability, elasticity, fault-tolerance, high availability, improved utilization rates, etc. Less obvious outcomes also include: reduced time for engineers to ramp-up new services at scale; reduced latency between batch and services, enabling new high-ROI use cases; and enabling dev/test apps to run on a production cluster without disrupting operations.
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Docker, Inc.
Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Google’s Borg, the Googleplex’s secret weapon of scalability and fault tolerance.
Mesos: The Operating System for your DatacenterDavid Greenberg
Maybe you’ve heard of Mesos—that thing that you can run Hadoop on. I think it powers Twitter? Isn’t it an Apache project, or something?
In this talk, we’ll learn all about Mesos—what it is, how you can leverage it to simplify your infrastructure and reduce AWS/cloud computing costs, and why you should develop your next application on top of it. This talk will give you the tools you need to understand whether Mesos is the right fit for your infrastructure, and several starting points for learning more about Mesos.
What is Apache Mesos and how to use it. A short introduction to distributed fault-tolerant systems with using ZooKeeper and Mesos. #installfest Prague 2014
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
O'Reilly Media - Strata SC 2014
Apache Mesos is an open source cluster manager that provides efficient resource isolation for distributed frameworks—similar to Google’s “Borg” and “Omega” projects for warehouse scale computing. It is based on isolation features in the modern kernel: “cgroups” in Linux, “zones” in Solaris.
Google’s “Omega” research paper shows that while 80% of the jobs on a given cluster may be batch (e.g., MapReduce), 55-60% of cluster resources go toward services. The batch jobs on a cluster are the easy part—services are much more complex to schedule efficiently. However by mixing workloads, the overall problem of scheduling resources can be greatly improved.
Given the use of Mesos as the kernel for a “data center OS”, two additional open source components Chronos (like Unix “cron”) and Marathon (like Unix “init.d”) serve as the building blocks for creating distributed, fault-tolerant, highly-available apps at scale.
This talk will examine case studies of Mesos uses in production at scale: ranging from Twitter (100% on prem) to Airbnb (100% cloud), plus MediaCrossing, Categorize, HubSpot, etc. How have these organizations leveraged Mesos to build better, more scalable and efficient distributed apps? Lessons from the Mesos developer community show that one can port an existing framework with a wrapper in approximately 100 line of code. Moreover, an important lesson from Spark is that based on “data center OS” building blocks one can rewrite a distributed system much like Hadoop to be 100x faster within a relatively small amount of source code.
These case studies illustrate the obvious benefits over prior approaches based on virtualization: scalability, elasticity, fault-tolerance, high availability, improved utilization rates, etc. Less obvious outcomes also include: reduced time for engineers to ramp-up new services at scale; reduced latency between batch and services, enabling new high-ROI use cases; and enabling dev/test apps to run on a production cluster without disrupting operations.
Building Web Scale Apps with Docker and Mesos by Alex Rukletsov (Mesosphere)Docker, Inc.
Operating apps at web scale has become the new normal, but has been out of reach for most companies. Join us as we show you how to deploy and manage your Docker containers at scale. See how easy it is to build highly-available, fault-tolerant web scale apps using Docker with the Mesos cluster scheduler. Docker plus Mesos is a new way to scale applications. Together they give you capabilities similar to Google’s Borg, the Googleplex’s secret weapon of scalability and fault tolerance.
Deploying Containers in Production and at ScaleMesosphere Inc.
This presentation was part of "Deploying Containers in Production and at Scale" by Sunil Shah (Engineer at Mesosphere) at ContainerCon 2015
Try Mesosphere for Free: https://mesosphere.com/try
Containers, cluster management, microservices, Kubernetes and many other buzzwords are flying around us all the time. Our team is building solutions that make it easy to cope with all the complexity around cluster infrastructure. In this talk we present the project we are working on, namely running Kubernetes on top of the Mesos cluster scheduler. Furthermore we show DCOS which makes it easy to deploy and run Kubernetes with a single command.
Federated mesos clusters for global data center designsKrishna-Kumar
This talk at MesosCon2016 gives a glimpse of how Mesos clusters can be federated across data centers using a specific way. The data in the slide deck is mainly based on the POC result and the actual production implementation may vary.
An overview of Mesos and Kubernetes ecosystem including overview, architecture, customers and partners. For a beginner it will give a good covering of all the basics!
1. Double Orchestration of Redis Enterprise cluster on Kubernetes
**************
In this session we'll display how we deploy a highly available database on Kubernetes. The considerations we took when deploying a stateful application, and the challenges of answering different clients' demands for different k8s environments.
2. Operators to the rescue: stateful applications made easy with operators
**************
Kubernetes 1.7 introduced an import feature called custom controllers. This allows you to customise your Kubernetes installation and add your own resources to be managed in the native Kubernetes manner.
The session will display the operator concept and cover our journey with developing the Redis Labs operator - why we chose it and how we use it.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Discover the NEW Mesosphere DC/OS 1.10 for more freedom of choice for container orchestration and data services. Now the most flexible platform for containerized, data-intensive applications.
To view the recorded demo on-demand, visit: http://bit.ly/2hwiWW3
Scaling Development Environments with DockerDocker, Inc.
We set out to solve the problems of quickly building high quality games for a fragmented mobile market. Taking advantage of HTML5 allowed a fast, familiar and highly iterative local development process, and a hybrid build process for native apps meant high performance games on mobile. Our product is designed to comprehensively handle complex UI flows, related server tasks as well as deep integrations with any social platform. This is necessarily complex piece of engineering, with dozens of large dependencies, and 5 local web servers powering a single user’s experience. When we set out to make this easily available to 3rd parties, we used Docker to solve to major challenges: 1) Fitting many users, each with a unique development environment, on to one machine; 2) Managing all of these development environments in a scalable way.
Karl Isenberg reviews the history of distributed computing, classifies multiple different platform layers, and performs a head-to-head comparison of several container orchestration solutions, including Kubernetes, Marathon, and Docker Swarm. Learn which features and qualities are critical for container orchestration and how you can apply this knowledge when evaluating platforms.
On Periscope: https://www.periscope.tv/mesosphere/1RDGlLplaeOGL
Presentation given at the OpenStack summit in Paris (Kilo) on Tue Nov 4th.
Last summit I had the pleasure to present a talk which encountered some success "Are enterprise ready for the OpenStack transformation?" (also published on SlideShare) . This talk is a follow up on what are the best practices that are successful in operating the transformation. We will first focus on identifying the right use cases for a generic enterprise, then define a roadmap with an organisational and a technical track, to finish with the definition what would be our success criterias for our group. This will happen as a workshop summary based on the multiple engagements eNovance has been delivering over the past 2 years.
Deploying Containers in Production and at ScaleMesosphere Inc.
This presentation was part of "Deploying Containers in Production and at Scale" by Sunil Shah (Engineer at Mesosphere) at ContainerCon 2015
Try Mesosphere for Free: https://mesosphere.com/try
Containers, cluster management, microservices, Kubernetes and many other buzzwords are flying around us all the time. Our team is building solutions that make it easy to cope with all the complexity around cluster infrastructure. In this talk we present the project we are working on, namely running Kubernetes on top of the Mesos cluster scheduler. Furthermore we show DCOS which makes it easy to deploy and run Kubernetes with a single command.
Federated mesos clusters for global data center designsKrishna-Kumar
This talk at MesosCon2016 gives a glimpse of how Mesos clusters can be federated across data centers using a specific way. The data in the slide deck is mainly based on the POC result and the actual production implementation may vary.
An overview of Mesos and Kubernetes ecosystem including overview, architecture, customers and partners. For a beginner it will give a good covering of all the basics!
1. Double Orchestration of Redis Enterprise cluster on Kubernetes
**************
In this session we'll display how we deploy a highly available database on Kubernetes. The considerations we took when deploying a stateful application, and the challenges of answering different clients' demands for different k8s environments.
2. Operators to the rescue: stateful applications made easy with operators
**************
Kubernetes 1.7 introduced an import feature called custom controllers. This allows you to customise your Kubernetes installation and add your own resources to be managed in the native Kubernetes manner.
The session will display the operator concept and cover our journey with developing the Redis Labs operator - why we chose it and how we use it.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Discover the NEW Mesosphere DC/OS 1.10 for more freedom of choice for container orchestration and data services. Now the most flexible platform for containerized, data-intensive applications.
To view the recorded demo on-demand, visit: http://bit.ly/2hwiWW3
Scaling Development Environments with DockerDocker, Inc.
We set out to solve the problems of quickly building high quality games for a fragmented mobile market. Taking advantage of HTML5 allowed a fast, familiar and highly iterative local development process, and a hybrid build process for native apps meant high performance games on mobile. Our product is designed to comprehensively handle complex UI flows, related server tasks as well as deep integrations with any social platform. This is necessarily complex piece of engineering, with dozens of large dependencies, and 5 local web servers powering a single user’s experience. When we set out to make this easily available to 3rd parties, we used Docker to solve to major challenges: 1) Fitting many users, each with a unique development environment, on to one machine; 2) Managing all of these development environments in a scalable way.
Karl Isenberg reviews the history of distributed computing, classifies multiple different platform layers, and performs a head-to-head comparison of several container orchestration solutions, including Kubernetes, Marathon, and Docker Swarm. Learn which features and qualities are critical for container orchestration and how you can apply this knowledge when evaluating platforms.
On Periscope: https://www.periscope.tv/mesosphere/1RDGlLplaeOGL
Presentation given at the OpenStack summit in Paris (Kilo) on Tue Nov 4th.
Last summit I had the pleasure to present a talk which encountered some success "Are enterprise ready for the OpenStack transformation?" (also published on SlideShare) . This talk is a follow up on what are the best practices that are successful in operating the transformation. We will first focus on identifying the right use cases for a generic enterprise, then define a roadmap with an organisational and a technical track, to finish with the definition what would be our success criterias for our group. This will happen as a workshop summary based on the multiple engagements eNovance has been delivering over the past 2 years.
Consolidation, cloud privé, cloud public, SQL As A Service etc. sont autant de scénarios de virtualisation possibles avec SQL Server. Cette session reposera les règles de bon usage de ce type de déploiement et les scénarios clés. Nous reviendrons sur quelques-unes des « Lessons learned from Azure ».
Consolidation, cloud privé, cloud public, SQL As A Service etc. sont autant de scénarios de virtualisation possibles avec SQL Server. Cette session reposera les règles de bon usage de ce type de déploiement et les scénarios clés. Nous reviendrons sur quelques-unes des « Lessons learned from Azure ».
Modernizing Applications with Microservices and DC/OS (Lightbend/Mesosphere c...Lightbend
**Featuring Aaron Williams, Head of Advocacy at Mesosphere, Inc. and Markus Eisele, Developer Advocate at Lightbend, Inc.**
The traditional architecture that enterprises run their businesses on has typically been delivered as monolithic applications running in a virtualized, on-premise infrastructure. Public and private cloud technologies have changed everything, but if the applications are not designed, or re-designed, appropriately, then it is impossible to take advantage of the advances in both distributed application services and hybrid infrastructure. Consequently, enterprise architects are looking to microservices-based architectures as a means to modernize their legacy applications.
This webinar with Lightbend and partner Mesosphere will introduce a new framework specifically designed to help developers modernize legacy Java EE applications into systems of microservices and then discuss exactly what is required to run these distributed systems at enterprise scale.
Network Automation Journey, A systems engineer NetOps perspectiveWalid Shaari
Network devices play a crucial role; they are not just in the Data Center. It's the Wifi, VOIP, WAN and recently underlays and overlays. Network teams are essential for operations. It's about time we highlight to the configuration management community the importance of Network teams and include them in our discussions. This talk describes the personal experience of systems engineer on how to kickstart a network team into automation. Most importantly, how and where to start, challenges faced, and progress made. The network team in question uses multi-vendor network devices in a large traditional enterprise.
NetDevOps, we do not hear that term as frequent as we should. Every time we hear about automation, or configuration management, it is usually the application, if not, it is the systems that host the applications. How about the network systems and devices that interconnect and protects our services? This talk aims to describe the journey a systems engineer had as part of an automation assignment with the network management team. Building from lessons learned and challenges faced with system automation, how one can kickstart an automation project and gain small wins quickly. Where and how to start the journey? What to avoid? What to prioritise? How to overcome the lack of network skills for the automation engineer and lack of automation and Linux/Unix skills for network engineers. What challenges were faced and how to overcome them? What fights to give up? Where do I see network automation and configuration management as a systems engineer? What are the status quo and future expectations?
OCCIware: Extensible and Standard-based XaaS Platform To Manage Everything in...OW2
The OCCIware project aims at managing in a unified manner all layers and domains of the Cloud (XaaS), by building on the Open Cloud Computing (OCCI) standard. OCCIware Metamodel formally specifies the main OCCI concepts. Today a first EMF metamodel is defined that adds to OCCI new concepts such as Extension, Configuration, and EDataType, addressing some limitations of OCCI.
This session highlights OCCIware platform two main components:
– The OCCIware Studio Factory, allowing to produce visually customizable diagram editors for any Cloud configuration business domain modeled in OCCI using the OCCI Extension Studio, such as the flagship Docker Studio ;
– The OCCIware Runtime, based on OW2 erocci project, including the tools for deployment, supervision and administration, and allowing to federate multiple XaaS Cloud runtimes, such as the Roboconf PaaS server and the ActiveEon Cloud Automation multi-IaaS connector.
This talk includes a demonstration of the Docker connector and of how to use the OCCIware Cloud Designer to configure a real life Cloud application (a Java API server on top of a MongoDB cluster)’s business, platform and infrastructure layers seamlessly on both VirtualBox and OpenStack infrastructure.
OCCIware, an extensible, standard-based XaaS consumer platform to manage ever...OCCIware
The OCCIware project aims at managing in a unified manner all layers and domains of the Cloud (XaaS), by building on the Open Cloud Computing (OCCI) standard. OCCIware Metamodel formally specifies the main OCCI concepts. Today a first EMF metamodel is defined that adds to OCCI new concepts such as Extension, Configuration, and EDataType, addressing some limitations of OCCI.
This session highlights OCCIware platform two main components:
– The OCCIware Studio Factory, allowing to produce visually customizable diagram editors for any Cloud configuration business domain modeled in OCCI using the OCCI Extension Studio, such as the flagship Docker Studio ;
– The OCCIware Runtime, based on OW2 erocci project, including the tools for deployment, supervision and administration, and allowing to federate multiple XaaS Cloud runtimes, such as the Roboconf PaaS server and the ActiveEon Cloud Automation multi-IaaS connector.
This talk includes a demonstration of the Docker connector and of how to use the OCCIware Cloud Designer to configure a real life Cloud application (a Java API server on top of a MongoDB cluster)’s business, platform and infrastructure layers seamlessly on both VirtualBox and OpenStack infrastructure.
OCCIware@POSS 2016 - an extensible, standard XaaS cloud consumer platformMarc Dutoo
OCCIware at Paris Open Source Summit 2016 - an extensible, standard XaaS cloud consumer platform - demos : Docker & Linked Data Studios, online playground
How jKool Analyzes Streaming Data in Real Time with DataStaxDataStax
In this webinar, Charles Rich, VP of Product Management at jKool will share their journey with DataStax; how jKool knew from the start that traditional relational databases wouldn’t work for the scalability and availability demands of time-series data, and why they turned to DataStax Enterprise for blazing performance and powerful enterprise search and analytics capabilities.
How jKool Analyzes Streaming Data in Real Time with DataStaxjKool
jKool provides an application analytics SaaS for DevOps. These slides illustrate some of the choices we had to make and the architectural decisions to build a system for both real-time and historical application analytics.
OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
Openstack Summit Tokyo 2015 - Building a private cloud to efficiently handle ...Pierre GRANDIN
What do you do when your usual setup or turnkey solution isn’t suited for your workload?
Most of the documentation and user feedback that you can find about OpenStack is written for the use-case of running a public facing cloud serving several external customers. When you want to host a single tenant with a single application the problem is completely different, you don't want publicly exposed APIs. You want to ensure optimal resource allocation to maximize your application performance. You want to leverage the fact that you own the infrastructure layer to optimize your instance placement strategy, and to get the best latency and to avoid creating SPOFs using affinity (or anti affinity rules).
This talk will focus on what we learned during a two years journey; from getting OpenStack up and running reliably, to investigating performance bottlenecks, to maximizing the performance of our private cloud.
Measure and Increase Developer Productivity with Help of Serverless at Server...Vadym Kazulkin
The goal of Serverless is to focus on writing the code that delivers business value and offload everything else to your trusted partners (like Cloud providers or SaaS vendors). You want to iterate quickly and today’s code quickly becomes tomorrow’s technical debt. In this talk we will show why Serverless adoption increases the developer productivity and how to measure it. We will also go through AWS Serverless architectures where you only glue together different Serverless managed services relying solely on configuration, minimizing the amount of the code written.
Stay productive while slicing up the monolithMarkus Eisele
Microservices-based architectures are in vogue. Over the last couple of years, we have learned how thought leaders implement them, and it seems like every other week we hear about how containers and platform-as-a-service offerings make them ultimately happen.
Tech Talent Night Copenhagen 11/22/17
https://greenticket.dk/techtalentnightcph
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Vadym Kazulkin
The goal of Serverless is to focus on writing the code that delivers business value and offload everything else to your trusted partners (like Cloud providers or SaaS vendors). You want to iterate quickly and today’s code quickly becomes tomorrow’s technical debt. In this talk we will show why Serverless adoption increases the developer productivity and how to measure it. We will also go through AWS Serverless architectures where you only glue together different Serverless managed services relying solely on configuration, minimizing the amount of the code written.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
3. Overview
● Project Synopsis
● History
● Ecosystem Questions
● More Detailed Description
● Interesting Use Cases
● Current “Shiny” Developments
4. What is Apache Mesos?
Apache Mesos is a cluster manager that
provides efficient resource isolation and sharing
across distributed applications, or frameworks.
It can run Hadoop, MPI, Hypertable, Spark,
Elastic Search, Storm, Aurora, Marathon ... and
other applications on a dynamically shared pool of
nodes.
5. Project Highlights
● Top-level Apache project ~ 1 year (mesos.apache.org)
● Scales to 10,000s of nodes
● Obviates the need for virtual machines
● Isolation for CPU, RAM, I/O, FS, etc.
● Fault-tolerant leader election based on Zookeeper
● API's in C++, Java/Scala, Python, Go, Erlang, Haskell.
● Web UI for inspecting state
● Available for Linux, OpenSolaris, Mac OSX
7. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
History
8. Understanding of Datacenter Computing
Google has been doing data center computing for
years to address the complexities of large-scale
data workflows:
● Leveraging the modern kernel isolation. (cgroups)
● Containerization !Virtualization (lmctfy - Docker)
● Most (>80) jobs are batch jobs, but the majority of
resources(55-80%) are allocated to service jobs.
● Mixed workloads, multi-tenancy
● Relatively high utilization rates
● JVM? Not so much...
● Reality: scheduling batch is simple;
– scheduling services is hard/expensive.
9. Refs.
● The Datacenter as a Computer: An Introduction
to the Design of Warehouse-Scale Machines
– http://research.google.com/pubs/pub35290.html
● GAFS Omega John Wilkes
– https://www.youtube.com/watch?v=0ZFMlO98Jkc
● Taming Latency Variability and Scaling Deep
Learning
– http://youtu.be/nK6daeTZGA8
10. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
Ecosystem
Questions?
11. Aren't There Several Other Existing
Solutions? - “Sounds Like” a grid
● IBM Platform Symphony
● Microsoft Autopilot
● Univa Grid Engine (SGE)
● Condor (Full Disclosure)
12. Follow Up:
What is the Gap?
● Many existing grid-solutions had architectural
deficiencies around a constraining model.
Everything is a “job”.
– Good at Batch, but tough for Services
– What happens when you want to write your own
distributed application? (no primitives)
– What happens when you want to write your own
scheduler (elastic service). Square wheel
reinvention.
13. What about Clouds?
● Can't you just “Cloud All the things”
– It's not very efficient
– Can be quite costly
● It doesn't actually solve the root cause of many of the
problems in applications, and in some cases a Cloud can
cause more issues, not less.
– Network Latency
– Data Gravity
– Multi-tenant Service Elasticity
...
14. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
Reality Check
15. The New Reality
● New applications need to be:
– Fault Tolerant (Withstand failure)
– Scalable (Doesn't crumble under it's own weight)
– Elastic (Can grow and shrink based on demand)
– Multi-tenent (It can't have it's own dedicated cluster)
● So what does that really mean?
16. Distributed Applications
● “There's Just No Getting Around It: You're Building a Distributed System” Mark Cavage
– queue.acm.org/detail.cfm?id=2482856
● Key takeaways on architecture:
– Decompose the business applications into discrete services on the boundaries of fault
domains, scaling, and data workload.
– Make as many things as possible stateless
– When dealing with state, deeply understand CAP, latency, throughput, and durability
requirements.
“Without practical experience working on successful—and failed—systems, most engineers take
a "hopefully it works" approach and attempt to string together off-the-shelf software, whether
open source or commercial, and often are unsuccessful at building a resilient, performant system.
In reality, building a distributed system requires a methodical approach to requirements along the
boundaries of failure domains, latency, throughput, durability, consistency, and desired SLAs for
the business application at all aspects of the application.”
17. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
Emerging
at Berkeley
18. Beyond Hadoop
Hadoop – an open source solution for fault-tolerant parallel
processing of batch jobs at scale, based on commodity hardware...
However, other priorities have emerged for analytics lifecycle:
●
Applications require integration beyond Hadoop
●
Multiple topologies, mixed workloads, mult-tenancy
●
Higher utilization
● Lower latency
●
High availability
●
More then “Just JVM” - e.g. Python ...
Next Generation Data Analytics Stack
20. Prior Practice: Dedicated Servers
• low utilization rates
• longer time to ramp up new services
DATACENTER
21. Prior Practice: Virtualization
DATACENTER PROVISIONED VMS
• even more machines to manage
• substantial performance decrease
due to virtualization
• VM licensing costs
22. Prior Practice: Static Partitioning
STATIC PARTITIONING
• even more machines to manage
• substantial performance decrease
due to virtualization
• VM licensing costs
• failures make static partitioning
more complex to manage
DATACENTER
23. What are the costs of Single Tenancy?
0%
25%
50%
75%
100%
RAILS CPU
LOAD
MEMCACHED
CPU LOAD
0%
25%
50%
75%
100%
HADOOP CPU
LOAD
0%
25%
50%
75%
100%
tt
0%
25%
50%
75%
100%
Rails
Memcached
Hadoop
COMBINED CPU LOAD (RAILS,
MEMCACHED, HADOOP)
24. MESOS
Mesos: One Large Pool of Resources
“We wanted people to be able to program
for the datacenter just like they program
for their laptop."
Ben Hindman
DATACENTER
25. MESOS
Mesos: One Large Pool of Resources
“We wanted people to be able to program
for the datacenter just like they program
for their laptop."
Ben Hindman
DATACENTER
26. Re-eval – What is Mesos?
● Mesos is a meta-scheduler
– Mesos is a distributed system to build and run distributed
systems.
● Microkernel for the datacenter.
– Common substrate, or programming abstractions, for
creating, or adapting distributed applications.
28. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
Use Cases
29. Case Study: Twitter (bare metal / on premise)
“Mesos is the cornerstone of our elastic compute infrastructure –
it’s how we build all our new services and is critical for Twitter’s
continued success at scale. It's one of the primary keys to our
data center efficiency."
Chris Fry, SVP Engineering
blog.twitter.com/2013/mesos-graduates-from-apache-incubation
wired.com/gadgetlab/2013/11/qa-with-chris-fry/
• key services run in production: analytics, typeahead, ads
• Twitter engineers rely on Mesos to build all new services
• instead of thinking about static machines, engineers think
about resources like CPU, memory and disk
• allows services to scale and leverage a shared pool of
servers across datacenters efficiently
• reduces the time between prototyping and launching
30. Case Study: Airbnb (fungible cloud infrastructure)
“We think we might be pushing data science in the field of travel
more so than anyone has ever done before… a smaller number
of engineers can have higher impact through automation on
Mesos."
Mike Curtis, VP Engineering
gigaom.com/2013/07/29/airbnb-is-engineering-itself-into-a-data...
• improves resource management and efficiency
• helps advance engineering strategy of building small teams
that can move fast
• key to letting engineers make the most of AWS-based
infrastructure beyond just Hadoop
• allowed company to migrate off Elastic MapReduce
• enables use of Hadoop along with Chronos, Spark, Storm, etc.
31. Case Study: HubSpot (cluster management)
Tom Petr
youtu.be/ROn14csiikw
• 500 deployable objects; 100 deploys/day to
production; 90 engineers; 3 devops on Mesos
cluster
• “Our QA cluster is now a fixed $10K/month —
that used to fluctuate”
32. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
The Shiny!
33. Dock-ah, dock-ah, dock-ah
● Enumerate hyperbolic awesome-sauce!
● Mesos had plugable add-ons for docker, now
they are being 1st classed into the core.
● Google Kubernetes Framework:
http://gigaom.com/2014/06/11/why-google-is-
sowing-the-seeds-of-container-based-
computing/
34. API's & More Plugability
● Always supported Protobuf API's compiled
against core libraries. Now native bindings are
occurring in the wild. (Go, Python, Java).
● Additional Plug-ins for Containerizers, Isolators,
...
36. Google Refs
● The Datacenter as a Computer: An Introduction to the Design
of Warehouse-Scale Machines
http://research.google.com/pubs/pub35290.html
● 2011 GAFS Omega John Wilkes:
http://youtu.be/0ZFMlO98Jkc
● Omega: flexible, scalable schedulers for large compute
clusters
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper
/Schwarzkopf.pdf
● Taming Latency Variability and Scaling Deep Learning
https://plus.google.com/u/0/+ResearchatGoogle/posts/C1dPh
QhcDRv
Thank You!
mesos.apache.org
@timothysc