This document discusses testing Kubernetes and OpenShift at scale. It describes installing large clusters of 1000+ nodes, using scalability test tools like the Kubernetes performance test repo and OpenShift SVT repo to load clusters and generate traffic. Sample results show loading clusters with thousands of pods and projects, and peaks in master node resource usage when loading and deleting hundreds of pods simultaneously.
Arkena's video-on-demand platform is used as backend by major european channels (TF1 / beIN SPORTS / Elisa) to propose a non-linear experience to their customers.
Previously hosted on Heroku, the number of our users is increasing constantly. In order to optimize resources we decided to move on a bare metal infrastructure powered by Kubernetes.
We'll share thoughts, feedbacks and technical details about this successful transition.
Sched Link:
Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools.
Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)
Learn from the dozens of large-scale deployments how to get the most out of your Kubernetes environment:
- Container images optimization
- Organizing namespaces
- Readiness and Liveness probes
- Resource requests and limits
- Failing with grace
- Mapping external services
- Upgrading clusters with zero downtime
Arkena's video-on-demand platform is used as backend by major european channels (TF1 / beIN SPORTS / Elisa) to propose a non-linear experience to their customers.
Previously hosted on Heroku, the number of our users is increasing constantly. In order to optimize resources we decided to move on a bare metal infrastructure powered by Kubernetes.
We'll share thoughts, feedbacks and technical details about this successful transition.
Sched Link:
Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools.
Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)
Learn from the dozens of large-scale deployments how to get the most out of your Kubernetes environment:
- Container images optimization
- Organizing namespaces
- Readiness and Liveness probes
- Resource requests and limits
- Failing with grace
- Mapping external services
- Upgrading clusters with zero downtime
In this deck from the Docker Workshop at ISC 2015, Andreas Schmidt from Cassini Consulting describes Docker in a Nutshell
"As the newest flavor of Linux Containers, Docker gained a lot of momentum in the last 12 months. With a very convenient and open API-driven architecture Docker is able to help decrease the complexity of operations and increase the productivity of computation. During the last two years Andreas, Christian, and Wolfgang gained a lot of experience with Docker and were thrilled by its possible impact early on. Andreas started working with Docker in mid-2013 and is interested in developing tools for solving Enterprise IT requirements on networking and security. In 2014 he held talks and workshops about these topics. Christian started using Docker in 2013 to virtualize a complete HPC cluster stack and since then held multiple talks about how Docker might impact HPC. Wolfgang and his partner Burak Yenier introduced Docker as a corner-stone of the UberCloud Marketplace to drastically improve and simplify access to HPC cloud resources. UberCloud just announced their new containers for computational fluid dynamics software like Fluent, STAR-CCM+ and OpenFOAM."
Watch the video presentation: http://wp.me/p3RLHQ-enP
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
A short presentation at a CSC internal workshop of the prospects of using container technologies, especially Docker, in the context of High Performance Computing (HPC).
Kubernetes and OpenStack at Scale at OpenStack Summit Boston 2017
Imagine being able to stand up thousands of tenants with thousands of apps, running thousands of Docker-formatted container images and routes, all on a self-healing cluster and elastic infrastructure. Now, take that one step further - all of those images being updatable through a single upload to the registry, and with zero downtime. In this session, you will see just that.
In this presentation, we will walk through a recent benchmarking deployment using Kubernetes and OpenStack on the Cloud Native Computing Foundation’s (CNCF's) 1,000 node cluster with OpenStack and Red Hat’s OpenShift Container Platform, the enterprise-ready Kubernetes for developers.
You'll also what's been happening in subsequent rounds of testing in Red Hat's own SCALE lab and the CNCF cluster and how we are working with the relevant open source communities including OpenStack, Kubernetes, and Ansible to continue to raise the bar for horizontal scaling of these platforms via community powered innovation.
DockerCon US 2016 - Extending Docker With APIs, Drivers, and PluginsArnaud Porterie
Anusha Ragunathan and Arnaud Porterie present different ways to extend the Docker Engine in increasing level of effort required: through the user-facing API, through plugins, and finally through execution drivers.
containerd the universal container runtimeDocker, Inc.
containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc..
containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.
containerd includes a daemon exposing gRPC API over a local UNIX socket. The API is a low-level one designed for higher layers to wrap and extend. It also includes a barebone CLI (ctr) designed specifically for development and debugging purpose. It uses runC to run containers according to the OCI specification. The code can be found on GitHub, and here are the contribution guidelines.
containerd is based on the Docker Engine’s core container runtime to benefit from its maturity and existing contributors.
Dockerizing OpenStack for High AvailabilityDaniel Krook
Presentation at the OpenStack Summit in Paris, France on November 4, 2014.
High availability in OpenStack can be achieved in many ways. In this session we will describe how Docker can be used to provide an active-active highly available OpenStack environment. We will focus the real world work that we have done to "Dockerize" OpenStack services, detail the advantages to this type of deployment (rapid deployment, rapid scale out, versioning, etc.), and walk through our design - from requirements, limitations, obstacles, and especially our decisions. We will use our experiences as examples to provide real world best practices, as well as showing a demonstration of the environment in action.
Manuel Silveyra - Senior Cloud Solutions Architect
Daniel Krook - Senior Certified IT Specialist
Shaun Murakami - Senior Cloud Solution Architect
Kalonji Bankole - Cloud Architect
Building stateful applications on Kubernetes with RookRoberto Hashioka
Deploying stateful applications such a Wordpress and Jenkins on top of Kubernetes or any other container orchestrator can be a challenging task. In this context, Rook will be used to showcase how to automatically manage the volume's lifecycle through the its Kubernetes operators (operator pattern approach) by leveraging the recently added CSI GA support.
- Introduction to Kubernetes features
- A look at Kubernetes Networking and Service Discovery
- New features in Kubernetes 1.6
- Kubernetes Installation options
To know more about our Kubernetes expertise, visit our center of excellence at: http://www.opcito.com/kubernetes/
Kubernetes has been a key component for many companies to reduce technical debt in infrastructure by:
• Fostering the Adoption of Docker
• Simplifying Container Management
• Onboarding Developers On Infrastructure
• Unlocking Continuous Integration and Delivery
During this meetup we are going to discuss the following topics and share some best practices
• What's new with Kubernetes 1.3
• Generate Cluster Configuration using CloudFormation
• Deploy Kubernetes Clusters on AWS
• Scaling the Cluster
• Integrating Ingress with Elastic Load Balancer
• Using Internal ELB's as Kubernetes' Service
• Using EBS for persistent volumes
• Integrating Route53
Presentation delivered at LinuxCon China 2017.
Operating systems need to move faster without sacrificing stability. New hardware, new software features, and bugfixes are making it into distribution components every day. To maintain stability, packagers and distribution developers are looking toward lessons learned in the DevOps movement to implement Continuous Integration/Continuous Delivery (CI/CD) workflows that provide quicker test feedback to developers.
This talk highlights some of the coming trends in Fedora such as: streamlined base package sets, userspace applications delivered as containers, continuous validation of individual distro components and the distro as a whole, and collaboration with the CentOS Project.
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...Wojciech Barczyński
I will tell you two stories about two different implementations of Kubernetes. One from Fashion mobile ecomerce. One from a Fintech. Kubernetes is not a silver bullet. But damn close ;).
KubeCon EU 2016: Multi-Tenant KubernetesKubeAcademy
Today Kubernetes is mostly employed in single tenant deployment, either private cloud, or as a COE on top of IaaS. By leveraging virtualized container like Hyper, Kubernetes will be the core of multi-tenant Container-as-a-Service. This talk will present Hypernetes, a secure Kubernetes distro focusing on the public container hosting service.
Sched Link: http://sched.co/6BYD
Thanks to tools like vagrant, puppet/chef, and Platform as a Service services like Heroku, developers are extremely used to being able to spin up a development environment that is the same every time. What if we could go a step further and make sure our development environment is not only using the same software, but 100% configured and set up like production. Docker will let us do that, and so much more. We’ll look at what Docker is, why you should look into using it, and all of the features that developers can take advantage of.
In this deck from the Docker Workshop at ISC 2015, Andreas Schmidt from Cassini Consulting describes Docker in a Nutshell
"As the newest flavor of Linux Containers, Docker gained a lot of momentum in the last 12 months. With a very convenient and open API-driven architecture Docker is able to help decrease the complexity of operations and increase the productivity of computation. During the last two years Andreas, Christian, and Wolfgang gained a lot of experience with Docker and were thrilled by its possible impact early on. Andreas started working with Docker in mid-2013 and is interested in developing tools for solving Enterprise IT requirements on networking and security. In 2014 he held talks and workshops about these topics. Christian started using Docker in 2013 to virtualize a complete HPC cluster stack and since then held multiple talks about how Docker might impact HPC. Wolfgang and his partner Burak Yenier introduced Docker as a corner-stone of the UberCloud Marketplace to drastically improve and simplify access to HPC cloud resources. UberCloud just announced their new containers for computational fluid dynamics software like Fluent, STAR-CCM+ and OpenFOAM."
Watch the video presentation: http://wp.me/p3RLHQ-enP
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
A short presentation at a CSC internal workshop of the prospects of using container technologies, especially Docker, in the context of High Performance Computing (HPC).
Kubernetes and OpenStack at Scale at OpenStack Summit Boston 2017
Imagine being able to stand up thousands of tenants with thousands of apps, running thousands of Docker-formatted container images and routes, all on a self-healing cluster and elastic infrastructure. Now, take that one step further - all of those images being updatable through a single upload to the registry, and with zero downtime. In this session, you will see just that.
In this presentation, we will walk through a recent benchmarking deployment using Kubernetes and OpenStack on the Cloud Native Computing Foundation’s (CNCF's) 1,000 node cluster with OpenStack and Red Hat’s OpenShift Container Platform, the enterprise-ready Kubernetes for developers.
You'll also what's been happening in subsequent rounds of testing in Red Hat's own SCALE lab and the CNCF cluster and how we are working with the relevant open source communities including OpenStack, Kubernetes, and Ansible to continue to raise the bar for horizontal scaling of these platforms via community powered innovation.
DockerCon US 2016 - Extending Docker With APIs, Drivers, and PluginsArnaud Porterie
Anusha Ragunathan and Arnaud Porterie present different ways to extend the Docker Engine in increasing level of effort required: through the user-facing API, through plugins, and finally through execution drivers.
containerd the universal container runtimeDocker, Inc.
containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc..
containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.
containerd includes a daemon exposing gRPC API over a local UNIX socket. The API is a low-level one designed for higher layers to wrap and extend. It also includes a barebone CLI (ctr) designed specifically for development and debugging purpose. It uses runC to run containers according to the OCI specification. The code can be found on GitHub, and here are the contribution guidelines.
containerd is based on the Docker Engine’s core container runtime to benefit from its maturity and existing contributors.
Dockerizing OpenStack for High AvailabilityDaniel Krook
Presentation at the OpenStack Summit in Paris, France on November 4, 2014.
High availability in OpenStack can be achieved in many ways. In this session we will describe how Docker can be used to provide an active-active highly available OpenStack environment. We will focus the real world work that we have done to "Dockerize" OpenStack services, detail the advantages to this type of deployment (rapid deployment, rapid scale out, versioning, etc.), and walk through our design - from requirements, limitations, obstacles, and especially our decisions. We will use our experiences as examples to provide real world best practices, as well as showing a demonstration of the environment in action.
Manuel Silveyra - Senior Cloud Solutions Architect
Daniel Krook - Senior Certified IT Specialist
Shaun Murakami - Senior Cloud Solution Architect
Kalonji Bankole - Cloud Architect
Building stateful applications on Kubernetes with RookRoberto Hashioka
Deploying stateful applications such a Wordpress and Jenkins on top of Kubernetes or any other container orchestrator can be a challenging task. In this context, Rook will be used to showcase how to automatically manage the volume's lifecycle through the its Kubernetes operators (operator pattern approach) by leveraging the recently added CSI GA support.
- Introduction to Kubernetes features
- A look at Kubernetes Networking and Service Discovery
- New features in Kubernetes 1.6
- Kubernetes Installation options
To know more about our Kubernetes expertise, visit our center of excellence at: http://www.opcito.com/kubernetes/
Kubernetes has been a key component for many companies to reduce technical debt in infrastructure by:
• Fostering the Adoption of Docker
• Simplifying Container Management
• Onboarding Developers On Infrastructure
• Unlocking Continuous Integration and Delivery
During this meetup we are going to discuss the following topics and share some best practices
• What's new with Kubernetes 1.3
• Generate Cluster Configuration using CloudFormation
• Deploy Kubernetes Clusters on AWS
• Scaling the Cluster
• Integrating Ingress with Elastic Load Balancer
• Using Internal ELB's as Kubernetes' Service
• Using EBS for persistent volumes
• Integrating Route53
Presentation delivered at LinuxCon China 2017.
Operating systems need to move faster without sacrificing stability. New hardware, new software features, and bugfixes are making it into distribution components every day. To maintain stability, packagers and distribution developers are looking toward lessons learned in the DevOps movement to implement Continuous Integration/Continuous Delivery (CI/CD) workflows that provide quicker test feedback to developers.
This talk highlights some of the coming trends in Fedora such as: streamlined base package sets, userspace applications delivered as containers, continuous validation of individual distro components and the distro as a whole, and collaboration with the CentOS Project.
Effective Kubernetes - Is Kubernetes the new Linux? Is the new Application Se...Wojciech Barczyński
I will tell you two stories about two different implementations of Kubernetes. One from Fashion mobile ecomerce. One from a Fintech. Kubernetes is not a silver bullet. But damn close ;).
KubeCon EU 2016: Multi-Tenant KubernetesKubeAcademy
Today Kubernetes is mostly employed in single tenant deployment, either private cloud, or as a COE on top of IaaS. By leveraging virtualized container like Hyper, Kubernetes will be the core of multi-tenant Container-as-a-Service. This talk will present Hypernetes, a secure Kubernetes distro focusing on the public container hosting service.
Sched Link: http://sched.co/6BYD
Thanks to tools like vagrant, puppet/chef, and Platform as a Service services like Heroku, developers are extremely used to being able to spin up a development environment that is the same every time. What if we could go a step further and make sure our development environment is not only using the same software, but 100% configured and set up like production. Docker will let us do that, and so much more. We’ll look at what Docker is, why you should look into using it, and all of the features that developers can take advantage of.
KubeCon NA, Seattle, 2016: Performance and Scalability Tuning Kubernetes for...Jeremy Eder
earn tips and tricks on how to best configure and tune your container infrastructure for maximum performance and scale. The Performance Engineering Group at Red Hat is responsible for performance of the complete container portfolio, including Docker, RHEL Atomic, Kubernetes and OpenShift. We will share: - Latest Performance Features in OpenShift, Docker and RHEL Atomic, tips and tricks on how to best configure and tune your system for maximum performance and scale - Latest performance and scale test results, using RHEL Atomic, OpenvSwitch, Cockpit multi-server container management - DevOps, Agile approach to Performance Analysis of OpenShift, Kubernetes, Docker and RHEL Atomic - Test harness code and example scripts
Audience
The audience is anyone interested in deploying containers to run performance sensitive workloads, as well as architecting highly scalable distributed systems for hosting those workloads. This includes workloads that require NUMA awareness, direct hardware access and kernel-bypass I/O.
We're really happy to say that today we made the first meetup about Kubernetes in Russia! Thanks to all speakers and guests! Join us: https://twitter.com/kubernetesMSK
We're really happy to say that today we made the first meetup about Kubernetes in Russia! Thanks to all speakers and guests! Join us: https://twitter.com/kubernetesMSK
With so many different tools at our disposable, how do you pick which ones to learn? At our latest meetup for Denver Code Club, we explored some best practices on evaluating new technology and how you can choose the right tools for you.
Bitnami, Deis, Google and the Kubernetes community have been working on developing Helm, a tool for streamlining the deployment of containerized applications on Kubernetes. Bitnami currently offers a set of Helm packages, known as charts, to make it easy to deploy your favorite open source applications on Kubernetes with a single command. Join our webinar to learn how to quickly get started with Helm:
In this webinar you will learn:
- How to deploy Kubernetes-native applications
- How to manage the lifecycle of applications on Kubernetes using Helm
- The benefits of using Bitnami Helm Charts
- The best practices we've learned while creating and configuring - Bitnami Helm charts
- How to get started with Bitnami Helm Charts
Kubernetes as Orchestrator for A10 Lightning ControllerAkshay Mathur
A10 Lightning Application Delivery System (ADS) supports hybrid environments by providing secure application services and advanced analytics across the entire deployment – from traditional on-premise data centers, to public and/or private clouds, or any combination thereof. A10 Lightning employs a controller-based architecture that can self-managed on-premise or in a private cloud, or utilized as a SaaS offering managed by A10, to enable management of heterogeneous workloads across physical hardware-based environments, as well as public, private, and hybrid clouds.
This presentation talks about our journey from a VM based Controller to a Kubernetes based Controller
DevFestMN 2017 - Learning Docker and Kubernetes with OpenshiftKeith Resar
Hands-on lab discovering containers (through docker), the need for container orchestration (using Kubernetes), and the place for a container PaaS (via OpenShift)
Bangalore Container Conference 2017 (BCC '17) is the first conference on container technologies in India. Organizations are increasingly adopting containers and related technologies in production. Hence, the main focus of this conference is “Containers in Production”. This one-day conference sets the perfect stage for container enthusiasts, developers, users and experts to meet together and learn from each others experiences. This deck provides details for sponsoring the conference.
This talk will focus on a brief history, including a demo and overview of how we at Superbalist use Kubernetes, and how Kubernetes uses Docker, does load balancing, deployments, and data migrations.
Talk from Cape Town DevOps meetup on Jun 21, 2016:
https://www.meetup.com/Cape-Town-DevOps/events/231530172/
Code: https://github.com/zoidbergwill/kubernetes-examples
Slides as markdown: http://www.zoidbergwill.com/presentations/2016/kubernetes-1.2-and-spread/index.md
Presentation slides from DevConf.cz 2017
Challenges, take-aways and recommendations on scaling up OpenShift's logging and metrics stack.
Authors:
Ricardo Lourenço:
https://www.linkedin.com/in/ricardopereira4it/
Elvir Kuric
https://www.linkedin.com/in/elvirkuric/
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas.
Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.
Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015Belmiro Moreira
Tips Tricks and Tactics with Cells and Scaling OpenStack
OpenStack Design Summit, Paris - May, 2015
Belmiro Moreira - CERN
Matt Van Winkle - Rackspace
Sam Morrison - NeCTAR, University of Melbourne
NetflixOSS Meetup S3 E1, covering latest components in Distributed Databases, Telemetry systems, Big Data tools and more. Speakers from Netflix, IBM Watson, Pivotal and Nike Digital
Capacity planning is a difficult challenge faced by most companies. If you have too few machines, you will not have enough compute resources available to deal with heavy loads. On the other hand, if you have too many machines, you are wasting money. This is why companies have started investing in automatically scaling services and infrastructure to minimize the amount of wasted money and resources.
In this talk, Nathan will describe how Yelp is using PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos, to automatically and gracefully scale services and the underlying cluster. He will go into detail about how this functionality was implemented and the design designs that were made while architecting the system. He will also provide a brief comparison of how this approach differs from existing solutions.
Historically, sharing a Linux server entailed all kinds of untenable compromises. In addition to the security concerns, there was simply no good way to keep one application from hogging resources and messing with the others. The classic “noisy neighbor” problem made shared systems the bargain-basement slums of the Internet, suitable only for small or throwaway projects.
Serious use-cases traditionally demanded dedicated systems. Over the past decade virtualization (in conjunction with Moore’s law) has democratized the availability of what amount to dedicated systems, and the result is hundreds of thousands of websites and applications deployed into VPS or cloud instances. It’s a step in the right direction, but still has glaring flaws.
Most of these websites are just piles of code sitting on a server somewhere. How did that code got there? How can it can be scaled? Secured? Maintained? It’s anybody’s guess. There simply isn’t enough SysAdmin talent in the world to meet the demands of managing all these apps with anything close to best practices without a better model.
Containers are a whole new ballgame. Unlike VMs, you skip the overhead of running an entire OS for every application environment. There’s also no need to provision a whole new machine to have a place to deploy, meaning you can spin up or scale your application with orders of magnitude more speed and accuracy.
Introduction to Container Storage Interface (CSI)Idan Atias
Among the cool stuff we do at Silk, my colleagues and I develop the Silk CSI Plugin for customers who use our system as the storage layer for their Kubernetes workloads.
Before deep diving into the code and as part of my ramp-up on this subject I prepared some slides that cover some basic and important information on this topic.
These slides start by recapping some basic storage principals in containers and Kubernetes, continues with some more advanced use cases (including an "offline demo" of persisting Redis data on EBS volumes), and ends with a detailed information on the CSI solution itself.
IMHO, reviewing these slides can improve your understanding on this matter and can get you started implementing your own CSI plugin.
The main sources of information I used for preparing these slides are:
* Official CSI docs
* Kubernetes Storage Lingo 101 - Saad Ali, Google
* Container Storage Interface: Present and Future - Jie Yu, Mesosphere, Inc.
Sanger OpenStack presentation March 2017Dave Holland
A description of the Sanger Institute's journey with OpenStack to date, covering RHOSP, Ceph, S3, user applications, and future plans. Given at the Sanger Institute's OpenStack Day.
Ever since the “CloudNative revolution” took over our development environment (devenv), we have never been more challenged (or more excited). With Kubernetes, Docker (Containerd) & many other microservice-related technologies, we have a handful of technologies to master before we write the first line of code.
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
Delivery of a new Bio-informatics infrastructure at the Wellcome Trust Sanger Center. We include how to programatically create, manage and provide providence for images used both at Sanger and elsewhere using open source tools and continuous integration.
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
In this presentation I talk about our motivation to converting our microservices to run on Kubernetes. I discuss many of the technical challenges we encountered along the way, including networking issues, Java issues, monitoring and alerting, and managing all of our resources!
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
2. Agenda
● Kubernetes/OpenShift runtimes & scalability goals
● OpenShift system testing: what does it cover?
● Installing large clusters
● Scalability test tools (the Kubernetes performance
test repo and the the OpenShift SVT repo)
● Sample results
3.
4. K8s and OpenShift runtimes
● Primarily targeted at cloud platforms
○ Amazon EC2, Google Cloud Platform, Microsoft Azure
○ Enterprise-hosted cloud offerings/infra
○ On-prem cloud infra such as OpenStack
○ Bare metal and other virtualization environments, too
● Cluster sizes from all-in-one dev/sandbox to
multi-master, 1000+ nodes or federated clusters
5. Persistent Volume StorageNodes
node
1
node
2
EBS
(Persistent
Volumes)
S3 (Registry)
node
1000
Control Plane
master1
+ etcd1
SSD
master2
+ etcd2
SSD
master3
+ etcd3
SSD
Infrastructure Group
infra2:
HAProxy router2
docker-registry2
infra1:
HAProxy router1
docker-registry1
Application
ELB
(Routes)
External
ELB
(Console)
Internet
Int
ELB
(Nodes)
What does a cluster look like?
AWS sample:
6. Kubernetes SIG-scale
● Scalability special interest group
○ https://github.com/kubernetes/community/tree/master/sig-scalability
● Container workload is what matters - listen to your applications
○ The numbers here are more “control plane” - think small pods/containers
● Stated future goals:
○ Assumption: core/node = 64 (higher in the future)
○ Pods/core = 10 (depends on workload)
○ Pods/node = 500 - 640 (depends on workload, these would be small pods)
○ nodes/cluster = 5000
○ pods/cluster = 500,000 (note: less than node x pods/node)
○ pod startup time < 5 seconds
○ Schedule 100 pods/second
9. System Test team in Red Hat
● Kubernetes and OpenShift Scalability
○ Cluster horizontal scale
■ # of nodes
■ # of running pods across all nodes
■ application traffic
○ Node vertical scale
■ # of pods running on a single node
■ workload that a single node can support (applications, builds, storage)
○ Application scalability
■ Scale # of application replicas up/down
10. System Test team in Red Hat
● Performance
○ Resource usage and response times for scenarios and workloads
■ Application workload and access performance
■ Builds (OpenShift)
■ Metrics and Log collection
○ OpenShift infrastructure performance
■ Resource usage of processes under load
■ Network (SDN) throughput
■ Routing
■ Storage (EBS, Ceph, Gluster, Cinder, etc)
11. System Test team in Red Hat
● Reliability
○ Simulated user workloads
■ monthly, weekly, daily, hourly, minute activities
■ accelerated to run faster than real-time
○ Run for extended periods and measure CPU, memory, I/O,
network over time
12. SVT Challenges/Fun
● Installation
○ 1000+ node installs are time consuming (multiple hours)
○ On public cloud providers, time = $$$. Maximize time testing
○ 500 node test cluster on AWS is around USD $1500 - 2000/day
● Verifying that a cluster is viable
○ Don’t waste time on buggy clusters
● Loading up a cluster with application containers
● Putting a workload on the cluster
● Collecting performance data in large clusters
18. Kubernetes e2e and perf test
● e2e (end-to-end) tests
○ https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-te
sts.md
○ Subset of e2e tests are tagged as Conformance.
○ Conformance = minimum supported functionality for operational cluster
○ OpenShift also adds some additional Conformance tests if you yum install
atomic-openshift-tests on top of OpenShift
● Performance tests
○ https://github.com/kubernetes/perf-tests
○ Work in progress
19. OpenShift SVT repo
● https://github.com/openshift/svt
● Tools for OpenShift performance, scale, reliability
○ cluster load-up
○ traffic generation
○ concurrent builds, deployments, pod start/stop
○ reliability testing
○ network performance
○ logging and metrics tests
● Automated and executed from Jenkins
20. Cluster load-up
● cluster-loader - python tool to quickly load clusters according to a YAML test
specification. Takes advantage of OpenShift’s template capabilities
● Can be used with Kubernetes or OpenShift
● SVT repository has sample YAML configurations for node vertical, cluster horizontal,
“Quick Start” applications with and without persistent storage.
“I want an environment with thousands of deployments, pods (with persistent storage), build
configurations, routes, services, secrets and more…”
projects:
- num: 1000
basename: nginx-explorer
tuning: default
templates:
- num: 10
file: cluster-loader/nginx.yaml
- num: 20
file: cluster-loader/explorer-pod.yaml
21. Cluster traffic generation
● cluster-loader can also run in traffic generation mode
● Runs a JMeter pod to generate traffic against applications (installed
by cluster-loader or otherwise)
● Hit rate, throughput, response codes, response times, etc
● Discovers applications, exposed routes, etc
● Currently OpenShift only, but working on an upstream version.
23. Performance Tools
● PBench: Performance and Benchmark Analysis
Framework
○ pbench-agent: collection agent and harness for running tests.
■ Collects data from sar, vmstat, iostat, pidstat, perf, etc
■ Extensible: additional data collectors can be added
■ Packages raw data from a test and ships it to pbench-server
○ pbench-server: processes raw data from all systems under test
○ web-server: provides visualization of data
https://github.com/distributed-system-analysis/pbench
26. Master 1 - is the controller leader for
most of the run
Master 2 - has to pick up controller
leader when Master 1 fails
Loading on OSP 8 cluster:
● 500 nodes
● 20K projects
● 52K pods
Masters are 40vCPU and peak out at
22 cores used.
27. Create/delete hundreds of pods : Amazon EBS IOPs credit exhaustion - AWS “I/O
cliff”
gp2 EBS volumes on EC2 can run “fast” until their IOPS credits are exhausted
After that, they are throttled to 3 iops/gb until credits build back up
