In 2011, CERN decided to start using Puppet as main tool for development, machines configuration and provisioning as replacement of Quattor.
Since then the infrastructure has changed a lot, the "Agile infrastructure" project evolved is a series of tools and softwares that currently allow more than 10.000 nodes to be configured and provisioned following custom definitions.
Foreman, Git, Openstack and our homemade librarian Jens are only a few of the tools that will be described during the talk, that aims to give an overview about the current workflow for machines lifecycle at CERN.
This talk will cover how Puppet allows us to deal with several hundred of installations a day and, at the same time, provide highly customizable machine configurations for service owners.
CERN, the European Organization for Nuclear Research, is one of the world’s largest centres for scientific research. Its business is fundamental physics, finding out what the universe is made of and how it works. At CERN, accelerators such as the 27km Large Hadron Collider, are used to study the basic constituents of matter. This talk reviews the challenges to record and analyse the 25 Petabytes/year produced by the experiments and the investigations into how OpenStack could help to deliver a more agile computing infrastructure.
Frank Ham from Cascade Technologies presented this deck at the Stanford HPC Conference.
"A spin-off of the Center for Turbulence Research at Stanford University, Cascade Technologies grew out of a need to bridge between fundamental research from institutions like Stanford University and its application in industries. In a continual push to improve the operability and performance of combustion devices, high-fidelity simulation methods for turbulent combustion are emerging as critical elements in the design process. Multiphysics based methodologies can accurately predict mixing, study flame structure and stability, and even predict product and pollutant concentrations at design and off-design conditions."
Watch the video: http://insidehpc.com/2017/02/best-practices-large-scale-multiphysics/
Learn more: http://www.cascadetechnologies.com
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
Designing HPC & Deep Learning Middleware for Exascale Systemsinside-BigData.com
DK Panda from Ohio State University presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"This talk will focus on challenges in designing runtime environments for exascale systems with millions of processors and accelerators to support various programming models. We will focus on MPI, PGAS (OpenSHMEM, CAF, UPC and UPC++) and Hybrid MPI+PGAS programming models by taking into account support for multi-core, high-performance networks, accelerators (GPGPUs and Intel MIC), virtualization technologies (KVM, Docker, and Singularity), and energy-awareness. Features and sample performance numbers from the MVAPICH2 libraries will be presented."
Watch the video: http://wp.me/p3RLHQ-glW
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
10 Years of OpenStack at CERN - From 0 to 300k coresBelmiro Moreira
CERN, the European Laboratory for Particle Physics, provides the infrastructure and resources to thousands of scientists all around the world to uncover the mysteries of the Universe. In the quest to build a private Cloud Infrastructure to support its users, CERN started early evaluating the OpenStack project, building several prototypes and engaging with the community. Finally, in 2013 CERN released its production Cloud Infrastructure using OpenStack. Since then we moved from a few hundred cores to a multi-cell deployment spread between different regions. After 7 years deploying and managing OpenStack in production at a large scale, we now look back and discuss the challenges of building a massive scale infrastructure from 0 to +300K cores. In this talk we will dive into the history, architecture, tools and technical decisions behind the CERN Cloud Infrastructure over the years.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
CERN, the European Organization for Nuclear Research, is one of the world’s largest centres for scientific research. Its business is fundamental physics, finding out what the universe is made of and how it works. At CERN, accelerators such as the 27km Large Hadron Collider, are used to study the basic constituents of matter. This talk reviews the challenges to record and analyse the 25 Petabytes/year produced by the experiments and the investigations into how OpenStack could help to deliver a more agile computing infrastructure.
Frank Ham from Cascade Technologies presented this deck at the Stanford HPC Conference.
"A spin-off of the Center for Turbulence Research at Stanford University, Cascade Technologies grew out of a need to bridge between fundamental research from institutions like Stanford University and its application in industries. In a continual push to improve the operability and performance of combustion devices, high-fidelity simulation methods for turbulent combustion are emerging as critical elements in the design process. Multiphysics based methodologies can accurately predict mixing, study flame structure and stability, and even predict product and pollutant concentrations at design and off-design conditions."
Watch the video: http://insidehpc.com/2017/02/best-practices-large-scale-multiphysics/
Learn more: http://www.cascadetechnologies.com
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
Designing HPC & Deep Learning Middleware for Exascale Systemsinside-BigData.com
DK Panda from Ohio State University presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"This talk will focus on challenges in designing runtime environments for exascale systems with millions of processors and accelerators to support various programming models. We will focus on MPI, PGAS (OpenSHMEM, CAF, UPC and UPC++) and Hybrid MPI+PGAS programming models by taking into account support for multi-core, high-performance networks, accelerators (GPGPUs and Intel MIC), virtualization technologies (KVM, Docker, and Singularity), and energy-awareness. Features and sample performance numbers from the MVAPICH2 libraries will be presented."
Watch the video: http://wp.me/p3RLHQ-glW
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
10 Years of OpenStack at CERN - From 0 to 300k coresBelmiro Moreira
CERN, the European Laboratory for Particle Physics, provides the infrastructure and resources to thousands of scientists all around the world to uncover the mysteries of the Universe. In the quest to build a private Cloud Infrastructure to support its users, CERN started early evaluating the OpenStack project, building several prototypes and engaging with the community. Finally, in 2013 CERN released its production Cloud Infrastructure using OpenStack. Since then we moved from a few hundred cores to a multi-cell deployment spread between different regions. After 7 years deploying and managing OpenStack in production at a large scale, we now look back and discuss the challenges of building a massive scale infrastructure from 0 to +300K cores. In this talk we will dive into the history, architecture, tools and technical decisions behind the CERN Cloud Infrastructure over the years.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
The next generation of research infrastructure and large scale scientific instruments will face new magnitudes of data.
This talk presents two flagship programmes: the next generation of the Large Hadron Collider (LHC) at CERN and the Square Kilometre Array (SKA) radio telescope. Each in their way will push infrastructure to the limit.
The LHC has been one of the significant users of OpenStack in scientific computing. The SKA is now working to a final software architecture design and is focusing on OpenStack as an underlying middleware function.
Together, we plan to develop a common platform for scaling science: to accommodate new applications and software services, to deliver high ingest rate real-time and batch processing, to integrate high performance storage and to unlock the potential of software defined networking.
CERN OpenStack Cloud Control Plane - From VMs to K8sBelmiro Moreira
CERN is the home of the Large Hadron Collider (LHC), a 27km circular proton accelerator that generates petabytes of physics data every year. To process all this data, CERN runs an OpenStack Cloud (>300K cores) that helps scientists all around the world to unveil the mysteries of the Universe. The Infrastructure is also used to run all the IT services of the Organization.
Delivering these services, with high performance and reliable service levels has been one of the major challenges for the CERN Cloud engineering team. We have been constantly iterating the architecture and deployment model of the Cloud control plane.
In this presentation we will describe the different control plane architecture models that we relied over the years. Finally, we will describe all the work done to move the OpenStack Cloud control plane from VMs into a kubernetes cluster. We will report about our experience running this architecture at scale, its advantages and challenges.
Why do containers suddenly matter so much when they have been around since 1998? Take a look at the potential of OpenStack's Magnum, Murano and Nova-Docker in the context leveraging the incredible interest in Linux Containers brought about by Docker.
Check out www.stackengine.com to learn more about our excellent container management solution.
CERN, the European Organization for Nuclear Research, is running for several years a large OpenStack Cloud that helps thousands of scientists to analyze the data from the LHC.
In 2012, early in the design phase of the CERN Cloud we decided to use Nova Cells to enable the infrastructure to scale to thousands of nodes. Now with more than 280K cores spread across 70 cells that are hosted in two data centres we were faced with the challenge to migrate to Nova Cells V2 required in the Pike release.
In this presentation, we will describe how Nova Cells allowed CERN to scale to thousands of nodes, its advantages and how we mitigate the implementation issues of Nova Cells V1. Next, we will cover how we upgraded Nova from Newton with Cells V1 to Pike with Cells V2. We will explain the steps that we followed and the issues that we faced during the upgrade. Finally, we will report our experience with Cells V2 at scale, its caveats and how we are mitigate them.
What can I expect to learn?
This presentation describes how CERN migrated from Cells V1 to Cells V2 when upgraded from Newton to Pike release.
You will learn the procedures followed by CERN in order to migrate Cells V1 to Cells V2 in a large production environment.
The issues found during the upgrade and how we mitigate them will be discussed.
Also, we will present how Cells V2 behaves in a large scale deployment with serveral thounsands nodes in 70 cells.
Multi-Cell OpenStack: How to Evolve Your Cloud to Scale - November, 2014Belmiro Moreira
Multi-Cell OpenStack: How to Evolve Your Cloud to Scale
OpenStack Design Summit, Paris - November, 2014
Belmiro Moreira - CERN
Matt Van Winkle - Rackspace
Sam Morrison - NeCTAR, University of Melbourne
During the OpenStack Tokyo Summit we provided an overview on how Workday started the production deployment with a very robust and efficient CI/CD process that it explained here.
Containers on Baremetal and Preemptible VMs at CERN and SKABelmiro Moreira
CERN the European Laboratory for Nuclear Research and SKA the Square Kilmeter Array are preparing the next generation of research infrastructure for the new large scale scientific instruments that will produce new magnitudes of data. In Sydney OpenStack Summit we presented the collaboration and the platform that we plan to develop for scaling science.
In this talk will present the work done related with Preemptible VMs and Containers on Baremetal.
Preemptible VMs are instances that use idle allocated resources in the infrastructure and can be terminated when this capacity is required. Containers in Baremetal eliminate the virtualization overhead enabling the container full performance required for scientific workloads.
We will present the current state, development and integration decisions and how these functionalities can be used in a common OpenStack infrastructure.
Cortex: Horizontally Scalable, Highly Available PrometheusGrafana Labs
In this talk we present Cortex - a horizontally scalable, highly available Prometheus implementation. Like Prometheus, Cortex is a CNCF (sandbox) project.
Cortex turns a lot of the Prometheus architectural assumptions on its head, by marrying a scale-out PromQL query engine with a storage layer based on NOSQL databases such as Bigtable, DynamoDB and Cassandra. We have disaggregated the Prometheus binary into a microservices-style architecture, with separate services for query, ingest, alerting and recording rules. By designing all these services as fungible replicas, this solution can be scaled out with ease and failure of any individual replica can be dealt with gracefully.
The Contrail Virtual Execution Platform (VEP) allows Cloud administrators to manage data centers and monitor the usage of resources. Users can manage their distributed applications on IaaS Cloud providers under the control of Service Level Agreements (SLA). VEP applications are packaged in the standard OVF format and they are deployed inside Constrained Execution Environments (CEE) derived from the SLA, to support the specification of SLA contracts between users and providers.
These CEE environments allow to define constraints concerning virtual hardware performance, localization and affinity allowing the administrator to configure the monitoring system in order to feed external SLA enforcement services. VEP integrates elasticity management capabilities which can be controlled by external SLA enforcement services. A resource allocator service is integrated to dispatch the virtual components on the physical resources of the provider in accordance with the SLA terms.
The first version of VEP is currently implemented on OpenNebula. This talk presents the implementation of VEP on OpenNebula and discusses some implementation choices such as the resource allocator.
OSDC 2018 | From batch to pipelines – why Apache Mesos and DC/OS are a soluti...NETWAYS
Apache Mesos is a distributed system for running other distributed systems, often described as a distributed kernel. It’s in use at massive scale at some of the worlds largest companies like Netflix, Uber and Yelp, abstracting entire data centres of hardware to allow for workloads to be distributed efficiently. DC/OS is an open source distribution of Mesos, which adds all the functionality to run Mesos in production across any substrate, both on-premise and in the cloud. In this talk, I’ll introduce both Mesos and DC/OS and talk about how they work under the hood, and what the benefits are of running these new kinds of systems for emerging cloud native workloads.
This presentation is about scaling container groups by integrating magnum and senlin service. Magnum provision container orchestration engines while senlin is used to scale the container groups and the VM groups used.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Puppet Camp Berlin 2015: Nicholas Corrarello | Puppet DemoNETWAYS
This is a flexible demo that will start by showing you all of the various technologies involved in using Puppet and Puppet Enterprise. It is meant to be interactive, so there should be plenty of time for additional demos to answer your specific questions about using Puppet and the related technologies.
Puppet Camp Berlin 2015: Martin Alfke | The Power of Puppet 4NETWAYS
The talk will guide you through the major changes which are brought by Puppet 4.x.
We will dig through deprecations and how to migrate and we will show new functionality like the built in data type validation and many more.
The next generation of research infrastructure and large scale scientific instruments will face new magnitudes of data.
This talk presents two flagship programmes: the next generation of the Large Hadron Collider (LHC) at CERN and the Square Kilometre Array (SKA) radio telescope. Each in their way will push infrastructure to the limit.
The LHC has been one of the significant users of OpenStack in scientific computing. The SKA is now working to a final software architecture design and is focusing on OpenStack as an underlying middleware function.
Together, we plan to develop a common platform for scaling science: to accommodate new applications and software services, to deliver high ingest rate real-time and batch processing, to integrate high performance storage and to unlock the potential of software defined networking.
CERN OpenStack Cloud Control Plane - From VMs to K8sBelmiro Moreira
CERN is the home of the Large Hadron Collider (LHC), a 27km circular proton accelerator that generates petabytes of physics data every year. To process all this data, CERN runs an OpenStack Cloud (>300K cores) that helps scientists all around the world to unveil the mysteries of the Universe. The Infrastructure is also used to run all the IT services of the Organization.
Delivering these services, with high performance and reliable service levels has been one of the major challenges for the CERN Cloud engineering team. We have been constantly iterating the architecture and deployment model of the Cloud control plane.
In this presentation we will describe the different control plane architecture models that we relied over the years. Finally, we will describe all the work done to move the OpenStack Cloud control plane from VMs into a kubernetes cluster. We will report about our experience running this architecture at scale, its advantages and challenges.
Why do containers suddenly matter so much when they have been around since 1998? Take a look at the potential of OpenStack's Magnum, Murano and Nova-Docker in the context leveraging the incredible interest in Linux Containers brought about by Docker.
Check out www.stackengine.com to learn more about our excellent container management solution.
CERN, the European Organization for Nuclear Research, is running for several years a large OpenStack Cloud that helps thousands of scientists to analyze the data from the LHC.
In 2012, early in the design phase of the CERN Cloud we decided to use Nova Cells to enable the infrastructure to scale to thousands of nodes. Now with more than 280K cores spread across 70 cells that are hosted in two data centres we were faced with the challenge to migrate to Nova Cells V2 required in the Pike release.
In this presentation, we will describe how Nova Cells allowed CERN to scale to thousands of nodes, its advantages and how we mitigate the implementation issues of Nova Cells V1. Next, we will cover how we upgraded Nova from Newton with Cells V1 to Pike with Cells V2. We will explain the steps that we followed and the issues that we faced during the upgrade. Finally, we will report our experience with Cells V2 at scale, its caveats and how we are mitigate them.
What can I expect to learn?
This presentation describes how CERN migrated from Cells V1 to Cells V2 when upgraded from Newton to Pike release.
You will learn the procedures followed by CERN in order to migrate Cells V1 to Cells V2 in a large production environment.
The issues found during the upgrade and how we mitigate them will be discussed.
Also, we will present how Cells V2 behaves in a large scale deployment with serveral thounsands nodes in 70 cells.
Multi-Cell OpenStack: How to Evolve Your Cloud to Scale - November, 2014Belmiro Moreira
Multi-Cell OpenStack: How to Evolve Your Cloud to Scale
OpenStack Design Summit, Paris - November, 2014
Belmiro Moreira - CERN
Matt Van Winkle - Rackspace
Sam Morrison - NeCTAR, University of Melbourne
During the OpenStack Tokyo Summit we provided an overview on how Workday started the production deployment with a very robust and efficient CI/CD process that it explained here.
Containers on Baremetal and Preemptible VMs at CERN and SKABelmiro Moreira
CERN the European Laboratory for Nuclear Research and SKA the Square Kilmeter Array are preparing the next generation of research infrastructure for the new large scale scientific instruments that will produce new magnitudes of data. In Sydney OpenStack Summit we presented the collaboration and the platform that we plan to develop for scaling science.
In this talk will present the work done related with Preemptible VMs and Containers on Baremetal.
Preemptible VMs are instances that use idle allocated resources in the infrastructure and can be terminated when this capacity is required. Containers in Baremetal eliminate the virtualization overhead enabling the container full performance required for scientific workloads.
We will present the current state, development and integration decisions and how these functionalities can be used in a common OpenStack infrastructure.
Cortex: Horizontally Scalable, Highly Available PrometheusGrafana Labs
In this talk we present Cortex - a horizontally scalable, highly available Prometheus implementation. Like Prometheus, Cortex is a CNCF (sandbox) project.
Cortex turns a lot of the Prometheus architectural assumptions on its head, by marrying a scale-out PromQL query engine with a storage layer based on NOSQL databases such as Bigtable, DynamoDB and Cassandra. We have disaggregated the Prometheus binary into a microservices-style architecture, with separate services for query, ingest, alerting and recording rules. By designing all these services as fungible replicas, this solution can be scaled out with ease and failure of any individual replica can be dealt with gracefully.
The Contrail Virtual Execution Platform (VEP) allows Cloud administrators to manage data centers and monitor the usage of resources. Users can manage their distributed applications on IaaS Cloud providers under the control of Service Level Agreements (SLA). VEP applications are packaged in the standard OVF format and they are deployed inside Constrained Execution Environments (CEE) derived from the SLA, to support the specification of SLA contracts between users and providers.
These CEE environments allow to define constraints concerning virtual hardware performance, localization and affinity allowing the administrator to configure the monitoring system in order to feed external SLA enforcement services. VEP integrates elasticity management capabilities which can be controlled by external SLA enforcement services. A resource allocator service is integrated to dispatch the virtual components on the physical resources of the provider in accordance with the SLA terms.
The first version of VEP is currently implemented on OpenNebula. This talk presents the implementation of VEP on OpenNebula and discusses some implementation choices such as the resource allocator.
OSDC 2018 | From batch to pipelines – why Apache Mesos and DC/OS are a soluti...NETWAYS
Apache Mesos is a distributed system for running other distributed systems, often described as a distributed kernel. It’s in use at massive scale at some of the worlds largest companies like Netflix, Uber and Yelp, abstracting entire data centres of hardware to allow for workloads to be distributed efficiently. DC/OS is an open source distribution of Mesos, which adds all the functionality to run Mesos in production across any substrate, both on-premise and in the cloud. In this talk, I’ll introduce both Mesos and DC/OS and talk about how they work under the hood, and what the benefits are of running these new kinds of systems for emerging cloud native workloads.
This presentation is about scaling container groups by integrating magnum and senlin service. Magnum provision container orchestration engines while senlin is used to scale the container groups and the VM groups used.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Puppet Camp Berlin 2015: Nicholas Corrarello | Puppet DemoNETWAYS
This is a flexible demo that will start by showing you all of the various technologies involved in using Puppet and Puppet Enterprise. It is meant to be interactive, so there should be plenty of time for additional demos to answer your specific questions about using Puppet and the related technologies.
Puppet Camp Berlin 2015: Martin Alfke | The Power of Puppet 4NETWAYS
The talk will guide you through the major changes which are brought by Puppet 4.x.
We will dig through deprecations and how to migrate and we will show new functionality like the built in data type validation and many more.
Puppet Camp Berlin 2015: Pedro Pessoa | Puppet at the center of everything - ...NETWAYS
Pedro shows a short overview on how they use puppet at Server Density to manage the entire infrastructure and code, content mostly taken from previous presentations.
The focus will be on how to reduce the 4-year-old code base while making more use of forge modules while migrating to Puppet Enterprise 3.
OSDC 2014: Michael Renner - Secure encryption in a wiretapped futureNETWAYS
Since the beginning of publications by Edward Snowden last year many of the presumedly exaggerated threat models in cryptography have become reality. When operating sensitive services it's more likely than not that communcation data will be tapped at large carriers as well as internet exchanges and stored indefinitily - this calls for strong and forward-secure encryption.
On the other hand we're faced with the problem that much of the software we're using in the datacenter today is not very secure when it comes to default encryption settings. On top of that, most developers and system administrators are not very fluent in the basic workings of encryption systems.
The talk will give an introduction to SSL/TLS and explain how to check for weaknesses in existing services with tools like nmap, sslscan and sslyze. For common daemons like apache, nginx, exim, postfix and dovecot best practice on improving cryptographic strength will be discussed.
OSDC 2014: Christian Kniep - Understand your data center by overlaying multi...NETWAYS
Today's data center managers are burdened by a lack of aligned information of multiple layers. Work-flow events like 'job starts' aligned with performance metrics and events extracted from log facilities are low-hanging fruit that is on the edge to become use-able due to open-source software like Graphite, StatsD, logstash and alike.
This talk aims to show off the benefits of merging multiple layers of information within an InfiniBand cluster by using use-cases for level 1/2/3 personnel.
OSDC 2014: Schlomo Schapiro - Test Driven InfrastructureNETWAYS
Common wisdom has it that the test effort should be related to the risk of a change. However, the reality is different: Developers build elaborate automated test chains to test every single commit of their application. Admins regularly “test” changes on the live platform in production. But which change carries a higher risk of taking the live platform down?
What about the software that runs at the “lower levels” of your platform, e.g. systems automation, provisioning, proxy configuration, mail server configuration, database systems etc. An outage of any of those systems can have a financial impact that is as severe as a bug in the “main” software!
One of the biggest learnings that any Ops person can learn from a Dev person is Test Driven Development. Easy to say - difficult to apply is my personal experience with the TDD challenge.
This talk throws some light on recent developments at ImmobilienScout24 that help us to develop the core of our infrastructure services with a test driven approach:
* How to do unit tests, integration tests and systems tests for infrastructure services?
* How to automatically verify Proxy, DNS, Postfix configurations before deploying them on live servers?
* How to test “dangerous” services like our PXE boot environment or the automated SAN mounting scripts?
* How to add a little bit of test coverage to everything we do.
* Test Driven: First write a failing test and then the code that fixes it.
OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2NETWAYS
Graylog2 is a free and open source log analysis tool that allows you to perform searches, analyse the data, build dashboards and set alarms using the streams system. Typical use cases range from debugging platform problems & monitoring exception counts to displaying average pizza delivery time per state on a dashboard.
In this talk I will go through the architecture of Graylog2, what you can do with it and how to get your data into it.
Puppet Camp Berlin 2015: Felix Frank | Rapid Testing Setups for PuppetNETWAYS
Puppet installations are usually quite robust and require low maintenance. The initial setup is not quite trivial, however. The Puppet master will also become quite a critical system once it is put to work in earnest.
As a result, it can become a somewhat daunting task to perform changes on the master, or conduct more intrusive debugging. This presentation shows how test instances of both the master and agent can be launched with little effort, and how Puppet can be run from source.
OSDC 2014: Nat Morris - Open Network Install EnvironmentNETWAYS
ONIE defines an open source “install environment” that runs on this management subsystem utilizing facilities in a Linux/BusyBox environment. This environment allows end-users and channel partners to install the target network OS as part of data center provisioning, in the fashion that servers are provisioned.
ONIE enables switch hardware suppliers, distributors and resellers to manage their operations based on a small number of hardware SKUs. This in turn creates economies of scale in manufacturing, distribution, stocking, and RMA enabling a thriving ecosystem of both network hardware and operating system alternatives.
Webinar Icinga 2: Enterprise Monitoring der nächsten Generation (22.07.2014)NETWAYS
Icinga 2 ist die vollständige Neuentwicklung der beliebten Open Source Monitoring Lösung Icinga. Neben der komplett überarbeiteten Architektur des Cores, wurden viele Anforderungen von IT-Administratoren wie automatische Check-Verteilung, integrierte Hochverfügbarkeit und Synchronisation der gesamten Konfiguration innerhalb einer Icinga 2 Umgebung direkt eingebaut. Darüber hinaus bietet das neue Konfigurationsschema eine wesentlich flexiblere Möglichkeit, seine Umgebung aufzubauen. Dabei helfen unter anderem die neuen Parameter „apply“, „assign“ und „ignore“.
Website: http://www.netways.de
Webinar: http://www.netways.de/de/webinare/archiv/icinga_2/#Icinga2_Enterprise_Monitoring_der_naechsten_Generation
YouTube: https://www.youtube.com/watch?v=HI4b24C7pVk
OpenNebula Conf 2014 | Deploying OpenNebula in a Snap using Configuration Man...NETWAYS
Configuration management and Cloud management tools are on the rage right now. Having both of them makes the life of us sysadmins a breeze which ultimately means we can spend more time doing something else rather than just tinkering around to fix a problem or deploying something new.
Foreman: Provisionierungswege (Webinar vom 31. März 2016)NETWAYS
Foreman ist ein Open Source Lifecycle Management Tool, welches einem bei der Verwaltung des gesamten IT-Lebenszyklus unterstützt. Darüber hinaus dient es unter anderem als Dashboard für die Configuration Management Lösung Puppet und erlaubt hier das anlegen und verwalten von unterschiedlichsten IT-Systemen. In diesem Webinar liegt der Fokus auf den möglichen Provisionierungswegen.
Webinare
Archiv Link: https://www.netways.de/webinare/webinare_aktuell/foreman_provisionierungswege/
Aktuell: https://www.netways.de/webinare/webinare_aktuell/
NETWAYS
Konferenzen: https://www.netways.de/events_schulungen/home/
Schulungen: https://www.netways.de/events_schulungen/schulungen/home/
Shop: https://shop.netways.de/
Blog: http://blog.netways.de/
Social Media
YouTube: https://www.youtube.com/channel/UC8nIBEFmjzXjXeJV_hkkeIQ
Facebook: https://www.facebook.com/netways
Google+: https://plus.google.com/+netways/
Twitter: https://twitter.com/netways
OSDC 2015: Timo Derstappen | Microservice orchestrationNETWAYS
There’s no doubt, everyone’s talking about container technologies these days. Especially Docker has established itself as a de facto standard here. These technologies are especially suited to develop, run, and deploy microservices. So there are a few things coming together. This talk will give an overview why things like immutable infrastructure, continuous delivery and small autonomous teams make a lot of sense in today’s web scale architectures. Further, we discuss how these technologies can be used to manage an infrastructure with a high amount of containers.
In many modern applications the database side is realized using polyglot persistence? Store each data format (graphs, documents, etc.) in an appropriate separate database.
This approach yields several benefits, databases are optimized for their specific duty, however there are also drawbacks:
keep all databases in sync
queries might require data from several databases
experts needed for all used systems
A multi-model database is not restricted to one data format, but can cope with several of them.
In this talk I will present how a multi-model database can be used in a polyglot persistence setup and how it will reduce the effort drastically.
OSDC 2015: Stephen Benjamin | Foreman in Your Data CenterNETWAYS
An introductory talk to Foreman, with an overview of how Foreman's plugin ecosystem can help you manage your data center. We'll talk about Discovery, Katello, Docker, and additional configuration management platforms beyond Puppet.
Kubermatic How to Migrate 100 Clusters from On-Prem to Google Cloud Without D...Tobias Schneck
Have you ever thought about migrating your Kubernetes clusters to Google Cloud to get your services closer to your customers? Yes? We too! Join us on an interactive journey to discover the main challenges of live migration at scale of etcd's, traffic routing and application workloads from your on-premise platform to GCP. The talk will discuss the current state of the technical concept, known problems and insides of the already proven migration steps for stateless workload.
As part of the journey, we'll see the differences between migrating one or one hundred clusters with productive workloads; What parts can be automated? What steps may need to be manual? Let's see how an automated solution could look like in the future and what steps are missing.
How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtimeloodse
Have you ever thought about migrating your Kubernetes clusters to Google Cloud to get your services closer to your customers? Yes? Us too! Join us on an interactive journey to discover the main challenges of live migration at scale of etcd’s, traffic routing and application workloads from your on-premise platform to GCP. The talk will discuss the current state of the technical concept, known problems and insides of the already proven migration steps for stateless workloads.
As part of the journey, we'll see
- The differences between migrating one or one hundred clusters with productive workloads
- What parts can be automated?
- What steps may need to be done manually?
Red Hat multi-cluster management & what's new in OpenShiftKangaroot
More and more organisations are not only using container platforms but starting to run multiple clusters of containers. And with that comes new headaches of maintaining, securing, and updating those multiple clusters. In this session we'll look into how Red Hat has solved multi-cluster management, covering cluster lifecycle, app lifecycle, and governance/risk/compliance.
DevOps Days Boston 2017: Real-world Kubernetes for DevOpsAmbassador Labs
DevOps Days Boston 2017
Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices?
In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.
Using Kubernetes to make cellular data plans cheaper for 50M usersMirantis
Use case of Kubernetes based NFV infrastructure used in production to run an open source evolved packet core. Presented by Facebook Connectivity and Mirantis at KubeCon + CloudNativeCon Europe 2020.
Top 10 present and future innovations in the NoSQL Cassandra ecosystem (2022)Cédrick Lunven
Are you new to Apache Cassandra® and wondering what all the excitement is about? Or a veteran Cassandra user interested in understanding what’s new in the project?
Attend our live webinar on October 18 to learn about the latest Cassandra release and why it represents a big step forward but also all the initiative and new projects rising in the ecosystem, DataStax Director of Developer Relations Cedrick Lunven will walk you through new features in version 4.1.
Get the inside scoop on how version 4.1 adds exciting new features for operators and improves the security posture, without compromising the stability achieved in Cassandra 4.0. Get some insights about projects actually in progress to make Cassandra more easy to use (Stargate) but also to deploy (K8ssandra).
You will learn:
System-wide Guardrails
Denylisting Partition Keys
Diagnostic events via CQL, not just JMX
CQLSH Auth support for LDAP, Kerberos and more
Lots of new, pluggable extension points
Also, celebrate our open source community with highlights from the 2022 Apache Cassandra World Party and a look ahead to Cassandra 5.0!
Architecting The Future - WeRise Women in TechnologyDaniel Barker
Kubernetes and Docker are two of the top open source projects, and they’re built around abstractions and metadata. These two concepts are the key to architecting in the future. Come with me as I dig a little deeper into these concepts within k8s and Docker and provide some examples from my own work.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, Universi...OpenStack
Audience Level
Intermediate
Synopsis
High performance computing and cloud computing have traditionally been seen as separate solutions to separate problems, dealing with issues of performance and flexibility respectively. In a diverse research environment however, both sets of compute requirements can occur. In addition to the administrative benefits in combining both requirements into a single unified system, opportunities are provided for incremental expansion.
The deployment of the Spartan cloud-HPC hybrid system at the University of Melbourne last year is an example of such a design. Despite its small size, it has attracted international attention due to its design features. This presentation, in addition to providing a grounding on why one would wish to build an HPC-cloud hybrid system and the results of the deployment, provides a complete technical overview of the design from the ground up, as well as problems encountered and planned future developments.
Speaker Bio
Lev Lafayette is the HPC and Training Officer at the University of Melbourne. Prior to that he worked at the Victorian Partnership for Advanced Computing for several years in a similar role.
An in depth overview of Kubernetes and it's various components.
NOTE: This is a fixed version of a previous presentation (a draft was uploaded with some errors)
Google has been running everything in containers for the past 15 years, but how do we orchestrate and manage all those containers? We've built and released the open source Kubernetes (http://kubernetes.io), which is based on years of running containers internally at Google. Join us for an introduction to containers and Kubernetes, followed by a hands-on workshop building and deploying your own Kubernetes cluster with multiple front end, database and caching instances.
Docker containers help solve the issue of process-level reproducibility by packaging up your apps and execution environments into a number of containers. But once you have a lot of containers running, you'll need to coordinate them across a cluster of machines while keeping them healthy and making sure they can find each other. This can quickly turn into an unmanageable mess! Wouldn't it be helpful if you could declare what wanted, and then have the cluster assign the resources to get it done and to recover from failures and scale on demand? Kubernetes is here to help!
Key takeaways
- Gentle introduction into containers: why and how
- Learn how Google manages applications using containers
- Intro to Kubernetes: managing applications and services
- Build and deploy your own multi-tier application using Kubernetes
Since 2012, CERN is running an Openstack private cloud with around 320k cores and supports not only the LHC, but also the services for the whole laboratory. We have been scaling up the infrastructure to cover these computing needs and at the same time we have also been increasing the service offering to include file shares, baremetal nodes and container orchestration clusters among others.
The key aspects that allowed us to scale quickly and be able to continuously adapt to our user needs are automation and integration into the CERN ecosystem. We will review the tools that allows us to offload most of the heavy-lifting tasks, further delegate administrative operations and react on monitoring alarms. It includes solutions for simplify project, resource management and support operations based on Mistral and Rundeck.
Finally we will look into ongoing work on services like Kubernetes jobs, Vitrage and Watcher that will increase even further the automation provided.
Similar to Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: Going Agile with Style (20)
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: Going Agile with Style
1.
2. Configuration Management @
CERN
Going Agile with Style
Andrea Giardini
CERN
andrea.giardini@cern.ch
24 April 2015 - PuppetCamp Berlin
Configuration Management @ CERN 2
3. Outline
Introduction
What is CERN
Datacenters Overview
Puppet @ CERN
Current Infrastructure
Modules, Hostgroups and Environments
Configuration Management
Managing Changes
Tools
Conclusions
Configuration Management @ CERN 3
4. What is CERN
European Organization for Nuclear
Research
Situated in the border between
Switzerland and France
21 Member states
Big challenges
Configuration Management @ CERN 4
9. Datacenters in Numbers
Two datacenters:
Budapest
Geneva
Two dedicated links:
2 x 100Gbps
The number of resources is growing year
by year. As today:
15k servers
100PB on tape
200PB on disk
Configuration Management @ CERN 9
10. Going Agile
Requirements started to grow
Agile approach was needed
Since a few years we started using Openstack to deploy virtual
machines for our users and Puppet to configure the services
Configuration Management @ CERN 10
11. Our Setup
We started using Puppet a few years ago and, since then, things evolved a lot . . .
We changed several time the configuration of our puppet masters in order to keep up
with the requests and we found out that:
Puppet scales horizontally quite well
The NFS filer underneath . . . does not
NFS is used to share configurations and Puppet code between different masters.
All the masters used to mount the same shared folder . . .
Configuration Management @ CERN 11
13. Few concepts
Modules (∼ 280)
The various modules available should be viewed as a library that your hostgroup
code can reuse.
Hostgroups (∼ 160)
Groups of nodes that are part of the same service and have some configurations in
common.
Environments (∼ 180)
Collections of modules and hostgroups at different development levels.
Configuration Management @ CERN 13
14. Environments allow us to . . .
Environment ”production” → All modules/hg from ”master” branch
Environment ”qa” → All modules/hg from ”qa” branch
Custom environments (for testing purpose):
Possibility to set a default branch
Specify specific branch for one or more modules/hostgroups
Configuration Management @ CERN 14
15. Manage changes
Three important concepts:
Modules
Hostgroups
Environments
A configuration change has to be
approved through a request in Jira.
Every git repo has at least two branches:
master
qa
Configuration Management @ CERN 15
17. Jens
Jens creates Puppet environments for the Puppet Masters
Using repository metadata and a list of environments definitions
Allows dynamic environments and isolates puppet code for different services
Has recently been opensourced on GitHub:
https://github.com/cernops/jens
Useful for those running different services under the same puppet infrastructure
Configuration Management @ CERN 17
18. Configuration Change Process
Configuration change process:
Modify a module on feature branch
Create a custom env and test the module
Open a ticket on Jira and announce the change
Merge to qa
After one week, merge to production
Service managers use the same module for different services: we need to be sure that
all the service managers are happy with the change before merging it to production.
Configuration Management @ CERN 18
19. Jenkins and Continuous Integration process
Machines are built and tested before
merging a change to production
More automation, less manual work
Still work in progress, but looks
promising
Configuration Management @ CERN 19
21. Automating procedures - RunDeck
Tedious prone-error tasks replaced
by executable code
Handing off operational tasks to
others
Procedures as a list of individual
and atomic steps
Ability to react to failures
Configuration Management @ CERN 21
23. Mcollective
Framework for server orchestration and
parallel job execution
Problems in the past with big clusters
> 3000 nodes
Latest improvements:
Direct addressing
New PuppetDB discovery method
Threaded Mode
Batched requests
Configuration Management @ CERN 23
24. Configuration Drifts
Configuration drifts started to be a problem:
Out of sync machines
Possibility for service managers to have snapshots
Possibility to freeze their environment
It’s not easy to keep all the configuration in sync
Configuration Management @ CERN 24
25. Package Inventory
Centralized service for package inventory:
Using Elasticsearch
Queryable using Cli
Compare a set of hosts
Reports differences and misalignments
Package History
Configuration Management @ CERN 25
26. Conclusions
Moving from a traditional infrastructure to an Agile one allowed us to:
Optimize our resources
Speed up the development cycle
Reduce interventions time
Have more free time :)
Configuration Management @ CERN 26
27. Conclusions
Puppet gives us the right combination between elasticity and efficiency
Big community
Active development
Highly customizable
Configuration Management @ CERN 27