This document discusses deploying software at scale through automation. It advocates treating infrastructure as code and using version control, continuous integration, and packaging tools. The key steps are to automate deployments, make them reproducible, and deploy changes frequently and consistently through a pipeline that checks code, runs tests, builds packages, and deploys to testing and production environments. This allows deploying changes safely and quickly while improving collaboration between developers and operations teams.
ContainerCon - Test Driven InfrastructureYury Tsarev
Great external coverage of this presentation can be found at https://www.cedric-meury.ch/2016/10/test-driven-infrastructure-with-puppet-docker-test-kitchen-and-serverspec-yury-tsarev-gooddata/
ContainerCon - Test Driven InfrastructureYury Tsarev
Great external coverage of this presentation can be found at https://www.cedric-meury.ch/2016/10/test-driven-infrastructure-with-puppet-docker-test-kitchen-and-serverspec-yury-tsarev-gooddata/
Puppet at DemonWare - Ruaidhri Power - Puppetcamp Dublin '12Puppet
This talk will describe the evolution of how we've used Puppet at Demonware, a subsidiary of Activision Blizzard, to run the infrastructure of some of the world's biggest games, supporting millions of concurrent users for titles such as Call of Duty.
Ruaidhri Power of DemonWare at PuppetCamp Dublin '12. http://www.puppetlabs.com
Deploying your Drupal site, Upgrading your Drupal Site, Scaling, Clustering and Monitoring it ... all topics Developers are often not involved with ...
Devops For Drupal explains the Devops problem, to a Drupal audience .
Manage your Windows Infrastructure with Puppet Bolt - August 26 - 2020Puppet
This two-hour workshop will focus on the following:
Improving operational efficiency for managing Windows infrastructure
Applying configuration baselines to Windows Server and IIS web servers
Utilizing PowerShell and Bolt to automate day to day management tasks
What's Bolt? Puppet Bolt is the easiest way to get started with DevOps and does not require Puppet knowledge. During this workshop you will utilize WinRM or SSH to communicate with your server environments.
You will leave this workshop with a working knowledge of Bolt, and your laptop equipped to start tackling automation challenges across your organization.
Dev to Delivery with Puppet, Vagrant and AWSPuppet
"Dev to Delivery with Puppet, Vagrant and AWS" by Sam Bashton of Bashton Ltd. at Puppet Camp London 2013. Find the video here: http://puppetlabs.com/community/puppet-camp
http://www.meetup.com/BruJUG/events/228994900/
During this session, you will presented a solution to the problem of scalability of continuous delivery in Jenkins, when your organisation has to deal with thousands of jobs, by introducing a self-service approach based on the "pipeline as code" principles.
Docker and Puppet for Continuous IntegrationGiacomo Vacca
Today developers want to change the code, build and deploy often, even several times per day.
New versions of software may need to be tested on different distributions, and with different configurations.
Achieving this with Virtual Machines it’s possible, but it’s very resource and time consuming. Docker provides an incredibly good solution for this, in particular if combined with Continuous Integration tools like Jenkins and Configuration Management tools like Puppet.
This presentation focuses on the opportunities to configure automatically Docker images, use Docker containers as disposable workers during your tests, and even running your Continuous Integration system inside Docker.
Towards Continuous Deployment with DjangoRoger Barnes
It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting.
This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.
Continuous integration for open source distros v 3.0Sriram Narayanan
These are the slides of my talk at Foss.IN 2012 on Continuous Integration for Open Source Distribution ecosystems. I have covered how existing Continuous Integration efforts can be improved to ensure greate collaboration between open source projects, developers and distributions to make earlier r
Puppet at DemonWare - Ruaidhri Power - Puppetcamp Dublin '12Puppet
This talk will describe the evolution of how we've used Puppet at Demonware, a subsidiary of Activision Blizzard, to run the infrastructure of some of the world's biggest games, supporting millions of concurrent users for titles such as Call of Duty.
Ruaidhri Power of DemonWare at PuppetCamp Dublin '12. http://www.puppetlabs.com
Deploying your Drupal site, Upgrading your Drupal Site, Scaling, Clustering and Monitoring it ... all topics Developers are often not involved with ...
Devops For Drupal explains the Devops problem, to a Drupal audience .
Manage your Windows Infrastructure with Puppet Bolt - August 26 - 2020Puppet
This two-hour workshop will focus on the following:
Improving operational efficiency for managing Windows infrastructure
Applying configuration baselines to Windows Server and IIS web servers
Utilizing PowerShell and Bolt to automate day to day management tasks
What's Bolt? Puppet Bolt is the easiest way to get started with DevOps and does not require Puppet knowledge. During this workshop you will utilize WinRM or SSH to communicate with your server environments.
You will leave this workshop with a working knowledge of Bolt, and your laptop equipped to start tackling automation challenges across your organization.
Dev to Delivery with Puppet, Vagrant and AWSPuppet
"Dev to Delivery with Puppet, Vagrant and AWS" by Sam Bashton of Bashton Ltd. at Puppet Camp London 2013. Find the video here: http://puppetlabs.com/community/puppet-camp
http://www.meetup.com/BruJUG/events/228994900/
During this session, you will presented a solution to the problem of scalability of continuous delivery in Jenkins, when your organisation has to deal with thousands of jobs, by introducing a self-service approach based on the "pipeline as code" principles.
Docker and Puppet for Continuous IntegrationGiacomo Vacca
Today developers want to change the code, build and deploy often, even several times per day.
New versions of software may need to be tested on different distributions, and with different configurations.
Achieving this with Virtual Machines it’s possible, but it’s very resource and time consuming. Docker provides an incredibly good solution for this, in particular if combined with Continuous Integration tools like Jenkins and Configuration Management tools like Puppet.
This presentation focuses on the opportunities to configure automatically Docker images, use Docker containers as disposable workers during your tests, and even running your Continuous Integration system inside Docker.
Towards Continuous Deployment with DjangoRoger Barnes
It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting.
This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.
Continuous integration for open source distros v 3.0Sriram Narayanan
These are the slides of my talk at Foss.IN 2012 on Continuous Integration for Open Source Distribution ecosystems. I have covered how existing Continuous Integration efforts can be improved to ensure greate collaboration between open source projects, developers and distributions to make earlier r
Thursday, June 12th 2014
Discussing strategies in Rails development for keeping multiple application environments as consistent as possible for the best development, testing, and deployment experience.
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Richard Bullington-McGuire
Have you ever needed to wrestle a legacy application onto a modern, scalable cloud platform, while increasing security test coverage? Sometimes real applications are not easily stuffed into a Docker container and deployed in a container orchestration system. In this talk, Modus Create Principal Architect Richard Bullington-McGuire will show how to compose Jenkins, Docker, Terraform, Packer, Ansible, Packer, Vagrant, Gauntlt, OpenSCAP, the CIS Benchmark for Linux, AWS CodeDeploy, Auto Scaling Groups, Application Load Balancers, and other AWS services to create a performant and scalable solution for deploying applications. A local development environment using Vagrant mirrors the cloud deployment environment to minimize surprises upon deployment.
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...NETWAYS
I gave a talk titled "Continuous Integration in data centers“ at OSDC in 2013, presenting ways how to realize continuous integration/delivery with Jenkins and related tools.Three years later we gained new tools in our continuous delivery pipeline, including Docker, Gerrit and Goss. Over the years we also had to deal with different problems caused by faster release cycles, a growing team and gaining new projects. We therefore established code review in our pipeline, improved our test infrastructure and invested in our infrastructure automation.In this talk I will discuss the lessons we learned over the last years, demonstrate how a proper continuous delivery pipeline can improve your life and how open source tools like Jenkins, Docker and Gerrit can be leveraged for setting up such an environment.
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Борис Зора
When you start your journey with µServices, you should be confident with your delivery lifecycle. In case of mistake, you should be able to navigate to appropriate tag in vcs to reproduce the bug with a test & go though pipeline within 3 hours to production with high confidence of quality.
We will discuss set of tools that could help you to achieve this within 3 months on your project. It does not include system decoupling suggestions. And in the same time, if you decide to break down monolith, it is better to do with dev & devOps best practices.
This talk will try to cover the most important techniques and best practices used when creating Django web application.
Overview of the topics covered:
- development general principles and goals
- python/django project initial setup - project layout, git&venv&pip&shell, settings
- central project shell command - contains all commands to manage project
- "IDE" - editor & shell
- edit/run/test cycle
- deploy/test-remotely cycle
Disclaimer: techniques and practices presented are current AUTHOR'S optimal choice used for usual django project.
Comment améliorer le quotidien des Développeurs PHP ?AFUP_Limoges
Conférence présentée lors du summer meetup de l'AFUP à Limoges le 19 juin 2018. Son objectif est de présenter plusieurs outils permettant de gagner rapidement en efficacité au quotidien.
Spenser Reinhardt's presentation on Detecting Security Breaches With Docker, Honeypots, & Nagios.
The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
Software Delivery in 2016 - A Continuous Delivery ApproachGiovanni Toraldo
The speech "Software Delivery in 2016" was held by Giovanni Toraldo (Lead Developer at ClouDesire) on July 1st 2016 in Pisa, Italy.
Event: Apericoder
Organizer: Coders TUG
Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.
2. Kris Buytaert
● I used to be a Dev,
● Then Became an Op
● Chief Trolling Officer and Open Source
Consultant @inuits.eu
● Everything is an effing DNS Problem
● Building Clouds since before the bookstore
● Some books, some papers, some blogs
● Evangelizing devops
3. Todays Goals
● A reproducable way to deploy and upgrade
software
● Automatically
● Fast
● Consistent
5. What's the problem ?
The community of developers whose work you
see on the Web, who probably don’t know what
ADO or UML or JPA even stand for, deploy better
systems at less cost in less time at lower risk
than we see in the Enterprise. This is true even
when you factor in the greater flexibility and
velocity of startups.
Tim Bray , on his blog January 2010
6. The Old Days
● “Put this Code Live, here's a tarball” NOW!
● What dependencies ?
● No machines available ?
● What database ?
● Security ?
● High Availability ?
● Scalability ?
● My computer can't install this ?
7. devops
● Culture
● (Lean)
● Automation
● Measurement
● Sharing
Damon Edwards and John Willis
Gene Kim
9. Nirvana
An “ecosystem” that supports continuous delivery, from
infrastructure, data and configuration management to
business.
Through automation of the build, deployment, and testing
process, and improved collaboration between developers,
testers, and operations, delivery teams can get changes
released in a matter of hours — sometimes even minutes–no
matter what the size of a project or the complexity of its code
base.
Continuous Delivery , Jez Humble
10.
11. How many times a day ?
● 10 @ Flickr
● Deployments used to be pain
● Nobody dared to deploy a site
● Practice makes perfect
● Knowing you can vs constantly doing it
12. " Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is to
enable the business goals. We strongly feel that
in order to do that you must have the ability to
deploy code quickly and safely. Even if the
business goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push all
the time), having a reliable and easy
deployment should be non-negotiable."
Etsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
15. Infrastructure as Code
● Treat configuration automation as code
● Development best practices
• Model your infrastructure
• Version your cookbooks / manifests
• Test your cookbooks/ manifests
• Dev/ test /uat / prod for your infra
● Model your infrastructure
● A working service = automated ( Application Code +
Infrastructure Code + Security + Monitoring )
● Think Puppet, Chef, Cfengine, Ansible , ....
16. Version Control
● Git !
● Version ALL the things:
• Source code Application
• Source code Infrastructure
• Builds
• Tests
• Pipelines
• Scripts
• Documentation
• Monitoring scripts
22. App Requirements
● Testable
● Configuration isolated
● Automated Deployments
● “If my computer can't install it , the installer is
borken” Luke Kanies at Fosdem (2007)
● Bulk provisioning of data
● http://www.krisbuytaert.be/blog/how-i-my-java
23. Why ops like to package
● Packages give you features
•Consistency, security, dependencies
● Uniquely identify where files come from
•Package or cfg-mgmt
● Source repo not always available
•Firewall / Cloud etc ..
● Weird deployment locations , no easy access
● Little overhead when you automate
● CONFIG does not belong in a package
28. fpm in action
● https://github.com/Inuits/build-gems
● Fork, pull
● Jenkins pulls , builds , pushes to repo
● (variants for Nagios Plugins / Jenkins Plugins
available)
29. A pipeline
● Checkout code ● Upload to Repo
● Syntax
● Style
● Code Coverage
● Tests
● Build
● More Tests
● Package
30. Pulp
Pulp is a Python application for managing
software repositories and their associated
content, such as packages, errata, and
distributions. It can replicate software
repositories from a variety of supported sources,
such as http/https, file system, ISO, and RHN, to
a local on-site repository. It provides mechanisms
for systems to gain access to these repositories,
providing centralized software installation.
31. Pulp
● Redhat Community
● Redhat Emerging Technology
● Part of Katello
32.
33. Pulp
● “manages” its own apache instance
● Symlinks , no copies
● Queues
•Syncing in the background
•No more screens ;)
● Actions are not instantly
•e.g. Add / sync / delete
● Hello mongodb :(
● v1 vs v2
● Only use repo functionality , cfgmgmt is in charge of packages
35. Version vs Latest
● Version your repos ?
ensure => latests
● Latest your environments ?
● Strict versioning in config ?
ensure => '0.98.4'
36. A pipeline
● Checkout code ● Upload to Repo
● Syntax ● Deploy on Test
● Style
● Code Coverage
● Tests
● Build
● More Tests
● Package
37. The Marionette Collective
● Distributed ssh ++
● What version of ssh do I have installed on my
servers ?
● On what servers is XYZ running ?
● Clean all my ssl certs ?
● Restart apache on all servers with fact X
38. mc-package
mc-package -W /dev/ status jdk
* [ ============================================================> ] 33 / 33
servicesdb01.dev.com version = -absent
services.dev.google.com version = jdk-1.6.0_13-fcs
drbdtest02.dev.google.com version = -absent
services3.dev.google.com version = jdk-1.6.0_20-fcs
um.dev.google.com version = jdk-1.5.0_19-fcs
devtools03.uat.com version = jdk-1.6.0_29-fcs
alexandria02.dev.google.com version = -absent
weblink01.dev.com version = -absent
wikitest.dev.google.com version = jdk-1.6.0_24-fcs
payment.dev.google.com version = jdk-1.5.0_17-fcs
tiff2pdf01.dev.com version = -absent
devdoos.dev.com version = jdk-1.6.0_30-fcs
wiki.dev.google.com version = jdk-1.6.0_24-fcs
reporting01.dev.com version = -absent
devtools01-dev.uat.com version = jdk-1.6.0_23-fcs
devtools02.uat.com version = jdk-1.6.0_29-fcs
drbdtest01.dev.google.com version = -absent
---- package agent summary ----
Nodes: 33/33
Versions: 1 * 1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs, 1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0
Elapsed Time: 1.73 s
39. What to Trigger ?
● Update Package
• Only updates package
● Trigger Puppet Run
• Updates config + package
40. A pipeline
● Checkout code ● Upload to Repo
● Syntax ● Deploy on Test
● Style ● More Tests
● Code Coverage ● Promote
● Tests ● Deploy on UAT
● Build ● More Tests
● More Tests ● Promote
● Package ● Deploy on Prod
41. Done ?
● Close the feedback loop,
● Send metric on deployment
echo "deployed.$package_name 1 `date +
%s`" > /dev/tcp/<%= graphite_host %>/2003
42. Done ?
A Software project is not done until your last
enduser is in his grave !
43. But remember
Everything is a Fscking DNS Problem
No really, Everything is a Fscking DNS Problem
If it's not a fucking DNS Problem ..
It's an arp problem
If it's not an arp problem...
It's a Full Filesystem Problem
If your filesystem isn't full
It's a Spanning Tree problem
If it's not a spanning Tree problem...
It's a USB problem
If it's not a USB Problem
It might be an ntp problem
If it's not an ntp problem
It's a sharing IRQ Problem
If it's not a sharing IRQ Problem
But most often .. its a Freaking Dns Problem !
Or someone playing tricks on you
Jan 2006