Development, Deployment and Collaboration at Etsy
•
4 likes•1,898 views
At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.
Report
Share
Report
Share
Download to read offline
Recommended
DevTools at Etsy
The DevTools team at Etsy consists of 5 engineers who build and maintain tools to help the 150 committers at Etsy. These tools include Deployinator for deploying code, DeveloperVMs for local development environments, Jenkins slaves for continuous integration, DevDB Proxy for accessing production data safely, and various monitoring and debugging tools like StatsD, Logster, Graphite, and Supergrep. The DevTools team works to foster collaboration between development and operations through these shared tools.
Mobile CI at Etsy
Etsy uses GitHub Enterprise for code reviews of code committed to their on-premise git repositories. They have over 350 Linux virtual machines called "The Bobs" that are used to build and test code. Etsy also has 25 Mac Minis that are used to build, test, and run linting on their mobile Android and iOS code using an automated process with Chef. The summary describes Etsy's implementation of continuous integration for mobile apps using GitHub, Linux VMs, Mac Minis, and automation with Chef to improve testing and visibility of their code.
Development, Deployment & Collaboration at Etsy
This document discusses development, deployment, and collaboration practices at Etsy. It describes that Etsy deploys changes 50 times per day on average and uses practices like continuous integration, developer VMs, the "Deployinator" tool, monitoring dashboards, on-call rotations, incident response procedures, post-mortem reviews, and knowledge sharing to facilitate frequent deployments in a collaborative environment. The goal is to share practices that work for Etsy and encourage learning and teaching across teams.
Continuous Deployment at Etsy: A Tale of Two Approaches
1. Etsy has transitioned from infrequent deployments that took weeks of work and often broke the site, to deploying up to 25 times per day with near effortless deploys.
2. By deploying frequently with small code changes and thorough testing, the probability and severity of degradations is reduced, allowing issues to be detected and resolved quickly.
3. Etsy's continuous deployment approach enables rapid experimentation and improvement through frequent analysis of deployment outcomes and re-examination of assumptions.
Data Driven Monitoring
At Etsy we are big fans of graphing and monitoring all the things. We deploy our main site several times a day and our monitoring provides us with the tight feedback loop we need to make this possible. The same goes for changes in the infrastructure which are deployed in a similar fashion of small and frequent changes. For this to work we have build up monitoring that tracks changes and possible problems in every nook and cranny of the Etsy stack, be it a network change, systems or application level performance or how bad the last week of on-call rotation was. The flipside of monitoring all the things however is that we have a myriad of graphs and alerts that can potentially be important and page the on-call engineer at any given time. The risk of running into alert fatigue and a normalization of deviance through not properly scoped checks is rising with this ever increasing size of the monitoring system. This is why we also continuously monitor our monitoring system and ask questions about whether we have all the information at hand when we get paged, if certain alerts actually need to wake someone up or if they are needed at all. I will give a quick overview of how our monitoring stack is built and then give insights into how we gather data about it and use it to make things better. For site operations and more importantly for the human getting paged when something does go wrong.
A Whirlwind Tour of Etsy's Monitoring Stack
Etsy uses a variety of open source and custom tools to monitor their infrastructure including Ganglia, StatsD, Graphite, Syslog-Ng, Logster, Splunk, Logstash, Eventinator, Chef, Nagios, and Nagios Herald. They collect system, application, and log metrics and events and store them for analysis. Their monitoring setup is fully configured through Chef to provide consistent deployments across environments with over 120 monitoring recipes.
My Top 5 Favorite Gems
My current top 5 favorite Ruby gems and why. Presented at Ruby Tuesdays #43 (https://www.meetup.com/ruby-malaysia/events/242979874/)
10 Deployments a day - A brief on extreme release protocols
A reflection on how we migrated from an era of production deployments once a week to more than 10 deployments a day. The story of transforming a mere sys-admin to super sophisticated DevOps team, armed with multiple tools, scripts and plugins for achieving automations, accuracy and invincible agility. As we continue on our endeavor for further improvisation, I believe its worth sharing the experience with community. Tentative topics outline:
1.Arming teams for their transformation to DevOps from mere sys-admins.
2.Killing manual deployments (manual deployment->capistrano->webistrano)
3.Automation (for daily backups,monitoring applications and servers deploying dependencies)
4.Saving up time on setting up new instances(using golden image).
5.Using LDAP to maintain servers with public key(avoiding individual logins to every server makes it less messy)
6.Using Puppet for instantiating multiple servers at once.
7.Commissioning our own Cloud Infrastructure.
Recommended
DevTools at Etsy
The DevTools team at Etsy consists of 5 engineers who build and maintain tools to help the 150 committers at Etsy. These tools include Deployinator for deploying code, DeveloperVMs for local development environments, Jenkins slaves for continuous integration, DevDB Proxy for accessing production data safely, and various monitoring and debugging tools like StatsD, Logster, Graphite, and Supergrep. The DevTools team works to foster collaboration between development and operations through these shared tools.
Mobile CI at Etsy
Etsy uses GitHub Enterprise for code reviews of code committed to their on-premise git repositories. They have over 350 Linux virtual machines called "The Bobs" that are used to build and test code. Etsy also has 25 Mac Minis that are used to build, test, and run linting on their mobile Android and iOS code using an automated process with Chef. The summary describes Etsy's implementation of continuous integration for mobile apps using GitHub, Linux VMs, Mac Minis, and automation with Chef to improve testing and visibility of their code.
Development, Deployment & Collaboration at Etsy
This document discusses development, deployment, and collaboration practices at Etsy. It describes that Etsy deploys changes 50 times per day on average and uses practices like continuous integration, developer VMs, the "Deployinator" tool, monitoring dashboards, on-call rotations, incident response procedures, post-mortem reviews, and knowledge sharing to facilitate frequent deployments in a collaborative environment. The goal is to share practices that work for Etsy and encourage learning and teaching across teams.
Continuous Deployment at Etsy: A Tale of Two Approaches
1. Etsy has transitioned from infrequent deployments that took weeks of work and often broke the site, to deploying up to 25 times per day with near effortless deploys.
2. By deploying frequently with small code changes and thorough testing, the probability and severity of degradations is reduced, allowing issues to be detected and resolved quickly.
3. Etsy's continuous deployment approach enables rapid experimentation and improvement through frequent analysis of deployment outcomes and re-examination of assumptions.
Data Driven Monitoring
At Etsy we are big fans of graphing and monitoring all the things. We deploy our main site several times a day and our monitoring provides us with the tight feedback loop we need to make this possible. The same goes for changes in the infrastructure which are deployed in a similar fashion of small and frequent changes. For this to work we have build up monitoring that tracks changes and possible problems in every nook and cranny of the Etsy stack, be it a network change, systems or application level performance or how bad the last week of on-call rotation was. The flipside of monitoring all the things however is that we have a myriad of graphs and alerts that can potentially be important and page the on-call engineer at any given time. The risk of running into alert fatigue and a normalization of deviance through not properly scoped checks is rising with this ever increasing size of the monitoring system. This is why we also continuously monitor our monitoring system and ask questions about whether we have all the information at hand when we get paged, if certain alerts actually need to wake someone up or if they are needed at all. I will give a quick overview of how our monitoring stack is built and then give insights into how we gather data about it and use it to make things better. For site operations and more importantly for the human getting paged when something does go wrong.
A Whirlwind Tour of Etsy's Monitoring Stack
Etsy uses a variety of open source and custom tools to monitor their infrastructure including Ganglia, StatsD, Graphite, Syslog-Ng, Logster, Splunk, Logstash, Eventinator, Chef, Nagios, and Nagios Herald. They collect system, application, and log metrics and events and store them for analysis. Their monitoring setup is fully configured through Chef to provide consistent deployments across environments with over 120 monitoring recipes.
My Top 5 Favorite Gems
My current top 5 favorite Ruby gems and why. Presented at Ruby Tuesdays #43 (https://www.meetup.com/ruby-malaysia/events/242979874/)
10 Deployments a day - A brief on extreme release protocols
A reflection on how we migrated from an era of production deployments once a week to more than 10 deployments a day. The story of transforming a mere sys-admin to super sophisticated DevOps team, armed with multiple tools, scripts and plugins for achieving automations, accuracy and invincible agility. As we continue on our endeavor for further improvisation, I believe its worth sharing the experience with community. Tentative topics outline:
1.Arming teams for their transformation to DevOps from mere sys-admins.
2.Killing manual deployments (manual deployment->capistrano->webistrano)
3.Automation (for daily backups,monitoring applications and servers deploying dependencies)
4.Saving up time on setting up new instances(using golden image).
5.Using LDAP to maintain servers with public key(avoiding individual logins to every server makes it less messy)
6.Using Puppet for instantiating multiple servers at once.
7.Commissioning our own Cloud Infrastructure.
Railsconf 2014 - Deploying Rails is Easier Thank It Looks
Introductory presentation on Deploying Rails Applications to your own servers using Chef and Capistrano 3
Ansible ALLTHETHINGS
Learn how in less than 6 months and with a 1-person team, they went from no infrastructure automation, to having all of their infrastructure automated with Ansible. Learn how BigPanda (http://bigpanda.io ) handles zero-downtime infrastructure updates and connects Ansible with their chat infrastructure, and some strategies on managing automation projects with very small teams.
Riak at Posterous
Posterous recently deployed Riak to serve as their content cache. In this talk, Julio Capote will cover why the engineering team chose Riak for the use case. He'll also share some details on the old post cache and its problems, what solutions they evaluated, and how they settled on Riak.
ConcourseCI love Minio
Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.
Slim PHP when you don't need the kitchen sink
Full stack frameworks can often be too much for an application. What if you have a one off project that doesn't need "everything and the kitchen sink"? What if you have a large project you want to build yourself to be as lean as possible?
Join us for an introduction into the Slim framework. We'll cover getting started, building a small application, and using components from bigger frameworks and other extraordinary sources.
Modern infrastructure as code with ansible cake fest 2021
This document discusses using Ansible to implement infrastructure as code. It provides an overview of Ansible concepts like modules, inventory, and playbooks. It then demonstrates how to create an Ansible playbook to install and configure PHP on a server. It also discusses best practices like separating environments, using variables, reusing roles from Ansible Galaxy, and secrets management.
My experience as Eclipse Contributor - ECE 2015
The Eclipse community consists of highly qualified professionals who decided to commit some of their time to grow and improve the Eclipse Project. Each of them - committers - began their careers by getting in touch with other people within the community; making small contributions to a project and then increasing the scope of their commitment.
This talk is to present my experience as a contributor, that is the initial stage of commitment needed to be an Eclipse guy. In this talk, I will quickly explain why I think joining the Eclipse community it is a very clever idea; I will outline some of the most important aspects to keep in mind while you are contributing, and finally I will highlight the major drivers and the most common pitfalls one can have when is contributing to Eclipse.
Concourse webhook
This document discusses using webhooks in Concourse pipelines to reduce API calls. It begins with an introduction and background on the author. It then provides an overview of Concourse architecture and the roles of the ATC, workers, and resources/jobs. Next, it explains how webhooks can be used resource-agnostically in Concourse by defining a webhook token. It describes how webhooks allow external services like GitHub to notify Concourse of changes instead of Concourse constantly checking. Finally, it mentions a demo, online resources, documentation, Slack community, and invites questions.
Building software by feature with immutable infrastructures on AWS
How serv.sg deliver features using AWS infrastructure: programatically building and provisioning a stack before destroying it. This talk will explain what they do, why they do it and the tools they use (packer.io, terraform.io, ansible etc.). Includes a live demo of the whole cycle.
Video of the talk: http://engineers.sg/v/474
VersaPay's Tools for Happyfficient Developers
Open source tools created by VersaPay to have Happyfficient developers!
http://www.versapay.com/developer-blog/versapay-development-flow/
Agile Barcamp Lyon
Modern infrastructure as code with ansible PyTN
Join us as we dive into Ansible, the server management tool designed to help developers keep servers configured properly and running smoothly. We’ll cover the basics and dive into some in-depth configuration options so you can stop treating your servers like pets: which you constantly have to feed and water. Join us as we learn how to turn those pets into cattle in your server farm and turn your infrastructure into code!
Ansible is an agentless server configuration manager which performs tasks over an SSH connection. There is nothing to install on your systems. We’ll cover examples of how to leverage Ansible modules to add users, configure services, setup configuration files, and deploy code to remote systems. Instead of manually performing these tasks, join us as we learn how to write Ansible playbooks to accomplish these tasks and turn our infrastructure into infrastructure as code!
Topic Outline:
Intro to Ansible
What Ansible is good for
What Ansible isn’t good for
Configuring Inventory & Environments
Building Playbooks
Running & Debugging Playbooks
Reuse other playbooks for your needs
Where to get help & other fantastic learning resources
Dev-Friendly Ops
1. The document discusses how OpsWorks has made the presenter's life easier as a developer who also handles operations. OpsWorks provides hosted infrastructure on AWS for deploying applications using Chef recipes.
2. It describes the main structures in OpsWorks - stacks, layers, apps, and instances. Stacks represent entire applications, layers define different parts like web servers, apps contain specific settings, and instances define the servers.
3. The presenter discusses using OpsWorks with Ruby on Rails applications, including customizing Chef recipes, deploying code, and integrating other AWS services for monitoring, security, and scaling. While documentation can be confusing, OpsWorks provides an easy way for developers to manage operations.
A Tale of Two Workflows - ChefConf 2014
Watch this talk here: https://www.youtube.com/watch?v=L__8o02od6Q
For an example of the code we used in our CI pipeline to make a Chef Environment from a Berksfile.lock - check out this project:
https://github.com/petecheslock/berks2env
One of the biggest advantages of Chef is it's flexibility, allowing you to customize it at-will to fit your infrastructure needs. While this makes Chef incredibly powerful, it can also be challenging to develop a workflow to manage the day-to-day usage of chef.
Should I use a single repo for all my cookbooks?
One cookbook per repo?
Berkshelf?
Librarian?
Test-Kitchen?
Where does Jenkins(CI) fit it?
What about Testing?
How does this work with my small team? What about my large team? What about my * Distributed Team?
Over the past few years I have been a part of two distinct Chef workflows that take opposite paths about how to solve issues around collaboration, versioning, testing, etc. During the course of this talk I will share:
Details about the requirements that lead us down these 2 paths.
What worked.
What didn't.
How we use many of the tools available to safely test code changes.
How we deploy cookbook changes safely and quickly (and keep uptime our highest priority).
An (Updated) Introduction to Gutenberg
An (Updated) Introduction to Gutenberg, WordPress's new editor. Presented by Andrew Marks at the Brisbane Northside WordPress Meetup on 19th September 2018.
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
This document discusses using Chef to automate server maintenance including patching and rebooting. It outlines:
- Current pain points with patching including unclear responsibilities and risk of simultaneous reboots.
- Proposed solution of using Chef to schedule patching and reboots on a periodic basis (e.g. weekends), with options to customize the schedule and opt out.
- Recommendations for testing the Chef cookbook gradually and allowing time for customization before full rollout.
Essential Plone development tools - Plone conf 2012
Slides for the talk given at Plone Conference 2012. In the following collective package further explaination can be found about the tools discussed: https://github.com/collective/collective.exampledevtools
Perfect workflow with vagrant
This document discusses Vagrant, an open source tool that allows users to configure and manage virtual machine environments. It introduces Vagrant and explains how it can be used to create consistent development environments across different machines. Key features of Vagrant like multi-machine support, provisioning, and integration with Vagrant Cloud are highlighted. The document provides commands for getting started with Vagrant and managing Vagrant machines.
Hacking on gems
This document provides instructions for contributing to Ruby gems hosted on GitHub. It outlines steps such as forking the gem's repository, making changes on a new branch, running tests, and submitting a pull request with a descriptive message to the gem maintainer. If the pull request is accepted, the new version of the gem will be published for others to benefit from the contribution. The overall process emphasizes making improvements through the open source community rather than maintaining private forks.
Intro to Netflix's Chaos Monkey
The basics you need to know to get up and running with Chaos Monkey in your Amazon Web Service's Cloud enviornment.
Links:
CloudFormation Template:
https://github.com/joehack3r/aws/blob/master/cloudformation/templates/chaosMonkey.json
Simian Army Quick Start Guide:
https://github.com/Netflix/SimianArmy/wiki/Quick-Start-Guide
Chaos Monkey Configuration:
https://github.com/Netflix/SimianArmy/wiki/Chaos-Settings
Chaos Monkey Army:
https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army
Avoiding surprises with Chef and Vagrant
This document discusses using Vagrant and Chef together to create consistent development environments that match production. It notes that development and production environments often differ, causing bugs. Vagrant allows creating and managing virtual machines easily. Chef automates server configuration through recipes and community cookbooks. The document provides basic instructions for installing Vagrant and VirtualBox then cloning a sample project using Vagrant, Chef and a Git submodule to provision a virtual machine. It concludes by inviting questions.
DevOps and Continuous Delivery Reference Architectures (including Nexus and o...
There are numerous examples of DevOps and Continuous Delivery reference architectures available, and each of them vary in levels of detail, tools highlighted, and processes followed. Yet, there is a constant theme among the tool sets: Jenkins, Maven, Sonatype Nexus, Subversion, Git, Docker, Puppet/Chef, Rundeck, ServiceNow, and Sonar seem to show up time and again.
DEV+OPS: How to automate infrastructure - Cloud Expo Europe 2016
L'histoire de l'informatique jeune: En moins de 50 ans l'informatique a révolutionné l'industrie et la société. En conséquence, la façon dont nous, informaticiens, travaillons est encore "artisanale" par endroit.
Si l'on considère les processus de développement aujourd'hui, nous pouvons parler d'usine logicielle. Les tâches fastidieuses peuvent être automatisées et ce du démarage d'un projet à l'intégration continue du service. Mais aujourd'hui tout le secteur de l'hébergement n'a pas encore complètement automatisé de façon similaire: industrialisé de bout en bout. La perte de productivité est phénoménale. Et c'est que ce Quentin ADAM va aborder au cours de cette conférence.
More Related Content
What's hot
Railsconf 2014 - Deploying Rails is Easier Thank It Looks
Introductory presentation on Deploying Rails Applications to your own servers using Chef and Capistrano 3
Ansible ALLTHETHINGS
Learn how in less than 6 months and with a 1-person team, they went from no infrastructure automation, to having all of their infrastructure automated with Ansible. Learn how BigPanda (http://bigpanda.io ) handles zero-downtime infrastructure updates and connects Ansible with their chat infrastructure, and some strategies on managing automation projects with very small teams.
Riak at Posterous
Posterous recently deployed Riak to serve as their content cache. In this talk, Julio Capote will cover why the engineering team chose Riak for the use case. He'll also share some details on the old post cache and its problems, what solutions they evaluated, and how they settled on Riak.
ConcourseCI love Minio
Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.
Slim PHP when you don't need the kitchen sink
Full stack frameworks can often be too much for an application. What if you have a one off project that doesn't need "everything and the kitchen sink"? What if you have a large project you want to build yourself to be as lean as possible?
Join us for an introduction into the Slim framework. We'll cover getting started, building a small application, and using components from bigger frameworks and other extraordinary sources.
Modern infrastructure as code with ansible cake fest 2021
This document discusses using Ansible to implement infrastructure as code. It provides an overview of Ansible concepts like modules, inventory, and playbooks. It then demonstrates how to create an Ansible playbook to install and configure PHP on a server. It also discusses best practices like separating environments, using variables, reusing roles from Ansible Galaxy, and secrets management.
My experience as Eclipse Contributor - ECE 2015
The Eclipse community consists of highly qualified professionals who decided to commit some of their time to grow and improve the Eclipse Project. Each of them - committers - began their careers by getting in touch with other people within the community; making small contributions to a project and then increasing the scope of their commitment.
This talk is to present my experience as a contributor, that is the initial stage of commitment needed to be an Eclipse guy. In this talk, I will quickly explain why I think joining the Eclipse community it is a very clever idea; I will outline some of the most important aspects to keep in mind while you are contributing, and finally I will highlight the major drivers and the most common pitfalls one can have when is contributing to Eclipse.
Concourse webhook
This document discusses using webhooks in Concourse pipelines to reduce API calls. It begins with an introduction and background on the author. It then provides an overview of Concourse architecture and the roles of the ATC, workers, and resources/jobs. Next, it explains how webhooks can be used resource-agnostically in Concourse by defining a webhook token. It describes how webhooks allow external services like GitHub to notify Concourse of changes instead of Concourse constantly checking. Finally, it mentions a demo, online resources, documentation, Slack community, and invites questions.
Building software by feature with immutable infrastructures on AWS
How serv.sg deliver features using AWS infrastructure: programatically building and provisioning a stack before destroying it. This talk will explain what they do, why they do it and the tools they use (packer.io, terraform.io, ansible etc.). Includes a live demo of the whole cycle.
Video of the talk: http://engineers.sg/v/474
VersaPay's Tools for Happyfficient Developers
Open source tools created by VersaPay to have Happyfficient developers!
http://www.versapay.com/developer-blog/versapay-development-flow/
Agile Barcamp Lyon
Modern infrastructure as code with ansible PyTN
Join us as we dive into Ansible, the server management tool designed to help developers keep servers configured properly and running smoothly. We’ll cover the basics and dive into some in-depth configuration options so you can stop treating your servers like pets: which you constantly have to feed and water. Join us as we learn how to turn those pets into cattle in your server farm and turn your infrastructure into code!
Ansible is an agentless server configuration manager which performs tasks over an SSH connection. There is nothing to install on your systems. We’ll cover examples of how to leverage Ansible modules to add users, configure services, setup configuration files, and deploy code to remote systems. Instead of manually performing these tasks, join us as we learn how to write Ansible playbooks to accomplish these tasks and turn our infrastructure into infrastructure as code!
Topic Outline:
Intro to Ansible
What Ansible is good for
What Ansible isn’t good for
Configuring Inventory & Environments
Building Playbooks
Running & Debugging Playbooks
Reuse other playbooks for your needs
Where to get help & other fantastic learning resources
Dev-Friendly Ops
1. The document discusses how OpsWorks has made the presenter's life easier as a developer who also handles operations. OpsWorks provides hosted infrastructure on AWS for deploying applications using Chef recipes.
2. It describes the main structures in OpsWorks - stacks, layers, apps, and instances. Stacks represent entire applications, layers define different parts like web servers, apps contain specific settings, and instances define the servers.
3. The presenter discusses using OpsWorks with Ruby on Rails applications, including customizing Chef recipes, deploying code, and integrating other AWS services for monitoring, security, and scaling. While documentation can be confusing, OpsWorks provides an easy way for developers to manage operations.
A Tale of Two Workflows - ChefConf 2014
Watch this talk here: https://www.youtube.com/watch?v=L__8o02od6Q
For an example of the code we used in our CI pipeline to make a Chef Environment from a Berksfile.lock - check out this project:
https://github.com/petecheslock/berks2env
One of the biggest advantages of Chef is it's flexibility, allowing you to customize it at-will to fit your infrastructure needs. While this makes Chef incredibly powerful, it can also be challenging to develop a workflow to manage the day-to-day usage of chef.
Should I use a single repo for all my cookbooks?
One cookbook per repo?
Berkshelf?
Librarian?
Test-Kitchen?
Where does Jenkins(CI) fit it?
What about Testing?
How does this work with my small team? What about my large team? What about my * Distributed Team?
Over the past few years I have been a part of two distinct Chef workflows that take opposite paths about how to solve issues around collaboration, versioning, testing, etc. During the course of this talk I will share:
Details about the requirements that lead us down these 2 paths.
What worked.
What didn't.
How we use many of the tools available to safely test code changes.
How we deploy cookbook changes safely and quickly (and keep uptime our highest priority).
An (Updated) Introduction to Gutenberg
An (Updated) Introduction to Gutenberg, WordPress's new editor. Presented by Andrew Marks at the Brisbane Northside WordPress Meetup on 19th September 2018.
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
This document discusses using Chef to automate server maintenance including patching and rebooting. It outlines:
- Current pain points with patching including unclear responsibilities and risk of simultaneous reboots.
- Proposed solution of using Chef to schedule patching and reboots on a periodic basis (e.g. weekends), with options to customize the schedule and opt out.
- Recommendations for testing the Chef cookbook gradually and allowing time for customization before full rollout.
Essential Plone development tools - Plone conf 2012
Slides for the talk given at Plone Conference 2012. In the following collective package further explaination can be found about the tools discussed: https://github.com/collective/collective.exampledevtools
Perfect workflow with vagrant
This document discusses Vagrant, an open source tool that allows users to configure and manage virtual machine environments. It introduces Vagrant and explains how it can be used to create consistent development environments across different machines. Key features of Vagrant like multi-machine support, provisioning, and integration with Vagrant Cloud are highlighted. The document provides commands for getting started with Vagrant and managing Vagrant machines.
Hacking on gems
This document provides instructions for contributing to Ruby gems hosted on GitHub. It outlines steps such as forking the gem's repository, making changes on a new branch, running tests, and submitting a pull request with a descriptive message to the gem maintainer. If the pull request is accepted, the new version of the gem will be published for others to benefit from the contribution. The overall process emphasizes making improvements through the open source community rather than maintaining private forks.
Intro to Netflix's Chaos Monkey
The basics you need to know to get up and running with Chaos Monkey in your Amazon Web Service's Cloud enviornment.
Links:
CloudFormation Template:
https://github.com/joehack3r/aws/blob/master/cloudformation/templates/chaosMonkey.json
Simian Army Quick Start Guide:
https://github.com/Netflix/SimianArmy/wiki/Quick-Start-Guide
Chaos Monkey Configuration:
https://github.com/Netflix/SimianArmy/wiki/Chaos-Settings
Chaos Monkey Army:
https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army
Avoiding surprises with Chef and Vagrant
This document discusses using Vagrant and Chef together to create consistent development environments that match production. It notes that development and production environments often differ, causing bugs. Vagrant allows creating and managing virtual machines easily. Chef automates server configuration through recipes and community cookbooks. The document provides basic instructions for installing Vagrant and VirtualBox then cloning a sample project using Vagrant, Chef and a Git submodule to provision a virtual machine. It concludes by inviting questions.
What's hot (20)
Railsconf 2014 - Deploying Rails is Easier Thank It Looks
Railsconf 2014 - Deploying Rails is Easier Thank It Looks
Modern infrastructure as code with ansible cake fest 2021
Modern infrastructure as code with ansible cake fest 2021
Building software by feature with immutable infrastructures on AWS
Building software by feature with immutable infrastructures on AWS
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
Essential Plone development tools - Plone conf 2012
Essential Plone development tools - Plone conf 2012
Viewers also liked
DevOps and Continuous Delivery Reference Architectures (including Nexus and o...
There are numerous examples of DevOps and Continuous Delivery reference architectures available, and each of them vary in levels of detail, tools highlighted, and processes followed. Yet, there is a constant theme among the tool sets: Jenkins, Maven, Sonatype Nexus, Subversion, Git, Docker, Puppet/Chef, Rundeck, ServiceNow, and Sonar seem to show up time and again.
DEV+OPS: How to automate infrastructure - Cloud Expo Europe 2016
L'histoire de l'informatique jeune: En moins de 50 ans l'informatique a révolutionné l'industrie et la société. En conséquence, la façon dont nous, informaticiens, travaillons est encore "artisanale" par endroit.
Si l'on considère les processus de développement aujourd'hui, nous pouvons parler d'usine logicielle. Les tâches fastidieuses peuvent être automatisées et ce du démarage d'un projet à l'intégration continue du service. Mais aujourd'hui tout le secteur de l'hébergement n'a pas encore complètement automatisé de façon similaire: industrialisé de bout en bout. La perte de productivité est phénoménale. Et c'est que ce Quentin ADAM va aborder au cours de cette conférence.
Understand immutable infrastructure, what? Why? how? - devops d day Marseill...
Why everybody is speaking about Immutability? Immutable infrastructure? The All IT automation ecosystem need to rely on the append only, remove historical management of servers. This talk explain what is immutable infrastructure, how to build it, and how to manage data in this infrastructure pattern. It will cover pattern to use it on containers or virtual machine world.
ATAGTR2017 Performance Automation in Dev-Ops
This document summarizes an agenda for a conference on performance testing in DevOps. It discusses automated performance testing using Jenkins, including developing plugins for tools like JMeter and Silk Performer. Approaches covered include identifying critical use cases, automating tests, running them through Jenkins, parsing results, and using the data to decide on production deployments. Advantages are reducing testing efforts and easily detecting regressions. Limitations include manually editing scripts and validating changes. The future plans discussed shifting performance testing left and automating resource utilization stats.
Dev and Ops Collaboration and Awareness at Etsy and Flickr
This is a talk I gave at QCon in SF, in 2010: http://qconsf.com/sf2010/presentation/Cooperation%2C+Collaboration%2C+and+Awareness
DevOps & Security: Here & Now
How do you integrate security within a Continuous Deployment (CD) environment - where every 5 minutes a feature, an enhancement, or a bug fix needs to be released?
Traditional application security tools which require lengthy periods of configuration, tuning and
application learning have become irrelevant in these fast-pace environments. Yet, falling back only on
the secure coding practices of the developer cannot be tolerated.
Secure coding requires a new approach where security tools become part of the development environment – and eliminate any unnecessary overhead. By collaborating with development teams, understanding their needs and requirements, you can pave the way to a secure deployment in minutes.
Continuous Security Testing with Devops - OWASP EU 2014
This document discusses continuous security testing in a DevOps environment. It advocates treating security testing as a form of quality testing that is automated and integrated into continuous delivery pipelines. The author presents the BDD-Security testing framework, which uses behavior-driven development and test automation tools like Selenium to write security tests against applications. The framework wraps security scanning tools like OWASP ZAP and integrates security testing into continuous integration pipelines like Jenkins. This allows security to keep up with DevOps practices like deploying code changes multiple times per day.
Who am i presentation
Lee Riches provides information about himself including his residence in North Carolina, interests in various sports played for many years like football and baseball, and dreams of having a happy family and well-paying job. Others describe him as genuine, thoughtful, honest, loyal, friendly, easy to talk to, diligent, compassionate, and introspective. He dislikes living in New York and has a fear of heights.
LasCon 2014 DevOoops
In a rare mash-up, DevOps is increasingly blending the work of both application and network security professionals. In a quest to move faster, organizations can end up creating security vulnerabilities using the tools and products meant to protect them. Both Chris Gates (carnal0wnage) and Ken Johnson (cktricky) will share their collaborative research into the technology driving DevOps as well as share their stories of what happens when these tools are used insecurely as well as when the tools are just insecure.
Technologies discussed will encompass AWS Technology, Chef, Puppet, Hudson/Jenkins, Vagrant, Kickstart and much, much more. Everything from common misconfigurations to remote code execution will be presented. This is research to bring awareness to those responsible for securing a DevOps environment.
Who am i presentation
Molly Rogers is a sophomore at BVHS who enjoys playing soccer, running cross country, and participating in debate and forensics. She lives in Washington D.C. with her mom, dad, and older brother Emmett. Some of Molly's accomplishments include her club soccer team doing well, earning medals in debate competitions, and finishing an entire tube of chapstick without losing it. In her free time, she plays soccer, volunteers, and attends debate tournaments, and also has two beagle dogs named Gracie and Mabel.
Architectures for open and scalable clouds
My presentation for 2012's Cloud Connect that goes over architectural and design patterns for open and scalable clouds. Technical deck targeted at business audiences with a technical bent.
The 2016 Top 50 Tech Pioneers, Australia and New Zealand
We are pleased to present the inaugural ‘Tech Pioneers Report’, which showcases the most exciting and pioneering startups from Australia and New Zealand. These are the companies at the leading edge of the ‘ideas boom’ and the drivers of the economy of the future.
The Tech Pioneers 50 are those companies using technology to the best advantage and driving disruption within a variety of industries and sectors. These companies have a commitment
to excellence, superior customer experience and a demonstrated ability to do one thing in a market better than anyone else.
The process for selecting the top 50 pioneers involved a wide search for pioneering companies to create a comprehensive list that was then ranked based on 5 factors: total capital raised, rate of capital raising, location and degree of sub industry disruption and the judging panel’s subjective rating of the degree of product, service, customer experience and business model innovation. A formula was then used to combine this data and determine the overall ranking.
The report has been prepared by H2 Ventures, one of Australia’s leading venture capital firms, with the support of Investec Australia, a distinctive specialist lender and asset manager.
DevOps beyond the Tools
DevOps is mainstream - at least the tools, the automation and the metrics. But what happened to DevOps Culture? Does it still matter? If yes - how do we achieve it?
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
This presentation describes my interpretation of the Why and How of DevOps, and the key findings from my 15 year study of high-performing IT organizations, and how they simultaneously deliver stellar service levels and rapid implementation of new features into the production environment.
Organizations employing DevOps practices such as Google, Amazon, Facebook, Etsy and Twitter are routinely deploying code into production hundreds, or even thousands, of times per day, while providing world-class availability, reliability and security. In contrast, most organizations struggle to do releases more every nine months.
He will present how these high-performing organizations achieve this fast flow of work through Product Management and Development, through QA and Infosec, and into IT Operations. By doing so, other organizations can now replicate the extraordinary culture and outcomes enabling their organization to win in the marketplace.
Adobe Presents Internal Service Delivery Platform at Velocity 13 Santa Clara
This document summarizes a presentation about Adobe's journey towards a DevOps model. It discusses how Adobe shifted to a cloud subscription business model and realized they needed to change from being toolsmiths to a service provider. They created the CDOT service to provide teams a clear and automated path to production. The presentation emphasizes that to be a successful internal service provider requires treating it like a real business through internal sales, marketing, creating a great user experience, and onboarding new customers.
Support and Initiate a DevOps Transformation
This document provides a guide to initiating and supporting a DevOps transformation through a 3-step process: 1) Build the "Why?" by understanding the business reasons for change, 2) Build organizational alignment by getting stakeholders on the same page through techniques like value stream mapping, and 3) Implement continuous improvement by developing a common DevOps vision focused on system flow and feedback. The value lies in improving productivity, quality and collaboration between development and operations teams through a cultural shift toward shared goals and automated processes.
Who am i slideshow
The document discusses the results of several surveys and assessments that provide insight into the person's interests, skills, values, and dreams. The surveys found that the person is a helper and persuader according to their interest profiler, an active listener and learner according to their basic skills survey, and values working conditions and achievement according to their work values survey. Their personal dreams include buying a kitten, traveling to Europe, volunteering, skydiving, meeting someone famous, and losing their fear of flying, while their professional dreams are not specified.
DevOps
The document discusses the roles and relationships between development (Dev) and operations (Ops) teams, and introduces the DevOps approach. It notes that traditionally there has been a disconnect between Devs and Ops that results in inefficiencies. DevOps aims to bridge this gap through a collaborative mindset and practices like automating infrastructure provisioning and deployments, implementing continuous integration/delivery, monitoring metrics, and breaking down silos between teams. Specific tools mentioned that support DevOps include Puppet for configuration management and Autobahn for continuous deployment.
Quelles changements de vision dans la cyber sécurité en 2017 ? - ADN OUEST, s...
Quentin Adam from Clever Cloud discusses changes in cybersecurity in 2017. Security is evolving from a fortress model to one based on authentication, encryption, and auditability for each server. Companies must implement auditing and monitoring, understand their systems and networks, invest properly in security rather than being miserly, and encrypt all outgoing data. Good security practices that people respect, like being nice, help as well.
What Product Leaders need to know about DevOps
DevOps is an organizational culture that aims to rapidly deliver high quality, resilient software at scale. It focuses on breaking down silos between development and operations teams through practices like continuous integration, delivery, and deployment. Adopting DevOps allows product leaders to build products faster, better, cheaper, and safer by leveraging techniques such as infrastructure as code, automated testing, and continuous experimentation. While DevOps is more about culture change than tools, product managers can help drive this culture by focusing on non-functional requirements, risk mitigation strategies, and ensuring their organizations can keep up with faster delivery through improved documentation and support.
Viewers also liked (20)
DevOps and Continuous Delivery Reference Architectures (including Nexus and o...
DevOps and Continuous Delivery Reference Architectures (including Nexus and o...
DEV+OPS: How to automate infrastructure - Cloud Expo Europe 2016
DEV+OPS: How to automate infrastructure - Cloud Expo Europe 2016
Understand immutable infrastructure, what? Why? how? - devops d day Marseill...
Understand immutable infrastructure, what? Why? how? - devops d day Marseill...
Dev and Ops Collaboration and Awareness at Etsy and Flickr
Dev and Ops Collaboration and Awareness at Etsy and Flickr
Continuous Security Testing with Devops - OWASP EU 2014
Continuous Security Testing with Devops - OWASP EU 2014
The 2016 Top 50 Tech Pioneers, Australia and New Zealand
The 2016 Top 50 Tech Pioneers, Australia and New Zealand
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Adobe Presents Internal Service Delivery Platform at Velocity 13 Santa Clara
Adobe Presents Internal Service Delivery Platform at Velocity 13 Santa Clara
Quelles changements de vision dans la cyber sécurité en 2017 ? - ADN OUEST, s...
Quelles changements de vision dans la cyber sécurité en 2017 ? - ADN OUEST, s...
Similar to Development, Deployment and Collaboration at Etsy
Continuous Integration, the minimum viable product
What does it mean to 'do' Continuous Integration? It used to be enough to execute your unit tests in CI. But the bar is steadily raising for engineering practices. In the last decade we've seen tremendous improvements inacceptance testing. JavaScript is now a platform in it's own right. Cloudcomputing is now vital. There's growing interest in deployment to prod.So Continuous Integration is under more pressure than ever. As the bar slowly raises for engineering practices, we ll present 2011's minimum viable feature set for Continuous Integration
Continuous Testing
Continuous Testing helps provide process improvements that can prevent future defects from occurring. It plays an important role in providing continuous feedback for your software.
Vagrant for Effective DevOps Culture
This document discusses how Vagrant was implemented at Wingify Engineering to help establish a DevOps culture. Previously, Wingify had issues with environments not matching production, difficult setups, and isolation between devs and ops. Vagrant provided developers similar environments to production using the same OS and configuration management. It simplified management through tools like "vagrant up", "vagrant ssh", and "vagrant destroy". This improved testing and reduced issues in production. It also improved collaboration by allowing ops to test configurations and devs to better understand infrastructure. Overall Vagrant helped establish closer alignment between devs and ops through shared responsibility of infrastructure.
Rails after 5 years
Short presentation given on the cool stuff in Rails 3. Impressions coming from a Developer who used Rails up to 0.12.1 and then came back to Rails 3.
Untangling spring week10
This document provides an agenda and instructions for setting up a server and database for a coding project. It discusses setting up a Node.js server on Digital Ocean, forking a sample project on GitHub, installing dependencies with NPM, running the server, and using EJS templates and MySQL Workbench to connect to and modify a database. Students are instructed to fork the sample project, modify it to use their own database, run it on their server, and submit the GitHub link and website URL for homework.
Take control of your Jenkins jobs via job DSL.
Jenkins jobs can be managed via the GUI, CLI, or script console which become unwieldy for large numbers of jobs. The Jenkins Job DSL plugin allows jobs to be defined and maintained as code using a Groovy domain-specific language. This provides benefits such as version control of job configuration, reuse of common job properties, and programmatic job creation. A seed job can be used to generate other jobs from DSL scripts checked into a code repository and run validation/testing.
Test like a_boss
The document discusses dbdeployer, a command line tool for deploying and testing MySQL database topologies. It can deploy single or multiple MySQL instances, as well as complex topologies like replication, group replication, and multi-source replication with a single command. Dbdeployer aims to make deploying and testing databases fast and easy by avoiding repetitive manual tasks. It has features for upgrading and importing existing databases.
Lessons Learnt in 2009
This document provides lessons learned from a presentation on Ruby on Rails development. It discusses using Ruby Enterprise Edition for faster tests. It recommends using factories over fixtures for test data and advocates writing more integration tests than unit tests. It also covers improving security with the rails_xss plugin, using MessageVerifier for authentication, and asynchronous job processing with Delayed Job. Methods for scaling file uploads using mod_porter and pagination without offsets are presented. The document aims to help Ruby web developers with performance, security, background processing, and scaling.
No More “Cowboy Coding”: A Best Practices Guide to Local Development & Migration
This hands-on Workshop will help you get started with running WordPress on your local laptop or desktop. We'll first talk about WHY you need to work locally and how to think about working more safely with your production websites. We’ll be talking ServerPress/Desktop Server and their ‘blueprint’ functionality.
We’ll also be talking about Migration strategies to your hosting company, whether you’re launching a new website to a blank WordPress install or ‘merging’ some new functionality into an existing website. Plugins & Functionalities discussed:
ServerPress & DesktopServer
WP Migrate DB (& WP Migrate DB Pro)
WPEngine’s Migration Utility
Production Ready WordPress #WPLDN
This document discusses using version control and continuous integration to deploy code. It recommends developing code locally, using distributed version control like Git, and deploying to a testing environment before production. The continuous integration workflow involves multiple developers sharing code through a central version control repository. Each code push is verified by automated builds to avoid integration issues and catch problems early. The document provides an example deployment script for Codeship that checks out code, installs dependencies, builds assets, commits changes, and pushes to the production remote. It also discusses testing deployments using Assertible and lessons learned around caching packages, installing dependencies, and using forceful Git pushes for deployment.
stackconf 2022: Infrastructure Automation (anti) patterns
Over the past 1.5 decade our industry has tried to adopt an increased amount of infrastructure automation. We called it Configuration Management, Infrastructure as Code, infrastructure as Software, Provisioning, Orchestration. We learned about Desired State, Idempotence, etc.. We have seen a number of tools become popular; we have seen a number of tools disappear. But over the years we have seen a number of patterns appear and reappear. Patterns that lead to actually getting great benefits out of automation, or just wasting time while missing out on goals. This talk will explain you a number of these patterns which we have frequently encountered in the wild, with their benefits and caveats. We will try to keep this tool agnostic. Your vision might be Clouded, and you might have to take this with a grain of Salt while you play the Chef from the Muppet show the story, all names, characters, and incidents portrayed in this production are fictitious. No identification with actual persons (living or deceased), places, buildings, and products are intended or should be inferred.
Infrastructure as Code Patterns
StackConf 2022 Talk about The Different Patterns and AntiPatterns
we've seen over the past decade of Automation
Frontend Engineer Toolbox
A collection of useful tools, techniques and workflow for modern frontend developers.
Included:
Yeoman, mocha, sass, compass, debugging,
Great Tools Heavily Used In Japan, You Don't Know.
The document discusses Japanese Perl developers who attended YAPC::EU 2015. It introduces many popular Perl modules created by Japanese developers, such as WebService::Simple for making web service requests, Riji for creating blogs, and GrowthForecast for visualizing metrics graphs. It encourages attendees to talk to the Japanese developers about their work or any questions. It emphasizes that Japanese developers prioritize speed and simplicity in their modules due to their culture of valuing efficiency.
Into The Box 2020 Keynote Day 1
This document provides an overview of a virtual ITB conference keynote on Day 1. It includes information about the hosts and speakers, as well as summaries of topics to be covered such as ColdBox 6 features including cbFutures, Task Executors, RestHandler, Singleton Renderer, and a new debugging experience called Whoops!. It also provides updates on projects like CommandBox, ForgeBox, and upcoming courses.
Into the Wild with Node.js and Web Apps
Node.js Presentation at Klout HQ, 2012
By Chris Bumgardner @ "Beyond the Closed Beta, Node.js from code to production (at scale)"
Production Ready WordPress - WC Utrecht 2017
This document discusses deploying code using version control and continuous integration. It recommends developing code locally, using distributed version control like Git, and deploying to a testing environment before production. The solution involves setting up continuous integration where multiple developers can share code and have automated builds and testing on each push. A workflow is proposed where code is developed locally, stored in a central version control repository, built through continuous integration pipelines, and deployed to servers. Tools like Composer, NPM, Gulp and Assertible are demonstrated in an example deployment script for automated testing and deployment.
Continuous Delivery of Puppet Manifests
This document discusses lessons learned from implementing continuous delivery of Puppet code. It describes starting with manual hacking directly on production servers, then using basic version control with Git. Improved approaches involved running Git on Puppet masters with automated pulls, using modules, and package management. The optimal solution uses a Jenkins pipeline for continuous integration testing and deployment, including syntax, style, testing, packaging, repository uploads, and deploying to test environments. Metrics are sent after deployment to close the feedback loop for continuous improvement.
Building a Simple Theme Framework
My talk from WordCamp Philly 2012. I talk about some things you can do to generalize your themes to make a nice framework as a place for you to start!
Steamlining your puppet development workflow
The document discusses ways to streamline a Puppet development workflow including using revision control, running Puppet in noop or automatic mode, moving changes slowly through testing and using branches, reporting on changes, and implementing testing strategies like unit testing with rspec-puppet and integration testing with serverspec. It also recommends tools like Foreman, Norman, Puppetfile, and Jenkins to improve testing and deployment.
Similar to Development, Deployment and Collaboration at Etsy (20)
Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
No More “Cowboy Coding”: A Best Practices Guide to Local Development & Migration
No More “Cowboy Coding”: A Best Practices Guide to Local Development & Migration
stackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patterns
Great Tools Heavily Used In Japan, You Don't Know.
Great Tools Heavily Used In Japan, You Don't Know.
More from Daniel Schauenberg
Human Factors and PostMortems
Our daily work takes place in a myriad of systems. They are comprised of software, hardware and humans. And everybody who has worked with complex systems at any scale knows: Failure is not an option, it's inevitable. At Etsy we are embracing the fact that failures happen and that the only way to understand how the accident happened is to investigate it without blaming the humans involved. This is why we have a blameless postmortem for every outage that occurs. It is an open meeting and everybody is invited to join and find out what happened and how we can make the system safer. This talk will explain how postmortems at Etsy are conducted and how we maintain and scale the process as the team grows and new people start. It will go over the tools we built and utilize to make postmortems efficient and also share the learnings from each one with all the people in the company.
Deploy, Collaborate and Listen
At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this? We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.
Scaling Deployment at Etsy
How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team.
Reprised version of the talk given at the Continuous Delivery NYC meetup October 2013.
Feature Flagging your Infrastructure for Fun and Profit
At Etsy we run our production stack exclusively on physical hardware and have (including developer and CI VMs) about 1000 nodes managed with a single Chef server running the same cookbooks across development and production nodes. We have built the knife-spork plugin to support our workflow and have 30 engineers making changes about 20 times a day. While this setup keeps configuration changes and package installations consistent across our network, it makes it harder to gradually roll out and test changes on a per node or role basis. In the talk our general workflow will be introduced and explained and how we enable engineers to frequently roll out small changes. Following the general introduction, our workflow for testing changes will be discussed and the library we wrote to enable config flags and gradual roll outs for our infrastructure within our chef recipes.
Scaling Deployment at Etsy
Etsy has scaled its deployment process to support around 150 engineers. Key aspects include running continuous integration tests across many virtual machines to avoid bottlenecks, coordinating deployments through an IRC channel with scheduled "push trains", and tools like Deployinator that automate deployments and ensure code quality and version consistency. The goal is to constantly improve deployment speed while maintaining robustness, bring development environments closer to production, and treat inability to deploy with the same urgency as a site outage.
StatsD Workshop Monitorama 2013
This document summarizes a StatsD workshop presented by Daniel Schauenberg at Monitorama 2013. The workshop covered what StatsD is, the different types of metrics it can handle, and how to set up and configure StatsD and its backends like Graphite. It also provided examples of sending metrics to StatsD and ideas for extending and customizing StatsD functionality.
Etsy chef-workflow
Etsy uses Chef for infrastructure configuration and management. They have around 1000 nodes managed by around 30 devops engineers making regular changes. Chef Server and GitHub Enterprise are used to manage configuration code and changes. Knife-spork is used as the deployment tool to promote changes through development, staging, and production environments. Monitoring and notification tools like IRC alerts are used for change management and failure reporting.
More from Daniel Schauenberg (7)
Feature Flagging your Infrastructure for Fun and Profit
Feature Flagging your Infrastructure for Fun and Profit
Recently uploaded
Operating System Used by Users in day-to-day life.pptx
Dive into the realm of operating systems (OS) with Pravash Chandra Das, a seasoned Digital Forensic Analyst, as your guide. 🚀 This comprehensive presentation illuminates the core concepts, types, and evolution of OS, essential for understanding modern computing landscapes.
Beginning with the foundational definition, Das clarifies the pivotal role of OS as system software orchestrating hardware resources, software applications, and user interactions. Through succinct descriptions, he delineates the diverse types of OS, from single-user, single-task environments like early MS-DOS iterations, to multi-user, multi-tasking systems exemplified by modern Linux distributions.
Crucial components like the kernel and shell are dissected, highlighting their indispensable functions in resource management and user interface interaction. Das elucidates how the kernel acts as the central nervous system, orchestrating process scheduling, memory allocation, and device management. Meanwhile, the shell serves as the gateway for user commands, bridging the gap between human input and machine execution. 💻
The narrative then shifts to a captivating exploration of prominent desktop OSs, Windows, macOS, and Linux. Windows, with its globally ubiquitous presence and user-friendly interface, emerges as a cornerstone in personal computing history. macOS, lauded for its sleek design and seamless integration with Apple's ecosystem, stands as a beacon of stability and creativity. Linux, an open-source marvel, offers unparalleled flexibility and security, revolutionizing the computing landscape. 🖥️
Moving to the realm of mobile devices, Das unravels the dominance of Android and iOS. Android's open-source ethos fosters a vibrant ecosystem of customization and innovation, while iOS boasts a seamless user experience and robust security infrastructure. Meanwhile, discontinued platforms like Symbian and Palm OS evoke nostalgia for their pioneering roles in the smartphone revolution.
The journey concludes with a reflection on the ever-evolving landscape of OS, underscored by the emergence of real-time operating systems (RTOS) and the persistent quest for innovation and efficiency. As technology continues to shape our world, understanding the foundations and evolution of operating systems remains paramount. Join Pravash Chandra Das on this illuminating journey through the heart of computing. 🌟
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Recently uploaded (20)
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Development, Deployment and Collaboration at Etsy
- 1. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz
- 3. @mrtazz
- 4. @mrtazz
- 8. avg n > m deploys/ day
- 9. How comfortable are you deploying a change right now?
- 13. “If this is your first day at Etsy, you deploy the site”
- 14. Developer VMs
- 15. Developer VMs • KVM • Every engineer has one • Fully Chef’d with the Etsy Stack • Different sizes and Chef roles @mrtazz
- 19. Continuous Integration • Run set of tests before each deploy • Full QA suite • Princess/Production smoker tests • Try (yup, there is one) @mrtazz
- 21. The Bobs • LXC virtualized hosts • 14/physical hosts • Spread over 3 SSDs • Most of them attached to try @mrtazz
- 24. Deployinator
- 25. Deployinator • 2 Buttons, no ambiguity • Overview of current state of deploy • Links to Logwatcher and Dashboards • Easy to add stacks for new tools to deploy @mrtazz
- 28. Monitoring
- 30. Monitoring • Devs do their feature monitoring • Everybody can access all the graphs • Dashboard All The Things! • Stream All The Logs! @mrtazz
- 34. On Call
- 35. If you are writing code, you are on-call
- 36. On-Call Schedules • ops on-call • dev on-call • payments on-call • support on-call @mrtazz
- 37. Dev On-Call • Scheduled for 6 months • On-call roughly every 4 weeks for 1 week • L1 and L2 escalations • L1 if it’s your first time @mrtazz
- 39. Incident Response • “This graph looks funny” • “Hey I just got paged for elevated error rate after deploys” • “Supergrep is going crazy!!” @mrtazz
- 41. #warroom
- 42. #warroom • only outage related conversations • coordinate investigation, communication, countermeasures and monitoring • good place to lurk for new engineers @mrtazz
- 43. Post Mortems
- 44. blameless
- 47. Summary
- 48. Summary • These are things that work for *us* • Culture is an on-going effort • Share everything • Encourage learning/teaching @mrtazz
- 49. Summary • Lunch ’n learns • DC visits • On-call for a day • Bootcamps/Senior rotations @mrtazz
- 51. Questions?
- 52. Development, Deployment and Collaboration at Etsy Daniel Schauenberg dschauenberg@etsy.com